From 779be788439048ac4ae6b8e5a341decfd0ccf464 Mon Sep 17 00:00:00 2001 From: Kellyvv Date: Wed, 8 Apr 2026 09:22:20 +0800 Subject: [PATCH] refactor: split MLX service, introduce skill type & runtime budgets MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Major restructuring of the agent stack to support content-type skills, data-driven runtime budgets, and the new translate skill. LLM/MLX/ - Split MLXLocalLLMService.swift (1500 → 700 lines) into Core/, Gemma4/, Installation/ subdirectories - New BundledModel: LinearBudgetFormula replaces tier tables, model parameters fully data-driven via ModelRuntimeProfile - New RuntimeBudgets: pure functions for text/thinking/multimodal budgets, zero hardcoded model IDs - Drop adjustedTextOutputTokens prepared-subtraction (chunked prefill made it obsolete and triggered false-positive truncation on long prompts) Skills/ - Add SkillType (.device / .content) metadata, parsed from SKILL.md frontmatter - New translate skill (type: content, no tools) with anaphora handling - Trim translate (4563→1804), device (2357→1680), calendar (2169→1790) - contacts SKILL.md adds multi-turn disambiguation + cancel guidance, semantics owned by body not framework - SkillRegistry validates allowed-tools against ToolRegistry at load time Agent/ - ToolCallParser becomes registry-aware: drops tool_call names not in ToolRegistry or protocol set (load_skill / list_skills); ToolRegistry is now the single white list, validation moves to parser boundary - Router sticky routing uses semantic user-turn boundary instead of fixed suffix(4); correctly handles long agent loops - AgentEngine system prompt template differentiates device vs content skills with explicit invocation rules - SYSPROMPT.md auto-migration: old single-placeholder template backed up to .bak when new dual-placeholder is detected - PromptBuilder.buildLoadedSkillPrompt receives availableTools list, explicitly tells model which tools (or none) are callable Tools/ - ToolRegistry split into Helpers/Stores/AppPermissions/Handlers/* for separation of concerns - Mark Tools/Handlers/* with explicit allowed argument signatures UI/ - Add long-press copy contextMenu on user and assistant bubbles - Enable .textSelection on user bubble Co-Authored-By: Claude Opus 4.6 (1M context) --- .gitignore | 4 + Agent/AgentEngine.swift | 2468 +---------------- Agent/Engine/ChatAttachments.swift | 200 ++ Agent/Engine/ChatMessage.swift | 51 + Agent/Engine/OutputCleaner.swift | 271 ++ Agent/Engine/Planner.swift | 561 ++++ Agent/Engine/Router.swift | 151 + Agent/Engine/SessionStore.swift | 214 ++ Agent/Engine/ToolCallParser.swift | 170 ++ Agent/Engine/ToolChain.swift | 579 ++++ LLM/MLX/Core/BundledModel.swift | 209 ++ LLM/MLX/Core/GPULifecycle.swift | 90 + LLM/MLX/{ => Core}/MLXTokenizersLoader.swift | 0 LLM/MLX/Core/MemoryStats.swift | 31 + LLM/MLX/Core/RuntimeBudgets.swift | 172 ++ .../Gemma4AudioFeatureExtractor.swift | 0 LLM/MLX/{ => Gemma4}/Gemma4AudioModel.swift | 0 LLM/MLX/{ => Gemma4}/Gemma4Config.swift | 0 .../{ => Gemma4}/Gemma4LanguageModel.swift | 0 LLM/MLX/{ => Gemma4}/Gemma4Model.swift | 72 +- LLM/MLX/{ => Gemma4}/Gemma4Processor.swift | 0 LLM/MLX/{ => Gemma4}/Gemma4Registration.swift | 0 LLM/MLX/{ => Gemma4}/Gemma4RoPE.swift | 0 LLM/MLX/{ => Gemma4}/Gemma4TextModel.swift | 0 LLM/MLX/{ => Gemma4}/Gemma4VisionModel.swift | 0 LLM/MLX/Installation/ModelDownloader.swift | 304 ++ LLM/MLX/Installation/ModelInstallState.swift | 67 + LLM/MLX/Installation/ModelInstaller.swift | 58 + LLM/MLX/Installation/ModelPaths.swift | 65 + LLM/MLX/MLXLocalLLMService.swift | 683 +++++ LLM/MLXLocalLLMService.swift | 1471 ---------- LLM/PromptBuilder.swift | 365 ++- PhoneClaw.xcodeproj/project.pbxproj | 192 +- .../xcshareddata/xcschemes/PhoneClaw.xcscheme | 79 + Skills/Library/calendar/SKILL.md | 67 + Skills/Library/clipboard/SKILL.md | 46 + Skills/Library/contacts/SKILL.md | 124 + Skills/Library/device/SKILL.md | 61 + Skills/Library/reminders/SKILL.md | 47 + Skills/Library/text/SKILL.md | 47 + Skills/Library/translate/SKILL.md | 60 + Skills/SkillLoader.swift | 555 +--- Skills/SkillRegistry.swift | 161 ++ Skills/Skills.swift | 5 +- Skills/ToolRegistry.swift | 1320 --------- Tools/AppPermissions.swift | 186 ++ Tools/Handlers/Calendar.swift | 87 + Tools/Handlers/Clipboard.swift | 146 + Tools/Handlers/Contacts.swift | 435 +++ Tools/Handlers/Device.swift | 179 ++ Tools/Handlers/Reminders.swift | 163 ++ Tools/Handlers/Text.swift | 47 + Tools/Helpers.swift | 94 + Tools/Stores.swift | 9 + Tools/ToolRegistry.swift | 189 ++ UI/ContentView.swift | 45 +- UI/SkillsManagerView.swift | 2 +- doc/architecture-decisions.md | 116 + 58 files changed, 7031 insertions(+), 5687 deletions(-) create mode 100644 Agent/Engine/ChatAttachments.swift create mode 100644 Agent/Engine/ChatMessage.swift create mode 100644 Agent/Engine/OutputCleaner.swift create mode 100644 Agent/Engine/Planner.swift create mode 100644 Agent/Engine/Router.swift create mode 100644 Agent/Engine/SessionStore.swift create mode 100644 Agent/Engine/ToolCallParser.swift create mode 100644 Agent/Engine/ToolChain.swift create mode 100644 LLM/MLX/Core/BundledModel.swift create mode 100644 LLM/MLX/Core/GPULifecycle.swift rename LLM/MLX/{ => Core}/MLXTokenizersLoader.swift (100%) create mode 100644 LLM/MLX/Core/MemoryStats.swift create mode 100644 LLM/MLX/Core/RuntimeBudgets.swift rename LLM/MLX/{ => Gemma4}/Gemma4AudioFeatureExtractor.swift (100%) rename LLM/MLX/{ => Gemma4}/Gemma4AudioModel.swift (100%) rename LLM/MLX/{ => Gemma4}/Gemma4Config.swift (100%) rename LLM/MLX/{ => Gemma4}/Gemma4LanguageModel.swift (100%) rename LLM/MLX/{ => Gemma4}/Gemma4Model.swift (77%) rename LLM/MLX/{ => Gemma4}/Gemma4Processor.swift (100%) rename LLM/MLX/{ => Gemma4}/Gemma4Registration.swift (100%) rename LLM/MLX/{ => Gemma4}/Gemma4RoPE.swift (100%) rename LLM/MLX/{ => Gemma4}/Gemma4TextModel.swift (100%) rename LLM/MLX/{ => Gemma4}/Gemma4VisionModel.swift (100%) create mode 100644 LLM/MLX/Installation/ModelDownloader.swift create mode 100644 LLM/MLX/Installation/ModelInstallState.swift create mode 100644 LLM/MLX/Installation/ModelInstaller.swift create mode 100644 LLM/MLX/Installation/ModelPaths.swift create mode 100644 LLM/MLX/MLXLocalLLMService.swift delete mode 100644 LLM/MLXLocalLLMService.swift create mode 100644 PhoneClaw.xcodeproj/xcshareddata/xcschemes/PhoneClaw.xcscheme create mode 100644 Skills/Library/calendar/SKILL.md create mode 100644 Skills/Library/clipboard/SKILL.md create mode 100644 Skills/Library/contacts/SKILL.md create mode 100644 Skills/Library/device/SKILL.md create mode 100644 Skills/Library/reminders/SKILL.md create mode 100644 Skills/Library/text/SKILL.md create mode 100644 Skills/Library/translate/SKILL.md create mode 100644 Skills/SkillRegistry.swift delete mode 100644 Skills/ToolRegistry.swift create mode 100644 Tools/AppPermissions.swift create mode 100644 Tools/Handlers/Calendar.swift create mode 100644 Tools/Handlers/Clipboard.swift create mode 100644 Tools/Handlers/Contacts.swift create mode 100644 Tools/Handlers/Device.swift create mode 100644 Tools/Handlers/Reminders.swift create mode 100644 Tools/Handlers/Text.swift create mode 100644 Tools/Helpers.swift create mode 100644 Tools/Stores.swift create mode 100644 Tools/ToolRegistry.swift create mode 100644 doc/architecture-decisions.md diff --git a/.gitignore b/.gitignore index 788aac6..68182d1 100644 --- a/.gitignore +++ b/.gitignore @@ -54,3 +54,7 @@ LiteRT-LM/ *.dylib *.a *.o + +# ─── PhoneClawCLI (本地开发测试工具, 不入仓库) ─────────────────── +PhoneClawCLI/ + diff --git a/Agent/AgentEngine.swift b/Agent/AgentEngine.swift index 84d4760..be29b76 100644 --- a/Agent/AgentEngine.swift +++ b/Agent/AgentEngine.swift @@ -7,18 +7,6 @@ func log(_ message: String) { print(message) } -private extension UserInput.Audio { - static func from(snapshot: AudioCaptureSnapshot) -> UserInput.Audio { - .pcm( - .init( - samples: snapshot.pcm, - sampleRate: snapshot.sampleRate, - channelCount: snapshot.channelCount - ) - ) - } -} - // MARK: - 模型/推理配置 @Observable @@ -42,269 +30,38 @@ class ModelConfig { private let kDefaultSystemPrompt = """ 你是 PhoneClaw，一个运行在本地设备上的私人 AI 助手。你完全离线运行，不联网，保护用户隐私。 -你拥有以下能力（Skill）： +你拥有以下两类能力（Skill）： -___SKILLS___ +【设备操作类】（访问 iPhone 硬件或系统数据） +___DEVICE_SKILLS___ -只有当用户明确要求执行某项设备内操作时，才调用 load_skill 加载该能力的详细指令。 -像"配置""信息""看看""帮我查一下"这类含糊词，不足以单独触发工具调用。 -如果用户只是普通聊天、追问上文、让你解释结果，直接回答，不要调用工具。 -如果确实需要某个能力，你必须自己调用 load_skill，不要让用户去"使用某个能力"或"打开某个 skill"。 +【内容处理类】（对文字做变换：翻译/总结/改写 等） +___CONTENT_SKILLS___ -当且仅当确实需要某个能力时，先调用 load_skill： +调用规则： + +▶ 设备操作类 skill： + - 只有用户明确要求执行某项设备操作时，才调用 load_skill。 + - "配置""信息""看看""帮我查一下"这类含糊词，不足以触发。 + - 闲聊、追问上文、解释已有结果时不调用。 + +▶ 内容处理类 skill： + - 只要用户意图是对文字做该类变换（翻译/总结/改写 等），立即调用 load_skill。 + - 即使用户用了"这段""刚才那段""上面"等指代词且没贴出源文本，也必须先调用 load_skill。 + 加载后的指令会告诉你如何从对话历史中定位源文本。**不要**先反问用户。 + +▶ 普通闲聊、追问设备操作结果、解释已经输出的内容：直接回答，不要调用任何 skill。 + +调用格式： {"name": "load_skill", "arguments": {"skill": "能力名"}} -在已经拿到工具结果后，优先直接给出最终答案，不要无谓追问。 +加载 skill 之后请按其指令执行；拿到工具结果后优先直接给最终答案，不要无谓追问。 用中文回答，简洁实用。 """ -// MARK: - 聊天消息 - -struct ChatImageAttachment: Identifiable, Codable { - let id: UUID - let data: Data - private static let storageMaxDimension: CGFloat = 1_024 - private static let compressionQuality: CGFloat = 0.78 - - init(id: UUID = UUID(), data: Data) { - self.id = id - self.data = data - } - - init?(image: UIImage) { - let prepared = Self.preparedImage(image, maxDimension: Self.storageMaxDimension) - if let jpeg = prepared.jpegData(compressionQuality: Self.compressionQuality) { - self.id = UUID() - self.data = jpeg - } else if let png = prepared.pngData() { - self.id = UUID() - self.data = png - } else { - return nil - } - } - - static func preparedImage(_ image: UIImage, maxDimension: CGFloat = storageMaxDimension) -> UIImage { - let originalSize = image.size - let longestSide = max(originalSize.width, originalSize.height) - guard longestSide > maxDimension, longestSide > 0 else { - return image - } - - let scale = maxDimension / longestSide - let targetSize = CGSize( - width: max(1, floor(originalSize.width * scale)), - height: max(1, floor(originalSize.height * scale)) - ) - - let format = UIGraphicsImageRendererFormat.default() - format.opaque = false - format.scale = 1 - let renderer = UIGraphicsImageRenderer(size: targetSize, format: format) - return renderer.image { _ in - image.draw(in: CGRect(origin: .zero, size: targetSize)) - } - } - - var uiImage: UIImage? { - UIImage(data: data) - } - - var ciImage: CIImage? { - if let image = CIImage( - data: data, - options: [.applyOrientationProperty: true] - ) { - return image - } - guard let uiImage else { return nil } - if let ciImage = uiImage.ciImage { - return ciImage - } - if let cgImage = uiImage.cgImage { - return CIImage(cgImage: cgImage) - } - return CIImage(image: uiImage) - } -} - -struct ChatAudioAttachment: Identifiable, Codable { - let id: UUID - let wavData: Data - let duration: TimeInterval - let sampleRate: Double - let waveform: [Float] - - init( - id: UUID = UUID(), - wavData: Data, - duration: TimeInterval, - sampleRate: Double, - waveform: [Float] - ) { - self.id = id - self.wavData = wavData - self.duration = duration - self.sampleRate = sampleRate - self.waveform = waveform - } - - init?(snapshot: AudioCaptureSnapshot) { - guard !snapshot.pcm.isEmpty, snapshot.sampleRate > 0 else { return nil } - self.id = UUID() - self.wavData = Self.makeWAVData( - pcm: snapshot.pcm, - sampleRate: snapshot.sampleRate, - channelCount: 1 - ) - self.duration = snapshot.duration - self.sampleRate = snapshot.sampleRate - self.waveform = Self.makeWaveform(from: snapshot.pcm) - } - - var formattedDuration: String { - let totalSeconds = Int(duration.rounded()) - let minutes = totalSeconds / 60 - let seconds = totalSeconds % 60 - return String(format: "%02d:%02d", minutes, seconds) - } - - private static func makeWaveform(from pcm: [Float], bucketCount: Int = 36) -> [Float] { - guard !pcm.isEmpty else { return Array(repeating: 0.12, count: bucketCount) } - let samplesPerBucket = max(pcm.count / bucketCount, 1) - var levels: [Float] = [] - levels.reserveCapacity(bucketCount) - - var index = 0 - while index < pcm.count { - let end = min(index + samplesPerBucket, pcm.count) - let slice = pcm[index.. bucketCount { - levels = Array(levels.prefix(bucketCount)) - } - - let maxLevel = max(levels.max() ?? 0, 0.001) - return levels.map { max($0 / maxLevel, 0.08) } - } - - private static func makeWAVData( - pcm: [Float], - sampleRate: Double, - channelCount: Int - ) -> Data { - let integerSampleRate = max(Int(sampleRate.rounded()), 1) - let clampedSamples = pcm.map { sample -> Int16 in - let limited = min(max(sample, -1), 1) - return Int16((limited * Float(Int16.max)).rounded()) - } - - let bytesPerSample = MemoryLayout.size - let dataChunkSize = clampedSamples.count * bytesPerSample - let riffChunkSize = 36 + dataChunkSize - let byteRate = integerSampleRate * channelCount * bytesPerSample - let blockAlign = channelCount * bytesPerSample - - var data = Data() - data.reserveCapacity(44 + dataChunkSize) - data.append("RIFF".data(using: .ascii)!) - append(UInt32(riffChunkSize), to: &data) - data.append("WAVE".data(using: .ascii)!) - data.append("fmt ".data(using: .ascii)!) - append(UInt32(16), to: &data) - append(UInt16(1), to: &data) - append(UInt16(channelCount), to: &data) - append(UInt32(integerSampleRate), to: &data) - append(UInt32(byteRate), to: &data) - append(UInt16(blockAlign), to: &data) - append(UInt16(bytesPerSample * 8), to: &data) - data.append("data".data(using: .ascii)!) - append(UInt32(dataChunkSize), to: &data) - - for sample in clampedSamples { - append(sample, to: &data) - } - return data - } - - private static func append(_ value: T, to data: inout Data) { - var littleEndian = value.littleEndian - withUnsafeBytes(of: &littleEndian) { bytes in - data.append(contentsOf: bytes) - } - } -} - -struct ChatMessage: Identifiable, Codable { - let id: UUID - var role: Role - var content: String - var images: [ChatImageAttachment] - var audios: [ChatAudioAttachment] - let timestamp: Date - var skillName: String? = nil - - init( - id: UUID = UUID(), - role: Role, - content: String, - images: [ChatImageAttachment] = [], - audios: [ChatAudioAttachment] = [], - timestamp: Date = Date(), - skillName: String? = nil - ) { - self.id = id - self.role = role - self.content = content - self.images = images - self.audios = audios - self.timestamp = timestamp - self.skillName = skillName - } - - mutating func update(content: String) { - guard self.content != content else { return } - self.content = content - } - - mutating func update(role: Role, content: String, skillName: String? = nil) { - self.role = role - self.content = content - self.skillName = skillName - } - - enum Role: String, Codable { - case user, assistant, system, skillResult - } -} - -struct ChatSessionSummary: Identifiable, Codable, Equatable { - let id: UUID - var title: String - var preview: String - var updatedAt: Date -} - -private struct ChatSessionRecord: Codable { - let id: UUID - var title: String - var preview: String - var updatedAt: Date - var messages: [ChatMessage] -} - // MARK: - Agent Engine @Observable @@ -324,23 +81,23 @@ class AgentEngine { var currentSessionID = UUID() // 文件驱动的 Skill 系统 - let skillLoader = SkillLoader() + let skillRegistry = SkillRegistry() let toolRegistry = ToolRegistry.shared // Skill 条目（给 UI 管理用，可开关） var skillEntries: [SkillEntry] = [] - private let thinkingOpenMarker = "[[PHONECLAW_THINK]]" - private let thinkingCloseMarker = "[[/PHONECLAW_THINK]]" - private let sessionsDirectoryName = "Sessions" - private let sessionsIndexFileName = "sessions_index.json" - private var sessionSaveTask: Task? + let sessionsDirectoryName = "Sessions" + let sessionsIndexFileName = "sessions_index.json" + let plannerRevision = "planner-v3-local-selection" + var sessionSaveTask: Task? var enabledSkillInfos: [SkillInfo] { skillEntries.filter(\.isEnabled).map { SkillInfo(name: $0.id, description: $0.description, - displayName: $0.name, icon: $0.icon, samplePrompt: $0.samplePrompt) + displayName: $0.name, icon: $0.icon, + type: $0.type, samplePrompt: $0.samplePrompt) } } @@ -355,8 +112,8 @@ class AgentEngine { ?? UUID() } - private func loadSkillEntries() { - let definitions = skillLoader.discoverSkills() + func loadSkillEntries() { + let definitions = skillRegistry.discoverSkills() self.skillEntries = definitions.map { SkillEntry(from: $0, registry: toolRegistry) } } @@ -366,1410 +123,40 @@ class AgentEngine { for i in skillEntries.indices { if let wasEnabled = enabledState[skillEntries[i].id] { skillEntries[i].isEnabled = wasEnabled - skillLoader.setEnabled(skillEntries[i].id, enabled: wasEnabled) + skillRegistry.setEnabled(skillEntries[i].id, enabled: wasEnabled) } } } // MARK: - Skill 查找（文件驱动） - private func findSkillId(for name: String) -> String? { - let resolvedName = skillLoader.canonicalSkillId(for: name) - if skillLoader.getDefinition(resolvedName) != nil { return resolvedName } - return skillLoader.findSkillId(forTool: name) + func findSkillId(for name: String) -> String? { + let resolvedName = skillRegistry.canonicalSkillId(for: name) + if skillRegistry.getDefinition(resolvedName) != nil { return resolvedName } + return skillRegistry.findSkillId(forTool: name) } - private func findDisplayName(for name: String) -> String { + func findDisplayName(for name: String) -> String { if let skillId = findSkillId(for: name), - let def = skillLoader.getDefinition(skillId) { + let def = skillRegistry.getDefinition(skillId) { return def.metadata.displayName } return name } - private func handleLoadSkill(skillName: String) -> String? { - let resolvedSkillName = skillLoader.canonicalSkillId(for: skillName) + func handleLoadSkill(skillName: String) -> String? { + let resolvedSkillName = skillRegistry.canonicalSkillId(for: skillName) guard let entry = skillEntries.first(where: { $0.id == resolvedSkillName }), entry.isEnabled else { return nil } - return skillLoader.loadBody(skillId: resolvedSkillName) + return skillRegistry.loadBody(skillId: resolvedSkillName) } - private func canonicalToolName(_ toolName: String, arguments: [String: Any]) -> String { - switch toolName { - case "contacts": - if arguments["action"] as? String == "delete" - || arguments["delete"] as? Bool == true { - return "contacts-delete" - } - if arguments["phone"] != nil - || arguments["company"] != nil - || arguments["notes"] != nil { - return "contacts-upsert" - } - if arguments["identifier"] != nil - || arguments["name"] != nil - || arguments["email"] != nil - || arguments["query"] != nil { - return "contacts-search" - } - return "contacts-search" - case "contacts_delete", "contacts-delete-contact": - return "contacts-delete" - case "contacts_upsert": - return "contacts-upsert" - case "contacts_search": - return "contacts-search" - default: - return toolName - } - } - - private func handleToolExecution(toolName: String, args: [String: Any]) async throws -> String { + func handleToolExecution(toolName: String, args: [String: Any]) async throws -> String { return try await toolRegistry.execute(name: toolName, args: args) } - private func autoToolCallForLoadedSkills( - skillIds: [String] - ) -> (name: String, arguments: [String: Any])? { - let uniqueSkillIds = Array(NSOrderedSet(array: skillIds)) as? [String] ?? skillIds - - guard uniqueSkillIds.count == 1, - let skillId = uniqueSkillIds.first, - let def = skillLoader.getDefinition(skillId), - def.isEnabled else { - return nil - } - - let uniqueToolNames = Array(NSOrderedSet(array: def.metadata.allowedTools)) as? [String] - ?? def.metadata.allowedTools - guard uniqueToolNames.count == 1, - let toolName = uniqueToolNames.first, - let tool = toolRegistry.find(name: toolName), - tool.parameters == "无" else { - return nil - } - - return (tool.name, [:]) - } - - private func registeredTools(for skillId: String) -> [RegisteredTool] { - if let def = skillLoader.getDefinition(skillId) { - let tools = toolRegistry.toolsFor(names: def.metadata.allowedTools) - if !tools.isEmpty { return tools } - } - - if let entry = skillEntries.first(where: { $0.id == skillId }) { - let tools = entry.tools.compactMap { toolRegistry.find(name: $0.name) } - if !tools.isEmpty { return tools } - } - - return [] - } - - private func inferFallbackToolCall( - skillIds: [String], - userQuestion: String - ) -> (name: String, arguments: [String: Any])? { - let uniqueSkillIds = Array(NSOrderedSet(array: skillIds)) as? [String] ?? skillIds - guard uniqueSkillIds.count == 1, - let skillId = uniqueSkillIds.first else { - return nil - } - - let tools = registeredTools(for: skillId) - guard !tools.isEmpty else { return nil } - - if tools.count == 1, tools[0].parameters == "无" { - return (tools[0].name, [:]) - } - - let normalizedQuestion = userQuestion.lowercased() - - func has(_ keyword: String) -> Bool { - normalizedQuestion.contains(keyword) - } - - let candidateNames: [String] - switch skillId { - case "device": - if has("系统版本") || has("ios 版本") || has("版本号") { - candidateNames = ["device-system-version", "device-info"] - } else if has("名字") || has("名称") || has("叫什么") { - candidateNames = ["device-name", "device-info"] - } else if has("型号") || has("机型") { - candidateNames = ["device-model", "device-info"] - } else if has("内存") || has("ram") { - candidateNames = ["device-memory", "device-info"] - } else if has("处理器") || has("核心") || has("cpu") { - candidateNames = ["device-processor-count", "device-info"] - } else { - candidateNames = ["device-info"] - } - case "contacts": - let searchKeywords = [ - "查", "检查", "查询", "看看", "电话", "手机号", "号码", - "联系方式", "邮箱", "mail", "email" - ] - let deleteKeywords = [ - "删除", "删掉", "删了", "删吗", "删", "移除", "去掉", "清除" - ] - let upsertKeywords = [ - "存", "保存", "添加", "新建", "创建", "记一下", "记住", "更新", "修改" - ] - - if deleteKeywords.contains(where: has) { - candidateNames = ["contacts-delete"] - } else if upsertKeywords.contains(where: has) { - candidateNames = ["contacts-upsert"] - } else if searchKeywords.contains(where: has) { - candidateNames = ["contacts-search"] - } else { - candidateNames = ["contacts-upsert", "contacts-search", "contacts-delete"] - } - case "clipboard": - let writeKeywords = [ - "复制", "拷贝", "写入", "放到剪贴板", "存到剪贴板", "复制到剪贴板" - ] - let readKeywords = [ - "剪贴板", "读一下", "读取", "看看", "看下", "查看", "内容" - ] - - if writeKeywords.contains(where: has), - let arguments = heuristicArgumentsForTool(toolName: "clipboard-write", userQuestion: userQuestion), - validateSingleToolArguments(toolName: "clipboard-write", arguments: arguments) { - return ("clipboard-write", arguments) - } - - if readKeywords.contains(where: has) { - candidateNames = ["clipboard-read"] - } else { - candidateNames = ["clipboard-read", "clipboard-write"] - } - case "calendar": - let actionKeywords = [ - "安排", "创建", "新建", "加入", "加到", "写进", "写入", "记到" - ] - let createIntent = - (has("日历") || has("日程") || has("会议") || has("约会")) - && actionKeywords.contains(where: has) - candidateNames = createIntent ? ["calendar-create-event"] : [] - case "reminders": - let actionKeywords = [ - "提醒", "提醒我", "记得", "待办", "创建", "新建", "添加" - ] - let createIntent = actionKeywords.contains(where: has) - candidateNames = createIntent ? ["reminders-create"] : [] - case "text": - if has("哈希") || has("hash") { - candidateNames = ["calculate-hash"] - } else if has("翻转") || has("反转") { - candidateNames = ["text-reverse"] - } else { - candidateNames = ["calculate-hash", "text-reverse"] - } - default: - candidateNames = [] - } - - for name in candidateNames { - guard let tool = tools.first(where: { $0.name == name }) else { continue } - if tool.parameters == "无" { - return (tool.name, [:]) - } - if let arguments = heuristicArgumentsForTool(toolName: tool.name, userQuestion: userQuestion), - validateSingleToolArguments(toolName: tool.name, arguments: arguments) { - return (tool.name, arguments) - } - } - - return nil - } - - private enum SingleToolExtractionOutcome { - case toolCall(name: String, arguments: [String: Any]) - case needsClarification(String) - case failed - } - - private func singleRegisteredToolForLoadedSkills(skillIds: [String]) -> RegisteredTool? { - let uniqueSkillIds = Array(NSOrderedSet(array: skillIds)) as? [String] ?? skillIds - guard uniqueSkillIds.count == 1, - let skillId = uniqueSkillIds.first else { - return nil - } - - let tools = registeredTools(for: skillId) - guard tools.count == 1 else { return nil } - return tools.first - } - - private func parseJSONObject(_ text: String) -> [String: Any]? { - let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) - guard !trimmed.isEmpty else { return nil } - - let candidates: [String] = { - if trimmed.hasPrefix("```json") || trimmed.hasPrefix("```") { - let stripped = trimmed - .replacingOccurrences(of: "```json", with: "") - .replacingOccurrences(of: "```", with: "") - .trimmingCharacters(in: .whitespacesAndNewlines) - return [stripped] - } - - if let start = trimmed.firstIndex(of: "{"), - let end = trimmed.lastIndex(of: "}"), - start <= end { - return [trimmed, String(trimmed[start...end])] - } - - return [trimmed] - }() - - for candidate in candidates { - guard let data = candidate.data(using: .utf8), - let object = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { - continue - } - return object - } - - return nil - } - - private func iso8601StringForModel(from date: Date) -> String { - let formatter = ISO8601DateFormatter() - formatter.timeZone = .current - formatter.formatOptions = [.withInternetDateTime] - return formatter.string(from: date) - } - - private func chineseNumberValue(_ token: String) -> Int? { - if let value = Int(token) { return value } - - let digits: [Character: Int] = [ - "零": 0, "〇": 0, "一": 1, "二": 2, "两": 2, "三": 3, "四": 4, - "五": 5, "六": 6, "七": 7, "八": 8, "九": 9 - ] - - if token == "十" { return 10 } - if token.hasPrefix("十"), let last = token.last, let digit = digits[last] { - return 10 + digit - } - if token.hasSuffix("十"), let first = token.first, let digit = digits[first] { - return digit * 10 - } - if token.count == 2 { - let chars = Array(token) - if let tens = digits[chars[0]], chars[1] == "十" { - return tens * 10 - } - } - if token.count == 3 { - let chars = Array(token) - if let tens = digits[chars[0]], chars[1] == "十", let ones = digits[chars[2]] { - return tens * 10 + ones - } - } - - return nil - } - - private func parseBasicChineseDate(from text: String) -> Date? { - let patterns = [ - "(今天|今日|今晚|今夜|明天|明晚|后天)(?:的)?(凌晨|早上|上午|中午|下午|晚上|傍晚)?([零〇一二两三四五六七八九十\\d]{1,3})点(?:(半)|([零〇一二两三四五六七八九十\\d]{1,3})分?)?", - "(凌晨|早上|上午|中午|下午|晚上|傍晚)([零〇一二两三四五六七八九十\\d]{1,3})点(?:(半)|([零〇一二两三四五六七八九十\\d]{1,3})分?)?" - ] - - for pattern in patterns { - guard let regex = try? NSRegularExpression(pattern: pattern) else { continue } - let range = NSRange(text.startIndex..., in: text) - guard let match = regex.firstMatch(in: text, range: range) else { continue } - - var dayToken: String? - var periodToken: String? - var hourToken: String? - var hasHalf = false - var minuteToken: String? - - if match.numberOfRanges >= 6 { - if let r = Range(match.range(at: 1), in: text), !r.isEmpty { - dayToken = String(text[r]) - } - if let r = Range(match.range(at: 2), in: text), !r.isEmpty { - periodToken = String(text[r]) - } - if let r = Range(match.range(at: 3), in: text), !r.isEmpty { - hourToken = String(text[r]) - } - if let r = Range(match.range(at: 4), in: text), !r.isEmpty { - hasHalf = true - } - if let r = Range(match.range(at: 5), in: text), !r.isEmpty { - minuteToken = String(text[r]) - } - } - - if hourToken == nil, match.numberOfRanges >= 5 { - if let r = Range(match.range(at: 1), in: text), !r.isEmpty { - periodToken = String(text[r]) - } - if let r = Range(match.range(at: 2), in: text), !r.isEmpty { - hourToken = String(text[r]) - } - if let r = Range(match.range(at: 3), in: text), !r.isEmpty { - hasHalf = true - } - if let r = Range(match.range(at: 4), in: text), !r.isEmpty { - minuteToken = String(text[r]) - } - } - - guard let hourToken, - var hour = chineseNumberValue(hourToken) else { - continue - } - - var minute = 0 - if hasHalf { - minute = 30 - } else if let minuteToken, let parsedMinute = chineseNumberValue(minuteToken) { - minute = parsedMinute - } - - if let periodToken { - switch periodToken { - case "下午", "晚上", "傍晚": - if hour < 12 { hour += 12 } - case "中午": - if hour < 11 { hour += 12 } - case "凌晨": - if hour == 12 { hour = 0 } - default: - break - } - } - - var dayOffset = 0 - switch dayToken { - case "明天", "明晚": - dayOffset = 1 - case "后天": - dayOffset = 2 - default: - dayOffset = 0 - } - - let calendar = Calendar.current - let baseDate = calendar.startOfDay(for: Date()) - guard let day = calendar.date(byAdding: .day, value: dayOffset, to: baseDate), - let finalDate = calendar.date( - bySettingHour: hour, - minute: minute, - second: 0, - of: day - ) else { - continue - } - return finalDate - } - - return nil - } - - private func detectDateInQuestion(_ text: String) -> Date? { - if let detector = try? NSDataDetector(types: NSTextCheckingResult.CheckingType.date.rawValue) { - let range = NSRange(text.startIndex..., in: text) - if let match = detector.firstMatch(in: text, options: [], range: range), - let date = match.date { - return date - } - } - return parseBasicChineseDate(from: text) - } - - private func heuristicArgumentsForTool( - toolName: String, - userQuestion: String - ) -> [String: Any]? { - let text = userQuestion.trimmingCharacters(in: .whitespacesAndNewlines) - guard !text.isEmpty else { return nil } - - switch toolName { - case "calculate-hash": - let patterns = [ - "(?:计算|求|生成|算一下|帮我算)(.+?)(?:的)?(?:哈希|hash)(?:值)?", - "(.+?)(?:的)?(?:哈希|hash)(?:值)?(?:是多少|是什么)?" - ] - - var extracted: String? - for pattern in patterns { - guard let regex = try? NSRegularExpression(pattern: pattern, options: [.caseInsensitive]) else { continue } - let range = NSRange(text.startIndex..., in: text) - if let match = regex.firstMatch(in: text, range: range), - let capture = Range(match.range(at: 1), in: text) { - let value = String(text[capture]) - .trimmingCharacters(in: .whitespacesAndNewlines) - .trimmingCharacters(in: CharacterSet(charactersIn: "“”\"' ")) - if !value.isEmpty { - extracted = value - break - } - } - } - - if extracted == nil { - let cleaned = text - .replacingOccurrences(of: "帮我", with: "") - .replacingOccurrences(of: "计算", with: "") - .replacingOccurrences(of: "求", with: "") - .replacingOccurrences(of: "生成", with: "") - .replacingOccurrences(of: "算一下", with: "") - .replacingOccurrences(of: "哈希值", with: "") - .replacingOccurrences(of: "哈希", with: "") - .replacingOccurrences(of: "hash", with: "", options: .caseInsensitive) - .trimmingCharacters(in: .whitespacesAndNewlines) - .trimmingCharacters(in: CharacterSet(charactersIn: "“”\"' ")) - if !cleaned.isEmpty { - extracted = cleaned - } - } - - guard let extracted, !extracted.isEmpty else { return nil } - return ["text": extracted] - - case "text-reverse": - let patterns = [ - "(?:把|将)(.+?)(?:翻转|反转)(?:过来|一下)?", - "(?:翻转|反转)(.+?)(?:过来|一下)?$" - ] - - var extracted: String? - for pattern in patterns { - guard let regex = try? NSRegularExpression(pattern: pattern) else { continue } - let range = NSRange(text.startIndex..., in: text) - if let match = regex.firstMatch(in: text, range: range), - let capture = Range(match.range(at: 1), in: text) { - let value = String(text[capture]) - .trimmingCharacters(in: .whitespacesAndNewlines) - .trimmingCharacters(in: CharacterSet(charactersIn: "“”\"' ")) - if !value.isEmpty { - extracted = value - break - } - } - } - - if extracted == nil { - let cleaned = text - .replacingOccurrences(of: "帮我", with: "") - .replacingOccurrences(of: "把", with: "") - .replacingOccurrences(of: "将", with: "") - .replacingOccurrences(of: "翻转过来", with: "") - .replacingOccurrences(of: "反转过来", with: "") - .replacingOccurrences(of: "翻转一下", with: "") - .replacingOccurrences(of: "反转一下", with: "") - .replacingOccurrences(of: "翻转", with: "") - .replacingOccurrences(of: "反转", with: "") - .trimmingCharacters(in: .whitespacesAndNewlines) - .trimmingCharacters(in: CharacterSet(charactersIn: "“”\"' ")) - if !cleaned.isEmpty { - extracted = cleaned - } - } - - guard let extracted, !extracted.isEmpty else { return nil } - return ["text": extracted] - - case "clipboard-write": - let patterns = [ - "(?:复制|拷贝|写入)(.+?)(?:到|进|到系统)?(?:剪贴板)", - "(?:把|将)(.+?)(?:复制|拷贝|写入)(?:到|进)?(?:剪贴板)" - ] - - var extracted: String? - for pattern in patterns { - guard let regex = try? NSRegularExpression(pattern: pattern) else { continue } - let range = NSRange(text.startIndex..., in: text) - if let match = regex.firstMatch(in: text, range: range), - let capture = Range(match.range(at: 1), in: text) { - let value = String(text[capture]) - .trimmingCharacters(in: .whitespacesAndNewlines) - .trimmingCharacters(in: CharacterSet(charactersIn: "“”\"' ")) - if !value.isEmpty { - extracted = value - break - } - } - } - - if extracted == nil { - let cleaned = text - .replacingOccurrences(of: "帮我", with: "") - .replacingOccurrences(of: "把", with: "") - .replacingOccurrences(of: "将", with: "") - .replacingOccurrences(of: "复制到剪贴板", with: "") - .replacingOccurrences(of: "拷贝到剪贴板", with: "") - .replacingOccurrences(of: "写入剪贴板", with: "") - .replacingOccurrences(of: "放到剪贴板", with: "") - .replacingOccurrences(of: "存到剪贴板", with: "") - .replacingOccurrences(of: "复制", with: "") - .replacingOccurrences(of: "拷贝", with: "") - .replacingOccurrences(of: "写入", with: "") - .replacingOccurrences(of: "剪贴板", with: "") - .trimmingCharacters(in: .whitespacesAndNewlines) - .trimmingCharacters(in: CharacterSet(charactersIn: "“”\"' ")) - if !cleaned.isEmpty { - extracted = cleaned - } - } - - guard let extracted, !extracted.isEmpty else { return nil } - return ["text": extracted] - - case "contacts-upsert": - let phone = text.firstMatch(of: /1[3-9]\d{9}/).map { String($0.0) } - var name: String? - - let patterns = [ - "(?:把|将)?(.+?)(?:的)?(?:电话|手机号|号码|联系方式)\\s*(?:1[3-9]\\d{9})\\s*(?:添加到|加到|存到|保存到)?(?:联系人|通讯录)", - "(?:帮我)?(?:存(?:一下)?|保存|记一下)(.+?)(?:的)?(?:电话|手机号|号码|联系方式)", - "(?:联系人|通讯录)(?:里)?(?:添加|保存)?(.+?)(?:的)?(?:电话|手机号|号码|联系方式)" - ] - for pattern in patterns { - guard let regex = try? NSRegularExpression(pattern: pattern) else { continue } - let range = NSRange(text.startIndex..., in: text) - if let match = regex.firstMatch(in: text, range: range), - let capture = Range(match.range(at: 1), in: text) { - name = String(text[capture]).trimmingCharacters(in: .whitespacesAndNewlines) - break - } - } - - if name == nil, - let phone, - let phoneRange = text.range(of: phone) { - let prefix = text[.. Bool { - switch toolName { - case "calendar-create-event": - return arguments["title"] is String && arguments["start"] is String - case "reminders-create": - return arguments["title"] is String - case "calculate-hash", "text-reverse": - return arguments["text"] is String - case "contacts-upsert": - return arguments["name"] is String - case "contacts-search", "contacts-delete": - return arguments["query"] is String - || arguments["identifier"] is String - || arguments["name"] is String - || arguments["phone"] is String - || arguments["email"] is String - case "clipboard-write": - return arguments["text"] is String - default: - return !arguments.isEmpty - } - } - - private func shouldSkipToolFollowUpModel(for toolName: String) -> Bool { - switch toolName { - case "clipboard-read", "clipboard-write": - return true - default: - return false - } - } - - private func preflightSkillLoadCall(for userQuestion: String) -> String? { - let normalizedQuestion = userQuestion.lowercased() - - func has(_ keyword: String) -> Bool { - normalizedQuestion.contains(keyword) - } - - if (has("哈希") || has("hash")), - let arguments = heuristicArgumentsForTool(toolName: "calculate-hash", userQuestion: userQuestion), - validateSingleToolArguments(toolName: "calculate-hash", arguments: arguments) { - return syntheticToolCallText( - name: "calculate-hash", - arguments: arguments - ) - } - - if (has("翻转") || has("反转")), - let arguments = heuristicArgumentsForTool(toolName: "text-reverse", userQuestion: userQuestion), - validateSingleToolArguments(toolName: "text-reverse", arguments: arguments) { - return syntheticToolCallText( - name: "text-reverse", - arguments: arguments - ) - } - - let mentionsClipboard = has("剪贴板") || has("clipboard") - if mentionsClipboard { - let readKeywords = ["读取", "读一下", "看看", "看下", "查看", "内容", "是什么"] - let writeKeywords = ["复制", "拷贝", "写入", "放到剪贴板", "存到剪贴板", "复制到剪贴板"] - - if writeKeywords.contains(where: has), - let arguments = heuristicArgumentsForTool(toolName: "clipboard-write", userQuestion: userQuestion), - validateSingleToolArguments(toolName: "clipboard-write", arguments: arguments) { - return syntheticToolCallText( - name: "clipboard-write", - arguments: arguments - ) - } - - if readKeywords.contains(where: has) || mentionsClipboard { - return syntheticToolCallText( - name: "load_skill", - arguments: ["skill": "clipboard"] - ) - } - } - - let calendarIntent = - (has("日历") || has("日程") || has("会议") || has("约会")) - && ["安排", "创建", "新建", "加入", "加到", "记到", "写进", "写入"].contains(where: has) - if calendarIntent { - if let arguments = heuristicArgumentsForTool(toolName: "calendar-create-event", userQuestion: userQuestion), - validateSingleToolArguments(toolName: "calendar-create-event", arguments: arguments) { - return syntheticToolCallText( - name: "calendar-create-event", - arguments: arguments - ) - } - return syntheticToolCallText( - name: "load_skill", - arguments: ["skill": "calendar"] - ) - } - - let reminderIntent = ["提醒我", "提醒", "记得", "待办", "提醒事项"].contains(where: has) - if reminderIntent { - if let arguments = heuristicArgumentsForTool(toolName: "reminders-create", userQuestion: userQuestion), - validateSingleToolArguments(toolName: "reminders-create", arguments: arguments) { - return syntheticToolCallText( - name: "reminders-create", - arguments: arguments - ) - } - return syntheticToolCallText( - name: "load_skill", - arguments: ["skill": "reminders"] - ) - } - - return nil - } - - private func extractToolCallForLoadedSkills( - originalPrompt: String, - userQuestion: String, - skillInstructions: String, - skillIds: [String], - images: [CIImage] - ) async -> SingleToolExtractionOutcome { - let uniqueSkillIds = Array(NSOrderedSet(array: skillIds)) as? [String] ?? skillIds - guard uniqueSkillIds.count == 1, - let skillId = uniqueSkillIds.first else { - return .failed - } - - let tools = registeredTools(for: skillId) - .filter { $0.parameters != "无" } - guard !tools.isEmpty else { - return .failed - } - - if tools.count == 1, let tool = tools.first { - if let heuristic = heuristicArgumentsForTool(toolName: tool.name, userQuestion: userQuestion), - validateSingleToolArguments(toolName: tool.name, arguments: heuristic) { - log("[Agent] load_skill heuristic 直接执行工具: \(tool.name)") - return .toolCall(name: tool.name, arguments: heuristic) - } - - let extractionPrompt = PromptBuilder.buildSingleToolArgumentsPrompt( - originalPrompt: originalPrompt, - userQuestion: userQuestion, - skillInstructions: skillInstructions, - toolName: tool.name, - toolParameters: tool.parameters, - currentImageCount: images.count - ) - - if let raw = await streamLLM(prompt: extractionPrompt, images: images) { - let cleaned = cleanOutput(raw) - if let payload = parseJSONObject(cleaned) { - if let clarification = payload["_needs_clarification"] as? String, - !clarification.trimmingCharacters(in: CharacterSet.whitespacesAndNewlines).isEmpty { - return .needsClarification(clarification) - } - - if validateSingleToolArguments(toolName: tool.name, arguments: payload) { - return .toolCall(name: tool.name, arguments: payload) - } - } - } - - return .failed - } - - let allowedToolsSummary = tools.map { - "- \($0.name): \($0.description)\n 参数: \($0.parameters)" - }.joined(separator: "\n") - - let extractionPrompt = PromptBuilder.buildSkillToolSelectionPrompt( - originalPrompt: originalPrompt, - userQuestion: userQuestion, - skillInstructions: skillInstructions, - allowedToolsSummary: allowedToolsSummary, - currentImageCount: images.count - ) - - if let raw = await streamLLM(prompt: extractionPrompt, images: images) { - let cleaned = cleanOutput(raw) - if let payload = parseJSONObject(cleaned) { - if let clarification = payload["_needs_clarification"] as? String, - !clarification.trimmingCharacters(in: CharacterSet.whitespacesAndNewlines).isEmpty { - return .needsClarification(clarification) - } - - if let rawName = payload["name"] as? String, - let arguments = payload["arguments"] as? [String: Any] { - let toolName = canonicalToolName(rawName, arguments: arguments) - if tools.contains(where: { $0.name == toolName }), - validateSingleToolArguments(toolName: toolName, arguments: arguments) { - return .toolCall(name: toolName, arguments: arguments) - } - } - } - } - - if let heuristic = inferFallbackToolCall(skillIds: skillIds, userQuestion: userQuestion), - tools.contains(where: { $0.name == heuristic.name }), - validateSingleToolArguments(toolName: heuristic.name, arguments: heuristic.arguments) { - return .toolCall(name: heuristic.name, arguments: heuristic.arguments) - } - - return .failed - } - - private func markSkillsDone(_ displayNames: [String]) { - guard !displayNames.isEmpty else { return } - for index in messages.indices { - guard messages[index].role == .system, - let skillName = messages[index].skillName, - displayNames.contains(skillName), - messages[index].content == "identified" || messages[index].content == "loaded" else { - continue - } - messages[index].update(role: .system, content: "done", skillName: skillName) - } - } - - private func looksLikeStructuredIntermediateOutput(_ text: String) -> Bool { - let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) - guard !trimmed.isEmpty else { return false } - - if trimmed.hasPrefix("```json") || trimmed.hasPrefix("```") { - return true - } - - if let regex = try? NSRegularExpression( - pattern: "\"[A-Za-z_][A-Za-z0-9_]*\"\\s*:", - options: [] - ) { - let matchCount = regex.numberOfMatches( - in: trimmed, - range: NSRange(trimmed.startIndex..., in: trimmed) - ) - if matchCount >= 2 && !trimmed.hasPrefix("{") { - return true - } - } - - let suspiciousFragments = [ - "tool_name\":", - "result_for_user_name\":", - "text_for_display\":", - "tool_operation_success\":", - "arguments_for_tool_no_skill\":", - "memory_user_power_conversion\":" - ] - if suspiciousFragments.filter({ trimmed.contains($0) }).count >= 2 { - return true - } - - guard let data = trimmed.data(using: .utf8), - let json = try? JSONSerialization.jsonObject(with: data) else { - return false - } - - if let dict = json as? [String: Any] { - if dict["name"] != nil { - return false - } - - let suspiciousKeys = [ - "final_answer", "tool_call", "arguments", "device_call", - "next_action", "action", "tool" - ] - return suspiciousKeys.contains { dict[$0] != nil } - } - - return false - } - - private func looksLikePromptEcho(_ text: String) -> Bool { - let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) - guard !trimmed.isEmpty else { return false } - - if trimmed.hasPrefix("user\n") || trimmed == "user" { - return true - } - - let suspiciousPhrases = [ - "根据已加载的 Skill", - "不要将任何关于工具、系统或该请求的描述变成 Markdown 代码或 JSON 模板", - "如果需要，请直接调用", - "package_name", - "text_for_user" - ] - - let hitCount = suspiciousPhrases.reduce(into: 0) { count, phrase in - if trimmed.contains(phrase) { count += 1 } - } - return hitCount >= 2 - } - - private func syntheticToolCallText( - name: String, - arguments: [String: Any] - ) -> String { - let jsonData = try? JSONSerialization.data(withJSONObject: [ - "name": name, - "arguments": arguments - ]) - let jsonString = jsonData.flatMap { String(data: $0, encoding: .utf8) } - ?? "{\"name\":\"\(name)\",\"arguments\":{}}" - return """ - - \(jsonString) - - """ - } - - private func parsedToolPayload(from toolResult: String) -> [String: Any]? { - guard let data = toolResult.data(using: .utf8), - let payload = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { - return nil - } - return payload - } - - private func toolResultSummaryForModel( - toolName: String, - toolResult: String - ) -> String { - let trimmed = toolResult.trimmingCharacters(in: .whitespacesAndNewlines) - guard !trimmed.isEmpty else { return "工具 \(toolName) 已执行，但没有返回内容。" } - - if let payload = parsedToolPayload(from: trimmed) { - if let success = payload["success"] as? Bool, - !success, - let error = payload["error"] as? String, - !error.isEmpty { - return "工具 \(toolName) 执行失败：\(error)" - } - - if let result = payload["result"] as? String { - let summary = result.trimmingCharacters(in: .whitespacesAndNewlines) - if !summary.isEmpty { return summary } - } - } - - if let rendered = renderToolResultLocally(toolName: toolName, toolResult: trimmed) { - return rendered - } - - return trimmed - } - - private func fallbackReplyForEmptyToolFollowUp(toolName: String, toolResult: String) -> String { - let trimmed = toolResult.trimmingCharacters(in: .whitespacesAndNewlines) - let summary = toolResultSummaryForModel(toolName: toolName, toolResult: trimmed) - if !summary.isEmpty, summary != trimmed { - return summary - } - - if trimmed.isEmpty { - return "工具 \(toolName) 已执行，但没有返回内容。" - } - - return """ - 工具 \(toolName) 已执行完成，但模型没有生成最终回答。 - 工具返回结果： - \(trimmed) - """ - } - - private func renderToolResultLocally( - toolName: String, - toolResult: String - ) -> String? { - guard let payload = parsedToolPayload(from: toolResult), - let success = payload["success"] as? Bool, - success else { - return nil - } - - func string(_ key: String) -> String? { - if let value = payload[key] as? String, !value.isEmpty { return value } - return nil - } - - func int(_ key: String) -> Int? { - if let value = payload[key] as? Int { return value } - if let value = payload[key] as? Double { return Int(value) } - if let value = payload[key] as? String, let intValue = Int(value) { return intValue } - return nil - } - - func double(_ key: String) -> Double? { - if let value = payload[key] as? Double { return value } - if let value = payload[key] as? Int { return Double(value) } - if let value = payload[key] as? String, let doubleValue = Double(value) { return doubleValue } - return nil - } - - switch toolName { - case "device-info": - var lines: [String] = [] - if let name = string("name") { - lines.append("设备名称：\(name)") - } - if let localizedModel = string("localized_model") ?? string("model") { - lines.append("设备类型：\(localizedModel)") - } - if let systemName = string("system_name"), - let systemVersion = string("system_version") { - lines.append("系统版本：\(systemName) \(systemVersion)") - } else if let systemVersion = string("system_version") { - lines.append("系统版本：\(systemVersion)") - } - if let memoryGB = double("memory_gb") { - lines.append(String(format: "物理内存：%.1f GB", memoryGB)) - } - if let processorCount = int("processor_count") { - lines.append("处理器核心数：\(processorCount)") - } - return lines.isEmpty ? nil : lines.joined(separator: "\n") - - case "device-name": - if let name = string("name") { - return "这台设备的名称是 \(name)。" - } - - case "device-model": - if let localizedModel = string("localized_model") ?? string("model") { - return "这台设备的官方设备类型是 \(localizedModel)。" - } - - case "device-system-version": - if let systemName = string("system_name"), - let systemVersion = string("system_version") { - return "当前系统版本是 \(systemName) \(systemVersion)。" - } - - case "device-memory": - if let memoryGB = double("memory_gb") { - return String(format: "这台设备的物理内存约为 %.1f GB。", memoryGB) - } - - case "device-processor-count": - if let processorCount = int("processor_count") { - return "这台设备的处理器核心数是 \(processorCount)。" - } - - case "device-identifier-for-vendor": - if let identifier = string("identifier_for_vendor") { - return "当前 App 在这台设备上的 identifierForVendor 是 \(identifier)。" - } - - case "clipboard-read": - if let content = string("content") { - return "剪贴板当前内容是：\(content)" - } - - case "clipboard-write": - if let copiedLength = int("copied_length") { - return "已写入剪贴板，共 \(copiedLength) 个字符。" - } - - case "text-reverse": - if let reversed = string("reversed") { - return "翻转结果：\(reversed)" - } - - case "calculate-hash": - if let hash = payload["hash"] { - return "哈希值是 \(hash)。" - } - - case "calendar-create-event": - if let title = string("title"), - let start = string("start") { - var parts = ["已创建日历事项“\(title)”", "开始时间是 \(start)"] - if let location = string("location") { - parts.append("地点是 \(location)") - } - return parts.joined(separator: "，") + "。" - } - - case "reminders-create": - if let title = string("title") { - if let due = string("due") { - return "已创建提醒事项“\(title)”，提醒时间是 \(due)。" - } - return "已创建提醒事项“\(title)”。" - } - - case "contacts-upsert": - if let name = string("name") { - let action = string("action") == "updated" ? "已更新" : "已创建" - var parts = ["\(action)联系人“\(name)”"] - if let phone = string("phone") { - parts.append("手机号是 \(phone)") - } - if let company = string("company") { - parts.append("公司是 \(company)") - } - return parts.joined(separator: "，") + "。" - } - - case "contacts-search": - if let result = string("result") { - return result - } - - case "contacts-delete": - if let result = string("result") { - return result - } - - default: - break - } - - return nil - } - - private func fallbackReplyForEmptySkillFollowUp(skillName: String) -> String { - "Skill \(skillName) 已加载，但模型没有继续生成工具调用或最终回答。请重试，或把问题说得更具体一些。" - } - - private func shouldUseToolingPrompt(for userQuestion: String) -> Bool { - let normalizedQuestion = userQuestion.lowercased() - guard !normalizedQuestion.isEmpty else { return false } - - let domainKeywords = [ - "日历", "提醒", "提醒事项", "通讯录", "联系人", "剪贴板", - "设备", "系统", "相册", "照片", "蓝牙", "wifi", "电量", - "亮度", "音量", "电话", "短信", "邮件" - ] - let actionKeywords = [ - "打开", "查看", "读取", "搜索", "查询", "创建", - "新建", "添加", "保存", "修改", "删除", "复制", - "拨打", "发送", "设置", "调高", "调低" - ] - - let mentionsDomain = domainKeywords.contains { normalizedQuestion.contains($0) } - let mentionsAction = actionKeywords.contains { normalizedQuestion.contains($0) } - if mentionsDomain && mentionsAction { - return true - } - - for entry in skillEntries where entry.isEnabled { - if normalizedQuestion.contains(entry.id.lowercased()) - || normalizedQuestion.contains(entry.name.lowercased()) - { - return true - } - - if entry.tools.contains(where: { normalizedQuestion.contains($0.name.lowercased()) }) { - return true - } - - if let definition = skillLoader.getDefinition(entry.id), - definition.metadata.triggers.contains(where: { trigger in - let normalizedTrigger = trigger.lowercased().trimmingCharacters(in: .whitespacesAndNewlines) - return !normalizedTrigger.isEmpty && normalizedQuestion.contains(normalizedTrigger) - }) - { - return true - } - } - - return mentionsDomain - } - // MARK: - 初始化 /// ConfigurationsView 的"Restore default"按钮使用。 @@ -1788,6 +175,8 @@ class AgentEngine { /// 从 ApplicationSupport/PhoneClaw/SYSPROMPT.md 读取 system prompt。 /// 文件不存在时自动写入 kDefaultSystemPrompt（供用户后续编辑）。 + /// 如果文件存在但缺少新的 ___DEVICE_SKILLS___ / ___CONTENT_SKILLS___ 占位符, + /// 视为旧版自动生成的模板, 备份后用新默认覆盖。 func loadSystemPrompt() { let fm = FileManager.default guard let supportDir = fm.urls(for: .applicationSupportDirectory, @@ -1802,8 +191,23 @@ class AgentEngine { if fm.fileExists(atPath: file.path), let content = try? String(contentsOf: file, encoding: .utf8), !content.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { - config.systemPrompt = content - print("[Agent] SYSPROMPT loaded (\(content.count) chars)") + + // 旧版模板检测: 缺新占位符且仍然包含旧扁平占位符 → 备份后覆盖 + let hasNewPlaceholders = content.contains("___DEVICE_SKILLS___") + || content.contains("___CONTENT_SKILLS___") + let hasOldPlaceholder = content.contains("___SKILLS___") + + if !hasNewPlaceholders && hasOldPlaceholder { + let backup = dir.appendingPathComponent("SYSPROMPT.md.bak") + try? fm.removeItem(at: backup) + try? fm.moveItem(at: file, to: backup) + try? kDefaultSystemPrompt.write(to: file, atomically: true, encoding: .utf8) + config.systemPrompt = kDefaultSystemPrompt + print("[Agent] SYSPROMPT migrated: 旧模板已备份到 SYSPROMPT.md.bak, 新默认已写入") + } else { + config.systemPrompt = content + print("[Agent] SYSPROMPT loaded (\(content.count) chars)") + } } else { try? kDefaultSystemPrompt.write(to: file, atomically: true, encoding: .utf8) config.systemPrompt = kDefaultSystemPrompt @@ -1890,12 +294,27 @@ class AgentEngine { applySamplingConfig() - let preflightToolCall = requiresMultimodal ? nil : preflightSkillLoadCall(for: normalizedText) + let matchedSkillIdsForTurn = requiresMultimodal ? [] : matchedSkillIds(for: normalizedText) + // TODO(Step 4): 接入 llm.selectedModel.supportsStructuredPlanning 后 gate Planner 入口 + let shouldUsePlanner = !requiresMultimodal && matchedSkillIdsForTurn.count >= 2 let shouldUseFullAgentPrompt = !requiresMultimodal - && (preflightToolCall != nil || shouldUseToolingPrompt(for: normalizedText)) - let activeSkillInfos = shouldUseFullAgentPrompt ? enabledSkillInfos : [] + && !shouldUsePlanner + && shouldUseToolingPrompt(for: normalizedText) + let activeSkillInfos: [SkillInfo] + if shouldUseFullAgentPrompt { + if matchedSkillIdsForTurn.isEmpty { + activeSkillInfos = enabledSkillInfos + } else { + let selectedIds = Set(matchedSkillIdsForTurn) + let matchedInfos = enabledSkillInfos.filter { selectedIds.contains($0.name) } + activeSkillInfos = matchedInfos.isEmpty ? enabledSkillInfos : matchedInfos + } + } else { + activeSkillInfos = [] + } let historyDepth = requiresMultimodal ? 0 : llm.safeHistoryDepth + let plannerHistoryDepth = shouldUsePlanner ? 0 : historyDepth print("[MEM] safeHistoryDepth=\(historyDepth), headroom=\(llm.availableHeadroomMB) MB") let promptImages = promptImages(historyDepth: historyDepth, currentImages: attachments) print( @@ -1903,6 +322,17 @@ class AgentEngine { + "audio=\(audioAttachment == nil ? 0 : 1)" ) + // [Metric] PhoneClaw 架构优化数据收集 + // 收集窗口：2026-04-08 ~ 2026-04-22（2 周） + // 到期后评估 P1-b / P2 是否执行，评估完可移除此埋点 + #warning("[Metric] PhoneClaw 架构优化埋点，2026-04-22 前评估并移除") + let routedPath = Self.decideRoute( + requiresMultimodal: requiresMultimodal, + shouldUsePlanner: shouldUsePlanner, + shouldUseFullAgentPrompt: shouldUseFullAgentPrompt + ) + log("[Metric] route=\(routedPath) skills=\(matchedSkillIdsForTurn.count) model=\(llm.selectedModel.id) multimodal=\(requiresMultimodal)") + messages.append(ChatMessage(role: .assistant, content: "▍")) let msgIndex = messages.count - 1 @@ -1956,7 +386,8 @@ class AgentEngine { history: messages, systemPrompt: config.systemPrompt, enableThinking: config.enableThinking, - historyDepth: historyDepth + historyDepth: historyDepth, + showListSkillsHint: matchedSkillIdsForTurn.isEmpty ) } else { prompt = PromptBuilder.buildLightweightTextPrompt( @@ -1964,27 +395,44 @@ class AgentEngine { history: messages, systemPrompt: config.systemPrompt, enableThinking: config.enableThinking, - historyDepth: historyDepth + historyDepth: shouldUsePlanner ? plannerHistoryDepth : historyDepth ) } log("[Agent] text prompt mode=\(shouldUseFullAgentPrompt ? "agent" : "light"), chars=\(prompt.count), skills=\(activeSkillInfos.count)") - if let preflightToolCall { - log("[Agent] preflight tool path triggered") + if shouldUsePlanner { + log("[Agent] planner path triggered revision=\(plannerRevision)") + let plannerHandled = await executePlannedSkillChainIfPossible( + prompt: prompt, + userQuestion: normalizedText, + images: promptImages + ) + + if plannerHandled { + if messages.indices.contains(msgIndex), + messages[msgIndex].role == .assistant, + messages[msgIndex].content == "▍" { + messages.remove(at: msgIndex) + } + return + } + if messages.indices.contains(msgIndex), messages[msgIndex].role == .assistant, messages[msgIndex].content == "▍" { messages.remove(at: msgIndex) } - await executeToolChain( - prompt: prompt, - fullText: preflightToolCall, - userQuestion: normalizedText, - images: promptImages + messages.append( + ChatMessage( + role: .assistant, + content: "⚠️ 已识别到多个 Skill，但当前无法完成编排。请补充更具体的信息后重试。" + ) ) + isProcessing = false return } + var detectedToolCall = false var buffer = "" var bufferFlushed = false @@ -2049,7 +497,7 @@ class AgentEngine { // MARK: - Skill 结果后的后续推理（支持多轮工具链） - private func streamLLM(prompt: String, images: [CIImage]) async -> String? { + func streamLLM(prompt: String, images: [CIImage]) async -> String? { return await withCheckedContinuation { (continuation: CheckedContinuation) in llm.generateStream(prompt: prompt, images: images, audios: []) { _ in } onComplete: { result in @@ -2065,7 +513,7 @@ class AgentEngine { } } - private func streamLLM(prompt: String, msgIndex: Int, images: [CIImage]) async -> String? { + func streamLLM(prompt: String, msgIndex: Int, images: [CIImage]) async -> String? { var buffer = "" return await withCheckedContinuation { (continuation: CheckedContinuation) in var toolCallDetected = false @@ -2111,286 +559,6 @@ class AgentEngine { } } - private func executeToolChain( - prompt: String, - fullText: String, - userQuestion: String, - images: [CIImage], - round: Int = 1, - maxRounds: Int = 10 - ) async { - guard round <= maxRounds else { - log("[Agent] 达到最大工具链轮数 \(maxRounds)") - isProcessing = false - return - } - - guard let parsedCall = parseToolCall(fullText) else { - let cleaned = cleanOutput(fullText) - if let lastAssistant = messages.lastIndex(where: { $0.role == .assistant }) { - messages[lastAssistant].update(content: cleaned.isEmpty ? "（无回复）" : cleaned) - } - isProcessing = false - return - } - - let call = ( - name: canonicalToolName(parsedCall.name, arguments: parsedCall.arguments), - arguments: parsedCall.arguments - ) - - log("[Agent] Round \(round): tool_call name=\(call.name)") - - // ── load_skill ── - if call.name == "load_skill" { - let allCalls = parseAllToolCalls(fullText) - let loadSkillCalls = allCalls.filter { $0.name == "load_skill" } - - var allInstructions = "" - var loadedDisplayNames: [String] = [] - var loadedSkillIds: [String] = [] - for lsCall in loadSkillCalls { - let requestedSkillName = (lsCall.arguments["skill"] as? String) - ?? (lsCall.arguments["name"] as? String) - ?? "" - let skillName = skillLoader.canonicalSkillId(for: requestedSkillName) - log("[Agent] load_skill: \(requestedSkillName)") - - let displayName = findDisplayName(for: skillName) - loadedDisplayNames.append(displayName) - messages.append(ChatMessage(role: .system, content: "identified", skillName: displayName)) - let cardIdx = messages.count - 1 - - guard let instructions = handleLoadSkill(skillName: skillName) else { - messages[cardIdx].update(role: .system, content: "done", skillName: displayName) - continue - } - - try? await Task.sleep(for: .milliseconds(300)) - messages[cardIdx].update(role: .system, content: "loaded", skillName: displayName) - messages.append(ChatMessage(role: .skillResult, content: instructions, skillName: skillName)) - allInstructions += instructions + "\n\n" - loadedSkillIds.append(skillName) - } - - guard !allInstructions.isEmpty else { - isProcessing = false - return - } - - if let autoCall = autoToolCallForLoadedSkills(skillIds: loadedSkillIds) - ?? inferFallbackToolCall(skillIds: loadedSkillIds, userQuestion: userQuestion) { - log("[Agent] load_skill 直接执行工具: \(autoCall.name)") - let syntheticToolCall = syntheticToolCallText( - name: autoCall.name, - arguments: autoCall.arguments - ) - await executeToolChain( - prompt: prompt, - fullText: syntheticToolCall, - userQuestion: userQuestion, - images: images, - round: round + 1, - maxRounds: maxRounds - ) - return - } - - let singleToolExtraction = await extractToolCallForLoadedSkills( - originalPrompt: prompt, - userQuestion: userQuestion, - skillInstructions: allInstructions, - skillIds: loadedSkillIds, - images: images - ) - switch singleToolExtraction { - case .toolCall(let name, let arguments): - log("[Agent] load_skill 参数提取后执行工具: \(name)") - let syntheticToolCall = syntheticToolCallText(name: name, arguments: arguments) - await executeToolChain( - prompt: prompt, - fullText: syntheticToolCall, - userQuestion: userQuestion, - images: images, - round: round + 1, - maxRounds: maxRounds - ) - return - - case .needsClarification(let clarification): - messages.append(ChatMessage(role: .assistant, content: clarification)) - markSkillsDone(loadedDisplayNames) - isProcessing = false - return - - case .failed: - break - } - - let followUpPrompt = PromptBuilder.buildLoadedSkillPrompt( - originalPrompt: prompt, - userQuestion: userQuestion, - skillInstructions: allInstructions, - currentImageCount: images.count - ) - - messages.append(ChatMessage(role: .assistant, content: "▍")) - let followUpIndex = messages.count - 1 - - guard let nextText = await streamLLM(prompt: followUpPrompt, msgIndex: followUpIndex, images: images) else { - isProcessing = false - return - } - - if parseToolCall(nextText) != nil { - log("[Agent] load_skill 后检测到 tool 调用 (round \(round + 1))") - messages[followUpIndex].update(content: "") - await executeToolChain( - prompt: followUpPrompt, fullText: nextText, - userQuestion: userQuestion, images: images, round: round + 1, maxRounds: maxRounds - ) - } else { - let cleaned = cleanOutput(nextText) - if cleaned.isEmpty - || looksLikeStructuredIntermediateOutput(cleaned) - || looksLikePromptEcho(cleaned) { - let retryPrompt = PromptBuilder.buildLoadedSkillPrompt( - originalPrompt: prompt, - userQuestion: userQuestion, - skillInstructions: allInstructions, - currentImageCount: images.count, - forceResponse: true - ) - - guard let retryText = await streamLLM(prompt: retryPrompt, msgIndex: followUpIndex, images: images) else { - isProcessing = false - return - } - - if parseToolCall(retryText) != nil { - log("[Agent] load_skill 重试后检测到 tool 调用 (round \(round + 1))") - messages[followUpIndex].update(content: "") - await executeToolChain( - prompt: retryPrompt, - fullText: retryText, - userQuestion: userQuestion, - images: images, - round: round + 1, - maxRounds: maxRounds - ) - } else { - let retryCleaned = cleanOutput(retryText) - let loadedSkillName = loadedDisplayNames.joined(separator: ", ").isEmpty - ? "已加载的能力" - : loadedDisplayNames.joined(separator: ", ") - let finalReply = retryCleaned.isEmpty - || looksLikeStructuredIntermediateOutput(retryCleaned) - || looksLikePromptEcho(retryCleaned) - ? fallbackReplyForEmptySkillFollowUp(skillName: loadedSkillName) - : retryCleaned - messages[followUpIndex].update(content: finalReply) - markSkillsDone(loadedDisplayNames) - isProcessing = false - } - } else { - messages[followUpIndex].update(content: cleaned) - markSkillsDone(loadedDisplayNames) - isProcessing = false - } - } - return - } - - // ── 具体 Tool 调用 ── - - let ownerSkillId = findSkillId(for: call.name) - let displayName = findDisplayName(for: call.name) - - let cardIndex: Int - if let idx = messages.lastIndex(where: { - $0.role == .system && ($0.skillName == displayName || $0.skillName == call.name) - && ($0.content == "identified" || $0.content == "loaded") - }) { - cardIndex = idx - } else { - messages.append(ChatMessage(role: .system, content: "identified", skillName: displayName)) - cardIndex = messages.count - 1 - } - - guard ownerSkillId != nil else { - messages[cardIndex].update(role: .system, content: "done", skillName: displayName) - messages.append(ChatMessage(role: .assistant, content: "⚠️ 未知工具: \(call.name)")) - isProcessing = false - return - } - - let enabledIds = Set(skillEntries.filter(\.isEnabled).map(\.id)) - guard enabledIds.contains(ownerSkillId!) else { - messages[cardIndex].update(role: .system, content: "done", skillName: displayName) - messages.append(ChatMessage(role: .assistant, content: "⚠️ Skill \(displayName) 未启用")) - isProcessing = false - return - } - - messages[cardIndex].update(role: .system, content: "executing:\(call.name)", skillName: displayName) - - do { - let toolResult = try await handleToolExecution(toolName: call.name, args: call.arguments) - let toolResultSummary = toolResultSummaryForModel(toolName: call.name, toolResult: toolResult) - messages[cardIndex].update(role: .system, content: "done", skillName: displayName) - messages.append(ChatMessage(role: .skillResult, content: toolResultSummary, skillName: call.name)) - log("[Agent] Tool \(call.name) round \(round) done") - - if shouldSkipToolFollowUpModel(for: call.name) { - messages.append(ChatMessage(role: .assistant, content: toolResultSummary)) - isProcessing = false - return - } - - let followUpPrompt = PromptBuilder.buildToolAnswerPrompt( - originalPrompt: prompt, - toolName: call.name, - toolResultSummary: toolResultSummary, - userQuestion: userQuestion, - currentImageCount: images.count - ) - - messages.append(ChatMessage(role: .assistant, content: "▍")) - let followUpIndex = messages.count - 1 - - guard let nextText = await streamLLM(prompt: followUpPrompt, msgIndex: followUpIndex, images: images) else { - isProcessing = false - return - } - - if !parseAllToolCalls(nextText).isEmpty { - log("[Agent] 检测到第 \(round + 1) 轮工具调用") - messages[followUpIndex].update(content: "") - await executeToolChain( - prompt: followUpPrompt, fullText: nextText, - userQuestion: userQuestion, images: images, round: round + 1, maxRounds: maxRounds - ) - } else { - let cleaned = cleanOutput(nextText) - if cleaned.isEmpty - || looksLikeStructuredIntermediateOutput(cleaned) - || looksLikePromptEcho(cleaned) { - messages[followUpIndex].update(content: fallbackReplyForEmptyToolFollowUp( - toolName: call.name, - toolResult: toolResult - )) - } else { - messages[followUpIndex].update(content: cleaned) - } - isProcessing = false - } - } catch { - messages[cardIndex].update(role: .system, content: "done", skillName: displayName) - messages.append(ChatMessage(role: .system, content: "❌ Tool 执行失败: \(error)")) - isProcessing = false - } - } - // MARK: - 工具 func clearMessages() { @@ -2489,228 +657,6 @@ class AgentEngine { // MARK: - 解析 - private func parseToolCall(_ text: String) -> (name: String, arguments: [String: Any])? { - return parseAllToolCalls(text).first - } - - private func parseAllToolCalls(_ text: String) -> [(name: String, arguments: [String: Any])] { - var results: [(name: String, arguments: [String: Any])] = [] - let patterns = [ - "\\s*(\\{.*?\\})\\s*", - "```json\\s*(\\{.*?\\})\\s*```", - "\\s*(\\{.*?\\})\\s*" - ] - for pattern in patterns { - guard let regex = try? NSRegularExpression(pattern: pattern, options: .dotMatchesLineSeparators) else { continue } - let range = NSRange(text.startIndex..., in: text) - let matches = regex.matches(in: text, range: range) - for match in matches { - if let jsonRange = Range(match.range(at: 1), in: text) { - let json = String(text[jsonRange]) - if let data = json.data(using: .utf8), - let dict = try? JSONSerialization.jsonObject(with: data) as? [String: Any], - let name = dict["name"] as? String { - results.append((name, dict["arguments"] as? [String: Any] ?? [:])) - } - } - } - if !results.isEmpty { break } - } - return results - } - - // MARK: - Session Persistence - - private func scheduleSessionSave() { - sessionSaveTask?.cancel() - let currentID = currentSessionID - sessionSaveTask = Task { [weak self] in - try? await Task.sleep(for: .milliseconds(350)) - guard !Task.isCancelled, let self, self.currentSessionID == currentID else { return } - self.saveCurrentSession() - } - } - - private func saveCurrentSession() { - guard !messages.isEmpty else { return } - - let summary = makeSessionSummary( - id: currentSessionID, - messages: messages - ) - let record = ChatSessionRecord( - id: currentSessionID, - title: summary.title, - preview: summary.preview, - updatedAt: summary.updatedAt, - messages: messages - ) - - let encoder = JSONEncoder() - encoder.outputFormatting = [.sortedKeys] - encoder.dateEncodingStrategy = .iso8601 - - do { - let directory = try ensureSessionsDirectory() - let data = try encoder.encode(record) - try data.write(to: sessionFileURL(for: currentSessionID), options: .atomic) - updateSessionSummary(summary) - persistSessionsIndex() - UserDefaults.standard.set(currentSessionID.uuidString, forKey: Self.currentSessionDefaultsKey) - _ = directory - } catch { - log("[History] save failed: \(error.localizedDescription)") - } - } - - private func loadPersistedSessions() { - do { - _ = try ensureSessionsDirectory() - } catch { - log("[History] setup failed: \(error.localizedDescription)") - } - - sessionSummaries = loadSessionsIndex().sorted { $0.updatedAt > $1.updatedAt } - - if let record = loadSessionRecord(id: currentSessionID) { - messages = record.messages - updateSessionSummary( - .init( - id: record.id, - title: record.title, - preview: record.preview, - updatedAt: record.updatedAt - ) - ) - return - } - - if let first = sessionSummaries.first, let record = loadSessionRecord(id: first.id) { - currentSessionID = first.id - UserDefaults.standard.set(currentSessionID.uuidString, forKey: Self.currentSessionDefaultsKey) - messages = record.messages - return - } - - currentSessionID = UUID() - UserDefaults.standard.set(currentSessionID.uuidString, forKey: Self.currentSessionDefaultsKey) - messages = [] - } - - private func makeSessionSummary(id: UUID, messages: [ChatMessage]) -> ChatSessionSummary { - let firstUser = messages.first(where: { $0.role == .user }) - let lastMeaningful = messages.reversed().first(where: { msg in - if !msg.content.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { - return true - } - return !msg.images.isEmpty || !msg.audios.isEmpty - }) - - let title = sessionTitle(from: firstUser) - let preview = sessionPreview(from: lastMeaningful ?? firstUser) - let updatedAt = (lastMeaningful ?? firstUser)?.timestamp ?? Date() - - return ChatSessionSummary( - id: id, - title: title, - preview: preview, - updatedAt: updatedAt - ) - } - - private func sessionTitle(from message: ChatMessage?) -> String { - guard let message else { return "新会话" } - let trimmed = message.content.trimmingCharacters(in: .whitespacesAndNewlines) - if !trimmed.isEmpty { - return String(trimmed.prefix(24)) - } - if !message.images.isEmpty && !message.audios.isEmpty { - return "图片与语音会话" - } - if !message.images.isEmpty { - return "图片会话" - } - if !message.audios.isEmpty { - return "语音会话" - } - return "新会话" - } - - private func sessionPreview(from message: ChatMessage?) -> String { - guard let message else { return "暂无内容" } - let trimmed = message.content.trimmingCharacters(in: .whitespacesAndNewlines) - if !trimmed.isEmpty { - return String(trimmed.prefix(80)) - } - if !message.images.isEmpty && !message.audios.isEmpty { - return "包含图片与语音" - } - if !message.images.isEmpty { - return "包含图片" - } - if !message.audios.isEmpty { - return "包含语音" - } - return "暂无内容" - } - - private func updateSessionSummary(_ summary: ChatSessionSummary) { - if let index = sessionSummaries.firstIndex(where: { $0.id == summary.id }) { - sessionSummaries[index] = summary - } else { - sessionSummaries.append(summary) - } - sessionSummaries.sort { $0.updatedAt > $1.updatedAt } - } - - private func persistSessionsIndex() { - let encoder = JSONEncoder() - encoder.outputFormatting = [.sortedKeys] - encoder.dateEncodingStrategy = .iso8601 - do { - let data = try encoder.encode(sessionSummaries) - try data.write(to: sessionsIndexURL(), options: .atomic) - } catch { - log("[History] index save failed: \(error.localizedDescription)") - } - } - - private func loadSessionsIndex() -> [ChatSessionSummary] { - let url = sessionsIndexURL() - guard let data = try? Data(contentsOf: url) else { return [] } - let decoder = JSONDecoder() - decoder.dateDecodingStrategy = .iso8601 - return (try? decoder.decode([ChatSessionSummary].self, from: data)) ?? [] - } - - private func loadSessionRecord(id: UUID) -> ChatSessionRecord? { - let url = sessionFileURL(for: id) - guard let data = try? Data(contentsOf: url) else { return nil } - let decoder = JSONDecoder() - decoder.dateDecodingStrategy = .iso8601 - return try? decoder.decode(ChatSessionRecord.self, from: data) - } - - private func ensureSessionsDirectory() throws -> URL { - let directory = sessionsDirectoryURL() - try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true) - return directory - } - - private func sessionsIndexURL() -> URL { - sessionsDirectoryURL().appendingPathComponent(sessionsIndexFileName) - } - - private func sessionFileURL(for id: UUID) -> URL { - sessionsDirectoryURL().appendingPathComponent("\(id.uuidString).json") - } - - private func sessionsDirectoryURL() -> URL { - let supportDir = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first! - let appDir = supportDir.appendingPathComponent("PhoneClaw", isDirectory: true) - return appDir.appendingPathComponent(sessionsDirectoryName, isDirectory: true) - } - private func extractSkillName(from text: String) -> String? { guard let regex = try? NSRegularExpression(pattern: "\"name\"\\s*:\\s*\"([^\"]+)\""), let match = regex.firstMatch(in: text, range: NSRange(text.startIndex..., in: text)), @@ -2720,182 +666,4 @@ class AgentEngine { return String(text[nameRange]) } - private func cleanOutputStreaming(_ text: String) -> String { - var result = preserveThinkingChannels(in: text) - - if let tcRange = result.range(of: "") { - result = String(result[result.startIndex..", "", ""] - for pat in endPatterns { - if let range = result.range(of: pat) { - result = String(result[result.startIndex...*?", options: .dotMatchesLineSeparators) { - result = regex.stringByReplacingMatches(in: result, range: NSRange(result.startIndex..., in: result), withTemplate: "") - } - - result = result.replacingOccurrences( - of: "<\\|?[/a-z_]+\\|?>", - with: "", - options: .regularExpression - ) - - if result.hasPrefix("model\n") { - result = String(result.dropFirst(6)) - } else if result == "model" { - return "" - } else if result.hasPrefix("user\n") { - result = String(result.dropFirst(5)) - } else if result == "user" { - return "" - } - - result = String(result.drop(while: { $0.isWhitespace || $0.isNewline })) - return normalizeSafetyTruncation(in: result) - } - - private func cleanOutput(_ text: String) -> String { - var result = preserveThinkingChannels(in: text) - - if let regex = try? NSRegularExpression(pattern: ".*?", options: .dotMatchesLineSeparators) { - result = regex.stringByReplacingMatches(in: result, range: NSRange(result.startIndex..., in: result), withTemplate: "") - } - - if let tcRange = result.range(of: "") { - result = String(result[result.startIndex..", "", ""] - for pat in endPatterns { - if let range = result.range(of: pat) { - result = String(result[result.startIndex..", - with: "", - options: .regularExpression - ) - - if let lastOpen = result.lastIndex(of: "<") { - let tail = String(result[lastOpen...]) - let tailBody = tail.dropFirst() - if !tailBody.isEmpty && tailBody.allSatisfy({ $0.isLetter || $0 == "_" || $0 == "/" || $0 == "|" }) { - result = String(result[result.startIndex.. String { - let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) - guard let warningRange = trimmed.range(of: "> ⚠️ ") else { - return trimmed - } - - let body = String(trimmed[.. String { - let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) - guard !trimmed.isEmpty else { return trimmed } - - if let paragraphBreak = trimmed.range(of: "\n\n", options: .backwards) { - let tailLength = trimmed.distance(from: paragraphBreak.upperBound, to: trimmed.endIndex) - if tailLength > 0 && tailLength <= 280 { - return String(trimmed[.. 0 && tailLength <= 220 { - return String(trimmed[.. String.Index? { - let sentenceEndings: Set = ["。", "！", "？", ".", "!", "?"] - var index = text.endIndex - while index > text.startIndex { - index = text.index(before: index) - if sentenceEndings.contains(text[index]) { - return text.index(after: index) - } - } - return nil - } - - private func preserveThinkingChannels(in text: String) -> String { - let openTokens = ["<|channel|>thought\n", "<|channel>thought\n"] - let closeToken = "" - - var result = "" - var cursor = text.startIndex - - while cursor < text.endIndex { - let nextOpen = openTokens - .compactMap { token -> (Range, String)? in - guard let range = text.range(of: token, range: cursor.. UserInput.Audio { + .pcm( + .init( + samples: snapshot.pcm, + sampleRate: snapshot.sampleRate, + channelCount: snapshot.channelCount + ) + ) + } +} + +struct ChatImageAttachment: Identifiable, Codable { + let id: UUID + let data: Data + private static let storageMaxDimension: CGFloat = 1_024 + private static let compressionQuality: CGFloat = 0.78 + + init(id: UUID = UUID(), data: Data) { + self.id = id + self.data = data + } + + init?(image: UIImage) { + let prepared = Self.preparedImage(image, maxDimension: Self.storageMaxDimension) + if let jpeg = prepared.jpegData(compressionQuality: Self.compressionQuality) { + self.id = UUID() + self.data = jpeg + } else if let png = prepared.pngData() { + self.id = UUID() + self.data = png + } else { + return nil + } + } + + static func preparedImage(_ image: UIImage, maxDimension: CGFloat = storageMaxDimension) -> UIImage { + let originalSize = image.size + let longestSide = max(originalSize.width, originalSize.height) + guard longestSide > maxDimension, longestSide > 0 else { + return image + } + + let scale = maxDimension / longestSide + let targetSize = CGSize( + width: max(1, floor(originalSize.width * scale)), + height: max(1, floor(originalSize.height * scale)) + ) + + let format = UIGraphicsImageRendererFormat.default() + format.opaque = false + format.scale = 1 + let renderer = UIGraphicsImageRenderer(size: targetSize, format: format) + return renderer.image { _ in + image.draw(in: CGRect(origin: .zero, size: targetSize)) + } + } + + var uiImage: UIImage? { + UIImage(data: data) + } + + var ciImage: CIImage? { + if let image = CIImage( + data: data, + options: [.applyOrientationProperty: true] + ) { + return image + } + guard let uiImage else { return nil } + if let ciImage = uiImage.ciImage { + return ciImage + } + if let cgImage = uiImage.cgImage { + return CIImage(cgImage: cgImage) + } + return CIImage(image: uiImage) + } +} + +struct ChatAudioAttachment: Identifiable, Codable { + let id: UUID + let wavData: Data + let duration: TimeInterval + let sampleRate: Double + let waveform: [Float] + + init( + id: UUID = UUID(), + wavData: Data, + duration: TimeInterval, + sampleRate: Double, + waveform: [Float] + ) { + self.id = id + self.wavData = wavData + self.duration = duration + self.sampleRate = sampleRate + self.waveform = waveform + } + + init?(snapshot: AudioCaptureSnapshot) { + guard !snapshot.pcm.isEmpty, snapshot.sampleRate > 0 else { return nil } + self.id = UUID() + self.wavData = Self.makeWAVData( + pcm: snapshot.pcm, + sampleRate: snapshot.sampleRate, + channelCount: 1 + ) + self.duration = snapshot.duration + self.sampleRate = snapshot.sampleRate + self.waveform = Self.makeWaveform(from: snapshot.pcm) + } + + var formattedDuration: String { + let totalSeconds = Int(duration.rounded()) + let minutes = totalSeconds / 60 + let seconds = totalSeconds % 60 + return String(format: "%02d:%02d", minutes, seconds) + } + + private static func makeWaveform(from pcm: [Float], bucketCount: Int = 36) -> [Float] { + guard !pcm.isEmpty else { return Array(repeating: 0.12, count: bucketCount) } + let samplesPerBucket = max(pcm.count / bucketCount, 1) + var levels: [Float] = [] + levels.reserveCapacity(bucketCount) + + var index = 0 + while index < pcm.count { + let end = min(index + samplesPerBucket, pcm.count) + let slice = pcm[index.. bucketCount { + levels = Array(levels.prefix(bucketCount)) + } + + let maxLevel = max(levels.max() ?? 0, 0.001) + return levels.map { max($0 / maxLevel, 0.08) } + } + + private static func makeWAVData( + pcm: [Float], + sampleRate: Double, + channelCount: Int + ) -> Data { + let integerSampleRate = max(Int(sampleRate.rounded()), 1) + let clampedSamples = pcm.map { sample -> Int16 in + let limited = min(max(sample, -1), 1) + return Int16((limited * Float(Int16.max)).rounded()) + } + + let bytesPerSample = MemoryLayout.size + let dataChunkSize = clampedSamples.count * bytesPerSample + let riffChunkSize = 36 + dataChunkSize + let byteRate = integerSampleRate * channelCount * bytesPerSample + let blockAlign = channelCount * bytesPerSample + + var data = Data() + data.reserveCapacity(44 + dataChunkSize) + data.append("RIFF".data(using: .ascii)!) + append(UInt32(riffChunkSize), to: &data) + data.append("WAVE".data(using: .ascii)!) + data.append("fmt ".data(using: .ascii)!) + append(UInt32(16), to: &data) + append(UInt16(1), to: &data) + append(UInt16(channelCount), to: &data) + append(UInt32(integerSampleRate), to: &data) + append(UInt32(byteRate), to: &data) + append(UInt16(blockAlign), to: &data) + append(UInt16(bytesPerSample * 8), to: &data) + data.append("data".data(using: .ascii)!) + append(UInt32(dataChunkSize), to: &data) + + for sample in clampedSamples { + append(sample, to: &data) + } + return data + } + + private static func append(_ value: T, to data: inout Data) { + var littleEndian = value.littleEndian + withUnsafeBytes(of: &littleEndian) { bytes in + data.append(contentsOf: bytes) + } + } +} diff --git a/Agent/Engine/ChatMessage.swift b/Agent/Engine/ChatMessage.swift new file mode 100644 index 0000000..6fe7dfe --- /dev/null +++ b/Agent/Engine/ChatMessage.swift @@ -0,0 +1,51 @@ +import Foundation + +struct ChatMessage: Identifiable, Codable { + let id: UUID + var role: Role + var content: String + var images: [ChatImageAttachment] + var audios: [ChatAudioAttachment] + let timestamp: Date + var skillName: String? = nil + + init( + id: UUID = UUID(), + role: Role, + content: String, + images: [ChatImageAttachment] = [], + audios: [ChatAudioAttachment] = [], + timestamp: Date = Date(), + skillName: String? = nil + ) { + self.id = id + self.role = role + self.content = content + self.images = images + self.audios = audios + self.timestamp = timestamp + self.skillName = skillName + } + + mutating func update(content: String) { + guard self.content != content else { return } + self.content = content + } + + mutating func update(role: Role, content: String, skillName: String? = nil) { + self.role = role + self.content = content + self.skillName = skillName + } + + enum Role: String, Codable { + case user, assistant, system, skillResult + } +} + +struct ChatSessionSummary: Identifiable, Codable, Equatable { + let id: UUID + var title: String + var preview: String + var updatedAt: Date +} diff --git a/Agent/Engine/OutputCleaner.swift b/Agent/Engine/OutputCleaner.swift new file mode 100644 index 0000000..ee43558 --- /dev/null +++ b/Agent/Engine/OutputCleaner.swift @@ -0,0 +1,271 @@ +import Foundation + +// 与 PromptBuilder / ChatModels 中的常量保持一致。 +// 仅供 OutputCleaner 内部使用，外部使用方分别在自己的文件里维护副本。 +private let thinkingOpenMarker = "[[PHONECLAW_THINK]]" +private let thinkingCloseMarker = "[[/PHONECLAW_THINK]]" + +extension AgentEngine { + + // MARK: - 中间输出/Prompt 回声识别 + + func looksLikeStructuredIntermediateOutput(_ text: String) -> Bool { + let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { return false } + + if trimmed.hasPrefix("```json") || trimmed.hasPrefix("```") { + return true + } + + if let regex = try? NSRegularExpression( + pattern: "\"[A-Za-z_][A-Za-z0-9_]*\"\\s*:", + options: [] + ) { + let matchCount = regex.numberOfMatches( + in: trimmed, + range: NSRange(trimmed.startIndex..., in: trimmed) + ) + if matchCount >= 2 && !trimmed.hasPrefix("{") { + return true + } + } + + let suspiciousFragments = [ + "tool_name\":", + "result_for_user_name\":", + "text_for_display\":", + "tool_operation_success\":", + "arguments_for_tool_no_skill\":", + "memory_user_power_conversion\":" + ] + if suspiciousFragments.filter({ trimmed.contains($0) }).count >= 2 { + return true + } + + guard let data = trimmed.data(using: .utf8), + let json = try? JSONSerialization.jsonObject(with: data) else { + return false + } + + if let dict = json as? [String: Any] { + if dict["name"] != nil { + return false + } + + let suspiciousKeys = [ + "final_answer", "tool_call", "arguments", "device_call", + "next_action", "action", "tool" + ] + return suspiciousKeys.contains { dict[$0] != nil } + } + + return false + } + + func looksLikePromptEcho(_ text: String) -> Bool { + let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { return false } + + if trimmed.hasPrefix("user\n") || trimmed == "user" { + return true + } + + let suspiciousPhrases = [ + "根据已加载的 Skill", + "不要将任何关于工具、系统或该请求的描述变成 Markdown 代码或 JSON 模板", + "如果需要，请直接调用", + "package_name", + "text_for_user" + ] + + let hitCount = suspiciousPhrases.reduce(into: 0) { count, phrase in + if trimmed.contains(phrase) { count += 1 } + } + return hitCount >= 2 + } + + // MARK: - 输出清洗 + + func cleanOutputStreaming(_ text: String) -> String { + var result = preserveThinkingChannels(in: text) + + if let tcRange = result.range(of: "") { + result = String(result[result.startIndex..", "", ""] + for pat in endPatterns { + if let range = result.range(of: pat) { + result = String(result[result.startIndex...*?", options: .dotMatchesLineSeparators) { + result = regex.stringByReplacingMatches(in: result, range: NSRange(result.startIndex..., in: result), withTemplate: "") + } + + result = result.replacingOccurrences( + of: "<\\|?[/a-z_]+\\|?>", + with: "", + options: .regularExpression + ) + + if result.hasPrefix("model\n") { + result = String(result.dropFirst(6)) + } else if result == "model" { + return "" + } else if result.hasPrefix("user\n") { + result = String(result.dropFirst(5)) + } else if result == "user" { + return "" + } + + result = String(result.drop(while: { $0.isWhitespace || $0.isNewline })) + return normalizeSafetyTruncation(in: result) + } + + func cleanOutput(_ text: String) -> String { + var result = preserveThinkingChannels(in: text) + + if let regex = try? NSRegularExpression(pattern: ".*?", options: .dotMatchesLineSeparators) { + result = regex.stringByReplacingMatches(in: result, range: NSRange(result.startIndex..., in: result), withTemplate: "") + } + + if let tcRange = result.range(of: "") { + result = String(result[result.startIndex..", "", ""] + for pat in endPatterns { + if let range = result.range(of: pat) { + result = String(result[result.startIndex..", + with: "", + options: .regularExpression + ) + + if let lastOpen = result.lastIndex(of: "<") { + let tail = String(result[lastOpen...]) + let tailBody = tail.dropFirst() + if !tailBody.isEmpty && tailBody.allSatisfy({ $0.isLetter || $0 == "_" || $0 == "/" || $0 == "|" }) { + result = String(result[result.startIndex.. String { + let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) + guard let warningRange = trimmed.range(of: "> ⚠️ ") else { + return trimmed + } + + let body = String(trimmed[.. String { + let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { return trimmed } + + if let paragraphBreak = trimmed.range(of: "\n\n", options: .backwards) { + let tailLength = trimmed.distance(from: paragraphBreak.upperBound, to: trimmed.endIndex) + if tailLength > 0 && tailLength <= 280 { + return String(trimmed[.. 0 && tailLength <= 220 { + return String(trimmed[.. String.Index? { + let sentenceEndings: Set = ["。", "！", "？", ".", "!", "?"] + var index = text.endIndex + while index > text.startIndex { + index = text.index(before: index) + if sentenceEndings.contains(text[index]) { + return text.index(after: index) + } + } + return nil + } + + // MARK: - Thinking 通道保留 + + private func preserveThinkingChannels(in text: String) -> String { + let openTokens = ["<|channel|>thought\n", "<|channel>thought\n"] + let closeToken = "" + + var result = "" + var cursor = text.startIndex + + while cursor < text.endIndex { + let nextOpen = openTokens + .compactMap { token -> (Range, String)? in + guard let range = text.range(of: token, range: cursor.. String { + let selectedIds = uniqueStringsPreservingOrder(skillIds) + let chosenEntries: [SkillEntry] + if selectedIds.isEmpty { + chosenEntries = skillEntries.filter(\.isEnabled) + } else { + let selectedSet = Set(selectedIds) + chosenEntries = skillEntries.filter { $0.isEnabled && selectedSet.contains($0.id) } + } + + return chosenEntries.map { entry in + if compact { + let tools = registeredTools(for: entry.id).map(\.name).joined(separator: "、") + return "- \(entry.id): \(tools)" + } else { + let tools = registeredTools(for: entry.id).map { + "\($0.name): \($0.description)" + }.joined(separator: "；") + return """ + - \(entry.id)(\(entry.name)): \(entry.description) + 可用工具: \(tools) + """ + } + }.joined(separator: "\n") + } + + func recentPlannerContextSummary(limit: Int = 2) -> String { + let toolNames = Set( + skillEntries + .filter(\.isEnabled) + .flatMap { registeredTools(for: $0.id).map(\.name) } + ) + guard !toolNames.isEmpty else { return "" } + + var blocks: [String] = [] + for message in messages.reversed() { + guard message.role == .skillResult, + let skillName = message.skillName, + toolNames.contains(skillName) else { + continue + } + + let trimmed = message.content.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { continue } + + let summary: String + if trimmed.count > 220 { + summary = String(trimmed.prefix(220)) + "..." + } else { + summary = trimmed + } + + blocks.append("- \(skillName): \(summary)") + if blocks.count >= limit { + break + } + } + + return blocks.reversed().joined(separator: "\n") + } + + // MARK: - 计划解析 + + func parseExecutionPlan(_ text: String) -> ExecutionPlan? { + guard let object = parseJSONObject(text) else { return nil } + let goal = object["goal"] as? String ?? "" + let needsClarification = object["needs_clarification"] as? String + let rawSteps = object["steps"] as? [[String: Any]] ?? [] + + let steps = rawSteps.compactMap { rawStep -> ExecutionPlanStep? in + guard let id = rawStep["id"] as? String, + let skill = rawStep["skill"] as? String, + let tool = rawStep["tool"] as? String, + let intent = rawStep["intent"] as? String else { + return nil + } + let dependsOn = rawStep["depends_on"] as? [String] ?? [] + return ExecutionPlanStep( + id: id, + skill: skill, + tool: tool, + intent: intent, + dependsOn: dependsOn + ) + } + + return ExecutionPlan(goal: goal, steps: steps, needsClarification: needsClarification) + } + + func parseSkillSelection(_ text: String) -> SkillSelection? { + guard let object = parseJSONObject(text) else { return nil } + let goal = object["goal"] as? String ?? "" + let requiredSkills = object["required_skills"] as? [String] ?? [] + let needsClarification = object["needs_clarification"] as? String + return SkillSelection( + goal: goal, + requiredSkills: requiredSkills, + needsClarification: needsClarification + ) + } + + func validateSkillSelection( + _ selection: SkillSelection, + candidateSkillIds: [String] + ) -> SkillSelection? { + let clarification = selection.needsClarification? + .trimmingCharacters(in: .whitespacesAndNewlines) + if let clarification, !clarification.isEmpty { + return SkillSelection(goal: selection.goal, requiredSkills: [], needsClarification: clarification) + } + + let normalizedSkills = uniqueStringsPreservingOrder( + selection.requiredSkills.compactMap { canonicalSkillSelectionEntry($0) } + ) + guard !normalizedSkills.isEmpty, normalizedSkills.count <= 3 else { + return nil + } + + let enabledSkillSet = Set(skillEntries.filter(\.isEnabled).map(\.id)) + let candidateSet = candidateSkillIds.isEmpty ? enabledSkillSet : Set(candidateSkillIds) + guard normalizedSkills.allSatisfy({ + enabledSkillSet.contains($0) && (candidateSet.contains($0) || candidateSkillIds.isEmpty) + }) else { + return nil + } + + return SkillSelection( + goal: selection.goal, + requiredSkills: normalizedSkills, + needsClarification: nil + ) + } + + func validateExecutionPlan( + _ plan: ExecutionPlan, + candidateSkillIds: [String] + ) -> ExecutionPlan? { + let clarification = plan.needsClarification? + .trimmingCharacters(in: .whitespacesAndNewlines) + if let clarification, !clarification.isEmpty, plan.steps.isEmpty { + return ExecutionPlan(goal: plan.goal, steps: [], needsClarification: clarification) + } + + guard !plan.steps.isEmpty, plan.steps.count <= 4 else { + return nil + } + + let enabledSkillSet = Set(skillEntries.filter(\.isEnabled).map(\.id)) + let candidateSet = candidateSkillIds.isEmpty ? enabledSkillSet : Set(candidateSkillIds) + var seenStepIds: Set = [] + var previousStepIds: Set = [] + + let uniqueSkillCount = Set(plan.steps.map(\.skill)).count + guard uniqueSkillCount <= 3 else { return nil } + + for step in plan.steps { + let stepID = step.id.trimmingCharacters(in: .whitespacesAndNewlines) + let skillID = skillRegistry.canonicalSkillId(for: step.skill) + let toolName = canonicalToolName( + step.tool, + arguments: [:], + preferredSkillId: step.skill + ) + + guard !stepID.isEmpty, + !step.intent.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty, + !seenStepIds.contains(stepID), + enabledSkillSet.contains(skillID), + candidateSet.contains(skillID) || candidateSkillIds.isEmpty else { + return nil + } + + let allowedToolNames = Set(registeredTools(for: skillID).map(\.name)) + guard allowedToolNames.contains(toolName) else { + return nil + } + + guard step.dependsOn.allSatisfy({ previousStepIds.contains($0) }) else { + return nil + } + + seenStepIds.insert(stepID) + previousStepIds.insert(stepID) + } + + let normalizedSteps = plan.steps.map { step in + ExecutionPlanStep( + id: step.id, + skill: skillRegistry.canonicalSkillId(for: step.skill), + tool: canonicalToolName( + step.tool, + arguments: [:], + preferredSkillId: step.skill + ), + intent: step.intent, + dependsOn: step.dependsOn + ) + } + + return ExecutionPlan(goal: plan.goal, steps: normalizedSteps, needsClarification: nil) + } + + func completedPlanSummary(_ completedSteps: [ExecutedPlanStep]) -> String { + completedSteps.map { completedStep in + let summary = completedStep.toolResultSummary.trimmingCharacters(in: .whitespacesAndNewlines) + let compactSummary: String + if summary.count > 160 { + compactSummary = String(summary.prefix(160)) + "..." + } else { + compactSummary = summary + } + + let block = """ + [\(completedStep.step.id)] \(completedStep.step.tool) + 目标:\(completedStep.step.intent) + 可直接给用户的结果:\(compactSummary) + """ + + return block + }.joined(separator: "\n\n") + } + + // MARK: - 多 Skill 编排主入口 + + @discardableResult + func executePlannedSkillChainIfPossible( + prompt: String, + userQuestion: String, + images: [CIImage] + ) async -> Bool { + let matchedSkills = matchedSkillIds(for: userQuestion) + log("[Agent] \(plannerRevision) matchedSkills=\(matchedSkills.joined(separator: ","))") + let candidateSkillIds = + matchedSkills.count >= 2 + ? matchedSkills + : skillEntries.filter(\.isEnabled).map(\.id) + let recentContextSummary = recentPlannerContextSummary() + let selectedSkillIds: [String] + if matchedSkills.count >= 2, matchedSkills.count <= 3 { + selectedSkillIds = matchedSkills + log("[Agent] skill selection satisfied locally skills=\(selectedSkillIds.joined(separator: ","))") + } else { + let selectionSkillsSummary = buildAvailableSkillsSummary(skillIds: candidateSkillIds) + guard !selectionSkillsSummary.isEmpty else { + messages.append(ChatMessage(role: .assistant, content: "⚠️ 当前没有可用于编排的 Skill。")) + isProcessing = false + return true + } + + let selectionPrompt = PromptBuilder.buildSkillSelectionPrompt( + originalPrompt: prompt, + userQuestion: userQuestion, + availableSkillsSummary: selectionSkillsSummary, + recentContextSummary: recentContextSummary, + currentImageCount: images.count + ) + log("[Agent] skill selection prompt chars=\(selectionPrompt.count), candidateSkills=\(candidateSkillIds.count)") + + guard let rawSelection = await streamLLM(prompt: selectionPrompt, images: images) else { + messages.append(ChatMessage(role: .assistant, content: "⚠️ 无法判断需要哪些 Skill，请重试。")) + isProcessing = false + return true + } + + let cleanedSelection = cleanOutput(rawSelection) + guard let parsedSelection = parseSkillSelection(cleanedSelection), + let validatedSelection = validateSkillSelection(parsedSelection, candidateSkillIds: candidateSkillIds) else { + messages.append(ChatMessage(role: .assistant, content: "⚠️ 当前无法判断需要哪些 Skill，请把需求说得更具体一些。")) + isProcessing = false + return true + } + + if let clarification = validatedSelection.needsClarification, + !clarification.isEmpty { + messages.append(ChatMessage(role: .assistant, content: clarification)) + isProcessing = false + return true + } + + guard validatedSelection.requiredSkills.count >= 2 else { + messages.append(ChatMessage(role: .assistant, content: "⚠️ 当前只识别到单一 Skill，请把组合操作说得更明确一些。")) + isProcessing = false + return true + } + + selectedSkillIds = validatedSelection.requiredSkills + log("[Agent] skill selection accepted skills=\(selectedSkillIds.joined(separator: ","))") + } + + var loadedInstructions: [String: String] = [:] + var loadedDisplayNames: [String: String] = [:] + var skillCardIndices: [String: Int] = [:] + var completedSteps: [ExecutedPlanStep] = [] + var toolResultsForAnswer: [(toolName: String, result: String)] = [] + var remainingSkillIds = selectedSkillIds + var planningPass = 0 + + func finishPlanning(with message: String? = nil) { + if let message, !message.isEmpty { + messages.append(ChatMessage(role: .assistant, content: message)) + } + markSkillsDone(Array(loadedDisplayNames.values)) + isProcessing = false + } + + while !remainingSkillIds.isEmpty, planningPass < 3 { + planningPass += 1 + + let combinedContextSummary = [recentContextSummary, completedPlanSummary(completedSteps)] + .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) } + .filter { !$0.isEmpty } + .joined(separator: "\n\n") + + let availableSkillsSummary = buildAvailableSkillsSummary( + skillIds: remainingSkillIds, + compact: true + ) + guard !availableSkillsSummary.isEmpty else { break } + + let planningPrompt = PromptBuilder.buildSkillPlanningPrompt( + originalPrompt: prompt, + userQuestion: userQuestion, + availableSkillsSummary: availableSkillsSummary, + recentContextSummary: combinedContextSummary, + currentImageCount: images.count + ) + log("[Agent] planner prompt chars=\(planningPrompt.count), candidateSkills=\(remainingSkillIds.count), pass=\(planningPass)") + + guard let rawPlan = await streamLLM(prompt: planningPrompt, images: images) else { + let message = completedSteps.isEmpty + ? "⚠️ 无法生成执行计划，请重试。" + : "⚠️ 无法继续规划剩余步骤，请重试。" + finishPlanning(with: message) + return true + } + + let cleanedPlan = cleanOutput(rawPlan) + guard let parsedPlan = parseExecutionPlan(cleanedPlan), + let validatedPlan = validateExecutionPlan(parsedPlan, candidateSkillIds: remainingSkillIds) else { + let message = completedSteps.isEmpty + ? "⚠️ 当前无法生成有效计划，请把需求说得更具体一些。" + : "⚠️ 当前无法继续规划剩余步骤，请把需求说得更具体一些。" + finishPlanning(with: message) + return true + } + + if let clarification = validatedPlan.needsClarification, + !clarification.isEmpty { + messages.append(ChatMessage(role: .assistant, content: clarification)) + isProcessing = false + return true + } + + guard !validatedPlan.steps.isEmpty else { + let message = completedSteps.isEmpty + ? "⚠️ 当前没有可执行步骤，请补充更具体的信息。" + : "⚠️ 当前无法继续规划剩余步骤，请补充更具体的信息。" + finishPlanning(with: message) + return true + } + + log("[Agent] planner accepted plan with \(validatedPlan.steps.count) steps") + + let executedSkillIdsThisPass = uniqueStringsPreservingOrder(validatedPlan.steps.map(\.skill)) + + for step in validatedPlan.steps { + if loadedInstructions[step.skill] == nil { + let displayName = findDisplayName(for: step.skill) + loadedDisplayNames[step.skill] = displayName + messages.append(ChatMessage(role: .system, content: "identified", skillName: displayName)) + let cardIndex = messages.count - 1 + skillCardIndices[step.skill] = cardIndex + + guard let instructions = handleLoadSkill(skillName: step.skill) else { + messages[cardIndex].update(role: .system, content: "done", skillName: displayName) + finishPlanning(with: "⚠️ 无法加载 Skill \(displayName)，已停止执行。") + return true + } + + try? await Task.sleep(for: .milliseconds(300)) + messages[cardIndex].update(role: .system, content: "loaded", skillName: displayName) + messages.append(ChatMessage(role: .skillResult, content: instructions, skillName: step.skill)) + loadedInstructions[step.skill] = instructions + } + + let displayName = loadedDisplayNames[step.skill] ?? findDisplayName(for: step.skill) + guard let cardIndex = skillCardIndices[step.skill], + let tool = toolRegistry.find(name: step.tool) else { + finishPlanning(with: "⚠️ 当前规划步骤无效，已停止执行。") + return true + } + + let arguments: [String: Any] + if tool.isParameterless { + arguments = [:] + } else { + let completedSummary = completedPlanSummary(completedSteps) + let argumentsPrompt = PromptBuilder.buildPlannedToolArgumentsPrompt( + originalPrompt: prompt, + userQuestion: userQuestion, + stepIntent: step.intent, + toolName: step.tool, + toolParameters: tool.parameters, + completedStepSummary: completedSummary, + currentImageCount: images.count + ) + + guard let rawArguments = await streamLLM(prompt: argumentsPrompt, images: images) else { + finishPlanning(with: "⚠️ 无法提取步骤参数，请重试。") + return true + } + + let cleanedArguments = cleanOutput(rawArguments) + guard let payload = parseJSONObject(cleanedArguments) else { + finishPlanning(with: "⚠️ 无法提取步骤参数，请把需求说得更具体一些。") + return true + } + + if let clarification = payload["_needs_clarification"] as? String, + !clarification.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { + finishPlanning(with: clarification) + return true + } + + guard toolRegistry.validatesArguments(payload, for: step.tool) else { + finishPlanning(with: "⚠️ 当前步骤缺少必要参数，请把需求说得更具体一些。") + return true + } + + arguments = payload + } + + messages[cardIndex].update( + role: .system, + content: "executing:\(step.tool)", + skillName: displayName + ) + + do { + let toolResult = try await handleToolExecution(toolName: step.tool, args: arguments) + let toolResultSummary = toolResultSummaryForModel( + toolName: step.tool, + toolResult: toolResult + ) + + messages[cardIndex].update(role: .system, content: "done", skillName: displayName) + messages.append(ChatMessage(role: .skillResult, content: toolResultSummary, skillName: step.tool)) + + if let payload = parsedToolPayload(from: toolResult), + let success = payload["success"] as? Bool, + !success { + let error = payload["error"] as? String ?? toolResultSummary + finishPlanning(with: error) + return true + } + + completedSteps.append( + ExecutedPlanStep( + step: step, + toolResult: toolResult, + toolResultSummary: toolResultSummary + ) + ) + toolResultsForAnswer.append((toolName: step.tool, result: toolResultSummary)) + } catch { + messages[cardIndex].update(role: .system, content: "done", skillName: displayName) + finishPlanning(with: "❌ Tool 执行失败:\(error.localizedDescription)") + return true + } + } + + remainingSkillIds.removeAll { executedSkillIdsThisPass.contains($0) } + } + + if !remainingSkillIds.isEmpty { + finishPlanning(with: "⚠️ 还缺少部分步骤未完成，请把需求说得更具体一些。") + return true + } + + let followUpPrompt = PromptBuilder.buildMultiToolAnswerPrompt( + originalPrompt: prompt, + toolResults: toolResultsForAnswer, + userQuestion: userQuestion, + currentImageCount: images.count + ) + + messages.append(ChatMessage(role: .assistant, content: "▍")) + let followUpIndex = messages.count - 1 + + guard let nextText = await streamLLM(prompt: followUpPrompt, msgIndex: followUpIndex, images: images) else { + markSkillsDone(Array(loadedDisplayNames.values)) + isProcessing = false + return true + } + + if !parseAllToolCalls(nextText).isEmpty { + log("[Agent] planner follow-up detected extra tool call") + messages[followUpIndex].update(content: "") + await executeToolChain( + prompt: followUpPrompt, + fullText: nextText, + userQuestion: userQuestion, + images: images + ) + return true + } + + let cleaned = cleanOutput(nextText) + let finalReply: String + if cleaned.isEmpty + || looksLikeStructuredIntermediateOutput(cleaned) + || looksLikePromptEcho(cleaned) { + finalReply = toolResultsForAnswer.map(\.result).joined(separator: "\n") + } else { + finalReply = cleaned + } + + messages[followUpIndex].update(content: finalReply) + markSkillsDone(Array(loadedDisplayNames.values)) + isProcessing = false + return true + } +} diff --git a/Agent/Engine/Router.swift b/Agent/Engine/Router.swift new file mode 100644 index 0000000..08d656f --- /dev/null +++ b/Agent/Engine/Router.swift @@ -0,0 +1,151 @@ +import Foundation + +extension AgentEngine { + + // MARK: - 通用工具 + + func uniqueStringsPreservingOrder(_ values: [String]) -> [String] { + Array(NSOrderedSet(array: values)) as? [String] ?? values + } + + // MARK: - Skill 触发匹配 + + /// 仅依赖 SKILL.md 的 triggers / allowedTools 字段, 零硬编关键词。 + /// + /// 支持 **sticky routing**: 如果当前消息不含任何 trigger, 但最近 history + /// 里有活跃的 skill 上下文 (skillResult 或系统卡片里带 skillName), + /// 认为用户在对同一个 skill 的多轮对话中做 follow-up, 继续路由到那个 skill。 + /// 这样 "明天下午14点的" 这种纯补全消息也能命中上一轮的 calendar skill, + /// 避免落到 light 路径丢失 skill 能力。 + func matchedSkillIds(for userQuestion: String) -> [String] { + let normalizedQuestion = userQuestion.lowercased() + guard !normalizedQuestion.isEmpty else { return [] } + + var matched: [String] = [] + for entry in skillEntries where entry.isEnabled { + let skillId = entry.id + let lowercasedNames = [ + skillId.lowercased(), + entry.name.lowercased() + ] + + var isMatch = lowercasedNames.contains { normalizedQuestion.contains($0) } + if !isMatch, + let definition = skillRegistry.getDefinition(skillId) { + isMatch = definition.metadata.triggers.contains { trigger in + let normalizedTrigger = trigger.lowercased().trimmingCharacters(in: .whitespacesAndNewlines) + return !normalizedTrigger.isEmpty && normalizedQuestion.contains(normalizedTrigger) + } || definition.metadata.allowedTools.contains { toolName in + normalizedQuestion.contains(toolName.lowercased()) + } + } + + if isMatch { + matched.append(skillId) + } + } + + // Sticky routing: 当前消息没命中任何 trigger, 但最近 history 里有 + // 活跃 skill 上下文 -> 继续使用该 skill。 + if matched.isEmpty, let stickySkillId = recentActiveSkillId() { + matched.append(stickySkillId) + } + + return uniqueStringsPreservingOrder(matched) + } + + /// 在"上一轮 user turn"范围内查找活跃的 skill 上下文。 + /// + /// 语义边界: 从最后一条 user message 倒着扫到上一条 user message 之间, + /// 这一段消息是"上一轮 user turn 触发的所有 agent 行为"。在这个范围内 + /// 找任何 .skillResult 或 .system(skillName) 消息, 第一个匹配即返回。 + /// + /// 跨越上一条 user message 后停止 — 再往前的 skill 上下文已经是更早 + /// 的对话, 不再相关。 + /// + /// 为什么不用固定窗口 (suffix(4))? + /// 一个完整 agent loop 会 append 6-10 条消息 (load_skill, identified, + /// loaded, skillResult, executing, done, follow-up assistant 等), + /// 固定窗口 4 经常错过 skill 上下文, 导致多轮对话失去 sticky 能力。 + /// 语义边界与 message 数量解耦, 任何长度的 agent loop 都能正确接住。 + /// + /// 这是纯框架层判定 — 不感知任何具体 skill 名, 不硬编任何业务字符串。 + private func recentActiveSkillId() -> String? { + var sawCurrentUser = false + for msg in messages.reversed() { + if msg.role == .user { + if sawCurrentUser { + // 跨越了上一条 user message, 停止扫描 + return nil + } + sawCurrentUser = true + continue + } + guard (msg.role == .skillResult || msg.role == .system), + let name = msg.skillName, !name.isEmpty else { + continue + } + + // name 可能是 skill id (如 "calendar") 或 tool name (如 "calendar-create-event")。 + // 优先作为 skill id 解析; 不行再作为 tool name 反查 skill id。 + let asSkillId = skillRegistry.canonicalSkillId(for: name) + if let def = skillRegistry.getDefinition(asSkillId), def.isEnabled { + return asSkillId + } + if let skillId = skillRegistry.findSkillId(forTool: name), + let def = skillRegistry.getDefinition(skillId), + def.isEnabled { + return skillId + } + } + return nil + } + + func canonicalSkillSelectionEntry(_ rawValue: String) -> String? { + let trimmed = rawValue.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { return nil } + + let directSkillId = skillRegistry.canonicalSkillId(for: trimmed) + if skillEntries.contains(where: { $0.isEnabled && $0.id == directSkillId }) { + return directSkillId + } + + let normalizedToolName = canonicalToolName( + trimmed + .replacingOccurrences(of: "_", with: "-") + .lowercased(), + arguments: [:] + ) + + for entry in skillEntries where entry.isEnabled { + let toolNames = Set(registeredTools(for: entry.id).map(\.name)) + if toolNames.contains(normalizedToolName) { + return entry.id + } + } + + return nil + } + + // MARK: - 路由决策 + + func shouldUseToolingPrompt(for userQuestion: String) -> Bool { + let normalizedQuestion = userQuestion.lowercased() + guard !normalizedQuestion.isEmpty else { return false } + // 完全依赖 SKILL.md 的 triggers 字段，不再硬编任何领域关键词 + return !matchedSkillIds(for: userQuestion).isEmpty + } + + /// 纯函数：根据已计算的条件变量确定 processInput 的路由路径。 + /// 可独立单元测试，也用于埋点日志。 + static func decideRoute( + requiresMultimodal: Bool, + shouldUsePlanner: Bool, + shouldUseFullAgentPrompt: Bool + ) -> String { + if requiresMultimodal { return "vlm" } + if shouldUsePlanner { return "planner" } + if shouldUseFullAgentPrompt { return "agent" } + return "light" + } +} diff --git a/Agent/Engine/SessionStore.swift b/Agent/Engine/SessionStore.swift new file mode 100644 index 0000000..f21592b --- /dev/null +++ b/Agent/Engine/SessionStore.swift @@ -0,0 +1,214 @@ +import Foundation + +// MARK: - SessionStore 内部数据类型 + +struct ChatSessionRecord: Codable { + let id: UUID + var title: String + var preview: String + var updatedAt: Date + var messages: [ChatMessage] +} + +extension AgentEngine { + + // MARK: - 保存调度 + + func scheduleSessionSave() { + sessionSaveTask?.cancel() + let currentID = currentSessionID + sessionSaveTask = Task { [weak self] in + try? await Task.sleep(for: .milliseconds(350)) + guard !Task.isCancelled, let self, self.currentSessionID == currentID else { return } + self.saveCurrentSession() + } + } + + func saveCurrentSession() { + guard !messages.isEmpty else { return } + + let summary = makeSessionSummary( + id: currentSessionID, + messages: messages + ) + let record = ChatSessionRecord( + id: currentSessionID, + title: summary.title, + preview: summary.preview, + updatedAt: summary.updatedAt, + messages: messages + ) + + let encoder = JSONEncoder() + encoder.outputFormatting = [.sortedKeys] + encoder.dateEncodingStrategy = .iso8601 + + do { + let directory = try ensureSessionsDirectory() + let data = try encoder.encode(record) + try data.write(to: sessionFileURL(for: currentSessionID), options: .atomic) + updateSessionSummary(summary) + persistSessionsIndex() + UserDefaults.standard.set(currentSessionID.uuidString, forKey: Self.currentSessionDefaultsKey) + _ = directory + } catch { + log("[History] save failed: \(error.localizedDescription)") + } + } + + // MARK: - 加载 + + func loadPersistedSessions() { + do { + _ = try ensureSessionsDirectory() + } catch { + log("[History] setup failed: \(error.localizedDescription)") + } + + sessionSummaries = loadSessionsIndex().sorted { $0.updatedAt > $1.updatedAt } + + if let record = loadSessionRecord(id: currentSessionID) { + messages = record.messages + updateSessionSummary( + .init( + id: record.id, + title: record.title, + preview: record.preview, + updatedAt: record.updatedAt + ) + ) + return + } + + if let first = sessionSummaries.first, let record = loadSessionRecord(id: first.id) { + currentSessionID = first.id + UserDefaults.standard.set(currentSessionID.uuidString, forKey: Self.currentSessionDefaultsKey) + messages = record.messages + return + } + + currentSessionID = UUID() + UserDefaults.standard.set(currentSessionID.uuidString, forKey: Self.currentSessionDefaultsKey) + messages = [] + } + + func loadSessionRecord(id: UUID) -> ChatSessionRecord? { + let url = sessionFileURL(for: id) + guard let data = try? Data(contentsOf: url) else { return nil } + let decoder = JSONDecoder() + decoder.dateDecodingStrategy = .iso8601 + return try? decoder.decode(ChatSessionRecord.self, from: data) + } + + // MARK: - 摘要 + + func makeSessionSummary(id: UUID, messages: [ChatMessage]) -> ChatSessionSummary { + let firstUser = messages.first(where: { $0.role == .user }) + let lastMeaningful = messages.reversed().first(where: { msg in + if !msg.content.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { + return true + } + return !msg.images.isEmpty || !msg.audios.isEmpty + }) + + let title = sessionTitle(from: firstUser) + let preview = sessionPreview(from: lastMeaningful ?? firstUser) + let updatedAt = (lastMeaningful ?? firstUser)?.timestamp ?? Date() + + return ChatSessionSummary( + id: id, + title: title, + preview: preview, + updatedAt: updatedAt + ) + } + + private func sessionTitle(from message: ChatMessage?) -> String { + guard let message else { return "新会话" } + let trimmed = message.content.trimmingCharacters(in: .whitespacesAndNewlines) + if !trimmed.isEmpty { + return String(trimmed.prefix(24)) + } + if !message.images.isEmpty && !message.audios.isEmpty { + return "图片与语音会话" + } + if !message.images.isEmpty { + return "图片会话" + } + if !message.audios.isEmpty { + return "语音会话" + } + return "新会话" + } + + private func sessionPreview(from message: ChatMessage?) -> String { + guard let message else { return "暂无内容" } + let trimmed = message.content.trimmingCharacters(in: .whitespacesAndNewlines) + if !trimmed.isEmpty { + return String(trimmed.prefix(80)) + } + if !message.images.isEmpty && !message.audios.isEmpty { + return "包含图片与语音" + } + if !message.images.isEmpty { + return "包含图片" + } + if !message.audios.isEmpty { + return "包含语音" + } + return "暂无内容" + } + + func updateSessionSummary(_ summary: ChatSessionSummary) { + if let index = sessionSummaries.firstIndex(where: { $0.id == summary.id }) { + sessionSummaries[index] = summary + } else { + sessionSummaries.append(summary) + } + sessionSummaries.sort { $0.updatedAt > $1.updatedAt } + } + + // MARK: - 索引文件 + + func persistSessionsIndex() { + let encoder = JSONEncoder() + encoder.outputFormatting = [.sortedKeys] + encoder.dateEncodingStrategy = .iso8601 + do { + let data = try encoder.encode(sessionSummaries) + try data.write(to: sessionsIndexURL(), options: .atomic) + } catch { + log("[History] index save failed: \(error.localizedDescription)") + } + } + + private func loadSessionsIndex() -> [ChatSessionSummary] { + let url = sessionsIndexURL() + guard let data = try? Data(contentsOf: url) else { return [] } + let decoder = JSONDecoder() + decoder.dateDecodingStrategy = .iso8601 + return (try? decoder.decode([ChatSessionSummary].self, from: data)) ?? [] + } + + // MARK: - URL helpers + + func ensureSessionsDirectory() throws -> URL { + let directory = sessionsDirectoryURL() + try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true) + return directory + } + + func sessionsIndexURL() -> URL { + sessionsDirectoryURL().appendingPathComponent(sessionsIndexFileName) + } + + func sessionFileURL(for id: UUID) -> URL { + sessionsDirectoryURL().appendingPathComponent("\(id.uuidString).json") + } + + func sessionsDirectoryURL() -> URL { + let supportDir = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first! + let appDir = supportDir.appendingPathComponent("PhoneClaw", isDirectory: true) + return appDir.appendingPathComponent(sessionsDirectoryName, isDirectory: true) + } +} diff --git a/Agent/Engine/ToolCallParser.swift b/Agent/Engine/ToolCallParser.swift new file mode 100644 index 0000000..cd3246e --- /dev/null +++ b/Agent/Engine/ToolCallParser.swift @@ -0,0 +1,170 @@ +import Foundation + +extension AgentEngine { + + // MARK: - 工具名归一化 + + func canonicalToolName( + _ toolName: String, + arguments: [String: Any], + preferredSkillId: String? = nil + ) -> String { + let normalizedToolName = toolName + .trimmingCharacters(in: .whitespacesAndNewlines) + .replacingOccurrences(of: "_", with: "-") + .lowercased() + + // 协议级 sentinel tool 的名称归一化。 + // load_skill / list_skills 不是注册在 ToolRegistry 的业务工具， + // 而是 AgentEngine 与 LLM 之间的协议入口（在 executeToolChain 内被特殊 + // hijack 处理），因此其名称归一化属于 caller 协议范畴而非 per-skill 硬编。 + switch normalizedToolName { + case "load-skill": + return "load_skill" + case "list-skills", "list-skill", "listskills": + return "list_skills" + default: + break + } + + // 委托 ToolRegistry 别名查询 + if let canonical = toolRegistry.canonicalName(for: normalizedToolName) { + return canonical + } + + // 按 preferredSkillId 做 suffix 匹配（通用逻辑，不涉及领域语义） + if let preferredSkillId { + let resolvedSkillId = skillRegistry.canonicalSkillId(for: preferredSkillId) + let allowedTools = registeredTools(for: resolvedSkillId).map(\.name) + + if allowedTools.contains(normalizedToolName) { + return normalizedToolName + } + + let suffixMatches = allowedTools.filter { allowedTool in + let normalizedAllowed = allowedTool.lowercased() + if normalizedAllowed.hasSuffix("-" + normalizedToolName) { + return true + } + let strippedPrefix = normalizedAllowed.replacingOccurrences( + of: resolvedSkillId.lowercased() + "-", + with: "" + ) + return strippedPrefix == normalizedToolName + } + if suffixMatches.count == 1, let match = suffixMatches.first { + return match + } + + if allowedTools.count == 1, let onlyTool = allowedTools.first { + let strippedPrefix = onlyTool + .lowercased() + .replacingOccurrences(of: resolvedSkillId.lowercased() + "-", with: "") + if strippedPrefix.contains(normalizedToolName) + || normalizedToolName.contains(strippedPrefix) { + return onlyTool + } + } + } + + return normalizedToolName + } + + // MARK: - JSON 解析 + + func parseJSONObject(_ text: String) -> [String: Any]? { + let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { return nil } + + let candidates: [String] = { + if trimmed.hasPrefix("```json") || trimmed.hasPrefix("```") { + let stripped = trimmed + .replacingOccurrences(of: "```json", with: "") + .replacingOccurrences(of: "```", with: "") + .trimmingCharacters(in: .whitespacesAndNewlines) + return [stripped] + } + + if let start = trimmed.firstIndex(of: "{"), + let end = trimmed.lastIndex(of: "}"), + start <= end { + return [trimmed, String(trimmed[start...end])] + } + + return [trimmed] + }() + + for candidate in candidates { + guard let data = candidate.data(using: .utf8), + let object = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { + continue + } + return object + } + + return nil + } + + // MARK: - tool_call 文本解析 + // + // Cage: parser 是框架边界, 同时承担"白名单校验"职责。任何不在 + // ToolRegistry (业务工具) 或 protocolTools (load_skill / list_skills 等 + // 框架内置 sentinel) 的名字一律 drop, 防止 LLM 幻觉穿透到 executeToolChain + // 创建 phantom card / 触发未知 tool 错误。 + // + // 这是 ToolRegistry 真正成为"白名单"的地方——之前 registry 只在 + // executeToolChain 末梢被消费, 校验缺位导致幻觉的 tool_call 在框架内 + // 多次易手才被丢弃。 + + /// 协议级 sentinel tool: 不在 ToolRegistry 注册, 但是框架与 LLM 的 + /// 通信协议入口, 由 executeToolChain 内特殊 hijack 处理。 + private static let protocolTools: Set = ["load_skill", "list_skills"] + + func parseToolCall(_ text: String) -> (name: String, arguments: [String: Any])? { + return parseAllToolCalls(text).first + } + + func parseAllToolCalls(_ text: String) -> [(name: String, arguments: [String: Any])] { + let raw = rawParseToolCalls(text) + return raw.compactMap { call in + let canonical = canonicalToolName(call.name, arguments: call.arguments) + if Self.protocolTools.contains(canonical) { + return (canonical, call.arguments) + } + if toolRegistry.find(name: canonical) != nil { + return (canonical, call.arguments) + } + print("[Parser] dropped invalid tool_call: \(call.name) (canonical: \(canonical))") + return nil + } + } + + /// 原始正则提取, 不做任何 registry 校验。 + /// 仅供 parseAllToolCalls 内部使用; 外部 caller 必须走 parseAllToolCalls + /// 以确保 cage 校验不被绕过。 + private func rawParseToolCalls(_ text: String) -> [(name: String, arguments: [String: Any])] { + var results: [(name: String, arguments: [String: Any])] = [] + let patterns = [ + "\\s*(\\{.*?\\})\\s*", + "```json\\s*(\\{.*?\\})\\s*```", + "\\s*(\\{.*?\\})\\s*" + ] + for pattern in patterns { + guard let regex = try? NSRegularExpression(pattern: pattern, options: .dotMatchesLineSeparators) else { continue } + let range = NSRange(text.startIndex..., in: text) + let matches = regex.matches(in: text, range: range) + for match in matches { + if let jsonRange = Range(match.range(at: 1), in: text) { + let json = String(text[jsonRange]) + if let data = json.data(using: .utf8), + let dict = try? JSONSerialization.jsonObject(with: data) as? [String: Any], + let name = dict["name"] as? String { + results.append((name, dict["arguments"] as? [String: Any] ?? [:])) + } + } + } + if !results.isEmpty { break } + } + return results + } +} diff --git a/Agent/Engine/ToolChain.swift b/Agent/Engine/ToolChain.swift new file mode 100644 index 0000000..b4fa5c9 --- /dev/null +++ b/Agent/Engine/ToolChain.swift @@ -0,0 +1,579 @@ +import CoreImage +import Foundation + +// MARK: - ToolChain 内部数据类型 + +enum SingleToolExtractionOutcome { + case toolCall(name: String, arguments: [String: Any]) + case needsClarification(String) + case failed +} + +extension AgentEngine { + + // MARK: - Tool 注册查询 + + func registeredTools(for skillId: String) -> [RegisteredTool] { + if let def = skillRegistry.getDefinition(skillId) { + let tools = toolRegistry.toolsFor(names: def.metadata.allowedTools) + if !tools.isEmpty { return tools } + } + + if let entry = skillEntries.first(where: { $0.id == skillId }) { + let tools = entry.tools.compactMap { toolRegistry.find(name: $0.name) } + if !tools.isEmpty { return tools } + } + + return [] + } + + // MARK: - 单 Skill 自动 / 引导式工具调用 + + func autoToolCallForLoadedSkills( + skillIds: [String] + ) -> (name: String, arguments: [String: Any])? { + let uniqueSkillIds = Array(NSOrderedSet(array: skillIds)) as? [String] ?? skillIds + + guard uniqueSkillIds.count == 1, + let skillId = uniqueSkillIds.first, + let def = skillRegistry.getDefinition(skillId), + def.isEnabled else { + return nil + } + + let uniqueToolNames = Array(NSOrderedSet(array: def.metadata.allowedTools)) as? [String] + ?? def.metadata.allowedTools + guard uniqueToolNames.count == 1, + let toolName = uniqueToolNames.first, + let tool = toolRegistry.find(name: toolName), + tool.isParameterless else { + return nil + } + + return (tool.name, [:]) + } + + func singleRegisteredToolForLoadedSkills(skillIds: [String]) -> RegisteredTool? { + let uniqueSkillIds = Array(NSOrderedSet(array: skillIds)) as? [String] ?? skillIds + guard uniqueSkillIds.count == 1, + let skillId = uniqueSkillIds.first else { + return nil + } + + let tools = registeredTools(for: skillId) + guard tools.count == 1 else { return nil } + return tools.first + } + + func extractToolCallForLoadedSkills( + originalPrompt: String, + userQuestion: String, + skillInstructions: String, + skillIds: [String], + images: [CIImage] + ) async -> SingleToolExtractionOutcome { + let uniqueSkillIds = Array(NSOrderedSet(array: skillIds)) as? [String] ?? skillIds + guard uniqueSkillIds.count == 1, + let skillId = uniqueSkillIds.first else { + return .failed + } + + let tools = registeredTools(for: skillId) + .filter { !$0.isParameterless } + guard !tools.isEmpty else { + return .failed + } + + if tools.count == 1, let tool = tools.first { + let extractionPrompt = PromptBuilder.buildSingleToolArgumentsPrompt( + originalPrompt: originalPrompt, + userQuestion: userQuestion, + skillInstructions: skillInstructions, + toolName: tool.name, + toolParameters: tool.parameters, + currentImageCount: images.count + ) + + if let raw = await streamLLM(prompt: extractionPrompt, images: images) { + let cleaned = cleanOutput(raw) + if let payload = parseJSONObject(cleaned) { + if let clarification = payload["_needs_clarification"] as? String, + !clarification.trimmingCharacters(in: CharacterSet.whitespacesAndNewlines).isEmpty { + return .needsClarification(clarification) + } + + if toolRegistry.validatesArguments(payload, for: tool.name) { + return .toolCall(name: tool.name, arguments: payload) + } + } + } + + return .failed + } + + let allowedToolsSummary = tools.map { + "- \($0.name): \($0.description)\n 参数: \($0.parameters)" + }.joined(separator: "\n") + + let extractionPrompt = PromptBuilder.buildSkillToolSelectionPrompt( + originalPrompt: originalPrompt, + userQuestion: userQuestion, + skillInstructions: skillInstructions, + allowedToolsSummary: allowedToolsSummary, + currentImageCount: images.count + ) + + if let raw = await streamLLM(prompt: extractionPrompt, images: images) { + let cleaned = cleanOutput(raw) + if let payload = parseJSONObject(cleaned) { + if let clarification = payload["_needs_clarification"] as? String, + !clarification.trimmingCharacters(in: CharacterSet.whitespacesAndNewlines).isEmpty { + return .needsClarification(clarification) + } + + if let rawName = payload["name"] as? String, + let arguments = payload["arguments"] as? [String: Any] { + let toolName = canonicalToolName(rawName, arguments: arguments) + if tools.contains(where: { $0.name == toolName }), + toolRegistry.validatesArguments(arguments, for: toolName) { + return .toolCall(name: toolName, arguments: arguments) + } + } + } + } + + return .failed + } + + // MARK: - synthetic / payload helpers + + func syntheticToolCallText( + name: String, + arguments: [String: Any] + ) -> String { + let jsonData = try? JSONSerialization.data(withJSONObject: [ + "name": name, + "arguments": arguments + ]) + let jsonString = jsonData.flatMap { String(data: $0, encoding: .utf8) } + ?? "{\"name\":\"\(name)\",\"arguments\":{}}" + return """ + + \(jsonString) + + """ + } + + func parsedToolPayload(from toolResult: String) -> [String: Any]? { + guard let data = toolResult.data(using: .utf8), + let payload = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { + return nil + } + return payload + } + + func toolResultSummaryForModel( + toolName: String, + toolResult: String + ) -> String { + let trimmed = toolResult.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { return "工具 \(toolName) 已执行，但没有返回内容。" } + + if let payload = parsedToolPayload(from: trimmed) { + if let success = payload["success"] as? Bool, + !success, + let error = payload["error"] as? String, + !error.isEmpty { + return "工具 \(toolName) 执行失败：\(error)" + } + + if let result = payload["result"] as? String { + let summary = result.trimmingCharacters(in: .whitespacesAndNewlines) + if !summary.isEmpty { return summary } + } + } + + return trimmed + } + + func fallbackReplyForEmptyToolFollowUp(toolName: String, toolResult: String) -> String { + let trimmed = toolResult.trimmingCharacters(in: .whitespacesAndNewlines) + let summary = toolResultSummaryForModel(toolName: toolName, toolResult: trimmed) + if !summary.isEmpty, summary != trimmed { + return summary + } + + if trimmed.isEmpty { + return "工具 \(toolName) 已执行，但没有返回内容。" + } + + return """ + 工具 \(toolName) 已执行完成，但模型没有生成最终回答。 + 工具返回结果： + \(trimmed) + """ + } + + func fallbackReplyForEmptySkillFollowUp(skillName: String) -> String { + "Skill \(skillName) 已加载，但模型没有继续生成工具调用或最终回答。请重试，或把问题说得更具体一些。" + } + + func markSkillsDone(_ displayNames: [String]) { + guard !displayNames.isEmpty else { return } + for index in messages.indices { + guard messages[index].role == .system, + let skillName = messages[index].skillName, + displayNames.contains(skillName), + messages[index].content == "identified" || messages[index].content == "loaded" else { + continue + } + messages[index].update(role: .system, content: "done", skillName: skillName) + } + } + + // MARK: - Tool 调用主循环 + + func executeToolChain( + prompt: String, + fullText: String, + userQuestion: String, + images: [CIImage], + round: Int = 1, + maxRounds: Int = 10 + ) async { + guard round <= maxRounds else { + log("[Agent] 达到最大工具链轮数 \(maxRounds)") + isProcessing = false + return + } + + guard let parsedCall = parseToolCall(fullText) else { + let cleaned = cleanOutput(fullText) + if let lastAssistant = messages.lastIndex(where: { $0.role == .assistant }) { + messages[lastAssistant].update(content: cleaned.isEmpty ? "（无回复）" : cleaned) + } + isProcessing = false + return + } + + let call = ( + name: canonicalToolName(parsedCall.name, arguments: parsedCall.arguments), + arguments: parsedCall.arguments + ) + + log("[Agent] Round \(round): tool_call name=\(call.name)") + + // ── list_skills ── + if call.name == "list_skills" { + let query = (call.arguments["query"] as? String ?? "").lowercased() + let results = skillEntries.filter(\.isEnabled).filter { entry in + guard !query.isEmpty else { return true } + return entry.id.lowercased().contains(query) + || entry.name.lowercased().contains(query) + || entry.description.lowercased().contains(query) + } + let listing = results.map { "\($0.id): \($0.description)" }.joined(separator: "\n") + let resultText = results.isEmpty + ? "没有找到匹配「\(query)」的能力。" + : "可用能力（\(results.count) 个）：\n\(listing)" + log("[Agent] list_skills query=\"\(query)\" results=\(results.count)") + + let toolResultSummary = toolResultSummaryForModel(toolName: "list_skills", toolResult: resultText) + messages.append(ChatMessage(role: .skillResult, content: toolResultSummary, skillName: "list_skills")) + + let followUpPrompt = PromptBuilder.buildToolAnswerPrompt( + originalPrompt: prompt, + toolName: "list_skills", + toolResultSummary: toolResultSummary, + userQuestion: userQuestion, + currentImageCount: images.count + ) + + messages.append(ChatMessage(role: .assistant, content: "▍")) + let followUpIndex = messages.count - 1 + + guard let nextText = await streamLLM(prompt: followUpPrompt, msgIndex: followUpIndex, images: images) else { + isProcessing = false + return + } + + if parseToolCall(nextText) != nil { + log("[Agent] list_skills 后检测到 tool 调用 (round \(round + 1))") + messages[followUpIndex].update(content: "") + await executeToolChain( + prompt: followUpPrompt, fullText: nextText, + userQuestion: userQuestion, images: images, round: round + 1, maxRounds: maxRounds + ) + } else { + let cleaned = cleanOutput(nextText) + messages[followUpIndex].update(content: cleaned.isEmpty ? "（无回复）" : cleaned) + isProcessing = false + } + return + } + + // ── load_skill ── + if call.name == "load_skill" { + let allCalls = parseAllToolCalls(fullText) + let loadSkillCalls = allCalls.filter { $0.name == "load_skill" } + + var allInstructions = "" + var loadedDisplayNames: [String] = [] + var loadedSkillIds: [String] = [] + for lsCall in loadSkillCalls { + let requestedSkillName = (lsCall.arguments["skill"] as? String) + ?? (lsCall.arguments["name"] as? String) + ?? "" + let skillName = skillRegistry.canonicalSkillId(for: requestedSkillName) + log("[Agent] load_skill: \(requestedSkillName)") + + let displayName = findDisplayName(for: skillName) + loadedDisplayNames.append(displayName) + messages.append(ChatMessage(role: .system, content: "identified", skillName: displayName)) + let cardIdx = messages.count - 1 + + guard let instructions = handleLoadSkill(skillName: skillName) else { + messages[cardIdx].update(role: .system, content: "done", skillName: displayName) + continue + } + + try? await Task.sleep(for: .milliseconds(300)) + messages[cardIdx].update(role: .system, content: "loaded", skillName: displayName) + messages.append(ChatMessage(role: .skillResult, content: instructions, skillName: skillName)) + allInstructions += instructions + "\n\n" + loadedSkillIds.append(skillName) + } + + guard !allInstructions.isEmpty else { + isProcessing = false + return + } + + if let autoCall = autoToolCallForLoadedSkills(skillIds: loadedSkillIds) { + let syntheticToolCall = syntheticToolCallText( + name: autoCall.name, + arguments: autoCall.arguments + ) + await executeToolChain( + prompt: prompt, + fullText: syntheticToolCall, + userQuestion: userQuestion, + images: images, + round: round + 1, + maxRounds: maxRounds + ) + return + } + + let singleToolExtraction = await extractToolCallForLoadedSkills( + originalPrompt: prompt, + userQuestion: userQuestion, + skillInstructions: allInstructions, + skillIds: loadedSkillIds, + images: images + ) + switch singleToolExtraction { + case .toolCall(let name, let arguments): + log("[Agent] load_skill 参数提取后执行工具: \(name)") + let syntheticToolCall = syntheticToolCallText(name: name, arguments: arguments) + await executeToolChain( + prompt: prompt, + fullText: syntheticToolCall, + userQuestion: userQuestion, + images: images, + round: round + 1, + maxRounds: maxRounds + ) + return + + case .needsClarification(let clarification): + messages.append(ChatMessage(role: .assistant, content: clarification)) + markSkillsDone(loadedDisplayNames) + isProcessing = false + return + + case .failed: + break + } + + // 计算所有 loaded skill 的 allowed-tools 并集 (去重) + // — 这是 Scaffold T2 disclosure 的输入: 告诉模型哪些工具实际可调 + let availableTools: [String] = { + var seen = Set() + var ordered: [String] = [] + for skillId in loadedSkillIds { + guard let def = skillRegistry.getDefinition(skillId) else { continue } + for toolName in def.metadata.allowedTools where !seen.contains(toolName) { + seen.insert(toolName) + ordered.append(toolName) + } + } + return ordered + }() + + let followUpPrompt = PromptBuilder.buildLoadedSkillPrompt( + originalPrompt: prompt, + userQuestion: userQuestion, + skillInstructions: allInstructions, + availableTools: availableTools, + currentImageCount: images.count + ) + + messages.append(ChatMessage(role: .assistant, content: "▍")) + let followUpIndex = messages.count - 1 + + guard let nextText = await streamLLM(prompt: followUpPrompt, msgIndex: followUpIndex, images: images) else { + isProcessing = false + return + } + + if parseToolCall(nextText) != nil { + log("[Agent] load_skill 后检测到 tool 调用 (round \(round + 1))") + messages[followUpIndex].update(content: "") + await executeToolChain( + prompt: followUpPrompt, fullText: nextText, + userQuestion: userQuestion, images: images, round: round + 1, maxRounds: maxRounds + ) + } else { + let cleaned = cleanOutput(nextText) + if cleaned.isEmpty + || looksLikeStructuredIntermediateOutput(cleaned) + || looksLikePromptEcho(cleaned) { + let retryPrompt = PromptBuilder.buildLoadedSkillPrompt( + originalPrompt: prompt, + userQuestion: userQuestion, + skillInstructions: allInstructions, + availableTools: availableTools, + currentImageCount: images.count, + forceResponse: true + ) + + guard let retryText = await streamLLM(prompt: retryPrompt, msgIndex: followUpIndex, images: images) else { + isProcessing = false + return + } + + if parseToolCall(retryText) != nil { + log("[Agent] load_skill 重试后检测到 tool 调用 (round \(round + 1))") + messages[followUpIndex].update(content: "") + await executeToolChain( + prompt: retryPrompt, + fullText: retryText, + userQuestion: userQuestion, + images: images, + round: round + 1, + maxRounds: maxRounds + ) + } else { + let retryCleaned = cleanOutput(retryText) + let loadedSkillName = loadedDisplayNames.joined(separator: ", ").isEmpty + ? "已加载的能力" + : loadedDisplayNames.joined(separator: ", ") + let finalReply = retryCleaned.isEmpty + || looksLikeStructuredIntermediateOutput(retryCleaned) + || looksLikePromptEcho(retryCleaned) + ? fallbackReplyForEmptySkillFollowUp(skillName: loadedSkillName) + : retryCleaned + messages[followUpIndex].update(content: finalReply) + markSkillsDone(loadedDisplayNames) + isProcessing = false + } + } else { + messages[followUpIndex].update(content: cleaned) + markSkillsDone(loadedDisplayNames) + isProcessing = false + } + } + return + } + + // ── 具体 Tool 调用 ── + + let ownerSkillId = findSkillId(for: call.name) + let displayName = findDisplayName(for: call.name) + + let cardIndex: Int + if let idx = messages.lastIndex(where: { + $0.role == .system && ($0.skillName == displayName || $0.skillName == call.name) + && ($0.content == "identified" || $0.content == "loaded") + }) { + cardIndex = idx + } else { + messages.append(ChatMessage(role: .system, content: "identified", skillName: displayName)) + cardIndex = messages.count - 1 + } + + guard ownerSkillId != nil else { + messages[cardIndex].update(role: .system, content: "done", skillName: displayName) + messages.append(ChatMessage(role: .assistant, content: "⚠️ 未知工具: \(call.name)")) + isProcessing = false + return + } + + let enabledIds = Set(skillEntries.filter(\.isEnabled).map(\.id)) + guard enabledIds.contains(ownerSkillId!) else { + messages[cardIndex].update(role: .system, content: "done", skillName: displayName) + messages.append(ChatMessage(role: .assistant, content: "⚠️ Skill \(displayName) 未启用")) + isProcessing = false + return + } + + messages[cardIndex].update(role: .system, content: "executing:\(call.name)", skillName: displayName) + + do { + let toolResult = try await handleToolExecution(toolName: call.name, args: call.arguments) + let toolResultSummary = toolResultSummaryForModel(toolName: call.name, toolResult: toolResult) + messages[cardIndex].update(role: .system, content: "done", skillName: displayName) + messages.append(ChatMessage(role: .skillResult, content: toolResultSummary, skillName: call.name)) + log("[Agent] Tool \(call.name) round \(round) done") + + if toolRegistry.shouldSkipFollowUp(for: call.name) { + messages.append(ChatMessage(role: .assistant, content: toolResultSummary)) + isProcessing = false + return + } + + let followUpPrompt = PromptBuilder.buildToolAnswerPrompt( + originalPrompt: prompt, + toolName: call.name, + toolResultSummary: toolResultSummary, + userQuestion: userQuestion, + currentImageCount: images.count + ) + + messages.append(ChatMessage(role: .assistant, content: "▍")) + let followUpIndex = messages.count - 1 + + guard let nextText = await streamLLM(prompt: followUpPrompt, msgIndex: followUpIndex, images: images) else { + isProcessing = false + return + } + + if !parseAllToolCalls(nextText).isEmpty { + log("[Agent] 检测到第 \(round + 1) 轮工具调用") + messages[followUpIndex].update(content: "") + await executeToolChain( + prompt: followUpPrompt, fullText: nextText, + userQuestion: userQuestion, images: images, round: round + 1, maxRounds: maxRounds + ) + } else { + let cleaned = cleanOutput(nextText) + if cleaned.isEmpty + || looksLikeStructuredIntermediateOutput(cleaned) + || looksLikePromptEcho(cleaned) { + messages[followUpIndex].update(content: fallbackReplyForEmptyToolFollowUp( + toolName: call.name, + toolResult: toolResult + )) + } else { + messages[followUpIndex].update(content: cleaned) + } + isProcessing = false + } + } catch { + messages[cardIndex].update(role: .system, content: "done", skillName: displayName) + messages.append(ChatMessage(role: .system, content: "❌ Tool 执行失败: \(error)")) + isProcessing = false + } + } +} diff --git a/LLM/MLX/Core/BundledModel.swift b/LLM/MLX/Core/BundledModel.swift new file mode 100644 index 0000000..11731f7 --- /dev/null +++ b/LLM/MLX/Core/BundledModel.swift @@ -0,0 +1,209 @@ +import Foundation + +// MARK: - Runtime Profile 类型 +// +// 设计目标: 把"哪个模型在 headroom X 时给 Y token"的所有业务参数, 从硬编码 +// switch / tier 表迁移成**连续的线性公式**。原因: +// 1. tier 表是 step function, 在边界附近 1 MB 差距导致几百 token 跳变, 不直观 +// 2. 标定 tier 表需要逐档调数字, 校准成本高; 公式只调 4 个参数 +// 3. chunked prefill 已让 prepared 长度对峰值内存影响极小, 可以用线性外推 +// +// 所有 token 预算 = clamp(min, max(0, headroom - safetyMargin) * tokensPerMB, max) +// +// 多模态和 historyDepth 仍保留 tier 表, 因为: +// - 多模态: imageSoftCap + maxOut 双约束, 而且有硬天花板 512, 不是单调线性 +// - historyDepth: 整数消息数, 只有 4 个离散值 + +/// 线性预算公式: +/// tokens = clamp(minTokens, max(0, headroom - safetyMarginMB) * tokensPerMB, maxTokens) +public struct LinearBudgetFormula: Sendable, Equatable { + public let safetyMarginMB: Int // headroom 减去这个量才参与计算 (留给系统/transient peak) + public let tokensPerMB: Double // 1 MB usable headroom 折算多少 token + public let minTokens: Int // 下限 (即使 headroom 极低也保证最少这么多) + public let maxTokens: Int // 上限 (避免 headroom 极大时给出荒诞的输出长度) + + public init(safetyMarginMB: Int, tokensPerMB: Double, minTokens: Int, maxTokens: Int) { + self.safetyMarginMB = safetyMarginMB + self.tokensPerMB = tokensPerMB + self.minTokens = minTokens + self.maxTokens = maxTokens + } + + public func evaluate(headroom: Int) -> Int { + let usable = max(0, headroom - safetyMarginMB) + let raw = Int((Double(usable) * tokensPerMB).rounded()) + return min(maxTokens, max(minTokens, raw)) + } +} + +/// 单档预算 (仅用于 historyDepth, 整数离散映射) +public struct BudgetTier: Sendable, Equatable { + public let headroomMaxMB: Int // headroom < 此值时命中此档 + public let tokens: Int // 此处 tokens 字段是"消息条数" + + public init(headroomMaxMB: Int, tokens: Int) { + self.headroomMaxMB = headroomMaxMB + self.tokens = tokens + } +} + +/// 多模态单档预算: 双约束 (image cap + max output) + 硬天花板 +public struct MultimodalTier: Sendable, Equatable { + public let headroomMaxMB: Int + public let imageSoftTokenCap: Int? + public let maxOutputTokens: Int + + public init(headroomMaxMB: Int, imageSoftTokenCap: Int?, maxOutputTokens: Int) { + self.headroomMaxMB = headroomMaxMB + self.imageSoftTokenCap = imageSoftTokenCap + self.maxOutputTokens = maxOutputTokens + } +} + +/// 模型运行时 profile +/// +/// 历史背景: +/// 早期版本有 textSequenceBudget / thinkingSequenceBudget 两个字段, 用来表达 +/// "prompt + output 的总序列预算"。这是 chunked prefill 之前的设计, 假设 KV +/// cache 与总序列长度线性增长, prompt 必须从 output 预算里扣减。 +/// +/// chunked prefill (windowSize=256) 上线后, 单次 forward 的 transient 内存 +/// 与 prompt 长度解耦, prepared 长度对峰值内存几乎无影响 (实测 prepared +/// 290 → 3319 之间, footprint Δ 完全不相关)。"总序列预算"概念失去意义, +/// 已删除。output 上限只受 textOutputBudget / thinkingOutputBudget 单一约束, +/// 它们已经是 headroom 的函数, 内存吃紧时自动收紧, 无需再做 prepared 扣减。 +public struct ModelRuntimeProfile: Sendable { + /// 触发 thinking 模式的 marker (nil = 不支持 thinking) + public let thinkingMarker: String? + + /// 普通文本输出 token 上限 (单次 generateStream 的硬上限, 不含 prompt) + public let textOutputBudget: LinearBudgetFormula + + /// thinking 输出 token 上限 + public let thinkingOutputBudget: LinearBudgetFormula + + /// 多模态输出: 保留 tier 表 (双约束 + 硬天花板, 不适合线性化) + public let multimodalOutputTiers: [MultimodalTier] + + /// 多模态严格下限 (headroom <= 此值直接 throw multimodalMemoryRisk) + /// 0 = 不做严格下限检查 + public let multimodalCriticalHeadroomMB: Int + + /// safeHistoryDepth: 整数离散映射, 保留 tier 表 + public let historyDepthTiers: [BudgetTier] + + /// 多模态失败时建议切换到的更轻量模型 ID + public let lighterAlternativeID: String? + + public init( + thinkingMarker: String?, + textOutputBudget: LinearBudgetFormula, + thinkingOutputBudget: LinearBudgetFormula, + multimodalOutputTiers: [MultimodalTier], + multimodalCriticalHeadroomMB: Int, + historyDepthTiers: [BudgetTier], + lighterAlternativeID: String? + ) { + self.thinkingMarker = thinkingMarker + self.textOutputBudget = textOutputBudget + self.thinkingOutputBudget = thinkingOutputBudget + self.multimodalOutputTiers = multimodalOutputTiers + self.multimodalCriticalHeadroomMB = multimodalCriticalHeadroomMB + self.historyDepthTiers = historyDepthTiers + self.lighterAlternativeID = lighterAlternativeID + } +} + +// MARK: - Gemma 4 Profiles +// +// 公式系数标定原则 (基于 2026-04-08 实测日志): +// - E4B baseline footprint = 4550 MB, 生成时峰值 +400 MB transient +// - jetsam = 6144 MB, 实测 headroom 稳定在 1500-1600 MB +// - safetyMarginMB = 300: 留 200 MB jetsam buffer + 100 MB transient overhead +// - tokensPerMB 约为 1.0-1.8: 1 MB usable headroom 大概折 1-2 个 token 的 KV cache +// - sequence > output: 序列预算包含 prompt + output, 系数比 output 更高 + +public enum MLXModelProfiles { + + // MARK: Gemma 4 E2B — 轻量, 26 layers (KV cache 较小) + + public static let gemma4_e2b = ModelRuntimeProfile( + thinkingMarker: "<|think|>", + + // 单次输出 token 上限 + // @500: 1.4*250=350→min384; @1000: 1.4*750=1050; @1500: 1.4*1250=1750; @2000: cap 2048 + textOutputBudget: LinearBudgetFormula( + safetyMarginMB: 250, tokensPerMB: 1.4, minTokens: 384, maxTokens: 2_048 + ), + + // thinking 输出比文本更紧 (避免思考占满预算把答案挤掉) + // @500: min; @1000: 1.1*750=825; @1500: 1.1*1250=1375; @2000: cap 1536 + thinkingOutputBudget: LinearBudgetFormula( + safetyMarginMB: 250, tokensPerMB: 1.1, minTokens: 256, maxTokens: 1_536 + ), + + // 多模态: E2B 不像 E4B 那么吃内存, 平表 + multimodalOutputTiers: [ + MultimodalTier(headroomMaxMB: .max, imageSoftTokenCap: 160, maxOutputTokens: 512), + ], + multimodalCriticalHeadroomMB: 0, + + historyDepthTiers: [ + BudgetTier(headroomMaxMB: 500, tokens: 0), + BudgetTier(headroomMaxMB: 900, tokens: 2), + BudgetTier(headroomMaxMB: 1_500, tokens: 4), + BudgetTier(headroomMaxMB: .max, tokens: 6), + ], + + lighterAlternativeID: nil + ) + + // MARK: Gemma 4 E4B — 重量, 42 layers (KV cache 大, 更紧) + + public static let gemma4_e4b = ModelRuntimeProfile( + thinkingMarker: "<|think|>", + + // 单次输出 token 上限 + // 实测验证 @1500: 1.0*1200=1200 vs 旧 tier 1280 (基本持平) + // @1700: 1.0*1400=1400; @2000: 1.0*1700=1700→cap 1500 + textOutputBudget: LinearBudgetFormula( + safetyMarginMB: 300, tokensPerMB: 1.0, minTokens: 256, maxTokens: 1_500 + ), + + // thinking 单独放更紧的系数 + thinkingOutputBudget: LinearBudgetFormula( + safetyMarginMB: 300, tokensPerMB: 0.85, minTokens: 192, maxTokens: 1_280 + ), + + // 多模态: E4B vision 激活内存大, 必须保留细分 tier + multimodalOutputTiers: [ + MultimodalTier(headroomMaxMB: 500, imageSoftTokenCap: 48, maxOutputTokens: 120), + MultimodalTier(headroomMaxMB: 700, imageSoftTokenCap: 64, maxOutputTokens: 200), + MultimodalTier(headroomMaxMB: 900, imageSoftTokenCap: 80, maxOutputTokens: 340), + MultimodalTier(headroomMaxMB: 1_100, imageSoftTokenCap: 96, maxOutputTokens: 512), + MultimodalTier(headroomMaxMB: 1_300, imageSoftTokenCap: 128, maxOutputTokens: 512), + MultimodalTier(headroomMaxMB: .max, imageSoftTokenCap: 160, maxOutputTokens: 512), + ], + multimodalCriticalHeadroomMB: 320, + + historyDepthTiers: [ + BudgetTier(headroomMaxMB: 700, tokens: 0), + BudgetTier(headroomMaxMB: 1_100, tokens: 2), + BudgetTier(headroomMaxMB: 1_700, tokens: 4), + BudgetTier(headroomMaxMB: .max, tokens: 6), + ], + + lighterAlternativeID: "gemma-4-e2b-it-4bit" + ) + + // MARK: - Lookup + + /// 根据 model.id 查 profile + public static func profile(for modelID: String) -> ModelRuntimeProfile? { + switch modelID { + case "gemma-4-e2b-it-4bit": return gemma4_e2b + case "gemma-4-e4b-it-4bit": return gemma4_e4b + default: return nil + } + } +} diff --git a/LLM/MLX/Core/GPULifecycle.swift b/LLM/MLX/Core/GPULifecycle.swift new file mode 100644 index 0000000..2f0b2d4 --- /dev/null +++ b/LLM/MLX/Core/GPULifecycle.swift @@ -0,0 +1,90 @@ +import Foundation +#if canImport(UIKit) +import UIKit +#endif + +// MARK: - MLXLocalLLMService GPU + app lifecycle extension +// +// iPhone 禁止后台继续跑 Metal compute, 否则 jetsam 直接 kill。 +// 这个 extension 监听 UIApplication 生命周期通知, 当 app 即将进入非活动状态时 +// 立即取消当前的 generate/load task, 并把 foregroundGPUAllowed flag 置 false。 +// ensureForegroundGPUExecution() 在关键路径前调用, 阻止任何后台提交。 + +extension MLXLocalLLMService { + + func ensureForegroundGPUExecution() async throws { + #if canImport(UIKit) + let isActive = await MainActor.run { + UIApplication.shared.applicationState == .active + } + setForegroundGPUAllowed(isActive) + guard isActive else { + throw MLXError.gpuExecutionRequiresForeground + } + #endif + } + + func configureLifecycleObservers() { + #if canImport(UIKit) + let center = NotificationCenter.default + lifecycleObserverTokens = [ + center.addObserver( + forName: UIApplication.willResignActiveNotification, + object: nil, + queue: nil + ) { [weak self] _ in + self?.handleApplicationLeavingForeground() + }, + center.addObserver( + forName: UIApplication.didEnterBackgroundNotification, + object: nil, + queue: nil + ) { [weak self] _ in + self?.handleApplicationLeavingForeground() + }, + center.addObserver( + forName: UIApplication.willEnterForegroundNotification, + object: nil, + queue: nil + ) { [weak self] _ in + self?.setForegroundGPUAllowed(true) + }, + center.addObserver( + forName: UIApplication.didBecomeActiveNotification, + object: nil, + queue: nil + ) { [weak self] _ in + self?.setForegroundGPUAllowed(true) + } + ] + + Task { [weak self] in + guard let self else { return } + let isActive = await MainActor.run { + UIApplication.shared.applicationState == .active + } + self.setForegroundGPUAllowed(isActive) + } + #endif + } + + func handleApplicationLeavingForeground() { + setForegroundGPUAllowed(false) + cancelled = true + currentGenerationTask?.cancel() + currentLoadTask?.cancel() + } + + func setForegroundGPUAllowed(_ allowed: Bool) { + foregroundStateLock.lock() + foregroundGPUAllowed = allowed + foregroundStateLock.unlock() + } + + func isForegroundGPUAllowed() -> Bool { + foregroundStateLock.lock() + let allowed = foregroundGPUAllowed + foregroundStateLock.unlock() + return allowed + } +} diff --git a/LLM/MLX/MLXTokenizersLoader.swift b/LLM/MLX/Core/MLXTokenizersLoader.swift similarity index 100% rename from LLM/MLX/MLXTokenizersLoader.swift rename to LLM/MLX/Core/MLXTokenizersLoader.swift diff --git a/LLM/MLX/Core/MemoryStats.swift b/LLM/MLX/Core/MemoryStats.swift new file mode 100644 index 0000000..b72aded --- /dev/null +++ b/LLM/MLX/Core/MemoryStats.swift @@ -0,0 +1,31 @@ +import Foundation +import Darwin + +// MARK: - MemoryStats +// +// task_vm_info 包装。从 MLXLocalLLMService.swift L733-745 原样迁移, +// 额外提供 headroomMB 便捷访问。 + +enum MemoryStats { + + /// (footprint MB, jetsam limit MB) via task_info. + static func footprintMB() -> (Double, Double) { + var info = task_vm_info_data_t() + var count = mach_msg_type_number_t(MemoryLayout.size / MemoryLayout.size) + let kr = withUnsafeMutablePointer(to: &info) { + $0.withMemoryRebound(to: integer_t.self, capacity: Int(count)) { + task_info(mach_task_self_, task_flavor_t(TASK_VM_INFO), $0, &count) + } + } + guard kr == KERN_SUCCESS else { return (0, 0) } + let footprint = Double(info.phys_footprint) / 1_048_576 + let limit = Double(info.limit_bytes_remaining) / 1_048_576 + footprint + return (footprint, limit) + } + + /// 当前可用内存 headroom (MB), 用于所有 budget 计算。 + static var headroomMB: Int { + let (footprint, limit) = footprintMB() + return max(0, Int(limit - footprint)) + } +} diff --git a/LLM/MLX/Core/RuntimeBudgets.swift b/LLM/MLX/Core/RuntimeBudgets.swift new file mode 100644 index 0000000..66d1b27 --- /dev/null +++ b/LLM/MLX/Core/RuntimeBudgets.swift @@ -0,0 +1,172 @@ +import Foundation +import MLXLMCommon + +// MARK: - RuntimeBudgets +// +// 纯函数 budget 计算, 零模型 ID 字符串, 零全局状态。 +// 输入: ModelRuntimeProfile + headroom + 少量上下文标志。 +// 输出: 各类 budget struct。 +// +// 文本/思考 budget 现在使用 LinearBudgetFormula 连续公式 (2026-04-08 重构), +// 多模态和 historyDepth 仍走 tier 表 (它们的语义不适合线性化)。 + +// MARK: - Budget 返回类型 + +public struct TextBudget: Sendable, Equatable { + public let maxOutputTokens: Int + public let headroomMB: Int +} + +public struct ThinkingBudget: Sendable, Equatable { + public let maxOutputTokens: Int + public let headroomMB: Int +} + +public struct MultimodalBudget: Sendable, Equatable { + public let imageSoftTokenCap: Int? + public let maxOutputTokens: Int + public let headroomMB: Int +} + +// MARK: - RuntimeBudgets + +public enum RuntimeBudgets { + + // MARK: 通用 tier 查表 (仅 historyDepth 用) + + /// 整数离散映射: 找第一个 headroom < tier.headroomMaxMB 的档。 + /// 最后一档通常用 .max 兜底。 + static func lookup(_ tiers: [BudgetTier], headroom: Int) -> Int { + for tier in tiers where headroom < tier.headroomMaxMB { + return tier.tokens + } + return tiers.last?.tokens ?? 0 + } + + static func lookupMultimodal(_ tiers: [MultimodalTier], headroom: Int) -> MultimodalTier? { + for tier in tiers where headroom < tier.headroomMaxMB { + return tier + } + return tiers.last + } + + // MARK: 文本输出 (线性公式) + + public static func text( + profile: ModelRuntimeProfile, + headroom: Int, + enabled: Bool + ) -> TextBudget? { + guard enabled else { return nil } + return TextBudget( + maxOutputTokens: profile.textOutputBudget.evaluate(headroom: headroom), + headroomMB: headroom + ) + } + + // MARK: Thinking 输出 (线性公式) + + public static func thinking( + profile: ModelRuntimeProfile, + headroom: Int, + enabled: Bool + ) -> ThinkingBudget? { + guard enabled else { return nil } + return ThinkingBudget( + maxOutputTokens: profile.thinkingOutputBudget.evaluate(headroom: headroom), + headroomMB: headroom + ) + } + + // MARK: 多模态输出 + + /// 对应原 dynamicMultimodalBudget(hasImages:hasAudio:)。 + /// - 无图无音: 返回 nil + /// - headroom <= multimodalCriticalHeadroomMB (且该字段 > 0): throw MLXError.multimodalMemoryRisk + /// - 其他情况: 按 tier 表返回 + public static func multimodal( + profile: ModelRuntimeProfile, + headroom: Int, + hasImages: Bool, + hasAudio: Bool, + modelDisplayName: String, + fallbackRecommendation: String + ) throws -> MultimodalBudget? { + guard hasImages || hasAudio else { return nil } + + // 严格下限保护 (原 L849 E4B 专属, 通过 critical=0 关闭 E2B) + if profile.multimodalCriticalHeadroomMB > 0, + headroom <= profile.multimodalCriticalHeadroomMB { + throw MLXError.multimodalMemoryRisk( + model: modelDisplayName, + headroomMB: headroom, + recommendation: fallbackRecommendation + ) + } + + let tier = lookupMultimodal(profile.multimodalOutputTiers, headroom: headroom) + // 原代码: hasImages ? cap : nil。tier 里存的是"有图时用什么 cap"。 + let imageSoftCap = hasImages ? tier?.imageSoftTokenCap : nil + // tier 理论上不会为空, 兜底取最后一档的 maxOutputTokens 是保守行为 + let maxOut = tier?.maxOutputTokens ?? profile.multimodalOutputTiers.last?.maxOutputTokens ?? 0 + + return MultimodalBudget( + imageSoftTokenCap: imageSoftCap, + maxOutputTokens: maxOut, + headroomMB: headroom + ) + } + + // adjustedTextOutputTokens 已在 2026-04-08 删除。 + // + // 它原本基于"prompt + output 必须 fit 在总序列预算里, 所以 output 必须扣减 + // prepared"的设计。但 chunked prefill (windowSize=256) 让 prepared 长度 + // 对峰值内存几乎无影响 (实测 prepared 290 → 3319 之间, footprint Δ 完全 + // 不相关), 这个扣减成了 false positive 截断的根源。 + // + // 现在的内存安全网完全靠 textOutputBudget / thinkingOutputBudget 两个公式, + // 它们已经是 headroom 的函数, 内存吃紧时自动收紧。无需再做 prepared 扣减。 + // + // 如果将来发现真的需要"context window 限制" (而不是内存限制), 应该独立 + // 表达 (比如 ModelRuntimeProfile.contextWindowTokens), 不要复活旧的 + // sequence budget 概念。 + + // MARK: 安全 history 深度 + + /// 对应原 safeHistoryDepth (L755-778)。 + public static func safeHistoryDepth( + profile: ModelRuntimeProfile, + headroom: Int + ) -> Int { + lookup(profile.historyDepthTiers, headroom: headroom) + } + + // MARK: Thinking 检测 + + /// 对应原 isThinkingEnabled(for:) (L901-924)。 + /// 本次 Phase 1 不在 validator 覆盖范围 (纯字符串匹配, 无数值风险)。 + public static func isThinkingEnabled( + input: UserInput, + profile: ModelRuntimeProfile + ) -> Bool { + if let enabled = input.additionalContext?["enable_thinking"] as? Bool, enabled { + return true + } + guard let marker = profile.thinkingMarker else { return false } + + switch input.prompt { + case .text(let text): + return text.contains(marker) + case .chat(let messages): + return messages.contains { $0.content.contains(marker) } + case .messages(let messages): + return messages.contains { msg in + if let s = msg["content"] as? String { return s.contains(marker) } + if let arr = msg["content"] as? [[String: any Sendable]] { + return arr.contains { ($0["text"] as? String)?.contains(marker) == true } + } + return false + } + } + } +} diff --git a/LLM/MLX/Gemma4AudioFeatureExtractor.swift b/LLM/MLX/Gemma4/Gemma4AudioFeatureExtractor.swift similarity index 100% rename from LLM/MLX/Gemma4AudioFeatureExtractor.swift rename to LLM/MLX/Gemma4/Gemma4AudioFeatureExtractor.swift diff --git a/LLM/MLX/Gemma4AudioModel.swift b/LLM/MLX/Gemma4/Gemma4AudioModel.swift similarity index 100% rename from LLM/MLX/Gemma4AudioModel.swift rename to LLM/MLX/Gemma4/Gemma4AudioModel.swift diff --git a/LLM/MLX/Gemma4Config.swift b/LLM/MLX/Gemma4/Gemma4Config.swift similarity index 100% rename from LLM/MLX/Gemma4Config.swift rename to LLM/MLX/Gemma4/Gemma4Config.swift diff --git a/LLM/MLX/Gemma4LanguageModel.swift b/LLM/MLX/Gemma4/Gemma4LanguageModel.swift similarity index 100% rename from LLM/MLX/Gemma4LanguageModel.swift rename to LLM/MLX/Gemma4/Gemma4LanguageModel.swift diff --git a/LLM/MLX/Gemma4Model.swift b/LLM/MLX/Gemma4/Gemma4Model.swift similarity index 77% rename from LLM/MLX/Gemma4Model.swift rename to LLM/MLX/Gemma4/Gemma4Model.swift index 807311e..31a9d27 100644 --- a/LLM/MLX/Gemma4Model.swift +++ b/LLM/MLX/Gemma4/Gemma4Model.swift @@ -158,16 +158,18 @@ public class Gemma4Model: Module, VLMModel, KVCacheDimensionProvider { ) throws -> PrepareResult { let convertedCache = cache.compactMap { $0 as KVCache } + // 文本路径: chunked prefill 限制单次 forward 的瞬时激活内存。 + // 详见 prepareTextChunked 注释。 guard input.image?.pixels != nil || input.audio?.features != nil else { - let result = languageModel( - input.text.tokens, + return prepareTextChunked( + tokens: input.text.tokens, cache: convertedCache, - inputsEmbeds: nil, - perLayerInputs: nil + windowSize: windowSize ) - return .logits(result) } + // 多模态路径: 图像/音频 embedding 必须整体进入, + // 它们是固定大小的视觉/音频前缀, 无法天然 chunk。 let inputEmbeddings = getInputEmbeddings( inputIds: input.text.tokens, pixelValues: input.image?.pixels, @@ -185,6 +187,66 @@ public class Gemma4Model: Module, VLMModel, KVCacheDimensionProvider { return .logits(result) } + /// Chunked prefill: 把 prompt 切成 windowSize 大小的 chunk 逐次 forward, + /// 每个 chunk 之后 eval(cache) 强制 materialize KV 并释放 compute graph。 + /// + /// **数学等价性**: 因果注意力 + KV cache 下, chunk 处理与单次 full prefill + /// 在最终 KV cache 状态和最后一个 token 的 logits 上完全一致。每个 chunk + /// 的 query 通过 cache 看到所有之前 token 的 K/V, 与一次性处理无差别。 + /// + /// **内存收益**: 单 chunk transient = O(windowSize² × layers × hiddenDim) + /// 而不是 O(promptLen² × ...)。Gemma 4 E4B (42 层, 8 head, 2560 hidden) + /// 在 800 token 单 forward 下瞬时 ~1.7 GB, 加 4.6 GB 已驻留 weights+KV + /// 会撞 iPhone jetsam 6.1 GB 上限。256 chunk 下瞬时 ~400 MB, 安全。 + /// + /// **框架层修复**: 不感知 prompt 内容、skill 数量、SKILL.md 格式或任何业务, + /// 只关心张量形状与内存预算。任何 prompt 长度都自动适用。 + private func prepareTextChunked( + tokens: MLXArray, + cache: [KVCache], + windowSize: Int? + ) -> PrepareResult { + let prefillStepSize = max(windowSize ?? 512, 1) + let totalSeqLen = tokens.dim(1) + + // 短 prompt: 单次 forward, 与重构前完全等价。 + if totalSeqLen <= prefillStepSize { + let result = languageModel( + tokens, + cache: cache, + inputsEmbeds: nil, + perLayerInputs: nil + ) + return .logits(result) + } + + // 长 prompt: 切 chunk 处理。每 chunk 跑 forward → eval(cache) → + // discard transient compute graph → 下一 chunk。 + var processed = 0 + while processed + prefillStepSize < totalSeqLen { + let chunk = tokens[0..., processed ..< (processed + prefillStepSize)] + _ = languageModel( + chunk, + cache: cache, + inputsEmbeds: nil, + perLayerInputs: nil + ) + eval(cache) + processed += prefillStepSize + } + + // 最后一段 (<= prefillStepSize tokens): forward 并把 logits 返回给 + // TokenIterator, 用于采样首个 generated token。 + let lastChunk = tokens[0..., processed ..< totalSeqLen] + let result = languageModel( + lastChunk, + cache: cache, + inputsEmbeds: nil, + perLayerInputs: nil + ) + return .logits(result) + } + public func callAsFunction(_ inputs: MLXArray, cache: [any KVCache]?) -> MLXArray { let cache = cache?.compactMap { $0 as? KVCache } let out = languageModel(inputs, cache: cache, inputsEmbeds: nil, perLayerInputs: nil) diff --git a/LLM/MLX/Gemma4Processor.swift b/LLM/MLX/Gemma4/Gemma4Processor.swift similarity index 100% rename from LLM/MLX/Gemma4Processor.swift rename to LLM/MLX/Gemma4/Gemma4Processor.swift diff --git a/LLM/MLX/Gemma4Registration.swift b/LLM/MLX/Gemma4/Gemma4Registration.swift similarity index 100% rename from LLM/MLX/Gemma4Registration.swift rename to LLM/MLX/Gemma4/Gemma4Registration.swift diff --git a/LLM/MLX/Gemma4RoPE.swift b/LLM/MLX/Gemma4/Gemma4RoPE.swift similarity index 100% rename from LLM/MLX/Gemma4RoPE.swift rename to LLM/MLX/Gemma4/Gemma4RoPE.swift diff --git a/LLM/MLX/Gemma4TextModel.swift b/LLM/MLX/Gemma4/Gemma4TextModel.swift similarity index 100% rename from LLM/MLX/Gemma4TextModel.swift rename to LLM/MLX/Gemma4/Gemma4TextModel.swift diff --git a/LLM/MLX/Gemma4VisionModel.swift b/LLM/MLX/Gemma4/Gemma4VisionModel.swift similarity index 100% rename from LLM/MLX/Gemma4VisionModel.swift rename to LLM/MLX/Gemma4/Gemma4VisionModel.swift diff --git a/LLM/MLX/Installation/ModelDownloader.swift b/LLM/MLX/Installation/ModelDownloader.swift new file mode 100644 index 0000000..ebd69f6 --- /dev/null +++ b/LLM/MLX/Installation/ModelDownloader.swift @@ -0,0 +1,304 @@ +import Foundation + +// MARK: - Download types (顶层, 从主类嵌套结构提出) + +struct DownloadSource: Sendable { + let label: String + let url: URL +} + +final class DownloadTaskClient: NSObject, URLSessionDownloadDelegate { + private let progressHandler: @Sendable (Int64, Int64?) -> Void + private var continuation: CheckedContinuation<(URL, URLResponse), Error>? + private var downloadedFileURL: URL? + private var response: URLResponse? + private lazy var session: URLSession = { + URLSession(configuration: .ephemeral, delegate: self, delegateQueue: nil) + }() + private var task: URLSessionDownloadTask? + + init(progressHandler: @escaping @Sendable (Int64, Int64?) -> Void) { + self.progressHandler = progressHandler + } + + func start(request: URLRequest) async throws -> (URL, URLResponse) { + try await withTaskCancellationHandler { + try await withCheckedThrowingContinuation { continuation in + self.continuation = continuation + let task = session.downloadTask(with: request) + self.task = task + task.resume() + } + } onCancel: { + self.cancel() + } + } + + func cancel() { + task?.cancel() + session.invalidateAndCancel() + } + + func urlSession( + _ session: URLSession, + downloadTask: URLSessionDownloadTask, + didWriteData bytesWritten: Int64, + totalBytesWritten: Int64, + totalBytesExpectedToWrite: Int64 + ) { + let expected = totalBytesExpectedToWrite > 0 ? totalBytesExpectedToWrite : nil + progressHandler(totalBytesWritten, expected) + } + + func urlSession( + _ session: URLSession, + downloadTask: URLSessionDownloadTask, + didFinishDownloadingTo location: URL + ) { + do { + let temporaryFileURL = FileManager.default.temporaryDirectory + .appendingPathComponent(UUID().uuidString) + .appendingPathExtension(location.pathExtension.isEmpty ? "tmp" : location.pathExtension) + + if FileManager.default.fileExists(atPath: temporaryFileURL.path) { + try FileManager.default.removeItem(at: temporaryFileURL) + } + + try FileManager.default.moveItem(at: location, to: temporaryFileURL) + downloadedFileURL = temporaryFileURL + response = downloadTask.response + } catch { + downloadedFileURL = nil + response = nil + continuation?.resume(throwing: error) + continuation = nil + self.task = nil + session.finishTasksAndInvalidate() + } + } + + func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) { + defer { + continuation = nil + downloadedFileURL = nil + response = nil + self.task = nil + session.finishTasksAndInvalidate() + } + + if let error { + continuation?.resume(throwing: error) + return + } + + guard let downloadedFileURL, let response else { + continuation?.resume(throwing: DownloadError.invalidResponse) + return + } + + continuation?.resume(returning: (downloadedFileURL, response)) + } +} + +// MARK: - MLXLocalLLMService download extension + +extension MLXLocalLLMService { + + func cleanupStalePartialDirectories() { + let fm = FileManager.default + for model in Self.availableModels { + let partialDirectory = ModelPaths.partial(for: model) + if fm.fileExists(atPath: partialDirectory.path) { + try? fm.removeItem(at: partialDirectory) + } + } + } + + func preferredDownloadSources(for model: BundledModelOption, file: String) -> [DownloadSource] { + // HF 系路径: "{repoID}/resolve/main/{file}" — 用于 hf-mirror / huggingface.co + let hfPath = "\(model.repositoryID)/resolve/main/\(file)" + .addingPercentEncoding(withAllowedCharacters: .urlPathAllowed)? + .replacingOccurrences(of: "%2F", with: "/") + + // ModelScope 路径: "{repoID}/resolve/master/{file}" — 结构和 HF 一样, 分支名是 master + // 这个端点对 LFS 和非 LFS 文件统一处理 (/api/v1/.../repo?FilePath= 的端点 LFS 会 404) + let msPath = "\(model.repositoryID)/resolve/master/\(file)" + .addingPercentEncoding(withAllowedCharacters: .urlPathAllowed)? + .replacingOccurrences(of: "%2F", with: "/") + + var sources: [DownloadSource] = [] + // 国内优先: ModelScope 无需 VPN, 已验证 mlx-community/gemma-4-* 在其上完整可下载 + if let msPath, let url = URL(string: "https://modelscope.cn/models/\(msPath)") { + sources.append(.init(label: "modelscope.cn", url: url)) + } + if let hfPath, let url = URL(string: "https://hf-mirror.com/\(hfPath)") { + sources.append(.init(label: "hf-mirror.com", url: url)) + } + if let hfPath, let url = URL(string: "https://huggingface.co/\(hfPath)") { + sources.append(.init(label: "huggingface.co", url: url)) + } + return sources + } + + func downloadFile( + with request: URLRequest, + progress: @escaping @Sendable (Int64, Int64?) -> Void + ) async throws -> (URL, URLResponse) { + let client = DownloadTaskClient(progressHandler: progress) + return try await client.start(request: request) + } + + public func downloadModel(id: String) async { + guard let model = Self.availableModels.first(where: { $0.id == id }) else { return } + if isModelAvailable(model) { + refreshModelInstallStates() + return + } + if currentDownloadTasks[id] != nil { + return + } + + let task = Task { [weak self] in + guard let self else { return } + let fm = FileManager.default + let modelsRoot = ModelPaths.documentsRoot() + let finalDirectory = ModelPaths.downloaded(for: model) + let partialDirectory = ModelPaths.partial(for: model) + + await MainActor.run { + self.modelInstallStates[id] = .checkingSource + } + + do { + if !fm.fileExists(atPath: modelsRoot.path) { + try fm.createDirectory(at: modelsRoot, withIntermediateDirectories: true) + } + if fm.fileExists(atPath: partialDirectory.path) { + try fm.removeItem(at: partialDirectory) + } + try fm.createDirectory(at: partialDirectory, withIntermediateDirectories: true) + + let totalFiles = model.requiredFiles.count + for (index, file) in model.requiredFiles.enumerated() { + let downloadSources = preferredDownloadSources(for: model, file: file) + guard !downloadSources.isEmpty else { + throw DownloadError.invalidURL(file) + } + + var downloadedResult: (URL, URLResponse)? + var lastError: Error? + + for source in downloadSources { + try Task.checkCancellation() + + await MainActor.run { + self.modelInstallStates[id] = .downloading( + completedFiles: index, + totalFiles: totalFiles, + currentFile: file + ) + self.modelDownloadMetrics[id] = .init( + bytesReceived: 0, + totalBytes: nil, + bytesPerSecond: nil, + sourceLabel: source.label + ) + } + + let request = URLRequest( + url: source.url, + cachePolicy: .reloadIgnoringLocalCacheData, + timeoutInterval: 1800 + ) + let startTime = Date() + + do { + let result = try await downloadFile(with: request) { received, expected in + let elapsed = max(Date().timeIntervalSince(startTime), 0.001) + let bytesPerSecond = Double(received) / elapsed + Task { @MainActor [weak self] in + self?.modelDownloadMetrics[id] = .init( + bytesReceived: received, + totalBytes: expected, + bytesPerSecond: bytesPerSecond, + sourceLabel: source.label + ) + } + } + // HTTP 状态码检查必须在循环内: URLSessionDownloadTask 对 4xx/5xx + // 也会成功调 didFinishDownloadingTo (下载错误页), 如果只在循环外检查, + // 某个 source 返回 404 会直接 throw, 永远不会试下一个 source。 + guard let http = result.1 as? HTTPURLResponse else { + lastError = DownloadError.invalidResponse + print("[Downloader] \(source.label) \(file): invalid response, trying next source") + continue + } + guard (200...299).contains(http.statusCode) else { + lastError = DownloadError.httpStatus(http.statusCode) + print("[Downloader] \(source.label) \(file): HTTP \(http.statusCode), trying next source") + continue + } + downloadedResult = result + break + } catch is CancellationError { + throw CancellationError() + } catch { + lastError = error + print("[Downloader] \(source.label) \(file): \(error.localizedDescription), trying next source") + } + } + + guard let (temporaryURL, _) = downloadedResult else { + throw lastError ?? DownloadError.invalidResponse + } + + let destinationURL = partialDirectory.appendingPathComponent(file) + let parentDirectory = destinationURL.deletingLastPathComponent() + if !fm.fileExists(atPath: parentDirectory.path) { + try fm.createDirectory(at: parentDirectory, withIntermediateDirectories: true) + } + if fm.fileExists(atPath: destinationURL.path) { + try fm.removeItem(at: destinationURL) + } + try fm.moveItem(at: temporaryURL, to: destinationURL) + } + + if fm.fileExists(atPath: finalDirectory.path) { + try fm.removeItem(at: finalDirectory) + } + try fm.moveItem(at: partialDirectory, to: finalDirectory) + + await MainActor.run { + self.modelInstallStates[id] = .downloaded + self.modelDownloadMetrics[id] = nil + self.refreshModelInstallStates() + } + } catch is CancellationError { + try? fm.removeItem(at: partialDirectory) + await MainActor.run { + self.modelInstallStates[id] = .notInstalled + self.modelDownloadMetrics[id] = nil + self.refreshModelInstallStates() + } + } catch { + try? fm.removeItem(at: partialDirectory) + await MainActor.run { + self.modelDownloadMetrics[id] = nil + self.modelInstallStates[id] = .failed(error.localizedDescription) + } + } + + await MainActor.run { + self.currentDownloadTasks[id] = nil + } + } + + currentDownloadTasks[id] = task + await task.value + } + + public func cancelModelDownload(id: String) { + guard currentDownloadTasks[id] != nil else { return } + currentDownloadTasks[id]?.cancel() + } +} diff --git a/LLM/MLX/Installation/ModelInstallState.swift b/LLM/MLX/Installation/ModelInstallState.swift new file mode 100644 index 0000000..51de5ca --- /dev/null +++ b/LLM/MLX/Installation/ModelInstallState.swift @@ -0,0 +1,67 @@ +import Foundation + +// MARK: - Install State + +public enum ModelInstallState: Equatable, Sendable { + case notInstalled + case checkingSource + case downloading(completedFiles: Int, totalFiles: Int, currentFile: String) + case downloaded + case bundled + case failed(String) +} + +public struct ModelDownloadMetrics: Equatable, Sendable { + public let bytesReceived: Int64 + public let totalBytes: Int64? + public let bytesPerSecond: Double? + public let sourceLabel: String? +} + +// MARK: - Errors + +enum MLXError: LocalizedError { + case modelNotLoaded + case modelDirectoryMissing(String) + case gpuExecutionRequiresForeground + case multimodalMemoryRisk(model: String, headroomMB: Int, recommendation: String) + + var errorDescription: String? { + switch self { + case .modelNotLoaded: + return "MLX model not loaded. Call load() first." + case .modelDirectoryMissing(let modelName): + return "\(modelName) 模型文件不存在，请先在配置页下载或重新安装。" + case .gpuExecutionRequiresForeground: + return "应用进入后台时，iPhone 不允许继续提交 GPU 推理任务。" + case .multimodalMemoryRisk(let model, let headroomMB, let recommendation): + return "\(model) 当前剩余内存仅约 \(headroomMB) MB，继续处理图片/音频很可能被系统直接杀掉。\(recommendation)" + } + } +} + +enum DownloadError: LocalizedError { + case invalidURL(String) + case invalidResponse + case httpStatus(Int) + + var errorDescription: String? { + switch self { + case .invalidURL(let file): + return "无法构造下载链接：\(file)" + case .invalidResponse: + return "下载源响应无效" + case .httpStatus(let statusCode): + switch statusCode { + case 401, 403: + return "下载源拒绝访问（\(statusCode)）" + case 404: + return "模型文件不存在（404）" + case 429: + return "下载过于频繁，请稍后重试（429）" + default: + return "下载失败，HTTP \(statusCode)" + } + } + } +} diff --git a/LLM/MLX/Installation/ModelInstaller.swift b/LLM/MLX/Installation/ModelInstaller.swift new file mode 100644 index 0000000..b373e44 --- /dev/null +++ b/LLM/MLX/Installation/ModelInstaller.swift @@ -0,0 +1,58 @@ +import Foundation + +// MARK: - MLXLocalLLMService install-state extension +// +// selectModel / isModelAvailable / installState / refreshModelInstallStates / +// downloadMetrics。纯 bookkeeping, 不触碰模型权重加载 (那在主类 load() 里)。 + +extension MLXLocalLLMService { + + public func selectModel(id: String) -> Bool { + guard let option = Self.availableModels.first(where: { $0.id == id }), + option != selectedModel else { + return false + } + + selectedModel = option + statusMessage = isLoaded + ? "已选择 \(option.displayName)，准备重新加载..." + : "已选择 \(option.displayName)，等待加载..." + return true + } + + public func isModelAvailable(_ model: BundledModelOption) -> Bool { + ModelPaths.bundled(for: model) != nil + || ModelPaths.hasRequiredFiles(model, at: ModelPaths.downloaded(for: model)) + } + + public func installState(for model: BundledModelOption) -> ModelInstallState { + if ModelPaths.bundled(for: model) != nil { + return .bundled + } + if ModelPaths.hasRequiredFiles(model, at: ModelPaths.downloaded(for: model)) { + return .downloaded + } + return modelInstallStates[model.id] ?? .notInstalled + } + + public func refreshModelInstallStates() { + cleanupStalePartialDirectories() + for model in Self.availableModels { + if ModelPaths.bundled(for: model) != nil { + modelInstallStates[model.id] = .bundled + } else if ModelPaths.hasRequiredFiles(model, at: ModelPaths.downloaded(for: model)) { + modelInstallStates[model.id] = .downloaded + } else if case .checkingSource = modelInstallStates[model.id] { + continue + } else if case .downloading = modelInstallStates[model.id] { + continue + } else { + modelInstallStates[model.id] = .notInstalled + } + } + } + + public func downloadMetrics(for modelID: String) -> ModelDownloadMetrics? { + modelDownloadMetrics[modelID] + } +} diff --git a/LLM/MLX/Installation/ModelPaths.swift b/LLM/MLX/Installation/ModelPaths.swift new file mode 100644 index 0000000..6b12d2a --- /dev/null +++ b/LLM/MLX/Installation/ModelPaths.swift @@ -0,0 +1,65 @@ +import Foundation + +// MARK: - ModelPaths +// +// 静态路径解析: bundle 路径优先, 否则落到 Documents/models/。 +// 从 MLXLocalLLMService.swift L278-332 原样迁移。 + +enum ModelPaths { + + static func resolve(for model: BundledModelOption) -> URL { + if let bundled = bundled(for: model) { + return bundled + } + return downloaded(for: model) + } + + static func documentsRoot() -> URL { + let documentsPath = FileManager.default.urls( + for: .documentDirectory, + in: .userDomainMask + ).first! + return documentsPath.appendingPathComponent("models", isDirectory: true) + } + + static func downloaded(for model: BundledModelOption) -> URL { + documentsRoot().appendingPathComponent(model.directoryName, isDirectory: true) + } + + static func partial(for model: BundledModelOption) -> URL { + documentsRoot().appendingPathComponent("\(model.directoryName).partial", isDirectory: true) + } + + static func bundled(for model: BundledModelOption) -> URL? { + guard let resourceURL = Bundle.main.resourceURL else { return nil } + + let directBundleDir = resourceURL.appendingPathComponent( + model.directoryName, + isDirectory: true + ) + if hasRequiredFiles(model, at: directBundleDir) { + return directBundleDir + } + + let nestedBundleDir = resourceURL + .appendingPathComponent("Models", isDirectory: true) + .appendingPathComponent(model.directoryName, isDirectory: true) + if hasRequiredFiles(model, at: nestedBundleDir) { + return nestedBundleDir + } + + return nil + } + + static func hasRequiredFiles(_ model: BundledModelOption, at directory: URL) -> Bool { + let fm = FileManager.default + var isDirectory: ObjCBool = false + guard fm.fileExists(atPath: directory.path, isDirectory: &isDirectory), + isDirectory.boolValue else { + return false + } + return model.requiredFiles.allSatisfy { file in + fm.fileExists(atPath: directory.appendingPathComponent(file).path) + } + } +} diff --git a/LLM/MLX/MLXLocalLLMService.swift b/LLM/MLX/MLXLocalLLMService.swift new file mode 100644 index 0000000..c7e32f4 --- /dev/null +++ b/LLM/MLX/MLXLocalLLMService.swift @@ -0,0 +1,683 @@ +import Foundation +import CoreImage +import MLX +import MLXLLM +import MLXLMCommon +import MLXVLM +#if canImport(UIKit) +import UIKit +#endif + +// MARK: - MLX Local LLM Service + +public struct BundledModelOption: Identifiable, Hashable, Sendable { + public let id: String + public let directoryName: String + public let displayName: String + public let repositoryID: String + public let requiredFiles: [String] + + /// Planner 及其他结构化 JSON 输出场景是否可用。 + /// false 时 Planner 入口会被跳过（具体降级策略见 architecture-decisions.md ADR-004）。 + public let supportsStructuredPlanning: Bool + + /// 运行时 budget / thinking / fallback 行为数据。所有 headroom→token 表都在这里, + /// 框架层 RuntimeBudgets 只查表, 不判断 model.id。 + public let runtimeProfile: ModelRuntimeProfile + + // Hashable / Equatable: 仅用 id (ModelRuntimeProfile 不 Hashable) + public static func == (lhs: BundledModelOption, rhs: BundledModelOption) -> Bool { + lhs.id == rhs.id + } + public func hash(into hasher: inout Hasher) { + hasher.combine(id) + } +} + +/// MLX GPU inference service for Gemma 4. +/// Forces MLX Metal GPU path — no CPU fallback. +@Observable +public class MLXLocalLLMService: LLMEngine { + static let availableModels: [BundledModelOption] = [ + .init( + id: "gemma-4-e2b-it-4bit", + directoryName: "gemma-4-e2b-it-4bit", + displayName: "Gemma 4 E2B", + repositoryID: "mlx-community/gemma-4-e2b-it-4bit", + requiredFiles: [ + "config.json", + "generation_config.json", + "model.safetensors", + "model.safetensors.index.json", + "processor_config.json", + "tokenizer.json", + "tokenizer_config.json", + "chat_template.jinja" + ], + supportsStructuredPlanning: false, + runtimeProfile: MLXModelProfiles.gemma4_e2b + ), + .init( + id: "gemma-4-e4b-it-4bit", + directoryName: "gemma-4-e4b-it-4bit", + displayName: "Gemma 4 E4B", + repositoryID: "mlx-community/gemma-4-e4b-it-4bit", + requiredFiles: [ + "config.json", + "generation_config.json", + "model.safetensors", + "model.safetensors.index.json", + "processor_config.json", + "tokenizer.json", + "tokenizer_config.json", + "chat_template.jinja" + ], + supportsStructuredPlanning: true, + runtimeProfile: MLXModelProfiles.gemma4_e4b + ) + ] + static let defaultModel = availableModels[0] + + // MARK: - State + + public private(set) var isLoaded = false + public private(set) var isLoading = false + public private(set) var isGenerating = false + public private(set) var stats = LLMStats() + public var statusMessage = "等待加载模型..." + public internal(set) var selectedModel = defaultModel + public internal(set) var loadedModel: BundledModelOption? + public var modelDisplayName: String { loadedModel?.displayName ?? selectedModel.displayName } + public var selectedModelID: String { selectedModel.id } + public var loadedModelID: String? { loadedModel?.id } + public internal(set) var modelInstallStates: [String: ModelInstallState] = [:] + public internal(set) var modelDownloadMetrics: [String: ModelDownloadMetrics] = [:] + + // MARK: - Compatibility Settings + + public var useGPU = true + public var samplingTopK: Int = 40 + public var samplingTopP: Float = 0.95 + public var samplingTemperature: Float = 1.0 + public var maxOutputTokens: Int = 4000 + + // Internal (not private) so extensions in ModelDownloader/Installer/GPULifecycle + // files can read/write these. Not part of the public API. + var modelContainer: ModelContainer? + var cancelled = false + var currentLoadTask: Task? + var currentGenerationTask: Task? + var currentDownloadTasks: [String: Task] = [:] + let foregroundStateLock = NSLock() + var foregroundGPUAllowed = true + var lifecycleObserverTokens: [NSObjectProtocol] = [] + var audioCapabilityEnabled = false + + /// Local path to the model directory + var modelPath: URL { + ModelPaths.resolve(for: selectedModel) + } + + // MARK: - Init + + public init(selectedModelID: String? = nil) { + if let selectedModelID, + let option = Self.availableModels.first(where: { $0.id == selectedModelID }) { + self.selectedModel = option + } + self.stats.backend = "mlx-gpu" + configureLifecycleObservers() + cleanupStalePartialDirectories() + refreshModelInstallStates() + } + + deinit { + for token in lifecycleObserverTokens { + NotificationCenter.default.removeObserver(token) + } + } + + /// Convenience init with default model location + public convenience init() { + self.init(selectedModelID: nil) + } + + func loadModel() { + currentLoadTask?.cancel() + currentLoadTask = Task { [weak self] in + guard let self else { return } + defer { self.currentLoadTask = nil } + do { + if self.isLoading { + return + } + try await load() + try await warmup() + } catch is CancellationError { + await MainActor.run { + if self.statusMessage.hasPrefix("正在加载") || self.statusMessage.hasPrefix("正在初始化") { + self.statusMessage = "已取消模型切换" + } + } + } catch { + if let mlxError = error as? MLXError, + case .modelDirectoryMissing = mlxError { + statusMessage = "请在配置中下载 \(self.selectedModel.displayName) 模型" + } else { + statusMessage = "❌ \(error.localizedDescription)" + } + self.isLoaded = false + self.loadedModel = nil + self.refreshModelInstallStates() + print("[MLX] Load failed: \(error.localizedDescription)") + } + } + } + + func generateStream( + prompt: String, + images: [CIImage] = [], + audios: [UserInput.Audio] = [], + onToken: @escaping (String) -> Void, + onComplete: @escaping (Result) -> Void + ) { + Task { + var fullResponse = "" + + do { + for try await token in generateStream(prompt: prompt, images: images, audios: audios) { + fullResponse += token + await MainActor.run { + onToken(token) + } + } + + await MainActor.run { + onComplete(.success(fullResponse)) + } + } catch { + await MainActor.run { + onComplete(.failure(error)) + } + } + } + } + + func generateStream( + chat: [Chat.Message], + onToken: @escaping (String) -> Void, + onComplete: @escaping (Result) -> Void + ) { + Task { + var fullResponse = "" + + do { + for try await token in generateStream(chat: chat) { + fullResponse += token + await MainActor.run { + onToken(token) + } + } + + await MainActor.run { + onComplete(.success(fullResponse)) + } + } catch { + await MainActor.run { + onComplete(.failure(error)) + } + } + } + } + + func generateStream( + chat: [Chat.Message], + additionalContext: [String: any Sendable]?, + onToken: @escaping (String) -> Void, + onComplete: @escaping (Result) -> Void + ) { + Task { + var fullResponse = "" + + do { + for try await token in generateStream(chat: chat, additionalContext: additionalContext) { + fullResponse += token + await MainActor.run { + onToken(token) + } + } + + await MainActor.run { + onComplete(.success(fullResponse)) + } + } catch { + await MainActor.run { + onComplete(.failure(error)) + } + } + } + } + + // MARK: - LLMEngine Protocol + + public func load() async throws { + if isLoading { + return + } + let model = selectedModel + let path = ModelPaths.resolve(for: model) + isLoading = true + defer { + isLoading = false + } + statusMessage = "正在初始化模型..." + await Gemma4Registration.setAudioCapabilityEnabled(audioCapabilityEnabled) + await Gemma4Registration.register() + + guard ModelPaths.hasRequiredFiles(model, at: path) else { + throw MLXError.modelDirectoryMissing(model.displayName) + } + + statusMessage = "正在加载 \(model.displayName)..." + let loadStart = CFAbsoluteTimeGetCurrent() + print("[MLX] load capability — audio=\(audioCapabilityEnabled ? 1 : 0)") + + // ── Memory diagnostics (read before load) ────────────────────────────── + let physMB = Double(ProcessInfo.processInfo.physicalMemory) / 1_048_576 + let (footprintBefore, limitBefore) = MemoryStats.footprintMB() + print("[MEM] Physical RAM: \(Int(physMB)) MB") + print("[MEM] Before load — footprint: \(Int(footprintBefore)) MB, jetsam limit: \(Int(limitBefore)) MB") + print("[MEM] MLX before — active: \(MLX.GPU.activeMemory / 1_048_576) MB, cache: \(MLX.GPU.cacheMemory / 1_048_576) MB") + + let container = try await VLMModelFactory.shared.loadContainer( + from: path, + using: MLXTokenizersLoader() + ) + + try Task.checkCancellation() + self.modelContainer = container + self.isLoaded = true + self.loadedModel = model + + // ── Memory diagnostics (read after load) ─────────────────────────────── + let (footprintAfter, _) = MemoryStats.footprintMB() + print("[MEM] After load — footprint: \(Int(footprintAfter)) MB") + print("[MEM] MLX after — active: \(MLX.GPU.activeMemory / 1_048_576) MB, cache: \(MLX.GPU.cacheMemory / 1_048_576) MB") + + let elapsed = (CFAbsoluteTimeGetCurrent() - loadStart) * 1000 + stats.loadTimeMs = elapsed + statusMessage = "模型已就绪 ✅ (\(Int(elapsed))ms)" + + print("[MLX] Model loaded in \(Int(elapsed))ms — backend: mlx-gpu — model: \(model.displayName)") + } + + private func ensureAudioCapability(hasAudio: Bool) async throws { + guard hasAudio != audioCapabilityEnabled || !isLoaded || modelContainer == nil else { + return + } + + audioCapabilityEnabled = hasAudio + print("[MLX] capability switch requested — audio=\(hasAudio ? 1 : 0)") + + if isLoaded || modelContainer != nil { + await prepareForReload(cancelCurrentGeneration: false, cancelCurrentLoad: false) + } + + try await load() + } + + /// 当前可用内存 headroom（MB）。Agent 用来动态调整 history 深度。 + public var availableHeadroomMB: Int { + MemoryStats.headroomMB + } + + /// 根据当前剩余内存推荐安全的 history 深度（消息条数）。 + /// Chunked prefill (LLM/MLX/Gemma4Model.swift) 把单次 forward 的 transient + /// 内存峰值钉死在 chunk² (windowSize=256), 不再随总序列长度线性增长。 + /// 因此 history 可以放更多: KV cache 增量是每 token ~14KB × layers, 几条 + /// 历史消息只多几十 MB, 远低于现在 ~1GB 的稳定 headroom。 + public var safeHistoryDepth: Int { + let profile = (loadedModel ?? selectedModel).runtimeProfile + return RuntimeBudgets.safeHistoryDepth(profile: profile, headroom: availableHeadroomMB) + } + + + public func warmup() async throws { + // Warmup skipped for E2B. + // + // E2B has 26 layers (E4B has 42). Running MLXLMCommon.generate() for the first time + // triggers Metal JIT shader compilation across all unique kernel variants + // (attention, MLP, PLE, RoPE ...). This compilation adds a temporary + // memory spike on top of the already-loaded 4.9 GB weights, which pushes + // the process past the jetsam limit on iPhone 17 Pro Max. + // + // Skipping warmup means the first user inference compiles shaders lazily + // (first response is ~2-3s slower) but avoids the OOM kill on startup. + print("[MLX] Warmup skipped — shaders will compile on first inference") + statusMessage = "模型已就绪 ✅" + } + + public func generateStream( + prompt: String, + images: [CIImage], + audios: [UserInput.Audio] + ) -> AsyncThrowingStream { + let input: UserInput + if images.isEmpty, audios.isEmpty { + input = UserInput(prompt: prompt) + } else { + input = UserInput( + chat: [ + .user( + prompt, + images: images.map { .ciImage($0) }, + audios: audios + ) + ] + ) + } + return generateStream(input: input, isMultimodal: !images.isEmpty || !audios.isEmpty) + } + + public func generateStream(chat: [Chat.Message]) -> AsyncThrowingStream { + generateStream(chat: chat, additionalContext: nil) + } + + public func generateStream( + chat: [Chat.Message], + additionalContext: [String: any Sendable]? + ) -> AsyncThrowingStream { + let input = UserInput(chat: chat, additionalContext: additionalContext) + let hasMedia = !input.images.isEmpty || !input.audios.isEmpty + return generateStream(input: input, isMultimodal: hasMedia) + } + + + private func currentMultimodalFallbackRecommendation() -> String { + let currentModel = loadedModel ?? selectedModel + if let lighterID = currentModel.runtimeProfile.lighterAlternativeID, + let lighter = Self.availableModels.first(where: { $0.id == lighterID }) { + if isModelAvailable(lighter) { + return "如仍失败，可手动切换到 \(lighter.displayName) 处理图片或音频。" + } + return "如仍失败，可先下载并手动切换到 \(lighter.displayName)。" + } + return "如仍失败，请改用更轻量的模型处理图片或音频。" + } + + + private func generateStream( + input: UserInput, + isMultimodal: Bool + ) -> AsyncThrowingStream { + AsyncThrowingStream { continuation in + let task = Task { [weak self] in + guard let self else { + continuation.finish(throwing: MLXError.modelNotLoaded) + return + } + do { + try await self.ensureAudioCapability(hasAudio: !input.audios.isEmpty) + } catch { + continuation.finish(throwing: error) + return + } + + guard let container = modelContainer else { + continuation.finish(throwing: MLXError.modelNotLoaded) + return + } + + // Free Metal buffers cached from previous inference before + // allocating the new computation graph. Critical on low-headroom devices: + // the follow-up prompt is longer than the first inference, + // and without clearing, residual cache + new activations + // exceed the 6GB jetsam limit on iPhone. + // + // NOTE: No prompt prefix caching is implemented. Every generateStream call + // rebuilds the full KV cache from scratch. Prompt prefix stability does NOT + // improve inference latency in the current architecture. Any future proposal + // to restructure prompts for "cache hit rate" should first implement actual + // prefix caching before assuming performance gains. + MLX.GPU.clearCache() + + let currentModel = self.loadedModel ?? self.selectedModel + let profile = currentModel.runtimeProfile + let headroom = MemoryStats.headroomMB + + let thinkingEnabled = RuntimeBudgets.isThinkingEnabled(input: input, profile: profile) + let textBudget = RuntimeBudgets.text(profile: profile, headroom: headroom, enabled: !isMultimodal) + let runtimeBudget: MultimodalBudget? + do { + runtimeBudget = try RuntimeBudgets.multimodal( + profile: profile, + headroom: headroom, + hasImages: !input.images.isEmpty, + hasAudio: !input.audios.isEmpty, + modelDisplayName: currentModel.displayName, + fallbackRecommendation: "请关闭后台应用后重试，或减少附件数量。\(self.currentMultimodalFallbackRecommendation())" + ) + } catch { + continuation.finish(throwing: error) + return + } + let thinkingBudget = RuntimeBudgets.thinking(profile: profile, headroom: headroom, enabled: thinkingEnabled) + + if let runtimeBudget { + print( + "[MEM] multimodal runtime budget — model=\(currentModel.displayName), " + + "headroom=\(headroom) MB, " + + "imageSoftTokenCap=\(runtimeBudget.imageSoftTokenCap.map(String.init) ?? "n/a"), " + + "maxOutputTokens=\(runtimeBudget.maxOutputTokens), " + + "audio=\(!input.audios.isEmpty ? 1 : 0)" + ) + } + if let thinkingBudget { + print("[MEM] thinking runtime budget — model=\(currentModel.displayName), headroom=\(headroom) MB, maxOutputTokens=\(thinkingBudget.maxOutputTokens)") + } + if let textBudget { + print("[MEM] text runtime budget — model=\(currentModel.displayName), headroom=\(headroom) MB, maxOutputTokens=\(textBudget.maxOutputTokens)") + } + + // TODO: 抽出 ModelAdapter 协议后移除 Gemma 专属耦合 + Gemma4Processor.setRuntimeImageSoftTokenCap(runtimeBudget?.imageSoftTokenCap) + defer { + Gemma4Processor.setRuntimeImageSoftTokenCap(nil) + } + let effectiveMaxOutputTokens: Int = { + let multimodalCap = isMultimodal + ? runtimeBudget?.maxOutputTokens ?? profile.multimodalOutputTiers.last?.maxOutputTokens ?? maxOutputTokens + : maxOutputTokens + let thinkingCap = thinkingBudget?.maxOutputTokens ?? maxOutputTokens + let textCap = textBudget?.maxOutputTokens ?? maxOutputTokens + return min(maxOutputTokens, multimodalCap, thinkingCap, textCap) + }() + var resolvedMaxOutputTokens = effectiveMaxOutputTokens + + self.isGenerating = true + self.cancelled = false + let genStart = CFAbsoluteTimeGetCurrent() + var firstTokenTime: Double? = nil + var tokenCount = 0 + var hitTokenCap = false + + let (fp, _) = MemoryStats.footprintMB() + print("[MEM] generateStream start — footprint: \(Int(fp)) MB, MLX active: \(MLX.GPU.activeMemory / 1_048_576) MB") + + do { + try await self.ensureForegroundGPUExecution() + _ = try await container.perform { context in + try await self.ensureForegroundGPUExecution() + if isMultimodal { + print("[VLM] multimodal budget — maxOutputTokens=\(resolvedMaxOutputTokens)") + } else if thinkingEnabled { + print("[LLM] thinking budget — baseMaxOutputTokens=\(resolvedMaxOutputTokens)") + } + let preparedInput = try await context.processor.prepare(input: input) + let preparedSequenceLength = preparedInput.text.tokens.dim(1) + if isMultimodal { + print("[VLM] prepared sequence length=\(preparedSequenceLength)") + } else { + // 不再基于 prepared 长度二次扣减 output 上限。 + // chunked prefill 让 prepared 长度对峰值内存几乎无影响, + // resolvedMaxOutputTokens 已由 textOutputBudget(headroom) + // 决定, 直接使用即可。 + print( + "[LLM] prepared sequence length=\(preparedSequenceLength), " + + "outputCap=\(resolvedMaxOutputTokens)" + ) + } + try await self.ensureForegroundGPUExecution() + + // prefillStepSize: chunked prefill window. 把长 prompt + // 切成 256 token / chunk 处理，每个 chunk 跑完调 eval(cache) + // 释放 compute graph，单 chunk transient 峰值控制在 ~400 MB + // 以内，避免长 prompt（800+ tokens）单次 forward 把 attention + // workspace 推过 iPhone 6.1 GB jetsam 上限。 + // + // MLX 默认 512 是为桌面 Apple Silicon 调的；iPhone E4B + // (42 layers) 在 512 chunk 下 transient 峰值约 1.7 GB, + // 加上 4.6 GB 已驻留 weights+KV 会撞 jetsam。 + // 这是框架层修复，与 prompt 内容、skill 数量、SKILL.md + // 格式完全无关。 + return try MLXLMCommon.generate( + input: preparedInput, + parameters: .init( + maxTokens: resolvedMaxOutputTokens, + temperature: samplingTemperature, + topP: samplingTopP, + topK: samplingTopK, + prefillStepSize: 256 + ), + context: context + ) { tokens in + if self.cancelled || !self.isForegroundGPUAllowed() { + return .stop + } + + tokenCount = tokens.count + if firstTokenTime == nil { + firstTokenTime = (CFAbsoluteTimeGetCurrent() - genStart) * 1000 + } + + // Stream the latest token + if let lastToken = tokens.last { + let text = context.tokenizer.decode(tokenIds: [lastToken]) + continuation.yield(text) + } + + // Multimodal path uses a tighter generation budget on iPhone. + // If we hit the cap, signal truncation so the caller can append a notice. + if tokens.count >= resolvedMaxOutputTokens { + hitTokenCap = true + return .stop + } + return .more + } + } + + let elapsed = CFAbsoluteTimeGetCurrent() - genStart + self.stats.ttftMs = firstTokenTime ?? 0 + self.stats.tokensPerSec = elapsed > 0 + ? Double(tokenCount) / elapsed : 0 + self.stats.totalTokens = tokenCount + + print( + "[MLX] Generated \(tokenCount) tokens in \(String(format: "%.1f", elapsed))s" + ) + print( + "[MLX] TTFT: \(String(format: "%.0f", self.stats.ttftMs))ms, " + + "Speed: \(String(format: "%.1f", self.stats.tokensPerSec)) tok/s") + + // 推理结束后立即释放 Metal activation 缓存， + // 确保下一轮有最大可用 headroom。 + MLX.GPU.clearCache() + let (fpEnd, _) = MemoryStats.footprintMB() + print("[MEM] generateStream end — footprint: \(Int(fpEnd)) MB, headroom: \(self.availableHeadroomMB) MB") + + // If we hit the token cap mid-sentence, append a visible notice. + // This makes truncation explicit rather than silently dropping content. + if hitTokenCap { + let isChinese = Locale.preferredLanguages.contains { $0.hasPrefix("zh") } + let modeLabel = isChinese + ? (thinkingEnabled ? "思考" : "输出") + : (thinkingEnabled ? "Thinking" : "Output") + continuation.yield("\n\n> ⚠️ \(modeLabel)已达单次输出上限（\(resolvedMaxOutputTokens) tokens），内容可能不完整。") + } + continuation.finish() + } catch { + continuation.finish(throwing: error) + } + + self.isGenerating = false + self.currentGenerationTask = nil + } + + currentGenerationTask = task + continuation.onTermination = { [weak self] _ in + task.cancel() + if self?.currentGenerationTask?.isCancelled == true { + self?.currentGenerationTask = nil + } + } + } + } + + public func cancel() { + cancelled = true + currentGenerationTask?.cancel() + currentLoadTask?.cancel() + } + + public func prepareForReload( + cancelCurrentGeneration: Bool = true, + cancelCurrentLoad: Bool = true + ) async { + cancelled = true + if cancelCurrentGeneration { + currentGenerationTask?.cancel() + } + if cancelCurrentLoad { + currentLoadTask?.cancel() + } + + while isGenerating || isLoading { + try? await Task.sleep(nanoseconds: 50_000_000) + } + + unload( + cancelCurrentGeneration: cancelCurrentGeneration, + cancelCurrentLoad: cancelCurrentLoad + ) + MLX.GPU.clearCache() + try? await Task.sleep(nanoseconds: 150_000_000) + } + + public func unload() { + unload(cancelCurrentGeneration: true, cancelCurrentLoad: true) + } + + public func unload( + cancelCurrentGeneration: Bool = true, + cancelCurrentLoad: Bool = true + ) { + if cancelCurrentGeneration { + currentGenerationTask?.cancel() + } + if cancelCurrentLoad { + currentLoadTask?.cancel() + } + modelContainer = nil + isLoaded = false + isLoading = false + isGenerating = false + loadedModel = nil + cancelled = false + stats = LLMStats() + stats.backend = "mlx-gpu" + MLX.GPU.clearCache() + statusMessage = "模型已卸载" + print("[MLX] Model unloaded") + } +} + diff --git a/LLM/MLXLocalLLMService.swift b/LLM/MLXLocalLLMService.swift deleted file mode 100644 index e490aec..0000000 --- a/LLM/MLXLocalLLMService.swift +++ /dev/null @@ -1,1471 +0,0 @@ -import Foundation -import CoreImage -import MLX -import MLXLLM -import MLXLMCommon -import MLXVLM -#if canImport(UIKit) -import UIKit -#endif - -// MARK: - MLX Local LLM Service - -public struct BundledModelOption: Identifiable, Hashable, Sendable { - public let id: String - public let directoryName: String - public let displayName: String - public let repositoryID: String - public let requiredFiles: [String] -} - -public enum ModelInstallState: Equatable, Sendable { - case notInstalled - case checkingSource - case downloading(completedFiles: Int, totalFiles: Int, currentFile: String) - case downloaded - case bundled - case failed(String) -} - -public struct ModelDownloadMetrics: Equatable, Sendable { - public let bytesReceived: Int64 - public let totalBytes: Int64? - public let bytesPerSecond: Double? - public let sourceLabel: String? -} - -/// MLX GPU inference service for Gemma 4. -/// Forces MLX Metal GPU path — no CPU fallback. -@Observable -public class MLXLocalLLMService: LLMEngine { - static let availableModels: [BundledModelOption] = [ - .init( - id: "gemma-4-e2b-it-4bit", - directoryName: "gemma-4-e2b-it-4bit", - displayName: "Gemma 4 E2B", - repositoryID: "mlx-community/gemma-4-e2b-it-4bit", - requiredFiles: [ - "config.json", - "generation_config.json", - "model.safetensors", - "model.safetensors.index.json", - "processor_config.json", - "tokenizer.json", - "tokenizer_config.json", - "chat_template.jinja" - ] - ), - .init( - id: "gemma-4-e4b-it-4bit", - directoryName: "gemma-4-e4b-it-4bit", - displayName: "Gemma 4 E4B", - repositoryID: "mlx-community/gemma-4-e4b-it-4bit", - requiredFiles: [ - "config.json", - "generation_config.json", - "model.safetensors", - "model.safetensors.index.json", - "processor_config.json", - "tokenizer.json", - "tokenizer_config.json", - "chat_template.jinja" - ] - ) - ] - static let defaultModel = availableModels[0] - private static let multimodalMaxOutputTokens = 512 - private static let e4bThinkingMaxOutputTokens = 1_024 - private static let e2bThinkingMaxOutputTokens = 768 - private static let e4bModelID = "gemma-4-e4b-it-4bit" - private static let e2bModelID = "gemma-4-e2b-it-4bit" - private static let e4bMultimodalCriticalHeadroomMB = 320 - - private struct MultimodalRuntimeBudget { - let imageSoftTokenCap: Int? - let maxOutputTokens: Int - let headroomMB: Int - } - - private struct ThinkingRuntimeBudget { - let maxOutputTokens: Int - let headroomMB: Int - } - - private struct TextRuntimeBudget { - let maxOutputTokens: Int - let headroomMB: Int - } - - // MARK: - State - - public private(set) var isLoaded = false - public private(set) var isLoading = false - public private(set) var isGenerating = false - public private(set) var stats = LLMStats() - public var statusMessage = "等待加载模型..." - public private(set) var selectedModel = defaultModel - public private(set) var loadedModel: BundledModelOption? - public var modelDisplayName: String { loadedModel?.displayName ?? selectedModel.displayName } - public var selectedModelID: String { selectedModel.id } - public var loadedModelID: String? { loadedModel?.id } - public private(set) var modelInstallStates: [String: ModelInstallState] = [:] - public private(set) var modelDownloadMetrics: [String: ModelDownloadMetrics] = [:] - - // MARK: - Compatibility Settings - - public var useGPU = true - public var samplingTopK: Int = 40 - public var samplingTopP: Float = 0.95 - public var samplingTemperature: Float = 1.0 - public var maxOutputTokens: Int = 4000 - - private var modelContainer: ModelContainer? - private var cancelled = false - private var currentLoadTask: Task? - private var currentGenerationTask: Task? - private var currentDownloadTasks: [String: Task] = [:] - private let foregroundStateLock = NSLock() - private var foregroundGPUAllowed = true - private var lifecycleObserverTokens: [NSObjectProtocol] = [] - private var audioCapabilityEnabled = false - - private struct DownloadSource: Sendable { - let label: String - let url: URL - } - - private final class DownloadTaskClient: NSObject, URLSessionDownloadDelegate { - private let progressHandler: @Sendable (Int64, Int64?) -> Void - private var continuation: CheckedContinuation<(URL, URLResponse), Error>? - private var downloadedFileURL: URL? - private var response: URLResponse? - private lazy var session: URLSession = { - URLSession(configuration: .ephemeral, delegate: self, delegateQueue: nil) - }() - private var task: URLSessionDownloadTask? - - init(progressHandler: @escaping @Sendable (Int64, Int64?) -> Void) { - self.progressHandler = progressHandler - } - - func start(request: URLRequest) async throws -> (URL, URLResponse) { - try await withTaskCancellationHandler { - try await withCheckedThrowingContinuation { continuation in - self.continuation = continuation - let task = session.downloadTask(with: request) - self.task = task - task.resume() - } - } onCancel: { - self.cancel() - } - } - - func cancel() { - task?.cancel() - session.invalidateAndCancel() - } - - func urlSession( - _ session: URLSession, - downloadTask: URLSessionDownloadTask, - didWriteData bytesWritten: Int64, - totalBytesWritten: Int64, - totalBytesExpectedToWrite: Int64 - ) { - let expected = totalBytesExpectedToWrite > 0 ? totalBytesExpectedToWrite : nil - progressHandler(totalBytesWritten, expected) - } - - func urlSession( - _ session: URLSession, - downloadTask: URLSessionDownloadTask, - didFinishDownloadingTo location: URL - ) { - do { - let temporaryFileURL = FileManager.default.temporaryDirectory - .appendingPathComponent(UUID().uuidString) - .appendingPathExtension(location.pathExtension.isEmpty ? "tmp" : location.pathExtension) - - if FileManager.default.fileExists(atPath: temporaryFileURL.path) { - try FileManager.default.removeItem(at: temporaryFileURL) - } - - try FileManager.default.moveItem(at: location, to: temporaryFileURL) - downloadedFileURL = temporaryFileURL - response = downloadTask.response - } catch { - downloadedFileURL = nil - response = nil - continuation?.resume(throwing: error) - continuation = nil - self.task = nil - session.finishTasksAndInvalidate() - } - } - - func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) { - defer { - continuation = nil - downloadedFileURL = nil - response = nil - self.task = nil - session.finishTasksAndInvalidate() - } - - if let error { - continuation?.resume(throwing: error) - return - } - - guard let downloadedFileURL, let response else { - continuation?.resume(throwing: DownloadError.invalidResponse) - return - } - - continuation?.resume(returning: (downloadedFileURL, response)) - } - } - - /// Local path to the model directory - private var modelPath: URL { - Self.resolveModelPath(for: selectedModel) - } - - // MARK: - Init - - public init(selectedModelID: String? = nil) { - if let selectedModelID, - let option = Self.availableModels.first(where: { $0.id == selectedModelID }) { - self.selectedModel = option - } - self.stats.backend = "mlx-gpu" - configureLifecycleObservers() - cleanupStalePartialDirectories() - refreshModelInstallStates() - } - - deinit { - for token in lifecycleObserverTokens { - NotificationCenter.default.removeObserver(token) - } - } - - /// Convenience init with default model location - public convenience init() { - self.init(selectedModelID: nil) - } - - public func selectModel(id: String) -> Bool { - guard let option = Self.availableModels.first(where: { $0.id == id }), - option != selectedModel else { - return false - } - - selectedModel = option - statusMessage = isLoaded - ? "已选择 \(option.displayName)，准备重新加载..." - : "已选择 \(option.displayName)，等待加载..." - return true - } - - private static func resolveModelPath(for model: BundledModelOption) -> URL { - if let bundledPath = bundledModelPath(for: model) { - return bundledPath - } - - return downloadedModelPath(for: model) - } - - private static func documentsModelsRoot() -> URL { - let documentsPath = FileManager.default.urls( - for: .documentDirectory, - in: .userDomainMask - ).first! - return documentsPath.appendingPathComponent("models", isDirectory: true) - } - - private static func downloadedModelPath(for model: BundledModelOption) -> URL { - documentsModelsRoot().appendingPathComponent(model.directoryName, isDirectory: true) - } - - private static func partialModelPath(for model: BundledModelOption) -> URL { - documentsModelsRoot().appendingPathComponent("\(model.directoryName).partial", isDirectory: true) - } - - private static func bundledModelPath(for model: BundledModelOption) -> URL? { - guard let resourceURL = Bundle.main.resourceURL else { return nil } - - let directBundleDir = resourceURL.appendingPathComponent( - model.directoryName, - isDirectory: true - ) - if hasRequiredFiles(for: model, at: directBundleDir) { - return directBundleDir - } - - let nestedBundleDir = resourceURL - .appendingPathComponent("Models", isDirectory: true) - .appendingPathComponent(model.directoryName, isDirectory: true) - if hasRequiredFiles(for: model, at: nestedBundleDir) { - return nestedBundleDir - } - - return nil - } - - private static func hasRequiredFiles(for model: BundledModelOption, at directory: URL) -> Bool { - let fm = FileManager.default - var isDirectory: ObjCBool = false - guard fm.fileExists(atPath: directory.path, isDirectory: &isDirectory), isDirectory.boolValue else { - return false - } - return model.requiredFiles.allSatisfy { file in - fm.fileExists(atPath: directory.appendingPathComponent(file).path) - } - } - - public func isModelAvailable(_ model: BundledModelOption) -> Bool { - Self.bundledModelPath(for: model) != nil - || Self.hasRequiredFiles(for: model, at: Self.downloadedModelPath(for: model)) - } - - public func installState(for model: BundledModelOption) -> ModelInstallState { - if Self.bundledModelPath(for: model) != nil { - return .bundled - } - if Self.hasRequiredFiles(for: model, at: Self.downloadedModelPath(for: model)) { - return .downloaded - } - return modelInstallStates[model.id] ?? .notInstalled - } - - public func refreshModelInstallStates() { - cleanupStalePartialDirectories() - for model in Self.availableModels { - if Self.bundledModelPath(for: model) != nil { - modelInstallStates[model.id] = .bundled - } else if Self.hasRequiredFiles(for: model, at: Self.downloadedModelPath(for: model)) { - modelInstallStates[model.id] = .downloaded - } else if case .checkingSource = modelInstallStates[model.id] { - continue - } else if case .downloading = modelInstallStates[model.id] { - continue - } else { - modelInstallStates[model.id] = .notInstalled - } - } - } - - public func downloadMetrics(for modelID: String) -> ModelDownloadMetrics? { - modelDownloadMetrics[modelID] - } - - private func cleanupStalePartialDirectories() { - let fm = FileManager.default - for model in Self.availableModels { - let partialDirectory = Self.partialModelPath(for: model) - if fm.fileExists(atPath: partialDirectory.path) { - try? fm.removeItem(at: partialDirectory) - } - } - } - - private func preferredDownloadSources(for model: BundledModelOption, file: String) -> [DownloadSource] { - let encodedPath = "\(model.repositoryID)/resolve/main/\(file)" - .addingPercentEncoding(withAllowedCharacters: .urlPathAllowed)? - .replacingOccurrences(of: "%2F", with: "/") - - guard let encodedPath else { return [] } - - return [ - .init(label: "hf-mirror.com", url: URL(string: "https://hf-mirror.com/\(encodedPath)")!), - .init(label: "huggingface.co", url: URL(string: "https://huggingface.co/\(encodedPath)")!) - ] - } - - private func downloadFile( - with request: URLRequest, - progress: @escaping @Sendable (Int64, Int64?) -> Void - ) async throws -> (URL, URLResponse) { - let client = DownloadTaskClient(progressHandler: progress) - return try await client.start(request: request) - } - - public func downloadModel(id: String) async { - guard let model = Self.availableModels.first(where: { $0.id == id }) else { return } - if isModelAvailable(model) { - refreshModelInstallStates() - return - } - if currentDownloadTasks[id] != nil { - return - } - - let task = Task { [weak self] in - guard let self else { return } - let fm = FileManager.default - let modelsRoot = Self.documentsModelsRoot() - let finalDirectory = Self.downloadedModelPath(for: model) - let partialDirectory = Self.partialModelPath(for: model) - - await MainActor.run { - self.modelInstallStates[id] = .checkingSource - } - - do { - if !fm.fileExists(atPath: modelsRoot.path) { - try fm.createDirectory(at: modelsRoot, withIntermediateDirectories: true) - } - if fm.fileExists(atPath: partialDirectory.path) { - try fm.removeItem(at: partialDirectory) - } - try fm.createDirectory(at: partialDirectory, withIntermediateDirectories: true) - - let totalFiles = model.requiredFiles.count - for (index, file) in model.requiredFiles.enumerated() { - let downloadSources = preferredDownloadSources(for: model, file: file) - guard !downloadSources.isEmpty else { - throw DownloadError.invalidURL(file) - } - - var downloadedResult: (URL, URLResponse)? - var lastError: Error? - - for source in downloadSources { - try Task.checkCancellation() - - await MainActor.run { - self.modelInstallStates[id] = .downloading( - completedFiles: index, - totalFiles: totalFiles, - currentFile: file - ) - self.modelDownloadMetrics[id] = .init( - bytesReceived: 0, - totalBytes: nil, - bytesPerSecond: nil, - sourceLabel: source.label - ) - } - - let request = URLRequest( - url: source.url, - cachePolicy: .reloadIgnoringLocalCacheData, - timeoutInterval: 1800 - ) - let startTime = Date() - - do { - let result = try await downloadFile(with: request) { received, expected in - let elapsed = max(Date().timeIntervalSince(startTime), 0.001) - let bytesPerSecond = Double(received) / elapsed - Task { @MainActor [weak self] in - self?.modelDownloadMetrics[id] = .init( - bytesReceived: received, - totalBytes: expected, - bytesPerSecond: bytesPerSecond, - sourceLabel: source.label - ) - } - } - downloadedResult = result - break - } catch is CancellationError { - throw CancellationError() - } catch { - lastError = error - } - } - - guard let (temporaryURL, response) = downloadedResult else { - throw lastError ?? DownloadError.invalidResponse - } - - guard let http = response as? HTTPURLResponse else { - throw DownloadError.invalidResponse - } - guard (200...299).contains(http.statusCode) else { - throw DownloadError.httpStatus(http.statusCode) - } - - let destinationURL = partialDirectory.appendingPathComponent(file) - let parentDirectory = destinationURL.deletingLastPathComponent() - if !fm.fileExists(atPath: parentDirectory.path) { - try fm.createDirectory(at: parentDirectory, withIntermediateDirectories: true) - } - if fm.fileExists(atPath: destinationURL.path) { - try fm.removeItem(at: destinationURL) - } - try fm.moveItem(at: temporaryURL, to: destinationURL) - } - - if fm.fileExists(atPath: finalDirectory.path) { - try fm.removeItem(at: finalDirectory) - } - try fm.moveItem(at: partialDirectory, to: finalDirectory) - - await MainActor.run { - self.modelInstallStates[id] = .downloaded - self.modelDownloadMetrics[id] = nil - self.refreshModelInstallStates() - } - } catch is CancellationError { - try? fm.removeItem(at: partialDirectory) - await MainActor.run { - self.modelInstallStates[id] = .notInstalled - self.modelDownloadMetrics[id] = nil - self.refreshModelInstallStates() - } - } catch { - try? fm.removeItem(at: partialDirectory) - await MainActor.run { - self.modelDownloadMetrics[id] = nil - self.modelInstallStates[id] = .failed(error.localizedDescription) - } - } - - await MainActor.run { - self.currentDownloadTasks[id] = nil - } - } - - currentDownloadTasks[id] = task - await task.value - } - - public func cancelModelDownload(id: String) { - guard currentDownloadTasks[id] != nil else { return } - currentDownloadTasks[id]?.cancel() - } - - func loadModel() { - currentLoadTask?.cancel() - currentLoadTask = Task { [weak self] in - guard let self else { return } - defer { self.currentLoadTask = nil } - do { - if self.isLoading { - return - } - try await load() - try await warmup() - } catch is CancellationError { - await MainActor.run { - if self.statusMessage.hasPrefix("正在加载") || self.statusMessage.hasPrefix("正在初始化") { - self.statusMessage = "已取消模型切换" - } - } - } catch { - if let mlxError = error as? MLXError, - case .modelDirectoryMissing = mlxError { - statusMessage = "请在配置中下载 \(self.selectedModel.displayName) 模型" - } else { - statusMessage = "❌ \(error.localizedDescription)" - } - self.isLoaded = false - self.loadedModel = nil - self.refreshModelInstallStates() - print("[MLX] Load failed: \(error.localizedDescription)") - } - } - } - - func generateStream( - prompt: String, - images: [CIImage] = [], - audios: [UserInput.Audio] = [], - onToken: @escaping (String) -> Void, - onComplete: @escaping (Result) -> Void - ) { - Task { - var fullResponse = "" - - do { - for try await token in generateStream(prompt: prompt, images: images, audios: audios) { - fullResponse += token - await MainActor.run { - onToken(token) - } - } - - await MainActor.run { - onComplete(.success(fullResponse)) - } - } catch { - await MainActor.run { - onComplete(.failure(error)) - } - } - } - } - - func generateStream( - chat: [Chat.Message], - onToken: @escaping (String) -> Void, - onComplete: @escaping (Result) -> Void - ) { - Task { - var fullResponse = "" - - do { - for try await token in generateStream(chat: chat) { - fullResponse += token - await MainActor.run { - onToken(token) - } - } - - await MainActor.run { - onComplete(.success(fullResponse)) - } - } catch { - await MainActor.run { - onComplete(.failure(error)) - } - } - } - } - - func generateStream( - chat: [Chat.Message], - additionalContext: [String: any Sendable]?, - onToken: @escaping (String) -> Void, - onComplete: @escaping (Result) -> Void - ) { - Task { - var fullResponse = "" - - do { - for try await token in generateStream(chat: chat, additionalContext: additionalContext) { - fullResponse += token - await MainActor.run { - onToken(token) - } - } - - await MainActor.run { - onComplete(.success(fullResponse)) - } - } catch { - await MainActor.run { - onComplete(.failure(error)) - } - } - } - } - - // MARK: - LLMEngine Protocol - - public func load() async throws { - if isLoading { - return - } - let model = selectedModel - let path = Self.resolveModelPath(for: model) - isLoading = true - defer { - isLoading = false - } - statusMessage = "正在初始化模型..." - await Gemma4Registration.setAudioCapabilityEnabled(audioCapabilityEnabled) - await Gemma4Registration.register() - - guard Self.hasRequiredFiles(for: model, at: path) else { - throw MLXError.modelDirectoryMissing(model.displayName) - } - - statusMessage = "正在加载 \(model.displayName)..." - let loadStart = CFAbsoluteTimeGetCurrent() - print("[MLX] load capability — audio=\(audioCapabilityEnabled ? 1 : 0)") - - // ── Memory diagnostics (read before load) ────────────────────────────── - let physMB = Double(ProcessInfo.processInfo.physicalMemory) / 1_048_576 - let (footprintBefore, limitBefore) = appMemoryFootprintMB() - print("[MEM] Physical RAM: \(Int(physMB)) MB") - print("[MEM] Before load — footprint: \(Int(footprintBefore)) MB, jetsam limit: \(Int(limitBefore)) MB") - print("[MEM] MLX before — active: \(MLX.GPU.activeMemory / 1_048_576) MB, cache: \(MLX.GPU.cacheMemory / 1_048_576) MB") - - let container = try await VLMModelFactory.shared.loadContainer( - from: path, - using: MLXTokenizersLoader() - ) - - try Task.checkCancellation() - self.modelContainer = container - self.isLoaded = true - self.loadedModel = model - - // ── Memory diagnostics (read after load) ─────────────────────────────── - let (footprintAfter, _) = appMemoryFootprintMB() - print("[MEM] After load — footprint: \(Int(footprintAfter)) MB") - print("[MEM] MLX after — active: \(MLX.GPU.activeMemory / 1_048_576) MB, cache: \(MLX.GPU.cacheMemory / 1_048_576) MB") - - let elapsed = (CFAbsoluteTimeGetCurrent() - loadStart) * 1000 - stats.loadTimeMs = elapsed - statusMessage = "模型已就绪 ✅ (\(Int(elapsed))ms)" - - print("[MLX] Model loaded in \(Int(elapsed))ms — backend: mlx-gpu — model: \(model.displayName)") - } - - private func ensureAudioCapability(hasAudio: Bool) async throws { - guard hasAudio != audioCapabilityEnabled || !isLoaded || modelContainer == nil else { - return - } - - audioCapabilityEnabled = hasAudio - print("[MLX] capability switch requested — audio=\(hasAudio ? 1 : 0)") - - if isLoaded || modelContainer != nil { - await prepareForReload(cancelCurrentGeneration: false, cancelCurrentLoad: false) - } - - try await load() - } - - /// Returns (footprint MB, jetsam limit MB) via task_info. - private func appMemoryFootprintMB() -> (Double, Double) { - var info = task_vm_info_data_t() - var count = mach_msg_type_number_t(MemoryLayout.size / MemoryLayout.size) - let kr = withUnsafeMutablePointer(to: &info) { - $0.withMemoryRebound(to: integer_t.self, capacity: Int(count)) { - task_info(mach_task_self_, task_flavor_t(TASK_VM_INFO), $0, &count) - } - } - guard kr == KERN_SUCCESS else { return (0, 0) } - let footprint = Double(info.phys_footprint) / 1_048_576 - let limit = Double(info.limit_bytes_remaining) / 1_048_576 + footprint - return (footprint, limit) - } - - /// 当前可用内存 headroom（MB）。Agent 用来动态调整 history 深度。 - public var availableHeadroomMB: Int { - let (footprint, limit) = appMemoryFootprintMB() - return max(0, Int(limit - footprint)) - } - - /// 根据当前剩余内存推荐安全的 history 深度（消息条数）。 - /// E4B 在 1 GB 左右 headroom 时对文本 KV cache 非常敏感，因此需要更激进地收紧历史。 - public var safeHistoryDepth: Int { - let h = availableHeadroomMB - let model = loadedModel ?? Self.availableModels.first(where: { $0.id == selectedModel.id }) - - if model?.id == Self.e4bModelID { - switch h { - case 1_700...: return 4 - case 1_100..<1_700: return 2 - default: return 0 - } - } else { - switch h { - case 1_500...: return 4 - case 900..<1_500: return 2 - default: return 0 - } - } - } - - - public func warmup() async throws { - // Warmup skipped for E2B. - // - // E2B has 26 layers (E4B has 42). Running MLXLMCommon.generate() for the first time - // triggers Metal JIT shader compilation across all unique kernel variants - // (attention, MLP, PLE, RoPE ...). This compilation adds a temporary - // memory spike on top of the already-loaded 4.9 GB weights, which pushes - // the process past the jetsam limit on iPhone 17 Pro Max. - // - // Skipping warmup means the first user inference compiles shaders lazily - // (first response is ~2-3s slower) but avoids the OOM kill on startup. - print("[MLX] Warmup skipped — shaders will compile on first inference") - statusMessage = "模型已就绪 ✅" - } - - public func generateStream( - prompt: String, - images: [CIImage], - audios: [UserInput.Audio] - ) -> AsyncThrowingStream { - let input: UserInput - if images.isEmpty, audios.isEmpty { - input = UserInput(prompt: prompt) - } else { - input = UserInput( - chat: [ - .user( - prompt, - images: images.map { .ciImage($0) }, - audios: audios - ) - ] - ) - } - return generateStream(input: input, isMultimodal: !images.isEmpty || !audios.isEmpty) - } - - public func generateStream(chat: [Chat.Message]) -> AsyncThrowingStream { - generateStream(chat: chat, additionalContext: nil) - } - - public func generateStream( - chat: [Chat.Message], - additionalContext: [String: any Sendable]? - ) -> AsyncThrowingStream { - let input = UserInput(chat: chat, additionalContext: additionalContext) - let hasMedia = !input.images.isEmpty || !input.audios.isEmpty - return generateStream(input: input, isMultimodal: hasMedia) - } - - private func dynamicMultimodalBudget( - hasImages: Bool, - hasAudio: Bool - ) throws -> MultimodalRuntimeBudget? { - guard hasImages || hasAudio else { return nil } - guard let model = loadedModel ?? Self.availableModels.first(where: { $0.id == selectedModel.id }) else { - return nil - } - let headroom = availableHeadroomMB - - guard model.id == Self.e4bModelID else { - return MultimodalRuntimeBudget( - imageSoftTokenCap: hasImages ? 160 : nil, - maxOutputTokens: Self.multimodalMaxOutputTokens, - headroomMB: headroom - ) - } - - guard headroom > Self.e4bMultimodalCriticalHeadroomMB else { - let recommendation = "请关闭后台应用后重试，或减少附件数量。\(currentMultimodalFallbackRecommendation())" - throw MLXError.multimodalMemoryRisk( - model: model.displayName, - headroomMB: headroom, - recommendation: recommendation - ) - } - - let imageSoftTokenCap: Int? - let maxOutputTokens: Int - - // Budget calibration (E4B, measured on-device): - // Actual activation cost ≈ 1 MB/output token - // (observed: 160 tokens consumed 167 MB headroom = 5311-5144 MB footprint delta) - // Safety margin formula: headroom − (imageSoftTokens × 2 MB) − (maxOut × 1 MB) > 300 MB - switch headroom { - case ..<500: - imageSoftTokenCap = hasImages ? 48 : nil - maxOutputTokens = 120 // was 64 — 48×2 + 120×1 + 300 = 516, fits <500 boundary - case ..<700: - imageSoftTokenCap = hasImages ? 64 : nil - maxOutputTokens = 200 // was 128 — 64×2 + 200 + 300 = 628 < 700 ✓ - case ..<900: - imageSoftTokenCap = hasImages ? 80 : nil - maxOutputTokens = 340 // was 192 — 80×2 + 340 + 300 = 800 < 900 ✓ - case ..<1_100: - imageSoftTokenCap = hasImages ? 96 : nil - maxOutputTokens = Self.multimodalMaxOutputTokens // 512: 96×2 + 512 + 300 = 1004 < 1100 ✓ - case ..<1_300: - imageSoftTokenCap = hasImages ? 128 : nil - maxOutputTokens = Self.multimodalMaxOutputTokens - default: - imageSoftTokenCap = hasImages ? 160 : nil - maxOutputTokens = Self.multimodalMaxOutputTokens - } - - print( - "[MEM] multimodal runtime budget — model=\(model.displayName), " - + "headroom=\(headroom) MB, " - + "imageSoftTokenCap=\(imageSoftTokenCap.map(String.init) ?? "n/a"), " - + "maxOutputTokens=\(maxOutputTokens), " - + "audio=\(hasAudio ? 1 : 0)" - ) - - return MultimodalRuntimeBudget( - imageSoftTokenCap: imageSoftTokenCap, - maxOutputTokens: maxOutputTokens, - headroomMB: headroom - ) - } - - private func isThinkingEnabled(for input: UserInput) -> Bool { - if let enabled = input.additionalContext?["enable_thinking"] as? Bool, enabled { - return true - } - - switch input.prompt { - case .text(let text): - return text.contains("<|think|>") - case .chat(let messages): - return messages.contains { $0.content.contains("<|think|>") } - case .messages(let messages): - return messages.contains { message in - if let content = message["content"] as? String { - return content.contains("<|think|>") - } - if let content = message["content"] as? [[String: any Sendable]] { - return content.contains { item in - (item["text"] as? String)?.contains("<|think|>") == true - } - } - return false - } - } - } - - private func dynamicThinkingBudget( - enabled: Bool - ) -> ThinkingRuntimeBudget? { - guard enabled else { return nil } - guard let model = loadedModel ?? Self.availableModels.first(where: { $0.id == selectedModel.id }) else { - return nil - } - - let headroom = availableHeadroomMB - let maxOutputTokens: Int - - if model.id == Self.e4bModelID { - switch headroom { - case ..<500: - maxOutputTokens = 128 - case ..<700: - maxOutputTokens = 192 - case ..<900: - maxOutputTokens = 256 - case ..<1_100: - maxOutputTokens = 384 - case ..<1_300: - maxOutputTokens = 512 - default: - maxOutputTokens = Self.e4bThinkingMaxOutputTokens - } - } else { - switch headroom { - case ..<500: - maxOutputTokens = 192 - case ..<800: - maxOutputTokens = 384 - case ..<1_200: - maxOutputTokens = 512 - default: - maxOutputTokens = Self.e2bThinkingMaxOutputTokens - } - } - - print( - "[MEM] thinking runtime budget — model=\(model.displayName), " - + "headroom=\(headroom) MB, " - + "maxOutputTokens=\(maxOutputTokens)" - ) - - return ThinkingRuntimeBudget( - maxOutputTokens: maxOutputTokens, - headroomMB: headroom - ) - } - - private func dynamicTextBudget( - enabled: Bool - ) -> TextRuntimeBudget? { - guard enabled else { return nil } - guard let model = loadedModel ?? Self.availableModels.first(where: { $0.id == selectedModel.id }) else { - return nil - } - - let headroom = availableHeadroomMB - let maxOutputTokens: Int - - if model.id == Self.e4bModelID { - switch headroom { - case ..<500: - maxOutputTokens = 256 - case ..<700: - maxOutputTokens = 384 - case ..<900: - maxOutputTokens = 512 - case ..<1_100: - maxOutputTokens = 768 - case ..<1_300: - maxOutputTokens = 1_024 - default: - maxOutputTokens = 1_280 - } - } else { - switch headroom { - case ..<500: - maxOutputTokens = 384 - case ..<800: - maxOutputTokens = 768 - case ..<1_200: - maxOutputTokens = 1_024 - default: - maxOutputTokens = 1_536 - } - } - - print( - "[MEM] text runtime budget — model=\(model.displayName), " - + "headroom=\(headroom) MB, " - + "maxOutputTokens=\(maxOutputTokens)" - ) - - return TextRuntimeBudget( - maxOutputTokens: maxOutputTokens, - headroomMB: headroom - ) - } - - private func adjustedTextOutputTokens( - baseMaxOutputTokens: Int, - preparedSequenceLength: Int, - thinkingEnabled: Bool - ) -> Int? { - guard let model = loadedModel ?? Self.availableModels.first(where: { $0.id == selectedModel.id }) else { - return nil - } - - let headroom = availableHeadroomMB - let totalSequenceBudget: Int - - if model.id == Self.e4bModelID { - if thinkingEnabled { - switch headroom { - case ..<500: - totalSequenceBudget = 320 - case ..<700: - totalSequenceBudget = 448 - case ..<900: - totalSequenceBudget = 576 - case ..<1_100: - totalSequenceBudget = 896 - case ..<1_300: - totalSequenceBudget = 1_152 - default: - totalSequenceBudget = 1_280 - } - } else { - switch headroom { - case ..<500: - totalSequenceBudget = 384 - case ..<700: - totalSequenceBudget = 512 - case ..<900: - totalSequenceBudget = 640 - case ..<1_100: - totalSequenceBudget = 768 - case ..<1_300: - totalSequenceBudget = 896 - default: - totalSequenceBudget = 1_024 - } - } - } else { - if thinkingEnabled { - switch headroom { - case ..<500: - totalSequenceBudget = 448 - case ..<800: - totalSequenceBudget = 640 - case ..<1_200: - totalSequenceBudget = 896 - default: - totalSequenceBudget = 1_152 - } - } else { - switch headroom { - case ..<500: - totalSequenceBudget = 512 - case ..<800: - totalSequenceBudget = 768 - case ..<1_200: - totalSequenceBudget = 1_024 - default: - totalSequenceBudget = 1_280 - } - } - } - - let minimumOutputTokens = thinkingEnabled ? 96 : 128 - let adjustedMaxOutputTokens = min( - baseMaxOutputTokens, - max(minimumOutputTokens, totalSequenceBudget - preparedSequenceLength) - ) - - print( - "[MEM] text sequence budget — model=\(model.displayName), " - + "headroom=\(headroom) MB, " - + "preparedTokens=\(preparedSequenceLength), " - + "totalSequenceBudget=\(totalSequenceBudget), " - + "adjustedMaxOutputTokens=\(adjustedMaxOutputTokens)" - ) - - return adjustedMaxOutputTokens - } - - private func currentMultimodalFallbackRecommendation() -> String { - if let e2b = Self.availableModels.first(where: { $0.id == Self.e2bModelID }) { - if isModelAvailable(e2b) { - return "如仍失败，可手动切换到 \(e2b.displayName) 处理图片或音频。" - } - return "如仍失败，可先下载并手动切换到 \(e2b.displayName)。" - } else { - return "如仍失败，请改用更轻量的模型处理图片或音频。" - } - } - - private func ensureForegroundGPUExecution() async throws { - #if canImport(UIKit) - let isActive = await MainActor.run { - UIApplication.shared.applicationState == .active - } - setForegroundGPUAllowed(isActive) - guard isActive else { - throw MLXError.gpuExecutionRequiresForeground - } - #endif - } - - private func configureLifecycleObservers() { - #if canImport(UIKit) - let center = NotificationCenter.default - lifecycleObserverTokens = [ - center.addObserver( - forName: UIApplication.willResignActiveNotification, - object: nil, - queue: nil - ) { [weak self] _ in - self?.handleApplicationLeavingForeground() - }, - center.addObserver( - forName: UIApplication.didEnterBackgroundNotification, - object: nil, - queue: nil - ) { [weak self] _ in - self?.handleApplicationLeavingForeground() - }, - center.addObserver( - forName: UIApplication.willEnterForegroundNotification, - object: nil, - queue: nil - ) { [weak self] _ in - self?.setForegroundGPUAllowed(true) - }, - center.addObserver( - forName: UIApplication.didBecomeActiveNotification, - object: nil, - queue: nil - ) { [weak self] _ in - self?.setForegroundGPUAllowed(true) - } - ] - - Task { [weak self] in - guard let self else { return } - let isActive = await MainActor.run { - UIApplication.shared.applicationState == .active - } - self.setForegroundGPUAllowed(isActive) - } - #endif - } - - private func handleApplicationLeavingForeground() { - setForegroundGPUAllowed(false) - cancelled = true - currentGenerationTask?.cancel() - currentLoadTask?.cancel() - } - - private func setForegroundGPUAllowed(_ allowed: Bool) { - foregroundStateLock.lock() - foregroundGPUAllowed = allowed - foregroundStateLock.unlock() - } - - private func isForegroundGPUAllowed() -> Bool { - foregroundStateLock.lock() - let allowed = foregroundGPUAllowed - foregroundStateLock.unlock() - return allowed - } - - private func generateStream( - input: UserInput, - isMultimodal: Bool - ) -> AsyncThrowingStream { - AsyncThrowingStream { continuation in - let task = Task { [weak self] in - guard let self else { - continuation.finish(throwing: MLXError.modelNotLoaded) - return - } - do { - try await self.ensureAudioCapability(hasAudio: !input.audios.isEmpty) - } catch { - continuation.finish(throwing: error) - return - } - - guard let container = modelContainer else { - continuation.finish(throwing: MLXError.modelNotLoaded) - return - } - - // Free Metal buffers cached from previous inference before - // allocating the new computation graph. Critical on low-headroom devices: - // the follow-up prompt is longer than the first inference, - // and without clearing, residual cache + new activations - // exceed the 6GB jetsam limit on iPhone. - MLX.GPU.clearCache() - - let thinkingEnabled = self.isThinkingEnabled(for: input) - let textBudget = self.dynamicTextBudget(enabled: !isMultimodal) - let runtimeBudget: MultimodalRuntimeBudget? - do { - runtimeBudget = try self.dynamicMultimodalBudget( - hasImages: !input.images.isEmpty, - hasAudio: !input.audios.isEmpty - ) - } catch { - continuation.finish(throwing: error) - return - } - let thinkingBudget = self.dynamicThinkingBudget(enabled: thinkingEnabled) - Gemma4Processor.setRuntimeImageSoftTokenCap(runtimeBudget?.imageSoftTokenCap) - defer { - Gemma4Processor.setRuntimeImageSoftTokenCap(nil) - } - let effectiveMaxOutputTokens: Int = { - let multimodalCap = isMultimodal - ? runtimeBudget?.maxOutputTokens ?? Self.multimodalMaxOutputTokens - : maxOutputTokens - let thinkingCap = thinkingBudget?.maxOutputTokens ?? maxOutputTokens - let textCap = textBudget?.maxOutputTokens ?? maxOutputTokens - return min(maxOutputTokens, multimodalCap, thinkingCap, textCap) - }() - var resolvedMaxOutputTokens = effectiveMaxOutputTokens - - self.isGenerating = true - self.cancelled = false - let genStart = CFAbsoluteTimeGetCurrent() - var firstTokenTime: Double? = nil - var tokenCount = 0 - var hitTokenCap = false - - let (fp, _) = appMemoryFootprintMB() - print("[MEM] generateStream start — footprint: \(Int(fp)) MB, MLX active: \(MLX.GPU.activeMemory / 1_048_576) MB") - - do { - try await self.ensureForegroundGPUExecution() - _ = try await container.perform { context in - try await self.ensureForegroundGPUExecution() - if isMultimodal { - print("[VLM] multimodal budget — maxOutputTokens=\(resolvedMaxOutputTokens)") - } else if thinkingEnabled { - print("[LLM] thinking budget — baseMaxOutputTokens=\(resolvedMaxOutputTokens)") - } - let preparedInput = try await context.processor.prepare(input: input) - if isMultimodal { - print("[VLM] prepared sequence length=\(preparedInput.text.tokens.dim(1))") - } else { - let preparedSequenceLength = preparedInput.text.tokens.dim(1) - print("[LLM] prepared sequence length=\(preparedSequenceLength)") - resolvedMaxOutputTokens = - self.adjustedTextOutputTokens( - baseMaxOutputTokens: resolvedMaxOutputTokens, - preparedSequenceLength: preparedSequenceLength, - thinkingEnabled: thinkingEnabled - ) ?? resolvedMaxOutputTokens - if textBudget != nil { - print("[LLM] text budget — maxOutputTokens=\(resolvedMaxOutputTokens)") - } else if thinkingEnabled { - print("[LLM] thinking budget — maxOutputTokens=\(resolvedMaxOutputTokens)") - } - } - try await self.ensureForegroundGPUExecution() - - return try MLXLMCommon.generate( - input: preparedInput, - parameters: .init( - maxTokens: resolvedMaxOutputTokens, - temperature: samplingTemperature, - topP: samplingTopP, - topK: samplingTopK - ), - context: context - ) { tokens in - if self.cancelled || !self.isForegroundGPUAllowed() { - return .stop - } - - tokenCount = tokens.count - if firstTokenTime == nil { - firstTokenTime = (CFAbsoluteTimeGetCurrent() - genStart) * 1000 - } - - // Stream the latest token - if let lastToken = tokens.last { - let text = context.tokenizer.decode(tokenIds: [lastToken]) - continuation.yield(text) - } - - // Multimodal path uses a tighter generation budget on iPhone. - // If we hit the cap, signal truncation so the caller can append a notice. - if tokens.count >= resolvedMaxOutputTokens { - hitTokenCap = true - return .stop - } - return .more - } - } - - let elapsed = CFAbsoluteTimeGetCurrent() - genStart - self.stats.ttftMs = firstTokenTime ?? 0 - self.stats.tokensPerSec = elapsed > 0 - ? Double(tokenCount) / elapsed : 0 - self.stats.totalTokens = tokenCount - - print( - "[MLX] Generated \(tokenCount) tokens in \(String(format: "%.1f", elapsed))s" - ) - print( - "[MLX] TTFT: \(String(format: "%.0f", self.stats.ttftMs))ms, " - + "Speed: \(String(format: "%.1f", self.stats.tokensPerSec)) tok/s") - - // 推理结束后立即释放 Metal activation 缓存， - // 确保下一轮有最大可用 headroom。 - MLX.GPU.clearCache() - let (fpEnd, _) = appMemoryFootprintMB() - print("[MEM] generateStream end — footprint: \(Int(fpEnd)) MB, headroom: \(self.availableHeadroomMB) MB") - - // If we hit the token cap mid-sentence, append a visible notice. - // This makes truncation explicit rather than silently dropping content. - if hitTokenCap { - let isChinese = Locale.preferredLanguages.contains { $0.hasPrefix("zh") } - let modeLabel = isChinese - ? (thinkingEnabled ? "思考" : "输出") - : (thinkingEnabled ? "Thinking" : "Output") - continuation.yield("\n\n> ⚠️ \(modeLabel)已达内存安全上限（\(resolvedMaxOutputTokens) tokens），内容可能不完整。") - } - continuation.finish() - } catch { - continuation.finish(throwing: error) - } - - self.isGenerating = false - self.currentGenerationTask = nil - } - - currentGenerationTask = task - continuation.onTermination = { [weak self] _ in - task.cancel() - if self?.currentGenerationTask?.isCancelled == true { - self?.currentGenerationTask = nil - } - } - } - } - - public func cancel() { - cancelled = true - currentGenerationTask?.cancel() - currentLoadTask?.cancel() - } - - public func prepareForReload( - cancelCurrentGeneration: Bool = true, - cancelCurrentLoad: Bool = true - ) async { - cancelled = true - if cancelCurrentGeneration { - currentGenerationTask?.cancel() - } - if cancelCurrentLoad { - currentLoadTask?.cancel() - } - - while isGenerating || isLoading { - try? await Task.sleep(nanoseconds: 50_000_000) - } - - unload( - cancelCurrentGeneration: cancelCurrentGeneration, - cancelCurrentLoad: cancelCurrentLoad - ) - MLX.GPU.clearCache() - try? await Task.sleep(nanoseconds: 150_000_000) - } - - public func unload() { - unload(cancelCurrentGeneration: true, cancelCurrentLoad: true) - } - - public func unload( - cancelCurrentGeneration: Bool = true, - cancelCurrentLoad: Bool = true - ) { - if cancelCurrentGeneration { - currentGenerationTask?.cancel() - } - if cancelCurrentLoad { - currentLoadTask?.cancel() - } - modelContainer = nil - isLoaded = false - isLoading = false - isGenerating = false - loadedModel = nil - cancelled = false - stats = LLMStats() - stats.backend = "mlx-gpu" - MLX.GPU.clearCache() - statusMessage = "模型已卸载" - print("[MLX] Model unloaded") - } -} - -// MARK: - Errors - -enum MLXError: LocalizedError { - case modelNotLoaded - case modelDirectoryMissing(String) - case gpuExecutionRequiresForeground - case multimodalMemoryRisk(model: String, headroomMB: Int, recommendation: String) - - var errorDescription: String? { - switch self { - case .modelNotLoaded: - return "MLX model not loaded. Call load() first." - case .modelDirectoryMissing(let modelName): - return "\(modelName) 模型文件不存在，请先在配置页下载或重新安装。" - case .gpuExecutionRequiresForeground: - return "应用进入后台时，iPhone 不允许继续提交 GPU 推理任务。" - case .multimodalMemoryRisk(let model, let headroomMB, let recommendation): - return "\(model) 当前剩余内存仅约 \(headroomMB) MB，继续处理图片/音频很可能被系统直接杀掉。\(recommendation)" - } - } -} - -enum DownloadError: LocalizedError { - case invalidURL(String) - case invalidResponse - case httpStatus(Int) - - var errorDescription: String? { - switch self { - case .invalidURL(let file): - return "无法构造下载链接：\(file)" - case .invalidResponse: - return "下载源响应无效" - case .httpStatus(let statusCode): - switch statusCode { - case 401, 403: - return "下载源拒绝访问（\(statusCode)）" - case 404: - return "模型文件不存在（404）" - case 429: - return "下载过于频繁，请稍后重试（429）" - default: - return "下载失败，HTTP \(statusCode)" - } - } - } -} diff --git a/LLM/PromptBuilder.swift b/LLM/PromptBuilder.swift index f2d1abb..1749c9b 100644 --- a/LLM/PromptBuilder.swift +++ b/LLM/PromptBuilder.swift @@ -32,11 +32,58 @@ struct PromptBuilder { return "\n" + Array(repeating: "<|image|>", count: count).joined(separator: "\n") } + /// 工具参数提取阶段需要的"当前时间锚点"。 + /// 模型必须知道"现在是何时"才能解析"明天/下午两点"等相对时间表达 + /// 并写出正确的 ISO 8601 字符串。 + /// 用本地时区(用户设备时区), 周几用中文,方便中文模型理解。 + private static func currentTimeAnchorBlock() -> String { + let formatter = DateFormatter() + formatter.locale = Locale(identifier: "zh_CN") + formatter.timeZone = .current + formatter.dateFormat = "yyyy-MM-dd EEEE HH:mm" + let now = formatter.string(from: Date()) + return "当前时间锚点(用于解析\"今天/明天/下午两点\"等相对时间): \(now)" + } + private static func extractSystemBlock(from prompt: String) -> String { + let raw: String if let turnEnd = prompt.range(of: "\n") { - return String(prompt[prompt.startIndex...turnEnd.upperBound]) + raw = String(prompt[prompt.startIndex...turnEnd.upperBound]) + } else { + raw = prompt } - return prompt + // 在 secondary prompt 的 system block 末尾追加"当前时间锚点"。 + // 这是 runtime data injection,不感知任何业务:任何调用 extractSystemBlock + // 的 prompt builder(load_skill 之后、tool answer、planner 各阶段等) + // 自动获得"现在是何时"的上下文,模型才能解析"明天/下午两点"等 + // 相对时间表达。 + return injectIntoSystemBlock(raw, extraInstructions: currentTimeAnchorBlock()) + } + + /// 从一个完整 prompt(由 PromptBuilder.build() 构造)里提取 + /// "system 块结束之后, 当前 user 消息开始之前"的所有历史 turn 块。 + /// + /// 用途: secondary 推理(load_skill 之后、tool 执行之后、planner 各阶段) + /// 自动获得和 first inference 同样的对话历史, 不再是无记忆地只看当前消息。 + /// + /// 实现是纯字符串切片, 不感知任何业务: 切的是 PromptBuilder.build() 自己 + /// 渲染的 turn 标签结构, 任何 secondary prompt builder 都能受益。 + private static func extractHistoryBlock(from prompt: String) -> String { + guard let systemEnd = prompt.range(of: "\n") else { return "" } + let afterSystem = systemEnd.upperBound + + // 找最后一个 "<|turn>user\n" - 那是当前 user message 的开头 + let searchRange = afterSystem..user\n", + options: .backwards, + range: searchRange + ) else { + return "" + } + + // 返回 system 结束 ~ 当前 user 开始 之间的所有历史 turn + return String(prompt[afterSystem.. String { let isMultimodalTurn = currentImageCount > 0 var prompt = "<|turn>system\n" @@ -103,25 +151,46 @@ struct PromptBuilder { : (systemPrompt ?? defaultSystemPrompt) // 构建 Skill 概要列表（只列名称 + 一句话描述，不暴露 Tool） - var skillListText = "" - for skill in tools { - skillListText += "- **\(skill.name)**: \(skill.description)\n" + // 按 SkillType 分两组, 给模型不同调用规则。 + let deviceSkills = tools.filter { $0.type == .device } + let contentSkills = tools.filter { $0.type == .content } + func renderList(_ list: [SkillInfo]) -> String { + if list.isEmpty { return "（无）\n" } + return list.map { "- **\($0.name)**: \($0.description)" }.joined(separator: "\n") + "\n" } + let deviceListText = renderList(deviceSkills) + let contentListText = renderList(contentSkills) + // 兼容旧版 SYSPROMPT.md (仅 ___SKILLS___) 的扁平列表 + let flatListText: String = { + var s = "" + for skill in tools { s += "- **\(skill.name)**: \(skill.description)\n" } + return s + }() if isMultimodalTurn { prompt += basePrompt + } else if basePrompt.contains("___DEVICE_SKILLS___") || basePrompt.contains("___CONTENT_SKILLS___") { + // 新版双占位符: 按类别分别注入 + var resolved = basePrompt + resolved = resolved.replacingOccurrences(of: "___DEVICE_SKILLS___", with: deviceListText) + resolved = resolved.replacingOccurrences(of: "___CONTENT_SKILLS___", with: contentListText) + prompt += resolved } else if basePrompt.contains("___SKILLS___") { - // 处理 ___SKILLS___ 占位符 - prompt += basePrompt.replacingOccurrences(of: "___SKILLS___", with: skillListText) + // 旧版扁平占位符: 保留向后兼容 + prompt += basePrompt.replacingOccurrences(of: "___SKILLS___", with: flatListText) } else { - // SYSPROMPT.md 不含 ___SKILLS___ 时的兜底：只追加技能列表，不追加指令。 - // 调用规则已在 SYSPROMPT.md 里定义，不在这里硬编。 + // SYSPROMPT.md 不含任何占位符时的兜底：只追加技能列表，不追加指令。 prompt += basePrompt if !tools.isEmpty { - prompt += "\n\n你拥有以下能力（Skill）：\n\n" + skillListText + prompt += "\n\n你拥有以下能力（Skill）：\n\n" + flatListText } } + // 仅全量注入（无匹配命中）时，提示模型可通过 list_skills 发现更多能力 + if showListSkillsHint && !isMultimodalTurn { + prompt += "\n如果以上列出的能力都不匹配用户需求，可以调用 list_skills 查询更多能力：\n\n{\"name\": \"list_skills\", \"arguments\": {\"query\": \"用户需求描述\"}}\n\n" + } + if enableThinking && !isMultimodalTurn { prompt += "\n\n" + thinkingLanguageInstruction } @@ -207,38 +276,58 @@ struct PromptBuilder { originalPrompt: String, userQuestion: String, skillInstructions: String, + availableTools: [String], currentImageCount: Int = 0, forceResponse: Bool = false ) -> String { + // Scaffold (T2 progressive disclosure): + // 显式告诉模型当前 skill 有哪些工具可调 (只列名字, 不列 schema — + // schema 属于 T3 暴露)。空列表时明确说"无工具", 防止模型幻觉编造 + // 不存在的工具名 (例如 "professional_translator")。 + let toolBlock: String + if availableTools.isEmpty { + toolBlock = """ + 当前 Skill **没有任何可调用的工具**。 + 按 Skill 指令直接给最终答案正文文本, 禁止输出 。 + """ + } else { + let listText = availableTools.map { "- `\($0)`" }.joined(separator: "\n") + toolBlock = """ + 当前 Skill 可调用的工具 (只允许这些名字): + \(listText) + 如果需要操作, 输出 {"name": "<上面列表中的名字>", "arguments": {...}}。 + 其他名字一律视为非法, 不要凭空编造。 + 如果不需要工具, 直接给最终答案正文文本。 + """ + } + let systemBlock = extractSystemBlock(from: originalPrompt) let systemInstructions = injectIntoSystemBlock( systemBlock, extraInstructions: """ - 对于当前这一个用户问题，你已经加载了所需的 Skill 指令。 - 不要再次调用 `load_skill`。 - 如果需要执行设备内操作，直接调用对应工具；如果不需要工具，直接回答。 + 对于当前这一个用户问题, 你已经加载了所需的 Skill 指令。不要再次调用 `load_skill`。 - 已加载的 Skill 指令： + 已加载的 Skill 指令: \(skillInstructions) + + \(toolBlock) """ ) var prompt = systemInstructions + prompt += extractHistoryBlock(from: originalPrompt) prompt += """ <|turn>user - 用户问题： + 用户问题: \(userQuestion)\(imagePromptSuffix(count: currentImageCount)) - 处理这个请求时，严格按以下顺序执行： - 1. 使用已经加载的 Skill 指令，不要再次调用 `load_skill`。 - 2. 如果需要设备内操作，直接调用对应工具。 - 3. 如果工具已经成功返回，或者已经足够回答，就只输出最终结果。 - - 不要让用户去“打开 skill”或“使用某个能力”，需要的话你自己直接调用工具。 - 你必须避免输出任何中间思考、状态更新、字段名、JSON 模板、代码块或规划草稿。 + 按上面的 Skill 指令处理这个请求。 + - 不要再次调用 load_skill。 + - 不要让用户去"打开 skill"或"使用某个能力"。 + - 不要输出中间思考/状态更新/字段名/JSON 模板/代码块/规划草稿。 \(forceResponse - ? "你的下一条回复必须是以下两种之一：1. 一个 `...` 2. 直接给用户的最终回答正文。禁止输出空白。" - : "如果需要工具就直接调用；如果已经足够回答，就直接给出最终答案正文。") + ? "你必须输出非空内容: 要么是合法的 ..., 要么是最终答案正文。" + : "如果不需要工具就直接给最终答案正文; 如果需要工具按上面规定的工具名调用。") <|turn>model @@ -257,7 +346,7 @@ struct PromptBuilder { ) -> String { let systemBlock = extractSystemBlock(from: originalPrompt) - return systemBlock + """ + return systemBlock + extractHistoryBlock(from: originalPrompt) + """ <|turn>user 用户原始问题： \(userQuestion)\(imagePromptSuffix(count: currentImageCount)) @@ -299,7 +388,7 @@ struct PromptBuilder { """ ) - return systemInstructions + """ + return systemInstructions + extractHistoryBlock(from: originalPrompt) + """ <|turn>user 用户问题： \(userQuestion)\(imagePromptSuffix(count: currentImageCount)) @@ -314,8 +403,9 @@ struct PromptBuilder { 3. 不要输出 Markdown、代码块、解释、字段草稿或多余文字。 4. 可选字段如果没有，就直接省略。 5. 时间字段必须转换成 ISO 8601，例如 `2026-04-07T20:00:00`。 - 6. 如果缺少必填参数，输出： - {"_needs_clarification":"请补充缺少的信息"} + 6. 如果缺少必填参数,只输出一个 JSON object: {"_needs_clarification": "..."}。 + "..." 部分必须用一句完整中文,直接陈述当前请求缺哪个具体参数, + 不能含尖括号、不能含占位符、不能复制本规则的任何字面文本。 <|turn>model @@ -342,7 +432,7 @@ struct PromptBuilder { """ ) - return systemInstructions + """ + return systemInstructions + extractHistoryBlock(from: originalPrompt) + """ <|turn>user 用户问题： \(userQuestion)\(imagePromptSuffix(count: currentImageCount)) @@ -362,8 +452,219 @@ struct PromptBuilder { 4. `arguments` 里只保留当前工具需要的参数；没有的可选参数直接省略。 5. 不要输出 Markdown、代码块、解释、草稿或多余文字。 6. 时间字段必须转换成 ISO 8601，例如 `2026-04-07T20:00:00`。 - 7. 如果缺少执行所需的关键信息，输出： - {"_needs_clarification":"请补充缺少的信息"} + 7. 如果缺少执行所需的关键信息,只输出一个 JSON object: {"_needs_clarification": "..."}。 + "..." 部分必须用一句完整中文,直接陈述当前请求缺哪个具体信息, + 不能含尖括号、不能含占位符、不能复制本规则的任何字面文本。 + + <|turn>model + + """ + } + + // MARK: - Planner v3 Prompt Builders + + static func buildSkillSelectionPrompt( + originalPrompt: String, + userQuestion: String, + availableSkillsSummary: String, + recentContextSummary: String = "", + currentImageCount: Int = 0 + ) -> String { + let systemBlock = extractSystemBlock(from: originalPrompt) + let systemInstructions = injectIntoSystemBlock( + systemBlock, + extraInstructions: """ + 对于当前这一个用户问题，你现在只负责判断需要哪些 Skill。 + 不要调用 `load_skill`，不要输出 ``，不要直接回答用户问题。 + + 可用的 Skill 与工具如下： + \(availableSkillsSummary) + """ + ) + + let recentContextBlock: String + if recentContextSummary.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { + recentContextBlock = "" + } else { + recentContextBlock = """ + + 最近已知的工具结果摘要（可作为当前规划的上下文）： + \(recentContextSummary) + """ + } + + return systemInstructions + extractHistoryBlock(from: originalPrompt) + """ + <|turn>user + 用户问题： + \(userQuestion)\(imagePromptSuffix(count: currentImageCount))\(recentContextBlock) + + 请输出一个 JSON object，格式必须是： + { + "required_skills": ["skill_id_1", "skill_id_2"], + "needs_clarification": null + } + + 严格遵守以下要求: + 1. 只输出 JSON object,不要输出 Markdown、代码块、解释或多余文字。 + 2. `required_skills` 里的每一项必须严格等于上面"可用的 Skill 与工具"段落中列出的 skill id 字符串本身,不能填该 skill 下属的工具名,不能自己拼接,不能翻译。 + 3. 如果任务需要先获取一个结果、再交给另一个 Skill 继续处理,涉及到的所有 Skill 都要列出来,不要只写最终那一个。 + 4. 如果"最近已知的工具结果摘要"已经提供了部分信息,也要据此补全后续需要的 Skill,不要漏掉。 + 5. 如果用户需求不需要任何 Skill,返回空数组 `[]`。 + 6. 如果无法判断需要哪些 Skill,返回: + {"required_skills": [], "needs_clarification": "请说明具体需要什么帮助"} + + <|turn>model + + """ + } + + static func buildSkillPlanningPrompt( + originalPrompt: String, + userQuestion: String, + availableSkillsSummary: String, + recentContextSummary: String = "", + currentImageCount: Int = 0 + ) -> String { + let systemBlock = extractSystemBlock(from: originalPrompt) + let systemInstructions = injectIntoSystemBlock( + systemBlock, + extraInstructions: """ + 对于当前这一个用户问题，你现在只负责生成完整的执行计划。 + 不要调用 `load_skill`，不要输出 ``，不要直接回答用户问题。 + + 本阶段只允许使用以下已选中的 Skill 与工具： + \(availableSkillsSummary) + """ + ) + + let recentContextBlock: String + if recentContextSummary.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { + recentContextBlock = "" + } else { + recentContextBlock = """ + + 最近已知的工具结果摘要（可作为当前规划的上下文）： + \(recentContextSummary) + """ + } + + return systemInstructions + extractHistoryBlock(from: originalPrompt) + """ + <|turn>user + 用户问题： + \(userQuestion)\(imagePromptSuffix(count: currentImageCount))\(recentContextBlock) + + 请输出一个 JSON object，格式必须是： + { + "goal": "一句话目标", + "steps": [ + { + "id": "s1", + "skill": "skill_id", + "tool": "tool-name", + "intent": "这一步要做什么", + "depends_on": [] + } + ], + "needs_clarification": null + } + + 严格遵守以下要求： + 1. 只输出 JSON object，不要输出 Markdown、代码块、解释或多余文字。 + 2. `skill` 必须是上面给出的 skill id 之一，`tool` 必须是该 skill 允许的工具之一，使用完整工具名。 + 3. step 最多 4 步，按执行顺序排列。每个已选 skill 至少规划一步。 + 4. `depends_on` 里只能引用前面步骤的 id。如果后续步骤需要前面步骤的结果，必须填写依赖。 + 5. 如果不需要任何技能或工具，返回 `steps: []`。 + 6. 如果后续步骤需要的信息可以通过前置步骤获得，或者已经出现在"最近已知的工具结果摘要"里，仍然要先把这些步骤规划出来，不要提前提问。 + 7. 只要 `steps` 里还能放入至少一个可执行步骤，`needs_clarification` 就必须是 null。 + 8. 只有在没有任何可行步骤可以获得关键缺失信息时，才返回： + {"goal":"", "steps": [], "needs_clarification": "请说明具体需要什么"} + + <|turn>model + + """ + } + + static func buildPlannedToolArgumentsPrompt( + originalPrompt: String, + userQuestion: String, + stepIntent: String, + toolName: String, + toolParameters: String, + completedStepSummary: String = "", + currentImageCount: Int = 0 + ) -> String { + let systemBlock = extractSystemBlock(from: originalPrompt) + let systemInstructions = injectIntoSystemBlock( + systemBlock, + extraInstructions: """ + 对于当前这一个用户问题，你现在只负责为工具提取参数。 + 不要调用 `load_skill`，不要输出 ``，不要直接回答用户问题。 + """ + ) + + let completedBlock: String + if completedStepSummary.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { + completedBlock = "" + } else { + completedBlock = "已完成步骤摘要：\n\(completedStepSummary)" + } + + return systemInstructions + extractHistoryBlock(from: originalPrompt) + """ + <|turn>user + 用户问题： + \(userQuestion)\(imagePromptSuffix(count: currentImageCount)) + + 当前步骤目标： + \(stepIntent) + + \(completedBlock) + + 你现在只负责为工具 `\(toolName)` 提取 arguments。 + 工具参数说明： + \(toolParameters) + + 严格遵守以下要求： + 1. 不要调用工具，不要输出 ``。 + 2. 只输出一个 JSON object，内容就是 arguments 本身。 + 3. 不要输出 Markdown、代码块、解释、字段草稿或多余文字。 + 4. 可选字段如果没有，就直接省略。 + 5. 如果上面的已完成步骤里已经包含当前工具需要的信息，可以直接引用那些结果来补齐参数。 + 6. 时间字段必须转换成 ISO 8601，例如 `2026-04-07T20:00:00`。 + 7. 如果缺少必填参数,只输出一个 JSON object: {"_needs_clarification": "..."}。 + "..." 部分必须用一句完整中文,直接陈述当前步骤缺哪个具体参数, + 不能含尖括号、不能含占位符、不能复制本规则的任何字面文本。 + + <|turn>model + + """ + } + + static func buildMultiToolAnswerPrompt( + originalPrompt: String, + toolResults: [(toolName: String, result: String)], + userQuestion: String, + currentImageCount: Int = 0 + ) -> String { + let systemBlock = extractSystemBlock(from: originalPrompt) + var resultsBlock = "" + for (toolName, result) in toolResults { + resultsBlock += "工具 \(toolName) 的执行结果：\(result)\n" + } + + return systemBlock + extractHistoryBlock(from: originalPrompt) + """ + <|turn>user + 用户原始问题： + \(userQuestion)\(imagePromptSuffix(count: currentImageCount)) + + 所有工具已执行完成： + \(resultsBlock) + + 请基于以上所有结果回答用户： + - 如果以上结果已经能回答用户问题，直接给出最终回答，不要重复调用已经成功的工具。 + - 如果还需要继续调用新的工具来补全答案，可以输出一个或多个 `...`。 + - 不要反问，不要提到工具名、Skill、status、result、arguments 等字段。 + - 不要输出 Markdown 代码块，也不要输出 JSON、键名、模板或中间步骤。 + - 不能输出空白。 <|turn>model diff --git a/PhoneClaw.xcodeproj/project.pbxproj b/PhoneClaw.xcodeproj/project.pbxproj index 370fbd5..1c75b4d 100644 --- a/PhoneClaw.xcodeproj/project.pbxproj +++ b/PhoneClaw.xcodeproj/project.pbxproj @@ -11,13 +11,30 @@ 2F4A6C8E0A1B2C3D4E5F6071 /* AudioCaptureService.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2F4A6C8E0A1B2C3D4E5F6070 /* AudioCaptureService.swift */; }; 45B7FEC6C4D055F44A87BFA9 /* PhoneClawApp.swift in Sources */ = {isa = PBXBuildFile; fileRef = 4DD9808FE93325C3CE4CFA00 /* PhoneClawApp.swift */; }; 5084E1CD2E831BC05F5E4374 /* PromptBuilder.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2E7C06DA449D1395217A42D5 /* PromptBuilder.swift */; }; + 6A7B8C9D0E1F203142536475 /* gemma-4-e4b-it-4bit in Resources */ = {isa = PBXBuildFile; fileRef = D5E6F7081A2B3C4D5E6F7086 /* gemma-4-e4b-it-4bit */; }; 9412304167DACB86E5D60D49 /* ContentView.swift in Sources */ = {isa = PBXBuildFile; fileRef = F5ED928BF14864713A9A9D16 /* ContentView.swift */; }; 97E96A2FBFFAF609F276A896 /* ConfigurationsView.swift in Sources */ = {isa = PBXBuildFile; fileRef = E62C3CEFC7AE63873D751E75 /* ConfigurationsView.swift */; }; + A31476EA2F856FD700318978 /* ChatAttachments.swift in Sources */ = {isa = PBXBuildFile; fileRef = A31476E82F856FD700318978 /* ChatAttachments.swift */; }; + A31476EB2F856FD700318978 /* ChatMessage.swift in Sources */ = {isa = PBXBuildFile; fileRef = A31476E92F856FD700318978 /* ChatMessage.swift */; }; + A31476F62F85741500318978 /* Planner.swift in Sources */ = {isa = PBXBuildFile; fileRef = A31476F02F85741500318978 /* Planner.swift */; }; + A31476F82F85741500318978 /* Router.swift in Sources */ = {isa = PBXBuildFile; fileRef = A31476F12F85741500318978 /* Router.swift */; }; + A31476F92F85741500318978 /* SessionStore.swift in Sources */ = {isa = PBXBuildFile; fileRef = A31476F22F85741500318978 /* SessionStore.swift */; }; + A31476FA2F85741500318978 /* ToolChain.swift in Sources */ = {isa = PBXBuildFile; fileRef = A31476F42F85741500318978 /* ToolChain.swift */; }; + A31476FB2F85741500318978 /* OutputCleaner.swift in Sources */ = {isa = PBXBuildFile; fileRef = A31476EF2F85741500318978 /* OutputCleaner.swift */; }; + A31476FC2F85741500318978 /* ToolCallParser.swift in Sources */ = {isa = PBXBuildFile; fileRef = A31476F32F85741500318978 /* ToolCallParser.swift */; }; AA1000000000000000000001 /* LLMEngine.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000001 /* LLMEngine.swift */; }; AA1000000000000000000002 /* MLXLocalLLMService.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000002 /* MLXLocalLLMService.swift */; }; AA1000000000000000000003 /* Gemma4Config.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000003 /* Gemma4Config.swift */; }; AA1000000000000000000004 /* Gemma4LanguageModel.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000004 /* Gemma4LanguageModel.swift */; }; AA1000000000000000000005 /* Gemma4Model.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000005 /* Gemma4Model.swift */; }; + AA1000000000000000000100 /* BundledModel.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000100 /* BundledModel.swift */; }; + AA1000000000000000000101 /* RuntimeBudgets.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000101 /* RuntimeBudgets.swift */; }; + AA1000000000000000000103 /* ModelInstallState.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000103 /* ModelInstallState.swift */; }; + AA1000000000000000000104 /* ModelPaths.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000104 /* ModelPaths.swift */; }; + AA1000000000000000000105 /* MemoryStats.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000105 /* MemoryStats.swift */; }; + AA1000000000000000000106 /* ModelDownloader.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000106 /* ModelDownloader.swift */; }; + AA1000000000000000000107 /* ModelInstaller.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000107 /* ModelInstaller.swift */; }; + AA1000000000000000000108 /* GPULifecycle.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000108 /* GPULifecycle.swift */; }; AA1000000000000000000006 /* Gemma4Registration.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000006 /* Gemma4Registration.swift */; }; AA1000000000000000000007 /* Gemma4RoPE.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000007 /* Gemma4RoPE.swift */; }; AA1000000000000000000008 /* Gemma4TextModel.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA2000000000000000000008 /* Gemma4TextModel.swift */; }; @@ -37,10 +54,21 @@ B8A84E4D0DDDE011B5929DA2 /* AgentEngine.swift in Sources */ = {isa = PBXBuildFile; fileRef = E18A47114A10132E25E220F8 /* AgentEngine.swift */; }; BD75E7E5620A8FF0A868E9F0 /* libPods-PhoneClaw.a in Frameworks */ = {isa = PBXBuildFile; fileRef = 4F4ACB3F4531FF61E0AADA07 /* libPods-PhoneClaw.a */; }; C1A2B3D4E5F607182930A1B2 /* SkillLoader.swift in Sources */ = {isa = PBXBuildFile; fileRef = C1A2B3D4E5F607182930A1B3 /* SkillLoader.swift */; }; - C2B3C4D5E6F718293041B2C3 /* ToolRegistry.swift in Sources */ = {isa = PBXBuildFile; fileRef = C2B3C4D5E6F718293041B2C4 /* ToolRegistry.swift */; }; + C1A2B3D4E5F607182930A1C2 /* SkillRegistry.swift in Sources */ = {isa = PBXBuildFile; fileRef = C1A2B3D4E5F607182930A1C3 /* SkillRegistry.swift */; }; + D3A1B2C3D4E5F60718290001 /* ToolRegistry.swift in Sources */ = {isa = PBXBuildFile; fileRef = D3A1B2C3D4E5F60718290101 /* ToolRegistry.swift */; }; + D3A1B2C3D4E5F60718290002 /* AppPermissions.swift in Sources */ = {isa = PBXBuildFile; fileRef = D3A1B2C3D4E5F60718290102 /* AppPermissions.swift */; }; + D3A1B2C3D4E5F60718290003 /* Stores.swift in Sources */ = {isa = PBXBuildFile; fileRef = D3A1B2C3D4E5F60718290103 /* Stores.swift */; }; + D3A1B2C3D4E5F60718290004 /* Helpers.swift in Sources */ = {isa = PBXBuildFile; fileRef = D3A1B2C3D4E5F60718290104 /* Helpers.swift */; }; + D3A1B2C3D4E5F60718290005 /* Clipboard.swift in Sources */ = {isa = PBXBuildFile; fileRef = D3A1B2C3D4E5F60718290105 /* Clipboard.swift */; }; + D3A1B2C3D4E5F60718290006 /* Device.swift in Sources */ = {isa = PBXBuildFile; fileRef = D3A1B2C3D4E5F60718290106 /* Device.swift */; }; + D3A1B2C3D4E5F60718290007 /* Text.swift in Sources */ = {isa = PBXBuildFile; fileRef = D3A1B2C3D4E5F60718290107 /* Text.swift */; }; + D3A1B2C3D4E5F60718290008 /* Calendar.swift in Sources */ = {isa = PBXBuildFile; fileRef = D3A1B2C3D4E5F60718290108 /* Calendar.swift */; }; + D3A1B2C3D4E5F60718290009 /* Reminders.swift in Sources */ = {isa = PBXBuildFile; fileRef = D3A1B2C3D4E5F60718290109 /* Reminders.swift */; }; + D3A1B2C3D4E5F60718290010 /* Contacts.swift in Sources */ = {isa = PBXBuildFile; fileRef = D3A1B2C3D4E5F60718290110 /* Contacts.swift */; }; C3C4D5E6F7081A2B3C52D3E4 /* ChatModels.swift in Sources */ = {isa = PBXBuildFile; fileRef = C3C4D5E6F7081A2B3C52D3E5 /* ChatModels.swift */; }; C4D5E6F70819A2B3C463E4F5 /* Theme.swift in Sources */ = {isa = PBXBuildFile; fileRef = C4D5E6F70819A2B3C463E4F6 /* Theme.swift */; }; ICONASSETS0000000000002 /* Assets.xcassets in Resources */ = {isa = PBXBuildFile; fileRef = ICONASSETS0000000000001 /* Assets.xcassets */; }; + B1A2C3D4E5F607182930B100 /* Library in Resources */ = {isa = PBXBuildFile; fileRef = B1A2C3D4E5F607182930B101 /* Library */; }; /* End PBXBuildFile section */ /* Begin PBXFileReference section */ @@ -54,11 +82,27 @@ 52F30C6348B834E5D157205F /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist; path = Info.plist; sourceTree = ""; }; 571908036E6DCED1D58C1E9E /* Skills.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Skills.swift; sourceTree = ""; }; A1AB7F250BEEFF938318AA75 /* Pods-PhoneClaw.debug.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-PhoneClaw.debug.xcconfig"; path = "Target Support Files/Pods-PhoneClaw/Pods-PhoneClaw.debug.xcconfig"; sourceTree = ""; }; + A31476E82F856FD700318978 /* ChatAttachments.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ChatAttachments.swift; sourceTree = ""; }; + A31476E92F856FD700318978 /* ChatMessage.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ChatMessage.swift; sourceTree = ""; }; + A31476EF2F85741500318978 /* OutputCleaner.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = OutputCleaner.swift; sourceTree = ""; }; + A31476F02F85741500318978 /* Planner.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Planner.swift; sourceTree = ""; }; + A31476F12F85741500318978 /* Router.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Router.swift; sourceTree = ""; }; + A31476F22F85741500318978 /* SessionStore.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SessionStore.swift; sourceTree = ""; }; + A31476F32F85741500318978 /* ToolCallParser.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ToolCallParser.swift; sourceTree = ""; }; + A31476F42F85741500318978 /* ToolChain.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ToolChain.swift; sourceTree = ""; }; AA2000000000000000000001 /* LLMEngine.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = LLMEngine.swift; sourceTree = ""; }; AA2000000000000000000002 /* MLXLocalLLMService.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MLXLocalLLMService.swift; sourceTree = ""; }; AA2000000000000000000003 /* Gemma4Config.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Gemma4Config.swift; sourceTree = ""; }; AA2000000000000000000004 /* Gemma4LanguageModel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Gemma4LanguageModel.swift; sourceTree = ""; }; AA2000000000000000000005 /* Gemma4Model.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Gemma4Model.swift; sourceTree = ""; }; + AA2000000000000000000100 /* BundledModel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = BundledModel.swift; sourceTree = ""; }; + AA2000000000000000000101 /* RuntimeBudgets.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RuntimeBudgets.swift; sourceTree = ""; }; + AA2000000000000000000103 /* ModelInstallState.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ModelInstallState.swift; sourceTree = ""; }; + AA2000000000000000000104 /* ModelPaths.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ModelPaths.swift; sourceTree = ""; }; + AA2000000000000000000105 /* MemoryStats.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MemoryStats.swift; sourceTree = ""; }; + AA2000000000000000000106 /* ModelDownloader.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ModelDownloader.swift; sourceTree = ""; }; + AA2000000000000000000107 /* ModelInstaller.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ModelInstaller.swift; sourceTree = ""; }; + AA2000000000000000000108 /* GPULifecycle.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = GPULifecycle.swift; sourceTree = ""; }; AA2000000000000000000006 /* Gemma4Registration.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Gemma4Registration.swift; sourceTree = ""; }; AA2000000000000000000007 /* Gemma4RoPE.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Gemma4RoPE.swift; sourceTree = ""; }; AA2000000000000000000008 /* Gemma4TextModel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Gemma4TextModel.swift; sourceTree = ""; }; @@ -68,7 +112,17 @@ AA2000000000000000000020 /* Gemma4AudioFeatureExtractor.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Gemma4AudioFeatureExtractor.swift; sourceTree = ""; }; AA2000000000000000000021 /* Gemma4AudioModel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Gemma4AudioModel.swift; sourceTree = ""; }; C1A2B3D4E5F607182930A1B3 /* SkillLoader.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SkillLoader.swift; sourceTree = ""; }; - C2B3C4D5E6F718293041B2C4 /* ToolRegistry.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ToolRegistry.swift; sourceTree = ""; }; + C1A2B3D4E5F607182930A1C3 /* SkillRegistry.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SkillRegistry.swift; sourceTree = ""; }; + D3A1B2C3D4E5F60718290101 /* ToolRegistry.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ToolRegistry.swift; sourceTree = ""; }; + D3A1B2C3D4E5F60718290102 /* AppPermissions.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AppPermissions.swift; sourceTree = ""; }; + D3A1B2C3D4E5F60718290103 /* Stores.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Stores.swift; sourceTree = ""; }; + D3A1B2C3D4E5F60718290104 /* Helpers.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Helpers.swift; sourceTree = ""; }; + D3A1B2C3D4E5F60718290105 /* Clipboard.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Clipboard.swift; sourceTree = ""; }; + D3A1B2C3D4E5F60718290106 /* Device.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Device.swift; sourceTree = ""; }; + D3A1B2C3D4E5F60718290107 /* Text.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Text.swift; sourceTree = ""; }; + D3A1B2C3D4E5F60718290108 /* Calendar.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Calendar.swift; sourceTree = ""; }; + D3A1B2C3D4E5F60718290109 /* Reminders.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Reminders.swift; sourceTree = ""; }; + D3A1B2C3D4E5F60718290110 /* Contacts.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Contacts.swift; sourceTree = ""; }; C3C4D5E6F7081A2B3C52D3E5 /* ChatModels.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ChatModels.swift; sourceTree = ""; }; C4D5E6F70819A2B3C463E4F6 /* Theme.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Theme.swift; sourceTree = ""; }; D269F8BFCAB68D82440F704B /* SkillsManagerView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SkillsManagerView.swift; sourceTree = ""; }; @@ -78,6 +132,7 @@ E62C3CEFC7AE63873D751E75 /* ConfigurationsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ConfigurationsView.swift; sourceTree = ""; }; F5ED928BF14864713A9A9D16 /* ContentView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ContentView.swift; sourceTree = ""; }; ICONASSETS0000000000001 /* Assets.xcassets */ = {isa = PBXFileReference; lastKnownFileType = folder.assetcatalog; path = Assets.xcassets; sourceTree = ""; }; + B1A2C3D4E5F607182930B101 /* Library */ = {isa = PBXFileReference; lastKnownFileType = folder; path = Library; sourceTree = ""; }; /* End PBXFileReference section */ /* Begin PBXFrameworksBuildPhase section */ @@ -131,31 +186,105 @@ 98EFD7CCA7E78715F07BABE3 /* Skills */ = { isa = PBXGroup; children = ( + B1A2C3D4E5F607182930B101 /* Library */, 571908036E6DCED1D58C1E9E /* Skills.swift */, C1A2B3D4E5F607182930A1B3 /* SkillLoader.swift */, - C2B3C4D5E6F718293041B2C4 /* ToolRegistry.swift */, + C1A2B3D4E5F607182930A1C3 /* SkillRegistry.swift */, ); path = Skills; sourceTree = ""; }; + D3A1B2C3D4E5F60718290200 /* Tools */ = { + isa = PBXGroup; + children = ( + D3A1B2C3D4E5F60718290101 /* ToolRegistry.swift */, + D3A1B2C3D4E5F60718290102 /* AppPermissions.swift */, + D3A1B2C3D4E5F60718290103 /* Stores.swift */, + D3A1B2C3D4E5F60718290104 /* Helpers.swift */, + D3A1B2C3D4E5F60718290300 /* Handlers */, + ); + path = Tools; + sourceTree = ""; + }; + D3A1B2C3D4E5F60718290300 /* Handlers */ = { + isa = PBXGroup; + children = ( + D3A1B2C3D4E5F60718290105 /* Clipboard.swift */, + D3A1B2C3D4E5F60718290106 /* Device.swift */, + D3A1B2C3D4E5F60718290107 /* Text.swift */, + D3A1B2C3D4E5F60718290108 /* Calendar.swift */, + D3A1B2C3D4E5F60718290109 /* Reminders.swift */, + D3A1B2C3D4E5F60718290110 /* Contacts.swift */, + ); + path = Handlers; + sourceTree = ""; + }; + A31476ED2F85741500318978 /* Engine */ = { + isa = PBXGroup; + children = ( + A31476EF2F85741500318978 /* OutputCleaner.swift */, + A31476F02F85741500318978 /* Planner.swift */, + A31476F12F85741500318978 /* Router.swift */, + A31476F22F85741500318978 /* SessionStore.swift */, + A31476F32F85741500318978 /* ToolCallParser.swift */, + A31476F42F85741500318978 /* ToolChain.swift */, + A31476E82F856FD700318978 /* ChatAttachments.swift */, + A31476E92F856FD700318978 /* ChatMessage.swift */, + ); + path = Engine; + sourceTree = ""; + }; AA3000000000000000000001 /* MLX */ = { isa = PBXGroup; children = ( - AA2000000000000000000003 /* Gemma4Config.swift */, - AA2000000000000000000004 /* Gemma4LanguageModel.swift */, - AA2000000000000000000020 /* Gemma4AudioFeatureExtractor.swift */, - AA2000000000000000000021 /* Gemma4AudioModel.swift */, - AA2000000000000000000005 /* Gemma4Model.swift */, - AA2000000000000000000006 /* Gemma4Registration.swift */, - AA2000000000000000000007 /* Gemma4RoPE.swift */, - AA2000000000000000000008 /* Gemma4TextModel.swift */, - AA2000000000000000000018 /* Gemma4VisionModel.swift */, - AA2000000000000000000019 /* Gemma4Processor.swift */, - AA2000000000000000000009 /* MLXTokenizersLoader.swift */, + AA3000000000000000000010 /* Core */, + AA3000000000000000000011 /* Installation */, + AA3000000000000000000012 /* Gemma4 */, + AA2000000000000000000002 /* MLXLocalLLMService.swift */, ); path = MLX; sourceTree = ""; }; + AA3000000000000000000010 /* Core */ = { + isa = PBXGroup; + children = ( + AA2000000000000000000100 /* BundledModel.swift */, + AA2000000000000000000101 /* RuntimeBudgets.swift */, + AA2000000000000000000105 /* MemoryStats.swift */, + AA2000000000000000000108 /* GPULifecycle.swift */, + AA2000000000000000000009 /* MLXTokenizersLoader.swift */, + ); + path = Core; + sourceTree = ""; + }; + AA3000000000000000000011 /* Installation */ = { + isa = PBXGroup; + children = ( + AA2000000000000000000104 /* ModelPaths.swift */, + AA2000000000000000000103 /* ModelInstallState.swift */, + AA2000000000000000000107 /* ModelInstaller.swift */, + AA2000000000000000000106 /* ModelDownloader.swift */, + ); + path = Installation; + sourceTree = ""; + }; + AA3000000000000000000012 /* Gemma4 */ = { + isa = PBXGroup; + children = ( + AA2000000000000000000003 /* Gemma4Config.swift */, + AA2000000000000000000005 /* Gemma4Model.swift */, + AA2000000000000000000004 /* Gemma4LanguageModel.swift */, + AA2000000000000000000008 /* Gemma4TextModel.swift */, + AA2000000000000000000018 /* Gemma4VisionModel.swift */, + AA2000000000000000000021 /* Gemma4AudioModel.swift */, + AA2000000000000000000020 /* Gemma4AudioFeatureExtractor.swift */, + AA2000000000000000000007 /* Gemma4RoPE.swift */, + AA2000000000000000000019 /* Gemma4Processor.swift */, + AA2000000000000000000006 /* Gemma4Registration.swift */, + ); + path = Gemma4; + sourceTree = ""; + }; AC5E0AA2711EA01E891DE3A7 /* Products */ = { isa = PBXGroup; children = ( @@ -169,7 +298,6 @@ children = ( AA2000000000000000000001 /* LLMEngine.swift */, AA3000000000000000000001 /* MLX */, - AA2000000000000000000002 /* MLXLocalLLMService.swift */, 2E7C06DA449D1395217A42D5 /* PromptBuilder.swift */, ); path = LLM; @@ -185,6 +313,7 @@ 6FA4C5AFA3123551D8F8F2F2 /* App */, AE5FF3067EFF33D507435528 /* LLM */, 98EFD7CCA7E78715F07BABE3 /* Skills */, + D3A1B2C3D4E5F60718290200 /* Tools */, EF4D07A808616D6F13771DBE /* UI */, ); name = PhoneClaw; @@ -193,6 +322,7 @@ E0660A5AE3C683B7B20600EE /* Agent */ = { isa = PBXGroup; children = ( + A31476ED2F85741500318978 /* Engine */, 2F4A6C8E0A1B2C3D4E5F6070 /* AudioCaptureService.swift */, E18A47114A10132E25E220F8 /* AgentEngine.swift */, ); @@ -287,7 +417,9 @@ isa = PBXResourcesBuildPhase; buildActionMask = 2147483647; files = ( + 6A7B8C9D0E1F203142536475 /* gemma-4-e4b-it-4bit in Resources */, ICONASSETS0000000000002 /* Assets.xcassets in Resources */, + B1A2C3D4E5F607182930B100 /* Library in Resources */, ); runOnlyForDeploymentPostprocessing = 0; }; @@ -332,7 +464,21 @@ AA1000000000000000000004 /* Gemma4LanguageModel.swift in Sources */, AA1000000000000000000020 /* Gemma4AudioFeatureExtractor.swift in Sources */, AA1000000000000000000021 /* Gemma4AudioModel.swift in Sources */, + A31476F62F85741500318978 /* Planner.swift in Sources */, + A31476F82F85741500318978 /* Router.swift in Sources */, + A31476F92F85741500318978 /* SessionStore.swift in Sources */, + A31476FA2F85741500318978 /* ToolChain.swift in Sources */, + A31476FB2F85741500318978 /* OutputCleaner.swift in Sources */, + A31476FC2F85741500318978 /* ToolCallParser.swift in Sources */, AA1000000000000000000005 /* Gemma4Model.swift in Sources */, + AA1000000000000000000100 /* BundledModel.swift in Sources */, + AA1000000000000000000101 /* RuntimeBudgets.swift in Sources */, + AA1000000000000000000103 /* ModelInstallState.swift in Sources */, + AA1000000000000000000104 /* ModelPaths.swift in Sources */, + AA1000000000000000000105 /* MemoryStats.swift in Sources */, + AA1000000000000000000106 /* ModelDownloader.swift in Sources */, + AA1000000000000000000107 /* ModelInstaller.swift in Sources */, + AA1000000000000000000108 /* GPULifecycle.swift in Sources */, AA1000000000000000000006 /* Gemma4Registration.swift in Sources */, AA1000000000000000000018 /* Gemma4VisionModel.swift in Sources */, AA1000000000000000000019 /* Gemma4Processor.swift in Sources */, @@ -344,10 +490,22 @@ 45B7FEC6C4D055F44A87BFA9 /* PhoneClawApp.swift in Sources */, 5084E1CD2E831BC05F5E4374 /* PromptBuilder.swift in Sources */, C1A2B3D4E5F607182930A1B2 /* SkillLoader.swift in Sources */, + C1A2B3D4E5F607182930A1C2 /* SkillRegistry.swift in Sources */, 20B8C55E032BA07B3E640E80 /* Skills.swift in Sources */, B881A2E7F9483D5ED2AC206B /* SkillsManagerView.swift in Sources */, + A31476EA2F856FD700318978 /* ChatAttachments.swift in Sources */, + A31476EB2F856FD700318978 /* ChatMessage.swift in Sources */, C4D5E6F70819A2B3C463E4F5 /* Theme.swift in Sources */, - C2B3C4D5E6F718293041B2C3 /* ToolRegistry.swift in Sources */, + D3A1B2C3D4E5F60718290001 /* ToolRegistry.swift in Sources */, + D3A1B2C3D4E5F60718290002 /* AppPermissions.swift in Sources */, + D3A1B2C3D4E5F60718290003 /* Stores.swift in Sources */, + D3A1B2C3D4E5F60718290004 /* Helpers.swift in Sources */, + D3A1B2C3D4E5F60718290005 /* Clipboard.swift in Sources */, + D3A1B2C3D4E5F60718290006 /* Device.swift in Sources */, + D3A1B2C3D4E5F60718290007 /* Text.swift in Sources */, + D3A1B2C3D4E5F60718290008 /* Calendar.swift in Sources */, + D3A1B2C3D4E5F60718290009 /* Reminders.swift in Sources */, + D3A1B2C3D4E5F60718290010 /* Contacts.swift in Sources */, ); runOnlyForDeploymentPostprocessing = 0; }; @@ -482,6 +640,7 @@ ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; CODE_SIGN_ENTITLEMENTS = PhoneClaw.entitlements; CURRENT_PROJECT_VERSION = 1; + DEVELOPMENT_TEAM = ZP4QNJ35TH; INFOPLIST_FILE = Info.plist; LD_RUNPATH_SEARCH_PATHS = ( "$(inherited)", @@ -501,6 +660,7 @@ ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; CODE_SIGN_ENTITLEMENTS = PhoneClaw.entitlements; CURRENT_PROJECT_VERSION = 1; + DEVELOPMENT_TEAM = ZP4QNJ35TH; INFOPLIST_FILE = Info.plist; LD_RUNPATH_SEARCH_PATHS = ( "$(inherited)", diff --git a/PhoneClaw.xcodeproj/xcshareddata/xcschemes/PhoneClaw.xcscheme b/PhoneClaw.xcodeproj/xcshareddata/xcschemes/PhoneClaw.xcscheme new file mode 100644 index 0000000..88e527a --- /dev/null +++ b/PhoneClaw.xcodeproj/xcshareddata/xcschemes/PhoneClaw.xcscheme @@ -0,0 +1,79 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Skills/Library/calendar/SKILL.md b/Skills/Library/calendar/SKILL.md new file mode 100644 index 0000000..8765149 --- /dev/null +++ b/Skills/Library/calendar/SKILL.md @@ -0,0 +1,67 @@ +--- +name: Calendar +name-zh: 日历 +description: '创建新的日历事项 (会议/约会/日程)。' +version: "1.0.0" +icon: calendar +disabled: false +type: device + +triggers: + - 日历 + - 日程 + - 会议 + - 约会 + - 安排 + +allowed-tools: + - calendar-create-event + +examples: + - query: "帮我创建明天下午两点的产品评审会议" + scenario: "新建日历事项" +--- + +# 日历事项创建 + +严格遵循下面的参数规则, 不要自由发挥, 不要反问多余问题。 + +## 参数 + +**硬参** (必填, 缺失才允许追问): +- `start`: ISO 8601 开始时间, 例如 `2026-04-07T14:00:00` + +**软参** (用户没说就省略字段, 永远不追问): +- `title`: 事项标题 +- `end`: 结束时间 +- `location`: 地点 +- `notes`: 备注 + +## start 提取规则 + +用户话语里**只要有任何时间线索就视为 start 已提供**: +- 相对时间: "明天下午两点" / "今晚八点" / "一小时后" / "后天中午" +- 绝对时间: "5月3日 15:00" / "4月10日晚上" + +结合 system prompt 的**当前时间锚点**算出 ISO 8601 字符串。**禁止**对已经给了相对时间的用户反问 "哪一天"。 + +只有用户完全没给时间 (如"安排个会议") 才允许输出: +``` +{"_needs_clarification": "请告诉我具体时间"} +``` + +## title 处理 + +- 用户话语里有名词短语 (如"产品评审会议") → 直接当 title +- 只有动作 (如"安排个会") → title 字段省略, 工具会用默认 +- **永远禁止**输出 "请告诉我标题" / "标题是什么" 这类反问 + +## 行为 + +拿到 start 立刻输出工具调用 JSON, 不解释、不反问、不追问 end/location/notes。 + +## 调用格式 + + +{"name": "calendar-create-event", "arguments": {"title": "产品评审会议", "start": "2026-04-07T14:00:00"}} + diff --git a/Skills/Library/clipboard/SKILL.md b/Skills/Library/clipboard/SKILL.md new file mode 100644 index 0000000..8baba7b --- /dev/null +++ b/Skills/Library/clipboard/SKILL.md @@ -0,0 +1,46 @@ +--- +name: Clipboard +name-zh: 剪贴板 +description: '读写系统剪贴板内容。当用户需要读取、复制或操作剪贴板时使用。' +version: "1.0.0" +icon: doc.on.clipboard +disabled: false +type: device + +triggers: + - 剪贴板 + - 粘贴 + - 复制 + - clipboard + +allowed-tools: + - clipboard-read + - clipboard-write + +examples: + - query: "读取我的剪贴板内容" + scenario: "读取剪贴板" + - query: "把这段文字复制到剪贴板" + scenario: "写入剪贴板" +--- + +# 剪贴板操作 + +你负责帮助用户读写系统剪贴板。 + +## 可用工具 + +- **clipboard-read**: 读取剪贴板当前内容（无参数） +- **clipboard-write**: 将文本写入剪贴板（参数: text — 要复制的文本） + +## 执行流程 + +1. 用户要求读取 → 调用 `clipboard-read` +2. 用户要求复制/写入 → 调用 `clipboard-write`，传入 text 参数 +3. 根据工具返回结果，简洁回答用户 + +## 调用格式 + + +{"name": "工具名", "arguments": {}} + diff --git a/Skills/Library/contacts/SKILL.md b/Skills/Library/contacts/SKILL.md new file mode 100644 index 0000000..6107ed8 --- /dev/null +++ b/Skills/Library/contacts/SKILL.md @@ -0,0 +1,124 @@ +--- +name: Contacts +name-zh: 通讯录 +description: '查询、创建、更新或删除联系人。当用户要查电话、看联系方式、存号码、补充联系人信息或删除联系人时使用。' +version: "1.1.0" +icon: person.crop.circle +disabled: false +type: device + +triggers: + - 联系人 + - 通讯录 + - 查电话 + - 联系电话 + - 存号码 + - 联系方式 + - 删除联系人 + +allowed-tools: + - contacts-search + - contacts-upsert + - contacts-delete + +examples: + - query: "把王总电话 13812345678 添加到联系人" + scenario: "新建或更新联系人" + - query: "检查下联系人张晓霞的电话多少" + scenario: "查询联系人电话" + - query: "把王总从联系人中删除" + scenario: "删除联系人" +--- + +# 联系人查询与维护 + +你负责帮助用户查询、创建、更新或删除通讯录联系人。 + +## 可用工具 + +- **contacts-search**: 查询联系人 + - `query`: 关键词，可用于模糊搜索 + - `name`: 联系人姓名 + - `phone`: 手机号 + - `email`: 邮箱 + - `identifier`: 联系人标识 +- **contacts-upsert**: 创建或更新联系人 + - `name`: 必填，联系人姓名 + - `phone`: 可选，手机号；如果提供，会优先按手机号查重 + - `company`: 可选，公司 + - `email`: 可选，邮箱 + - `notes`: 可选，备注 +- **contacts-delete**: 删除联系人 + - `query`: 关键词，可用于模糊搜索 + - `name`: 联系人姓名 + - `phone`: 手机号 + - `email`: 邮箱 + - `identifier`: 联系人标识 + +## 执行流程 + +1. 如果用户在查询电话、邮箱、联系方式，优先调用 `contacts-search` +2. 如果用户在删除、移除、删掉联系人，调用 `contacts-delete` +3. 如果用户在保存、添加或更新联系人，调用 `contacts-upsert` +4. 查询或删除时优先提取 `name`，提取不到再用 `query` +5. 保存或更新时提取姓名、手机号、公司、邮箱、备注 +6. 如果缺少保存联系人所需的 `name`，先简短追问 +7. 工具成功后，直接用中文给出简洁结果 + +## 多轮澄清处理 + +### 找到多个匹配时 + +调用 `contacts-search` 或 `contacts-delete` 后，如果工具结果显示有多个候选（matches > 1），不要直接报错或乱选一个。按以下格式问用户： + +> 找到多个 [name]： +> (1) [phone1] · [extra info] +> (2) [phone2] · [extra info] +> +> 要操作哪一个？回复编号、电话号码后几位，或"全部"。 + +把这些候选信息**保留**在你的回答里，下一轮用户回应时你需要参考。 + +### 用户回答澄清后（关键） + +如果上一轮你刚问过用户"要操作哪一个"，**当前用户消息就是答案**。不要再问一次，按答案语义解析后**重新调用同一个工具**： + +| 用户说什么 | 含义 | 怎么调 | +|---|---|---| +| 完整电话 `15212345678` | 精确指定 | 用 `phone` 参数加完整号码 | +| 尾号 `5678` / "尾号 5678" | 模糊定位 | 用 `query` 参数加尾号 | +| 编号 `1` / `(1)` / "第一个" | 选候选列表第 N 个 | 取上一轮列出的第 N 个的电话作为 `phone` | +| "全部" / "都删" / "两个都" | 操作所有候选 | 对每个候选**依次**调用工具，每次用 `phone` 精确定位一个 | +| 名字 + 公司 / 备注 | 用区分性信息 | 加 `name` + `notes` 等参数 | + +调用例（用户回答 "152123458"）： + +{"name": "contacts-delete", "arguments": {"name": "张总", "phone": "152123458"}} + + +### 用户取消时 + +如果用户在多轮澄清过程中表达了**放弃意图**——例如说"算了"、"不删了"、"取消"、"停"、"nevermind"，或任何自然语言里表示不想继续的意思——**直接给一句简短确认**（如"好的，已取消"），**不要 emit 任何 tool_call**。 + +判断"是否在表达放弃"由你自己理解上下文，不要依赖任何固定关键词列表。模型有自然语言理解能力，请用它。 + +### 不要捏造执行结果 + +如果你**没有真正调用 tool**，**绝对不要**说"已经删除"、"已添加"、"已更新"。 +- 要么 emit 一个真实的 `` +- 要么如实告诉用户你需要更多信息或操作已取消 +- **不允许**只输出"已完成"这种文本而不调工具 + +## 调用格式 + + +{"name": "contacts-search", "arguments": {"name": "张晓霞"}} + + + +{"name": "contacts-upsert", "arguments": {"name": "王总", "phone": "13812345678", "company": "字节"}} + + + +{"name": "contacts-delete", "arguments": {"name": "王总"}} + diff --git a/Skills/Library/device/SKILL.md b/Skills/Library/device/SKILL.md new file mode 100644 index 0000000..edd7ca5 --- /dev/null +++ b/Skills/Library/device/SKILL.md @@ -0,0 +1,61 @@ +--- +name: Device +name-zh: 设备 +description: '查询当前设备的名称、机型、系统版本、内存、处理器数量等系统信息 (基于 iOS 公开 API)。' +version: "1.0.0" +icon: desktopcomputer +disabled: false +type: device + +triggers: + - 设备 + - 系统信息 + - 当前设备 + - 本机 + - 系统版本 + - 内存 + - 处理器 + +allowed-tools: + - device-info + - device-name + - device-model + - device-system-version + - device-memory + - device-processor-count + - device-identifier-for-vendor + +examples: + - query: "这台手机的设备信息是什么" + scenario: "整体汇总" + - query: "系统版本是多少" + scenario: "查单项" + - query: "内存多大" + scenario: "查单项" +--- + +# 设备信息查询 + +## 工具选择 + +- 用户问"整体配置 / 全部信息 / 这台手机什么样" → `device-info` +- 用户问单个属性 → 用最小的专用工具: + - 名字 → `device-name` + - 机型 → `device-model` + - 系统版本 → `device-system-version` + - 内存 → `device-memory` + - CPU 核数 → `device-processor-count` + - 厂商标识 → `device-identifier-for-vendor` + +## 行为约束 + +1. 必须用户**明确**问当前设备/本机/手机的属性才调用。"配置""信息"这种含糊词不触发。 +2. iOS 公开 API 只能返回通用机型 (如 `iPhone`)，**不要**自己映射成 "iPhone 17 Pro Max" 这种营销名。如果用户问"什么型号", 如实说明限制。 +3. 工具返回 JSON 后, 转成自然中文描述, 不要照搬原始字段名。 +4. 拿到结果直接答, 不要反问 "你指的是什么配置"。 + +## 调用格式 + + +{"name": "device-info", "arguments": {}} + diff --git a/Skills/Library/reminders/SKILL.md b/Skills/Library/reminders/SKILL.md new file mode 100644 index 0000000..61a05c9 --- /dev/null +++ b/Skills/Library/reminders/SKILL.md @@ -0,0 +1,47 @@ +--- +name: Reminders +name-zh: 提醒事项 +description: '创建新的提醒事项。当用户需要记得做某事、设置待办或提醒时使用。' +version: "1.0.0" +icon: bell +disabled: false +type: device + +triggers: + - 提醒 + - 待办 + - 记得 + - 提示 + +allowed-tools: + - reminders-create + +examples: + - query: "提醒我今晚八点发文件" + scenario: "新建提醒事项" +--- + +# 提醒事项创建 + +你负责帮助用户创建新的提醒事项。 + +## 可用工具 + +- **reminders-create**: 创建提醒事项 + - `title`: 必填，提醒标题 + - `due`: 可选，ISO 8601 提醒时间，例如 `2026-04-07T20:00:00` + - `notes`: 可选，备注 + +## 执行流程 + +1. 只有当用户明确要设置提醒或待办时才调用工具 +2. 由你提取标题、时间、备注 +3. 如果有时间，必须转换成 ISO 8601 字符串 +4. 如果缺少 `title`，先简短追问 +5. 工具成功后，直接告诉用户提醒已创建 + +## 调用格式 + + +{"name": "reminders-create", "arguments": {"title": "发文件", "due": "2026-04-07T20:00:00"}} + diff --git a/Skills/Library/text/SKILL.md b/Skills/Library/text/SKILL.md new file mode 100644 index 0000000..f568484 --- /dev/null +++ b/Skills/Library/text/SKILL.md @@ -0,0 +1,47 @@ +--- +name: Text +name-zh: 文本 +description: '文本处理工具：哈希计算、翻转等。当用户需要对文本进行处理或转换时使用。' +version: "1.0.0" +icon: textformat +disabled: false +type: device + +triggers: + - 哈希 + - hash + - 翻转 + - 反转 + - 文本处理 + +allowed-tools: + - calculate-hash + - text-reverse + +examples: + - query: "计算 Hello World 的哈希值" + scenario: "哈希计算" + - query: "把这段文字翻转过来" + scenario: "文本翻转" +--- + +# 文本处理 + +你负责帮助用户进行文本处理操作。 + +## 可用工具 + +- **calculate-hash**: 计算文本的哈希值（参数: text — 要计算哈希的文本） +- **text-reverse**: 翻转文本（参数: text — 要翻转的文本） + +## 执行流程 + +1. 判断用户需要哪种文本操作 +2. 调用对应工具，传入 text 参数 +3. 返回处理结果 + +## 调用格式 + + +{"name": "工具名", "arguments": {"text": "要处理的文本"}} + diff --git a/Skills/Library/translate/SKILL.md b/Skills/Library/translate/SKILL.md new file mode 100644 index 0000000..ae269db --- /dev/null +++ b/Skills/Library/translate/SKILL.md @@ -0,0 +1,60 @@ +--- +name: Translate +name-zh: 翻译 +description: '专业翻译助手, 支持任意语种互译。' +version: "1.1.0" +icon: character.bubble +disabled: false +type: content + +triggers: + - 翻译 + - 译成 + - 翻成 + - translate + - translation + - 中译英 + - 英译中 + - 译为 + +allowed-tools: [] + +examples: + - query: "把下面这段话翻译成英文: 今天天气真好" + scenario: "中译英" + - query: "translate to Chinese: The early bird catches the worm" + scenario: "英译中" + - query: "把刚才那段译成日语" + scenario: "指代历史" +--- + +# 专业翻译 + +直接输出译文, 不加前缀、不解释过程、不列备选。 + +## 识别源文本 + +1. 用户消息里有引号或冒号后的内容 → 取该段 +2. 用户用了"这段 / 刚才那段 / 上面 / 前面"等指代词 → 取对话历史中最近一条 assistant 消息的正文 (跳过 ⚠️ 警告行和"好的"之类单句确认) +3. 以上都没有 → 才允许追问 "请提供需要翻译的文本" + +## 识别目标语言 + +- 用户明示 (如 "译成英文") → 按用户要求 +- 用户问 "是什么意思" → 默认译成中文 +- 源文本是中文且未指定 → 默认译成英文 + +## 翻译准则 + +**信达雅**: 忠实原意 + 符合目标语言语法习惯 + 匹配原文语域 (正式/口语/文学/技术)。 + +具体: +- 成语用等效表达, 不字面直译 ("画蛇添足" → "gild the lily") +- 中译英要补全省略的主语, 英译中按中文习惯省略 +- 日韩 → 中/英 调整 SOV→SVO 语序 +- 标点用目标语言体系 (中文全角, 英文半角) +- 专有名词 (人名/地名/品牌) 保留或用公认译法 + +## 输出 + +只输出译文文本本身。源文本歧义且需要上下文澄清时, 先给最可能译文, 下一段简短补一句"另一种可能是 ..."。 diff --git a/Skills/SkillLoader.swift b/Skills/SkillLoader.swift index 97df08f..8aac22b 100644 --- a/Skills/SkillLoader.swift +++ b/Skills/SkillLoader.swift @@ -1,12 +1,19 @@ import Foundation import Yams -// MARK: - SKILL.md 解析器 + 加载器 +// MARK: - SKILL.md 解析 + Skill Registry // -// 参考 Vera 项目的 skill_loader.py，实现 Swift 版。 -// 渐进式加载： -// 1. 启动时：只加载 YAML frontmatter（元数据） -// 2. load_skill 时：加载完整 body（指令体） +// 拆分 (对称 ToolRegistry): +// - SkillLoader : 无状态, 纯解析 + bundle 读取 (static helpers) +// - SkillRegistry: 有状态, 持有 [id: SkillDefinition], 提供注册/查询/启用/保存 +// +// 注册制架构: +// - Skills 在 app 启动时显式 register, 不再扫描文件系统 +// - Built-in skills 从 Bundle.main/Library//SKILL.md 读取 +// - 用户编辑 -> saveSkill 写 Application Support/skills//SKILL.md +// 作为 override; 下次启动注册时优先使用 override (读不到才回 bundle) +// - 未来 download / import / in-app 创建 复用同一个 register API +// (给出 SkillDefinition 即可, 内容来源无感知) // MARK: - 数据模型 @@ -15,6 +22,19 @@ struct SkillExample { let scenario: String } +/// Skill 类别。决定 system prompt 给模型的调用规则。 +/// +/// - device : 访问 iPhone 硬件或系统数据 (clipboard/calendar/contacts/...). +/// 只有当用户明确要求执行设备操作时才 load_skill。 +/// 含糊词不触发, 闲聊不触发。 +/// - content: 对文本做变换 (translate/summarize/rewrite/...). +/// 只要用户意图是该类操作就立即 load_skill, 包括"翻译这段" +/// 这种带指代词的请求 (skill body 会指导从历史定位源文本)。 +enum SkillType: String, Sendable { + case device + case content +} + struct SkillMetadata { let id: String // 目录名 "clipboard" let name: String // 默认英文名 / 回退显示名 @@ -23,6 +43,7 @@ struct SkillMetadata { let version: String let icon: String let disabled: Bool + let type: SkillType // 类别 (frontmatter `type:` 必填, 缺省视为 .device) let triggers: [String] let allowedTools: [String] let examples: [SkillExample] @@ -39,9 +60,9 @@ struct SkillMetadata { struct SkillDefinition: Identifiable { let id: String - let filePath: URL + let filePath: URL // 内容来源 URL (bundle 或 override) let metadata: SkillMetadata - var body: String? // Markdown body（懒加载） + var body: String? // Markdown body (register 时已解析填充) var isEnabled: Bool /// 完整的 SKILL.md 原始内容 @@ -50,140 +71,68 @@ struct SkillDefinition: Identifiable { } } -// MARK: - Skill Loader +// MARK: - Skill Loader (无状态: 解析 + bundle 读取) +// +// 有状态的注册表见 SkillRegistry.swift。 -class SkillLoader { - private static let skillAliases: [String: String] = [ - "contacts_delete": "contacts", - "contacts-delete": "contacts" - ] +enum SkillLoader { - let skillsDirectory: URL - private var cache: [String: SkillDefinition] = [:] - - init() { - let appSupport = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first! - self.skillsDirectory = appSupport.appendingPathComponent("PhoneClaw/skills", isDirectory: true) - ensureDefaultSkills() - } - - // MARK: - 公开接口 - - /// 发现并加载所有 Skill 的元数据 - func discoverSkills() -> [SkillDefinition] { - cache.removeAll() - guard let items = try? FileManager.default.contentsOfDirectory( - at: skillsDirectory, - includingPropertiesForKeys: [.isDirectoryKey], - options: .skipsHiddenFiles - ) else { return [] } - - var results: [SkillDefinition] = [] - for item in items.sorted(by: { $0.lastPathComponent < $1.lastPathComponent }) { - var isDir: ObjCBool = false - FileManager.default.fileExists(atPath: item.path, isDirectory: &isDir) - guard isDir.boolValue else { continue } - - let skillFile = item.appendingPathComponent("SKILL.md") - guard FileManager.default.fileExists(atPath: skillFile.path) else { continue } - - let skillId = item.lastPathComponent - if Self.skillAliases[skillId] != nil { - continue - } - if let def = loadDefinition(skillId: skillId, file: skillFile) { - cache[skillId] = def - results.append(def) - } + /// 从 Bundle.main/Library//SKILL.md 读取并解析为 SkillDefinition + static func loadFromBundle(id: String) -> SkillDefinition? { + guard let bundleURL = Bundle.main.url( + forResource: "SKILL", + withExtension: "md", + subdirectory: "Library/\(id)" + ) else { + print("[SkillLoader] skill '\(id)' 未找到: Bundle.main/Library/\(id)/SKILL.md") + return nil } - return results - } - - /// 完整加载 Skill（包括 body）— load_skill 时调用 - func loadBody(skillId: String) -> String? { - let resolvedSkillId = canonicalSkillId(for: skillId) - if let cached = cache[resolvedSkillId], cached.body != nil { - return cached.body + guard let content = try? String(contentsOf: bundleURL, encoding: .utf8) else { + print("[SkillLoader] skill '\(id)' 读取失败") + return nil } - let skillFile = skillsDirectory - .appendingPathComponent(resolvedSkillId, isDirectory: true) - .appendingPathComponent("SKILL.md") - - guard let content = try? String(contentsOf: skillFile, encoding: .utf8) else { return nil } - let body = parseBody(content) - cache[resolvedSkillId]?.body = body - return body + return parseDefinition(id: id, content: content, filePath: bundleURL) } - /// 保存 SKILL.md（编辑后写回） - func saveSkill(skillId: String, content: String) throws { - let skillFile = skillsDirectory - .appendingPathComponent(skillId, isDirectory: true) - .appendingPathComponent("SKILL.md") - try content.write(to: skillFile, atomically: true, encoding: .utf8) - // 清缓存，下次重新解析 - cache.removeValue(forKey: skillId) + /// 从任意 URL 读取并解析 + static func loadFromFile(id: String, url: URL) -> SkillDefinition? { + guard let content = try? String(contentsOf: url, encoding: .utf8) else { return nil } + return parseDefinition(id: id, content: content, filePath: url) } - /// 重新加载所有（热更新入口） - func reloadAll() -> [SkillDefinition] { - return discoverSkills() - } - - /// 根据工具名反查 Skill ID - func findSkillId(forTool toolName: String) -> String? { - for (id, def) in cache { - if def.metadata.allowedTools.contains(toolName) { - return id - } - } - return nil - } - - /// 获取缓存的 SkillDefinition - func getDefinition(_ skillId: String) -> SkillDefinition? { - cache[canonicalSkillId(for: skillId)] - } - - /// 更新启用状态 - func setEnabled(_ skillId: String, enabled: Bool) { - cache[canonicalSkillId(for: skillId)]?.isEnabled = enabled - } - - func canonicalSkillId(for skillId: String) -> String { - Self.skillAliases[skillId] ?? skillId - } - - // MARK: - 解析 - - private func loadDefinition(skillId: String, file: URL) -> SkillDefinition? { - guard let content = try? String(contentsOf: file, encoding: .utf8) else { return nil } + /// 解析 SKILL.md 文本为 SkillDefinition + static func parseDefinition(id: String, content: String, filePath: URL) -> SkillDefinition? { guard let frontmatter = parseFrontmatter(content) else { return nil } + let typeRaw = (frontmatter["type"] as? String)?.lowercased() ?? "device" + let type = SkillType(rawValue: typeRaw) ?? .device + let metadata = SkillMetadata( - id: skillId, - name: frontmatter["name"] as? String ?? skillId, + id: id, + name: frontmatter["name"] as? String ?? id, localizedNameZh: frontmatter["name-zh"] as? String, description: frontmatter["description"] as? String ?? "", version: frontmatter["version"] as? String ?? "1.0.0", icon: frontmatter["icon"] as? String ?? "wrench", disabled: frontmatter["disabled"] as? Bool ?? false, + type: type, triggers: frontmatter["triggers"] as? [String] ?? [], allowedTools: frontmatter["allowed-tools"] as? [String] ?? [], examples: parseExamples(frontmatter["examples"]) ) return SkillDefinition( - id: skillId, - filePath: file, + id: id, + filePath: filePath, metadata: metadata, - body: nil, // 懒加载 + body: parseBody(content), isEnabled: !metadata.disabled ) } - private func parseFrontmatter(_ content: String) -> [String: Any]? { - // 匹配 --- ... --- 包裹的 YAML + // MARK: - 解析 helpers + + private static func parseFrontmatter(_ content: String) -> [String: Any]? { guard let regex = try? NSRegularExpression( pattern: "^---\\s*\\n(.*?)\\n---\\s*\\n", options: .dotMatchesLineSeparators @@ -194,12 +143,11 @@ class SkillLoader { let yamlRange = Range(match.range(at: 1), in: content) else { return nil } let yamlString = String(content[yamlRange]) - // 用 Yams 解析 guard let parsed = try? Yams.load(yaml: yamlString) as? [String: Any] else { return nil } return parsed } - private func parseBody(_ content: String) -> String { + private static func parseBody(_ content: String) -> String { guard let regex = try? NSRegularExpression( pattern: "^---\\s*\\n.*?\\n---\\s*\\n(.*)$", options: .dotMatchesLineSeparators @@ -213,7 +161,7 @@ class SkillLoader { return content.trimmingCharacters(in: .whitespacesAndNewlines) } - private func parseExamples(_ raw: Any?) -> [SkillExample] { + private static func parseExamples(_ raw: Any?) -> [SkillExample] { guard let list = raw as? [[String: Any]] else { return [] } return list.compactMap { dict in guard let query = dict["query"] as? String, @@ -221,376 +169,5 @@ class SkillLoader { return SkillExample(query: query, scenario: scenario) } } - - // MARK: - 首次启动：写入默认 Skill - - private func ensureDefaultSkills() { - let fm = FileManager.default - if !fm.fileExists(atPath: skillsDirectory.path) { - try? fm.createDirectory(at: skillsDirectory, withIntermediateDirectories: true) - } - - for (dirName, content) in Self.defaultSkills { - let dir = skillsDirectory.appendingPathComponent(dirName, isDirectory: true) - let file = dir.appendingPathComponent("SKILL.md") - let normalized = content.hasSuffix("\n") ? content : content + "\n" - let current = try? String(contentsOf: file, encoding: .utf8) - if current == normalized { continue } - try? fm.createDirectory(at: dir, withIntermediateDirectories: true) - try? normalized.write(to: file, atomically: true, encoding: .utf8) - } - } - - // MARK: - 内置默认 SKILL.md - - static let defaultSkills: [(String, String)] = [ - ("clipboard", """ - --- - name: Clipboard - name-zh: 剪贴板 - description: '读写系统剪贴板内容。当用户需要读取、复制或操作剪贴板时使用。' - version: "1.0.0" - icon: doc.on.clipboard - disabled: false - - triggers: - - 剪贴板 - - 粘贴 - - 复制 - - clipboard - - allowed-tools: - - clipboard-read - - clipboard-write - - examples: - - query: "读取我的剪贴板内容" - scenario: "读取剪贴板" - - query: "把这段文字复制到剪贴板" - scenario: "写入剪贴板" - --- - - # 剪贴板操作 - - 你负责帮助用户读写系统剪贴板。 - - ## 可用工具 - - - **clipboard-read**: 读取剪贴板当前内容（无参数） - - **clipboard-write**: 将文本写入剪贴板（参数: text — 要复制的文本） - - ## 执行流程 - - 1. 用户要求读取 → 调用 `clipboard-read` - 2. 用户要求复制/写入 → 调用 `clipboard-write`，传入 text 参数 - 3. 根据工具返回结果，简洁回答用户 - - ## 调用格式 - - - {"name": "工具名", "arguments": {}} - - """), - - ("device", """ - --- - name: Device - name-zh: 设备 - description: '使用 iOS 官方公开 API 查询当前设备名称、设备类型、系统版本、内存和处理器数量。' - version: "1.0.0" - icon: desktopcomputer - disabled: false - - triggers: - - 设备 - - 系统信息 - - 当前设备 - - 本机 - - allowed-tools: - - device-info - - device-name - - device-model - - device-system-version - - device-memory - - device-processor-count - - device-identifier-for-vendor - - examples: - - query: "这台手机的设备信息是什么" - scenario: "查看官方设备信息汇总" - - query: "系统版本是多少" - scenario: "查看系统版本" - - query: "这台设备叫什么名字" - scenario: "查看设备名称" - - query: "内存多大" - scenario: "查看物理内存" - - query: "处理器核心数是多少" - scenario: "查看处理器数量" - --- - - # 设备信息查询 - - 你负责帮助用户查看当前这台设备的系统与硬件基础信息。 - - ## 可用工具 - - - **device-info**: 汇总查询当前设备名称、设备类型、系统版本、物理内存、处理器数量 - - **device-name**: 查询当前设备名称 - - **device-model**: 查询当前设备类型（官方 `UIDevice.model` / `localizedModel`） - - **device-system-version**: 查询系统名称和系统版本 - - **device-memory**: 查询物理内存大小 - - **device-processor-count**: 查询处理器核心数 - - **device-identifier-for-vendor**: 查询当前 App 的 `identifierForVendor` - - ## 执行流程 - - 1. 只有当用户明确询问当前设备、本机、手机、系统版本、内存、处理器等信息时，才调用这些工具 - 2. 如果用户只是泛泛提到“配置”或“信息”，但没有明确在问当前设备，不要调用这个工具 - 3. 能用单个专用工具回答时，优先使用最小 sufficient 的那个工具；只有用户想要整体参数时，才调用 `device-info` - 4. 工具返回后，直接把 JSON 转成用户友好的中文描述 - 5. 这些工具只使用 iOS 官方公开 API，不要自行映射成具体营销机型名 - 6. 如果用户问“手机型号”，要如实说明官方 API 只能返回通用设备类型，例如 `iPhone` - 7. 如果工具结果已经足够，直接回答，不要反问用户“你指的是什么配置” - - ## 调用格式 - - - {"name": "device-info", "arguments": {}} - - """), - - ("text", """ - --- - name: Text - name-zh: 文本 - description: '文本处理工具：哈希计算、翻转等。当用户需要对文本进行处理或转换时使用。' - version: "1.0.0" - icon: textformat - disabled: false - - triggers: - - 哈希 - - hash - - 翻转 - - 反转 - - 文本处理 - - allowed-tools: - - calculate-hash - - text-reverse - - examples: - - query: "计算 Hello World 的哈希值" - scenario: "哈希计算" - - query: "把这段文字翻转过来" - scenario: "文本翻转" - --- - - # 文本处理 - - 你负责帮助用户进行文本处理操作。 - - ## 可用工具 - - - **calculate-hash**: 计算文本的哈希值（参数: text — 要计算哈希的文本） - - **text-reverse**: 翻转文本（参数: text — 要翻转的文本） - - ## 执行流程 - - 1. 判断用户需要哪种文本操作 - 2. 调用对应工具，传入 text 参数 - 3. 返回处理结果 - - ## 调用格式 - - - {"name": "工具名", "arguments": {"text": "要处理的文本"}} - - """), - - ("calendar", """ - --- - name: Calendar - name-zh: 日历 - description: '创建新的日历事项。当用户需要安排日程、会议、约会或写入日历时使用。' - version: "1.0.0" - icon: calendar - disabled: false - - triggers: - - 日历 - - 日程 - - 会议 - - 约会 - - 安排 - - allowed-tools: - - calendar-create-event - - examples: - - query: "帮我创建一个明天下午两点的会议" - scenario: "新建日历事项" - --- - - # 日历事项创建 - - 你负责帮助用户创建新的日历事项。 - - ## 可用工具 - - - **calendar-create-event**: 创建日历事项 - - `title`: 必填，事项标题 - - `start`: 必填，ISO 8601 开始时间，例如 `2026-04-07T14:00:00` - - `end`: 可选，ISO 8601 结束时间；不传时默认开始后一小时 - - `location`: 可选，地点 - - `notes`: 可选，备注 - - ## 执行流程 - - 1. 只有当用户明确要新建/安排日历事项时才调用工具 - 2. 由你从用户话语中提取参数，工具层不做自然语言解析 - 3. 传给工具前，必须把时间整理成 ISO 8601 字符串 - 4. 如果缺少 `title` 或 `start`，先简短追问，不要猜测 - 5. 工具成功后，直接告诉用户已创建什么事项和时间 - - ## 调用格式 - - - {"name": "calendar-create-event", "arguments": {"title": "会议", "start": "2026-04-07T14:00:00"}} - - """), - - ("reminders", """ - --- - name: Reminders - name-zh: 提醒事项 - description: '创建新的提醒事项。当用户需要记得做某事、设置待办或提醒时使用。' - version: "1.0.0" - icon: bell - disabled: false - - triggers: - - 提醒 - - 待办 - - 记得 - - 提示 - - allowed-tools: - - reminders-create - - examples: - - query: "提醒我今晚八点发文件" - scenario: "新建提醒事项" - --- - - # 提醒事项创建 - - 你负责帮助用户创建新的提醒事项。 - - ## 可用工具 - - - **reminders-create**: 创建提醒事项 - - `title`: 必填，提醒标题 - - `due`: 可选，ISO 8601 提醒时间，例如 `2026-04-07T20:00:00` - - `notes`: 可选，备注 - - ## 执行流程 - - 1. 只有当用户明确要设置提醒或待办时才调用工具 - 2. 由你提取标题、时间、备注 - 3. 如果有时间，必须转换成 ISO 8601 字符串 - 4. 如果缺少 `title`，先简短追问 - 5. 工具成功后，直接告诉用户提醒已创建 - - ## 调用格式 - - - {"name": "reminders-create", "arguments": {"title": "发文件", "due": "2026-04-07T20:00:00"}} - - """), - - ("contacts", """ - --- - name: Contacts - name-zh: 通讯录 - description: '查询、创建、更新或删除联系人。当用户要查电话、看联系方式、存号码、补充联系人信息或删除联系人时使用。' - version: "1.1.0" - icon: person.crop.circle - disabled: false - - triggers: - - 联系人 - - 通讯录 - - 查电话 - - 联系电话 - - 存号码 - - 联系方式 - - 删除联系人 - - allowed-tools: - - contacts-search - - contacts-upsert - - contacts-delete - - examples: - - query: "把王总电话 13812345678 添加到联系人" - scenario: "新建或更新联系人" - - query: "检查下联系人张晓霞的电话多少" - scenario: "查询联系人电话" - - query: "把王总从联系人中删除" - scenario: "删除联系人" - --- - - # 联系人查询与维护 - - 你负责帮助用户查询、创建、更新或删除通讯录联系人。 - - ## 可用工具 - - - **contacts-search**: 查询联系人 - - `query`: 关键词，可用于模糊搜索 - - `name`: 联系人姓名 - - `phone`: 手机号 - - `email`: 邮箱 - - `identifier`: 联系人标识 - - **contacts-upsert**: 创建或更新联系人 - - `name`: 必填，联系人姓名 - - `phone`: 可选，手机号；如果提供，会优先按手机号查重 - - `company`: 可选，公司 - - `email`: 可选，邮箱 - - `notes`: 可选，备注 - - **contacts-delete**: 删除联系人 - - `query`: 关键词，可用于模糊搜索 - - `name`: 联系人姓名 - - `phone`: 手机号 - - `email`: 邮箱 - - `identifier`: 联系人标识 - - ## 执行流程 - - 1. 如果用户在查询电话、邮箱、联系方式，优先调用 `contacts-search` - 2. 如果用户在删除、移除、删掉联系人，调用 `contacts-delete` - 3. 如果用户在保存、添加或更新联系人，调用 `contacts-upsert` - 4. 查询或删除时优先提取 `name`，提取不到再用 `query` - 5. 保存或更新时提取姓名、手机号、公司、邮箱、备注 - 6. 如果缺少保存联系人所需的 `name`，先简短追问 - 7. 删除时如果匹配到多个联系人，不要猜测，应提示用户说得更具体 - 8. 工具成功后，直接用中文给出简洁结果 - - ## 调用格式 - - - {"name": "contacts-search", "arguments": {"name": "张晓霞"}} - - - - {"name": "contacts-upsert", "arguments": {"name": "王总", "phone": "13812345678", "company": "字节"}} - - - - {"name": "contacts-delete", "arguments": {"name": "王总"}} - - """), - ] } + diff --git a/Skills/SkillRegistry.swift b/Skills/SkillRegistry.swift new file mode 100644 index 0000000..79943d0 --- /dev/null +++ b/Skills/SkillRegistry.swift @@ -0,0 +1,161 @@ +import Foundation + +// MARK: - Skill Registry (有状态: 注册/查询/启用/保存) +// +// 对称 ToolRegistry。持有 [id: SkillDefinition], 提供注册制 API。 +// 解析和 bundle 读取由无状态的 SkillLoader (SkillLoader.swift) 提供。 + +class SkillRegistry { + private static let skillAliases: [String: String] = [ + "contacts_delete": "contacts", + "contacts-delete": "contacts" + ] + + /// 用户编辑的 override 目录 (runtime 可写) + let overridesDirectory: URL + + /// Registry: skill id -> SkillDefinition + private var skills: [String: SkillDefinition] = [:] + + init() { + let appSupport = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first! + self.overridesDirectory = appSupport.appendingPathComponent("PhoneClaw/skills", isDirectory: true) + try? FileManager.default.createDirectory(at: overridesDirectory, withIntermediateDirectories: true) + registerBuiltInSkills() + } + + // MARK: - 公开 API + + /// 返回所有已注册的 skill。 + /// 名字沿用 discoverSkills() 以保持向后兼容, 实际是 "all registered"。 + func discoverSkills() -> [SkillDefinition] { + Array(skills.values).sorted { $0.id < $1.id } + } + + /// 返回 skill body (register 时已解析, 直接从 cache 返回) + func loadBody(skillId: String) -> String? { + let id = canonicalSkillId(for: skillId) + return skills[id]?.body + } + + /// 用户保存 SKILL.md。写到 override 目录, 重新 parse 并更新 registry。 + func saveSkill(skillId: String, content: String) throws { + let dir = overridesDirectory.appendingPathComponent(skillId, isDirectory: true) + try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + let file = dir.appendingPathComponent("SKILL.md") + try content.write(to: file, atomically: true, encoding: .utf8) + + // 重新 parse 并更新 registry (保持原 isEnabled) + if var def = SkillLoader.parseDefinition(id: skillId, content: content, filePath: file) { + if let existing = skills[skillId] { + def.isEnabled = existing.isEnabled + } + skills[skillId] = def + } + } + + /// 重载所有 (清空 registry 并重新注册) + func reloadAll() -> [SkillDefinition] { + skills.removeAll() + registerBuiltInSkills() + return discoverSkills() + } + + /// 根据工具名反查 skill id + func findSkillId(forTool toolName: String) -> String? { + for (id, def) in skills where def.metadata.allowedTools.contains(toolName) { + return id + } + return nil + } + + /// 获取注册的 SkillDefinition + func getDefinition(_ skillId: String) -> SkillDefinition? { + skills[canonicalSkillId(for: skillId)] + } + + /// 更新启用状态 + func setEnabled(_ skillId: String, enabled: Bool) { + skills[canonicalSkillId(for: skillId)]?.isEnabled = enabled + } + + /// 别名归一化 (例如 "contacts_delete" -> "contacts") + func canonicalSkillId(for skillId: String) -> String { + Self.skillAliases[skillId] ?? skillId + } + + // MARK: - 注册 API (core) + + /// 核心注册 API。传入一个 SkillDefinition 就注册进 registry。 + /// + /// 来源无感知: bundle / 下载 / import / in-app 创建 都走同一个入口。 + /// 未来支持运行时动态添加 skill 时, caller 构造 SkillDefinition 后直接调用。 + func register(_ definition: SkillDefinition) { + skills[definition.id] = definition + } + + // MARK: - Built-in 注册 (从 bundle) + + /// 注册一个内置 skill: 优先用户 override, 否则从 bundle 加载。 + private func registerBuiltIn(id: String) { + // 1. 优先 override + let overrideFile = overridesDirectory + .appendingPathComponent(id, isDirectory: true) + .appendingPathComponent("SKILL.md") + + if FileManager.default.fileExists(atPath: overrideFile.path), + let def = SkillLoader.loadFromFile(id: id, url: overrideFile) { + register(def) + return + } + + // 2. 从 bundle 读取 + if let def = SkillLoader.loadFromBundle(id: id) { + register(def) + } + } + + /// Built-in skill 注册列表。对称 ToolRegistry.registerDefaultTools 的模式。 + /// + /// 加新内置 skill 的流程: + /// 1. 在 `Skills/Library//` 创建 SKILL.md + /// 2. 在 Xcode 里把 Library/ 更新后的文件加到 PhoneClaw target (bundle) + /// 3. 在下面加一行 `registerBuiltIn(id: "")` + private func registerBuiltInSkills() { + registerBuiltIn(id: "clipboard") + registerBuiltIn(id: "device") + registerBuiltIn(id: "text") + registerBuiltIn(id: "calendar") + registerBuiltIn(id: "reminders") + registerBuiltIn(id: "contacts") + registerBuiltIn(id: "translate") + + validateRegisteredSkills() + } + + // MARK: - Load-time validation + // + // 校验所有 SKILL.md 的 `allowed-tools` 字段引用的 tool 名都真实存在于 + // ToolRegistry。SKILL.md 是手工维护的 frontmatter, 没有编译时检查; + // 这里在启动时一次性 cross-check, 写错的 tool 名会立刻在控制台暴露, + // 而不是等运行时模型 emit tool_call 然后被 cage 静默丢弃。 + // + // 这是 ToolRegistry "白名单"角色的另一个体现: 不只在运行时丢弃非法 + // 调用, 也在加载时拒绝非法引用。 + + private func validateRegisteredSkills() { + let registry = ToolRegistry.shared + var hadError = false + for (id, def) in skills { + for toolName in def.metadata.allowedTools { + if registry.find(name: toolName) == nil { + print("[SkillRegistry] ⚠️ skill '\(id)' 的 allowed-tools 引用了未注册的工具 '\(toolName)'") + hadError = true + } + } + } + if !hadError { + print("[SkillRegistry] ✓ 所有 \(skills.count) 个 skill 的 allowed-tools 校验通过") + } + } +} diff --git a/Skills/Skills.swift b/Skills/Skills.swift index 4d37e0e..0782f79 100644 --- a/Skills/Skills.swift +++ b/Skills/Skills.swift @@ -5,7 +5,7 @@ import Foundation // 文件驱动架构： // - SKILL.md 定义 Skill 元数据 + 指令体（热更新） // - ToolRegistry.swift 注册原生工具实现（编译时） -// - SkillLoader.swift 解析和加载 SKILL.md +// - SkillLoader.swift 解析 SKILL.md (无状态) + SkillRegistry 注册/查询 (有状态) // // 以下仅为给 UI 和 PromptBuilder 使用的精简数据结构。 @@ -22,6 +22,7 @@ struct SkillEntry: Identifiable { var name: String // display name, e.g. "Clipboard" var description: String var icon: String + var type: SkillType var samplePrompt: String var tools: [ToolInfo] = [] var isEnabled: Bool = true @@ -33,6 +34,7 @@ struct SkillEntry: Identifiable { self.name = def.metadata.displayName self.description = def.metadata.description self.icon = def.metadata.icon + self.type = def.metadata.type self.samplePrompt = def.metadata.examples.first?.query ?? "" self.isEnabled = def.isEnabled self.filePath = def.filePath @@ -50,5 +52,6 @@ struct SkillInfo { let description: String var displayName: String = "" var icon: String = "wrench" + var type: SkillType = .device var samplePrompt: String = "" } diff --git a/Skills/ToolRegistry.swift b/Skills/ToolRegistry.swift deleted file mode 100644 index cd99e17..0000000 --- a/Skills/ToolRegistry.swift +++ /dev/null @@ -1,1320 +0,0 @@ -import AVFoundation -import Contacts -import EventKit -import Foundation -import UIKit - -// MARK: - 原生工具注册表 -// -// 所有原生 API 封装集中注册在这里。 -// SKILL.md 通过 allowed-tools 字段引用工具名。 - -enum AppPermissionKind: String, CaseIterable, Identifiable { - case microphone - case calendar - case reminders - case contacts - - var id: String { rawValue } - - var title: String { - switch self { - case .microphone: return "麦克风" - case .calendar: return "日历" - case .reminders: return "提醒事项" - case .contacts: return "通讯录" - } - } - - var description: String { - switch self { - case .microphone: return "允许录音并采集实时音频输入" - case .calendar: return "允许创建和写入日历事项" - case .reminders: return "允许创建提醒和待办" - case .contacts: return "允许查询、保存和删除联系人" - } - } - - var icon: String { - switch self { - case .microphone: return "mic" - case .calendar: return "calendar" - case .reminders: return "bell" - case .contacts: return "person.crop.circle" - } - } -} - -enum AppPermissionStatus: Equatable { - case notDetermined - case denied - case restricted - case granted - - var label: String { - switch self { - case .notDetermined: return "未请求" - case .denied: return "已拒绝" - case .restricted: return "受限制" - case .granted: return "已授权" - } - } - - var detail: String { - switch self { - case .notDetermined: return "首次使用时会弹出系统授权框" - case .denied: return "请到系统设置里手动开启权限" - case .restricted: return "当前设备限制了这项权限" - case .granted: return "可以直接执行相关 Skill" - } - } - - var isGranted: Bool { - self == .granted - } -} - -struct RegisteredTool { - let name: String - let description: String - let parameters: String - let execute: ([String: Any]) async throws -> String -} - -class ToolRegistry { - static let shared = ToolRegistry() - - private var tools: [String: RegisteredTool] = [:] - private let eventStore = EKEventStore() - private let contactStore = CNContactStore() - - private init() { - registerBuiltInTools() - } - - // MARK: - 公开接口 - - func register(_ tool: RegisteredTool) { - tools[tool.name] = tool - } - - func find(name: String) -> RegisteredTool? { - tools[name] - } - - func execute(name: String, args: [String: Any]) async throws -> String { - guard let tool = tools[name] else { - return "{\"success\": false, \"error\": \"未知工具: \(name)\"}" - } - return try await tool.execute(args) - } - - /// 根据名称列表获取工具（用于 SKILL.md 的 allowed-tools） - func toolsFor(names: [String]) -> [RegisteredTool] { - names.compactMap { tools[$0] } - } - - /// 根据工具名反查：它属于哪些 allowed-tools 列表 - /// 返回 true 如果该工具已注册 - func hasToolNamed(_ name: String) -> Bool { - tools[name] != nil - } - - /// 所有已注册的工具名 - var allToolNames: [String] { - Array(tools.keys).sorted() - } - - func authorizationStatus(for kind: AppPermissionKind) -> AppPermissionStatus { - switch kind { - case .microphone: - switch AVAudioSession.sharedInstance().recordPermission { - case .granted: - return .granted - case .denied: - return .denied - case .undetermined: - return .notDetermined - @unknown default: - return .restricted - } - - case .calendar: - let status = EKEventStore.authorizationStatus(for: .event) - switch status { - case .fullAccess, .writeOnly, .authorized: - return .granted - case .notDetermined: - return .notDetermined - case .denied: - return .denied - case .restricted: - return .restricted - @unknown default: - return .restricted - } - - case .reminders: - let status = EKEventStore.authorizationStatus(for: .reminder) - switch status { - case .fullAccess, .authorized: - return .granted - case .notDetermined: - return .notDetermined - case .denied: - return .denied - case .restricted, .writeOnly: - return .restricted - @unknown default: - return .restricted - } - - case .contacts: - let status = CNContactStore.authorizationStatus(for: .contacts) - switch status { - case .authorized: - return .granted - case .limited: - return .granted - case .notDetermined: - return .notDetermined - case .denied: - return .denied - case .restricted: - return .restricted - @unknown default: - return .restricted - } - } - } - - func allPermissionStatuses() -> [AppPermissionKind: AppPermissionStatus] { - Dictionary(uniqueKeysWithValues: AppPermissionKind.allCases.map { - ($0, authorizationStatus(for: $0)) - }) - } - - func requestAccess(for kind: AppPermissionKind) async throws -> Bool { - switch kind { - case .microphone: - return try await requestMicrophoneAccess() - case .calendar: - return try await requestCalendarWriteAccess() - case .reminders: - return try await requestRemindersAccess() - case .contacts: - return try await requestContactsAccess() - } - } - - private func requestMicrophoneAccess() async throws -> Bool { - await withCheckedContinuation { continuation in - AVAudioSession.sharedInstance().requestRecordPermission { granted in - continuation.resume(returning: granted) - } - } - } - - private func requestCalendarWriteAccess() async throws -> Bool { - try await withCheckedThrowingContinuation { continuation in - eventStore.requestWriteOnlyAccessToEvents { granted, error in - if let error { - continuation.resume(throwing: error) - } else { - continuation.resume(returning: granted) - } - } - } - } - - private func requestRemindersAccess() async throws -> Bool { - try await withCheckedThrowingContinuation { continuation in - eventStore.requestFullAccessToReminders { granted, error in - if let error { - continuation.resume(throwing: error) - } else { - continuation.resume(returning: granted) - } - } - } - } - - private func requestContactsAccess() async throws -> Bool { - try await withCheckedThrowingContinuation { continuation in - contactStore.requestAccess(for: .contacts) { granted, error in - if let error { - continuation.resume(throwing: error) - } else { - continuation.resume(returning: granted) - } - } - } - } - - private func parseISO8601Date(_ raw: String) -> Date? { - let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines) - guard !trimmed.isEmpty else { return nil } - - let isoFormatters: [ISO8601DateFormatter] = [ - { - let formatter = ISO8601DateFormatter() - formatter.timeZone = .current - formatter.formatOptions = [.withInternetDateTime, .withFractionalSeconds] - return formatter - }(), - { - let formatter = ISO8601DateFormatter() - formatter.timeZone = .current - formatter.formatOptions = [.withInternetDateTime] - return formatter - }() - ] - - for formatter in isoFormatters { - if let date = formatter.date(from: trimmed) { - return date - } - } - - let formats = [ - "yyyy-MM-dd'T'HH:mm:ss", - "yyyy-MM-dd'T'HH:mm", - "yyyy-MM-dd HH:mm:ss", - "yyyy-MM-dd HH:mm" - ] - - for format in formats { - let formatter = DateFormatter() - formatter.locale = Locale(identifier: "en_US_POSIX") - formatter.timeZone = .current - formatter.dateFormat = format - if let date = formatter.date(from: trimmed) { - return date - } - } - - return nil - } - - private func writableEventCalendar() -> EKCalendar? { - if let calendar = eventStore.defaultCalendarForNewEvents, - calendar.allowsContentModifications { - return calendar - } - - return eventStore.calendars(for: .event) - .first(where: \.allowsContentModifications) - } - - private func writableReminderCalendar() -> EKCalendar? { - if let calendar = eventStore.defaultCalendarForNewReminders(), - calendar.allowsContentModifications { - return calendar - } - - return eventStore.calendars(for: .reminder) - .first(where: \.allowsContentModifications) - } - - private func newReminderListTitle() -> String { - let prefersChinese = Locale.preferredLanguages.contains { $0.hasPrefix("zh") } - return prefersChinese ? "PhoneClaw 提醒事项" : "PhoneClaw Reminders" - } - - private func reminderCalendarCreationSources() -> [EKSource] { - let existingReminderSources = Set( - eventStore.calendars(for: .reminder) - .map(\.source.sourceIdentifier) - ) - - func priority(for source: EKSource) -> Int? { - switch source.sourceType { - case .local: - return existingReminderSources.contains(source.sourceIdentifier) ? 0 : 1 - case .mobileMe: - return existingReminderSources.contains(source.sourceIdentifier) ? 2 : 3 - case .calDAV: - return existingReminderSources.contains(source.sourceIdentifier) ? 4 : 5 - case .exchange: - return existingReminderSources.contains(source.sourceIdentifier) ? 6 : 7 - case .subscribed, .birthdays: - return nil - @unknown default: - return existingReminderSources.contains(source.sourceIdentifier) ? 8 : 9 - } - } - - let prioritizedSources: [(priority: Int, source: EKSource)] = eventStore.sources.compactMap { source -> (priority: Int, source: EKSource)? in - guard let priority = priority(for: source) else { return nil } - return (priority, source) - } - - return prioritizedSources - .sorted { lhs, rhs in - if lhs.priority == rhs.priority { - return lhs.source.title.localizedCaseInsensitiveCompare(rhs.source.title) == .orderedAscending - } - return lhs.priority < rhs.priority - } - .map(\.source) - } - - private func ensureWritableReminderCalendar() throws -> EKCalendar? { - if let calendar = writableReminderCalendar() { - return calendar - } - - var lastError: Error? - for source in reminderCalendarCreationSources() { - let reminderList = EKCalendar(for: .reminder, eventStore: eventStore) - reminderList.title = newReminderListTitle() - reminderList.source = source - - do { - try eventStore.saveCalendar(reminderList, commit: true) - if reminderList.allowsContentModifications { - return reminderList - } - if let saved = eventStore.calendar(withIdentifier: reminderList.calendarIdentifier), - saved.allowsContentModifications { - return saved - } - } catch { - lastError = error - } - } - - if let lastError { - throw lastError - } - - return nil - } - - private func iso8601String(from date: Date) -> String { - let formatter = ISO8601DateFormatter() - formatter.timeZone = .current - formatter.formatOptions = [.withInternetDateTime] - return formatter.string(from: date) - } - - private func reminderDateComponents(from date: Date) -> DateComponents { - Calendar.current.dateComponents( - in: .current, - from: date - ) - } - - private func contactKeysToFetch() -> [CNKeyDescriptor] { - [ - CNContactIdentifierKey as CNKeyDescriptor, - CNContactGivenNameKey as CNKeyDescriptor, - CNContactFamilyNameKey as CNKeyDescriptor, - CNContactMiddleNameKey as CNKeyDescriptor, - CNContactNicknameKey as CNKeyDescriptor, - CNContactJobTitleKey as CNKeyDescriptor, - CNContactPhoneNumbersKey as CNKeyDescriptor, - CNContactOrganizationNameKey as CNKeyDescriptor, - CNContactEmailAddressesKey as CNKeyDescriptor - ] - } - - private func findExistingContact(phone: String) throws -> CNContact? { - let trimmed = phone.trimmingCharacters(in: .whitespacesAndNewlines) - guard !trimmed.isEmpty else { return nil } - - let predicate = CNContact.predicateForContacts( - matching: CNPhoneNumber(stringValue: trimmed) - ) - return try contactStore.unifiedContacts( - matching: predicate, - keysToFetch: contactKeysToFetch() - ).first - } - - private func allContacts() throws -> [CNContact] { - var contacts: [CNContact] = [] - let request = CNContactFetchRequest(keysToFetch: contactKeysToFetch()) - request.sortOrder = .userDefault - try contactStore.enumerateContacts(with: request) { contact, _ in - contacts.append(contact) - } - return contacts - } - - private func formattedContactName(_ contact: CNContact) -> String { - let manual = [contact.familyName, contact.middleName, contact.givenName] - .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) } - .filter { !$0.isEmpty } - .joined() - if !manual.isEmpty { - return manual - } - - let nickname = contact.nickname.trimmingCharacters(in: .whitespacesAndNewlines) - if !nickname.isEmpty { - return nickname - } - - let organization = contact.organizationName.trimmingCharacters(in: .whitespacesAndNewlines) - if !organization.isEmpty { - return organization - } - - return "未命名联系人" - } - - private func clipboardTextPreview( - from text: String, - maxCharacters: Int = 500 - ) -> (preview: String, truncated: Bool)? { - let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) - guard !trimmed.isEmpty else { return nil } - - let endIndex = trimmed.index( - trimmed.startIndex, - offsetBy: maxCharacters, - limitedBy: trimmed.endIndex - ) ?? trimmed.endIndex - - return ( - preview: String(trimmed[.. [String] { - [ - formattedContactName(contact), - contact.familyName, - contact.middleName, - contact.givenName, - contact.nickname, - contact.organizationName, - contact.jobTitle - ] - .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) } - .filter { !$0.isEmpty } - } - - private func relaxedSearchAliases(for raw: String) -> [String] { - let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines) - guard !trimmed.isEmpty else { return [] } - - var aliases = [trimmed] - let suffixes = ["总经理", "经理", "总监", "老板", "老师", "医生", "主任", "总", "哥", "姐"] - for suffix in suffixes where trimmed.hasSuffix(suffix) { - let candidate = String(trimmed.dropLast(suffix.count)) - .trimmingCharacters(in: .whitespacesAndNewlines) - if candidate.count >= 2 { - aliases.append(candidate) - } - } - - let prefixes = ["老", "小", "阿"] - for prefix in prefixes where trimmed.hasPrefix(prefix) { - let candidate = String(trimmed.dropFirst(prefix.count)) - .trimmingCharacters(in: .whitespacesAndNewlines) - if candidate.count >= 2 { - aliases.append(candidate) - } - } - - return Array(NSOrderedSet(array: aliases)) as? [String] ?? aliases - } - - private func primaryPhone(_ contact: CNContact) -> String? { - contact.phoneNumbers - .map { $0.value.stringValue.trimmingCharacters(in: .whitespacesAndNewlines) } - .first(where: { !$0.isEmpty }) - } - - private func primaryEmail(_ contact: CNContact) -> String? { - contact.emailAddresses - .map { String($0.value).trimmingCharacters(in: .whitespacesAndNewlines) } - .first(where: { !$0.isEmpty }) - } - - private func contactSummaryDictionary(_ contact: CNContact) -> [String: Any] { - [ - "identifier": contact.identifier, - "name": formattedContactName(contact), - "phone": primaryPhone(contact) ?? "", - "company": contact.organizationName, - "email": primaryEmail(contact) ?? "" - ] - } - - private func contactSummaryText(_ contact: CNContact) -> String { - var parts = [formattedContactName(contact)] - if let phone = primaryPhone(contact) { - parts.append("电话 \(phone)") - } - if let email = primaryEmail(contact) { - parts.append("邮箱 \(email)") - } - let company = contact.organizationName.trimmingCharacters(in: .whitespacesAndNewlines) - if !company.isEmpty { - parts.append("公司 \(company)") - } - return parts.joined(separator: "，") - } - - private func searchContacts( - identifier: String? = nil, - name: String? = nil, - phone: String? = nil, - email: String? = nil, - query: String? = nil - ) throws -> [CNContact] { - let identifier = identifier?.trimmingCharacters(in: .whitespacesAndNewlines) - let name = name?.trimmingCharacters(in: .whitespacesAndNewlines) - let phone = phone?.trimmingCharacters(in: .whitespacesAndNewlines) - let email = email?.trimmingCharacters(in: .whitespacesAndNewlines) - let query = query?.trimmingCharacters(in: .whitespacesAndNewlines) - - let candidates: [CNContact] - if let identifier, !identifier.isEmpty { - candidates = try contactStore.unifiedContacts( - matching: CNContact.predicateForContacts(withIdentifiers: [identifier]), - keysToFetch: contactKeysToFetch() - ) - } else { - candidates = try allContacts() - } - - let matches = candidates.filter { contact in - if let identifier, !identifier.isEmpty, contact.identifier != identifier { - return false - } - - if let name, !name.isEmpty { - let aliases = relaxedSearchAliases(for: name) - let searchTexts = contactSearchTexts(contact) - let matched = aliases.contains { alias in - searchTexts.contains { $0.localizedCaseInsensitiveContains(alias) } - } - if !matched { - return false - } - } - - if let phone, !phone.isEmpty, - !contact.phoneNumbers.contains(where: { - $0.value.stringValue.localizedCaseInsensitiveContains(phone) - }) { - return false - } - - if let email, !email.isEmpty, - !contact.emailAddresses.contains(where: { - String($0.value).localizedCaseInsensitiveContains(email) - }) { - return false - } - - if let query, !query.isEmpty { - let aliases = relaxedSearchAliases(for: query) - let textMatch = aliases.contains { alias in - contactSearchTexts(contact).contains { - $0.localizedCaseInsensitiveContains(alias) - } - } - let phoneMatch = contact.phoneNumbers.contains { - $0.value.stringValue.localizedCaseInsensitiveContains(query) - } - let emailMatch = contact.emailAddresses.contains { - String($0.value).localizedCaseInsensitiveContains(query) - } - if !(textMatch || phoneMatch || emailMatch) { - return false - } - } - - return true - } - - return matches.sorted { - formattedContactName($0).localizedCaseInsensitiveCompare(formattedContactName($1)) == .orderedAscending - } - } - - // MARK: - 内置工具注册 - - private func registerBuiltInTools() { - func successPayload( - result: String, - extras: [String: Any] = [:] - ) -> String { - var payload = extras - payload["success"] = true - payload["status"] = "succeeded" - payload["result"] = result - return jsonString(payload) - } - - func failurePayload(error: String, extras: [String: Any] = [:]) -> String { - var payload = extras - payload["success"] = false - payload["status"] = "failed" - payload["error"] = error - return jsonString(payload) - } - - func officialDevicePayload() async -> [String: Any] { - let info = ProcessInfo.processInfo - let device = await MainActor.run { - ( - UIDevice.current.name, - UIDevice.current.model, - UIDevice.current.localizedModel, - UIDevice.current.systemName, - UIDevice.current.systemVersion, - UIDevice.current.identifierForVendor?.uuidString - ) - } - - var payload: [String: Any] = [ - "success": true, - "name": device.0, - "model": device.1, - "localized_model": device.2, - "system_name": device.3, - "system_version": device.4, - "memory_bytes": Double(info.physicalMemory), - "memory_gb": Double(info.physicalMemory) / 1_073_741_824.0, - "processor_count": info.processorCount - ] - - if let identifierForVendor = device.5, !identifierForVendor.isEmpty { - payload["identifier_for_vendor"] = identifierForVendor - } - - return payload - } - - // ── Clipboard ── - register(RegisteredTool( - name: "clipboard-read", - description: "读取剪贴板当前内容", - parameters: "无" - ) { _ in - let snapshot = await MainActor.run { () -> [String: Any] in - let pasteboard = UIPasteboard.general - - if pasteboard.numberOfItems == 0 { - return ["kind": "empty"] - } - - if pasteboard.hasImages { - return [ - "kind": "image", - "item_count": pasteboard.numberOfItems - ] - } - - if pasteboard.hasURLs, - let urlText = pasteboard.url?.absoluteString, - let preview = self.clipboardTextPreview(from: urlText, maxCharacters: 500) { - return [ - "kind": "url", - "content": preview.preview, - "truncated": preview.truncated - ] - } - - if pasteboard.hasStrings, - let raw = pasteboard.string, - let preview = self.clipboardTextPreview(from: raw, maxCharacters: 500) { - return [ - "kind": "text", - "content": preview.preview, - "truncated": preview.truncated - ] - } - - return [ - "kind": "unsupported", - "item_count": pasteboard.numberOfItems - ] - } - - switch snapshot["kind"] as? String { - case "text": - let preview = snapshot["content"] as? String ?? "" - let truncated = snapshot["truncated"] as? Bool ?? false - let suffix = truncated ? "（内容较长，已截断显示）" : "" - return successPayload( - result: "剪贴板当前文本内容是：\(preview)\(suffix)", - extras: [ - "type": "text", - "content": preview, - "truncated": truncated - ] - ) - - case "url": - let preview = snapshot["content"] as? String ?? "" - let truncated = snapshot["truncated"] as? Bool ?? false - let suffix = truncated ? "（内容较长，已截断显示）" : "" - return successPayload( - result: "剪贴板当前是链接：\(preview)\(suffix)", - extras: [ - "type": "url", - "content": preview, - "truncated": truncated - ] - ) - - case "image": - let itemCount = snapshot["item_count"] as? Int ?? 1 - return successPayload( - result: "剪贴板当前是图片内容。为避免额外内存占用，暂不直接解码图片。", - extras: [ - "type": "image", - "item_count": itemCount - ] - ) - - case "unsupported": - let itemCount = snapshot["item_count"] as? Int ?? 1 - return successPayload( - result: "剪贴板当前包含 \(itemCount) 项非文本内容，暂不直接读取。", - extras: [ - "type": "unsupported", - "item_count": itemCount - ] - ) - - default: - return failurePayload(error: "剪贴板为空") - } - }) - - register(RegisteredTool( - name: "clipboard-write", - description: "将文本写入剪贴板", - parameters: "text: 要复制的文本内容" - ) { args in - guard let text = args["text"] as? String else { - return failurePayload(error: "缺少 text 参数") - } - await MainActor.run { UIPasteboard.general.string = text } - return successPayload( - result: "已写入剪贴板，共 \(text.count) 个字符。", - extras: ["copied_length": text.count] - ) - }) - - // ── Device ── - register(RegisteredTool( - name: "device-info", - description: "使用 iOS 官方公开 API 汇总获取当前设备名称、设备类型、系统版本、内存和处理器数量", - parameters: "无" - ) { _ in - let payload = await officialDevicePayload() - let name = payload["name"] as? String ?? "" - let localizedModel = (payload["localized_model"] as? String)?.isEmpty == false - ? (payload["localized_model"] as? String ?? "") - : (payload["model"] as? String ?? "") - let systemName = payload["system_name"] as? String ?? "" - let systemVersion = payload["system_version"] as? String ?? "" - let memoryGB = payload["memory_gb"] as? Double ?? 0 - let processorCount = payload["processor_count"] as? Int ?? 0 - - let summary = [ - name.isEmpty ? nil : "设备名称：\(name)", - localizedModel.isEmpty ? nil : "设备类型：\(localizedModel)", - systemVersion.isEmpty ? nil : "系统版本：\(systemName.isEmpty ? "" : systemName + " ")\(systemVersion)", - memoryGB > 0 ? String(format: "物理内存：%.1f GB", memoryGB) : nil, - processorCount > 0 ? "处理器核心数：\(processorCount)" : nil - ].compactMap { $0 }.joined(separator: "\n") - - var enriched = payload - enriched["result"] = summary - enriched["status"] = "succeeded" - return jsonString(enriched) - }) - - register(RegisteredTool( - name: "device-name", - description: "使用 UIDevice.current.name 获取当前设备名称", - parameters: "无" - ) { _ in - let payload = await officialDevicePayload() - let name = payload["name"] as? String ?? "" - return successPayload( - result: "这台设备的名称是 \(name)。", - extras: ["name": name] - ) - }) - - register(RegisteredTool( - name: "device-model", - description: "使用 UIDevice.current.model 和 localizedModel 获取当前设备类型", - parameters: "无" - ) { _ in - let payload = await officialDevicePayload() - let model = payload["model"] as? String ?? "" - let localizedModel = payload["localized_model"] as? String ?? "" - return jsonString([ - "success": true, - "status": "succeeded", - "result": "这台设备的官方设备类型是 \((localizedModel.isEmpty ? model : localizedModel))。", - "model": model, - "localized_model": localizedModel - ]) - }) - - register(RegisteredTool( - name: "device-system-version", - description: "使用 UIDevice.current.systemName 和 systemVersion 获取系统版本", - parameters: "无" - ) { _ in - let payload = await officialDevicePayload() - let systemName = payload["system_name"] as? String ?? "" - let systemVersion = payload["system_version"] as? String ?? "" - return jsonString([ - "success": true, - "status": "succeeded", - "result": "当前系统版本是 \(systemName) \(systemVersion)。", - "system_name": systemName, - "system_version": systemVersion - ]) - }) - - register(RegisteredTool( - name: "device-memory", - description: "使用 ProcessInfo.processInfo.physicalMemory 获取设备物理内存", - parameters: "无" - ) { _ in - let payload = await officialDevicePayload() - let memoryBytes = payload["memory_bytes"] as? Double ?? 0 - let memoryGB = payload["memory_gb"] as? Double ?? 0 - return jsonString([ - "success": true, - "status": "succeeded", - "result": String(format: "这台设备的物理内存约为 %.1f GB。", memoryGB), - "memory_bytes": memoryBytes, - "memory_gb": memoryGB - ]) - }) - - register(RegisteredTool( - name: "device-processor-count", - description: "使用 ProcessInfo.processInfo.processorCount 获取处理器核心数", - parameters: "无" - ) { _ in - let payload = await officialDevicePayload() - let processorCount = payload["processor_count"] as? Int ?? 0 - return jsonString([ - "success": true, - "status": "succeeded", - "result": "这台设备的处理器核心数是 \(processorCount)。", - "processor_count": processorCount - ]) - }) - - register(RegisteredTool( - name: "device-identifier-for-vendor", - description: "使用 UIDevice.current.identifierForVendor 获取当前 App 在该设备上的 vendor 标识", - parameters: "无" - ) { _ in - let payload = await officialDevicePayload() - let identifier = payload["identifier_for_vendor"] as? String ?? "" - return jsonString([ - "success": true, - "status": "succeeded", - "result": "当前 App 在这台设备上的 identifierForVendor 是 \(identifier)。", - "identifier_for_vendor": identifier - ]) - }) - - // ── Text ── - register(RegisteredTool( - name: "calculate-hash", - description: "计算文本的哈希值", - parameters: "text: 要计算哈希的文本" - ) { args in - guard let text = args["text"] as? String else { - return failurePayload(error: "缺少 text 参数") - } - let hash = text.hashValue - return successPayload( - result: "文本“\(text)”的哈希值是 \(hash)。", - extras: [ - "input": text, - "hash": hash - ] - ) - }) - - register(RegisteredTool( - name: "text-reverse", - description: "翻转文本", - parameters: "text: 要翻转的文本" - ) { args in - guard let text = args["text"] as? String else { - return failurePayload(error: "缺少 text 参数") - } - let reversed = String(text.reversed()) - return successPayload( - result: "翻转结果：\(reversed)", - extras: [ - "original": text, - "reversed": reversed - ] - ) - }) - - // ── Calendar / Reminders / Contacts ── - register(RegisteredTool( - name: "calendar-create-event", - description: "创建新的日历事项，可写入标题、开始时间、结束时间、地点和备注", - parameters: "title: 事件标题, start: ISO 8601 开始时间, end: ISO 8601 结束时间（可选）, location: 地点（可选）, notes: 备注（可选）" - ) { args in - guard let rawTitle = args["title"] as? String else { - return failurePayload(error: "缺少 title 参数") - } - let title = rawTitle.trimmingCharacters(in: .whitespacesAndNewlines) - guard !title.isEmpty else { - return failurePayload(error: "缺少 title 参数") - } - guard let startRaw = args["start"] as? String, - let startDate = self.parseISO8601Date(startRaw) else { - return failurePayload(error: "缺少有效的 start 参数，必须是 ISO 8601 时间字符串") - } - - let endRaw = (args["end"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - let endDate = endRaw.flatMap(self.parseISO8601Date) ?? startDate.addingTimeInterval(3600) - guard endDate >= startDate else { - return failurePayload(error: "end 不能早于 start") - } - - let location = (args["location"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - let notes = (args["notes"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - - do { - guard try await self.requestCalendarWriteAccess() else { - return failurePayload(error: "未获得日历写入权限") - } - - guard let calendar = self.writableEventCalendar() else { - return failurePayload(error: "没有可用于新建事项的可写日历，请先在系统日历中启用或创建一个日历") - } - - let event = EKEvent(eventStore: self.eventStore) - event.calendar = calendar - event.title = title - event.startDate = startDate - event.endDate = endDate - if let location, !location.isEmpty { - event.location = location - } - if let notes, !notes.isEmpty { - event.notes = notes - } - - try self.eventStore.save(event, span: .thisEvent, commit: true) - - return successPayload( - result: "已创建日历事项“\(title)”，开始时间为 \(self.iso8601String(from: startDate))。", - extras: [ - "eventId": event.eventIdentifier ?? "", - "title": title, - "start": self.iso8601String(from: startDate), - "end": self.iso8601String(from: endDate), - "location": location ?? "", - "notes": notes ?? "" - ] - ) - } catch { - return failurePayload(error: "创建日历事项失败：\(error.localizedDescription)") - } - }) - - register(RegisteredTool( - name: "reminders-create", - description: "创建新的提醒事项，可写入标题、到期时间和备注", - parameters: "title: 提醒标题, due: ISO 8601 到期时间（可选）, notes: 备注（可选）" - ) { args in - guard let rawTitle = args["title"] as? String else { - return failurePayload(error: "缺少 title 参数") - } - let title = rawTitle.trimmingCharacters(in: .whitespacesAndNewlines) - guard !title.isEmpty else { - return failurePayload(error: "缺少 title 参数") - } - - let dueRaw = (args["due"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - if let dueRaw, !dueRaw.isEmpty, self.parseISO8601Date(dueRaw) == nil { - return failurePayload(error: "due 必须是有效的 ISO 8601 时间字符串") - } - - let dueDate = dueRaw.flatMap(self.parseISO8601Date) - let notes = (args["notes"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - - do { - guard try await self.requestRemindersAccess() else { - return failurePayload(error: "未获得提醒事项权限") - } - - guard let calendar = try self.ensureWritableReminderCalendar() else { - return failurePayload(error: "没有可用于新建提醒事项的可写列表，且无法自动创建提醒列表，请先在系统提醒事项 App 中启用或创建一个列表") - } - - let reminder = EKReminder(eventStore: self.eventStore) - reminder.calendar = calendar - reminder.title = title - if let dueDate { - reminder.dueDateComponents = self.reminderDateComponents(from: dueDate) - reminder.addAlarm(EKAlarm(absoluteDate: dueDate)) - } - if let notes, !notes.isEmpty { - reminder.notes = notes - } - - try self.eventStore.save(reminder, commit: true) - - return successPayload( - result: dueDate != nil - ? "已创建提醒事项“\(title)”，提醒时间为 \(self.iso8601String(from: dueDate!))。" - : "已创建提醒事项“\(title)”。", - extras: [ - "calendarItemId": reminder.calendarItemIdentifier, - "title": title, - "due": dueDate.map { self.iso8601String(from: $0) } ?? "", - "notes": notes ?? "" - ] - ) - } catch { - return failurePayload(error: "创建提醒事项失败：\(error.localizedDescription)") - } - }) - - register(RegisteredTool( - name: "contacts-upsert", - description: "创建或更新联系人；若提供手机号则优先按手机号查重再更新", - parameters: "name: 联系人姓名, phone: 手机号（可选）, company: 公司（可选）, email: 邮箱（可选）, notes: 备注（可选）" - ) { args in - guard let rawName = args["name"] as? String else { - return failurePayload(error: "缺少 name 参数") - } - let name = rawName.trimmingCharacters(in: .whitespacesAndNewlines) - guard !name.isEmpty else { - return failurePayload(error: "缺少 name 参数") - } - - let phone = (args["phone"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - let company = (args["company"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - let email = (args["email"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - let notes = (args["notes"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - - do { - guard try await self.requestContactsAccess() else { - return failurePayload(error: "未获得通讯录权限") - } - - let existingContact = phone.flatMap { try? self.findExistingContact(phone: $0) } - let mutableContact: CNMutableContact - let action: String - - if let existingContact { - mutableContact = existingContact.mutableCopy() as! CNMutableContact - action = "updated" - } else { - mutableContact = CNMutableContact() - action = "created" - } - - mutableContact.givenName = name - mutableContact.familyName = "" - - if let phone, !phone.isEmpty { - mutableContact.phoneNumbers = [ - CNLabeledValue( - label: CNLabelPhoneNumberMobile, - value: CNPhoneNumber(stringValue: phone) - ) - ] - } - if let company, !company.isEmpty { - mutableContact.organizationName = company - } - if let email, !email.isEmpty { - mutableContact.emailAddresses = [ - CNLabeledValue(label: CNLabelWork, value: email as NSString) - ] - } - if let notes, !notes.isEmpty { - mutableContact.note = notes - } - - let saveRequest = CNSaveRequest() - if existingContact != nil { - saveRequest.update(mutableContact) - } else { - saveRequest.add(mutableContact, toContainerWithIdentifier: nil) - } - try self.contactStore.execute(saveRequest) - - let actionText = action == "updated" ? "已更新" : "已创建" - return successPayload( - result: "\(actionText)联系人“\(name)”。", - extras: [ - "action": action, - "name": name, - "phone": phone ?? "", - "company": company ?? "", - "email": email ?? "", - "notes": notes ?? "" - ] - ) - } catch { - return failurePayload(error: "保存联系人失败：\(error.localizedDescription)") - } - }) - - register(RegisteredTool( - name: "contacts-search", - description: "搜索联系人，可按姓名、手机号、邮箱、identifier 或关键词查询联系方式", - parameters: "query: 搜索关键词（可选）, identifier: 联系人标识（可选）, name: 姓名（可选）, phone: 手机号（可选）, email: 邮箱（可选）" - ) { args in - let identifier = (args["identifier"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - let name = (args["name"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - let phone = (args["phone"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - let email = (args["email"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - let query = (args["query"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - - guard identifier?.isEmpty == false - || name?.isEmpty == false - || phone?.isEmpty == false - || email?.isEmpty == false - || query?.isEmpty == false else { - return failurePayload(error: "请至少提供 query、name、phone、email 或 identifier 其中一个参数") - } - - do { - guard try await self.requestContactsAccess() else { - return failurePayload(error: "未获得通讯录权限") - } - - let matches = Array(try self.searchContacts( - identifier: identifier, - name: name, - phone: phone, - email: email, - query: query - ).prefix(5)) - - let items = matches.map(self.contactSummaryDictionary) - if matches.isEmpty { - return successPayload( - result: "未找到匹配的联系人。", - extras: [ - "count": 0, - "items": items - ] - ) - } - - let lines = matches.map(self.contactSummaryText) - return successPayload( - result: "找到 \(matches.count) 个联系人：\(lines.joined(separator: "；"))。", - extras: [ - "count": matches.count, - "items": items - ] - ) - } catch { - return failurePayload(error: "搜索联系人失败：\(error.localizedDescription)") - } - }) - - register(RegisteredTool( - name: "contacts-delete", - description: "删除联系人，可按姓名、手机号、邮箱、identifier 或关键词匹配后删除", - parameters: "query: 搜索关键词（可选）, identifier: 联系人标识（可选）, name: 姓名（可选）, phone: 手机号（可选）, email: 邮箱（可选）" - ) { args in - let identifier = (args["identifier"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - let rawName = (args["name"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - let phone = (args["phone"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - let email = (args["email"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - let query = (args["query"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) - let name = rawName? - .replacingOccurrences(of: "的电话", with: "") - .replacingOccurrences(of: "电话", with: "") - .replacingOccurrences(of: "手机号", with: "") - .replacingOccurrences(of: "号码", with: "") - .replacingOccurrences(of: "联系方式", with: "") - .trimmingCharacters(in: CharacterSet(charactersIn: "，。,？！!? ")) - - guard identifier?.isEmpty == false - || name?.isEmpty == false - || phone?.isEmpty == false - || email?.isEmpty == false - || query?.isEmpty == false else { - return failurePayload(error: "请至少提供 query、name、phone、email 或 identifier 其中一个参数") - } - - do { - guard try await self.requestContactsAccess() else { - return failurePayload(error: "未获得通讯录权限") - } - - let matches = try self.searchContacts( - identifier: identifier, - name: name, - phone: phone, - email: email, - query: query - ) - - if matches.isEmpty { - return failurePayload(error: "未找到匹配的联系人") - } - - if matches.count > 1 { - let previews = matches.prefix(5).map(self.contactSummaryText).joined(separator: "；") - return failurePayload(error: "匹配到多个联系人，请提供更具体的信息：\(previews)") - } - - let contact = matches[0] - let mutableContact = contact.mutableCopy() as! CNMutableContact - let saveRequest = CNSaveRequest() - saveRequest.delete(mutableContact) - try self.contactStore.execute(saveRequest) - - return successPayload( - result: "已删除联系人“\(self.formattedContactName(contact))”。", - extras: [ - "identifier": contact.identifier, - "name": self.formattedContactName(contact), - "phone": self.primaryPhone(contact) ?? "", - "email": self.primaryEmail(contact) ?? "" - ] - ) - } catch { - return failurePayload(error: "删除联系人失败：\(error.localizedDescription)") - } - }) - } -} - -// MARK: - Helpers - -func jsonEscape(_ str: String) -> String { - str.replacingOccurrences(of: "\\", with: "\\\\") - .replacingOccurrences(of: "\"", with: "\\\"") - .replacingOccurrences(of: "\n", with: "\\n") - .replacingOccurrences(of: "\t", with: "\\t") -} - -func jsonString(_ object: Any) -> String { - guard JSONSerialization.isValidJSONObject(object), - let data = try? JSONSerialization.data(withJSONObject: object), - let string = String(data: data, encoding: .utf8) else { - return "{\"success\": false, \"error\": \"JSON 编码失败\"}" - } - return string -} diff --git a/Tools/AppPermissions.swift b/Tools/AppPermissions.swift new file mode 100644 index 0000000..74a0ceb --- /dev/null +++ b/Tools/AppPermissions.swift @@ -0,0 +1,186 @@ +import AVFoundation +import Contacts +import EventKit +import Foundation + +// MARK: - 权限模型 + +enum AppPermissionKind: String, CaseIterable, Identifiable { + case microphone + case calendar + case reminders + case contacts + + var id: String { rawValue } + + var title: String { + switch self { + case .microphone: return "麦克风" + case .calendar: return "日历" + case .reminders: return "提醒事项" + case .contacts: return "通讯录" + } + } + + var description: String { + switch self { + case .microphone: return "允许录音并采集实时音频输入" + case .calendar: return "允许创建和写入日历事项" + case .reminders: return "允许创建提醒和待办" + case .contacts: return "允许查询、保存和删除联系人" + } + } + + var icon: String { + switch self { + case .microphone: return "mic" + case .calendar: return "calendar" + case .reminders: return "bell" + case .contacts: return "person.crop.circle" + } + } +} + +enum AppPermissionStatus: Equatable { + case notDetermined + case denied + case restricted + case granted + + var label: String { + switch self { + case .notDetermined: return "未请求" + case .denied: return "已拒绝" + case .restricted: return "受限制" + case .granted: return "已授权" + } + } + + var detail: String { + switch self { + case .notDetermined: return "首次使用时会弹出系统授权框" + case .denied: return "请到系统设置里手动开启权限" + case .restricted: return "当前设备限制了这项权限" + case .granted: return "可以直接执行相关 Skill" + } + } + + var isGranted: Bool { + self == .granted + } +} + +// MARK: - 权限查询与请求 + +extension ToolRegistry { + + func authorizationStatus(for kind: AppPermissionKind) -> AppPermissionStatus { + switch kind { + case .microphone: + switch AVAudioSession.sharedInstance().recordPermission { + case .granted: + return .granted + case .denied: + return .denied + case .undetermined: + return .notDetermined + @unknown default: + return .restricted + } + + case .calendar: + let status = EKEventStore.authorizationStatus(for: .event) + switch status { + case .fullAccess, .writeOnly, .authorized: + return .granted + case .notDetermined: + return .notDetermined + case .denied: + return .denied + case .restricted: + return .restricted + @unknown default: + return .restricted + } + + case .reminders: + let status = EKEventStore.authorizationStatus(for: .reminder) + switch status { + case .fullAccess, .authorized: + return .granted + case .notDetermined: + return .notDetermined + case .denied: + return .denied + case .restricted, .writeOnly: + return .restricted + @unknown default: + return .restricted + } + + case .contacts: + let status = CNContactStore.authorizationStatus(for: .contacts) + switch status { + case .authorized: + return .granted + case .limited: + return .granted + case .notDetermined: + return .notDetermined + case .denied: + return .denied + case .restricted: + return .restricted + @unknown default: + return .restricted + } + } + } + + func allPermissionStatuses() -> [AppPermissionKind: AppPermissionStatus] { + Dictionary(uniqueKeysWithValues: AppPermissionKind.allCases.map { + ($0, authorizationStatus(for: $0)) + }) + } + + func requestAccess(for kind: AppPermissionKind) async throws -> Bool { + switch kind { + case .microphone: + return await withCheckedContinuation { continuation in + AVAudioSession.sharedInstance().requestRecordPermission { granted in + continuation.resume(returning: granted) + } + } + case .calendar: + return try await withCheckedThrowingContinuation { continuation in + SystemStores.event.requestWriteOnlyAccessToEvents { granted, error in + if let error { + continuation.resume(throwing: error) + } else { + continuation.resume(returning: granted) + } + } + } + case .reminders: + return try await withCheckedThrowingContinuation { continuation in + SystemStores.event.requestFullAccessToReminders { granted, error in + if let error { + continuation.resume(throwing: error) + } else { + continuation.resume(returning: granted) + } + } + } + case .contacts: + return try await withCheckedThrowingContinuation { continuation in + SystemStores.contacts.requestAccess(for: .contacts) { granted, error in + if let error { + continuation.resume(throwing: error) + } else { + continuation.resume(returning: granted) + } + } + } + } + } +} diff --git a/Tools/Handlers/Calendar.swift b/Tools/Handlers/Calendar.swift new file mode 100644 index 0000000..c140acf --- /dev/null +++ b/Tools/Handlers/Calendar.swift @@ -0,0 +1,87 @@ +import EventKit +import Foundation + +enum CalendarTools { + + static func register(into registry: ToolRegistry) { + + // ── calendar-create-event ── + registry.register(RegisteredTool( + name: "calendar-create-event", + description: "创建新的日历事项，可写入标题、开始时间、结束时间、地点和备注", + parameters: "title: 事件标题, start: ISO 8601 开始时间, end: ISO 8601 结束时间（可选）, location: 地点（可选）, notes: 备注（可选）", + // 只有 start 是硬参 (EventKit API 强制要求)。 + // title 是软参,缺失时 handler 用默认值,不强制 caller 提供。 + requiredParameters: ["start"] + ) { args in + // title 是软参: 没传或为空时使用默认标题, 不阻断流程 + let rawTitle = (args["title"] as? String)? + .trimmingCharacters(in: .whitespacesAndNewlines) ?? "" + let title = rawTitle.isEmpty ? "新日历事项" : rawTitle + + guard let startRaw = args["start"] as? String, + let startDate = parseISO8601Date(startRaw) else { + return failurePayload(error: "缺少有效的 start 参数，必须是 ISO 8601 时间字符串") + } + + let endRaw = (args["end"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) + let endDate = endRaw.flatMap(parseISO8601Date) ?? startDate.addingTimeInterval(3600) + guard endDate >= startDate else { + return failurePayload(error: "end 不能早于 start") + } + + let location = (args["location"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) + let notes = (args["notes"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) + + do { + guard try await ToolRegistry.shared.requestAccess(for: .calendar) else { + return failurePayload(error: "未获得日历写入权限") + } + + guard let calendar = writableEventCalendar() else { + return failurePayload(error: "没有可用于新建事项的可写日历，请先在系统日历中启用或创建一个日历") + } + + let event = EKEvent(eventStore: SystemStores.event) + event.calendar = calendar + event.title = title + event.startDate = startDate + event.endDate = endDate + if let location, !location.isEmpty { + event.location = location + } + if let notes, !notes.isEmpty { + event.notes = notes + } + + try SystemStores.event.save(event, span: .thisEvent, commit: true) + + return successPayload( + result: "已创建日历事项\u{201C}\(title)\u{201D}，开始时间为 \(iso8601String(from: startDate))。", + extras: [ + "eventId": event.eventIdentifier ?? "", + "title": title, + "start": iso8601String(from: startDate), + "end": iso8601String(from: endDate), + "location": location ?? "", + "notes": notes ?? "" + ] + ) + } catch { + return failurePayload(error: "创建日历事项失败：\(error.localizedDescription)") + } + }) + } + + // MARK: - Private Helpers + + private static func writableEventCalendar() -> EKCalendar? { + if let calendar = SystemStores.event.defaultCalendarForNewEvents, + calendar.allowsContentModifications { + return calendar + } + + return SystemStores.event.calendars(for: .event) + .first(where: \.allowsContentModifications) + } +} diff --git a/Tools/Handlers/Clipboard.swift b/Tools/Handlers/Clipboard.swift new file mode 100644 index 0000000..fbe416b --- /dev/null +++ b/Tools/Handlers/Clipboard.swift @@ -0,0 +1,146 @@ +import UIKit + +enum ClipboardTools { + + static func register(into registry: ToolRegistry) { + + // ── clipboard-read ── + registry.register(RegisteredTool( + name: "clipboard-read", + description: "读取剪贴板当前内容", + parameters: "无", + isParameterless: true, + skipFollowUp: true + ) { _ in + let snapshot = await MainActor.run { () -> [String: Any] in + let pasteboard = UIPasteboard.general + + if pasteboard.numberOfItems == 0 { + return ["kind": "empty"] + } + + if pasteboard.hasImages { + return [ + "kind": "image", + "item_count": pasteboard.numberOfItems + ] + } + + if pasteboard.hasURLs, + let urlText = pasteboard.url?.absoluteString, + let preview = textPreview(from: urlText, maxCharacters: 500) { + return [ + "kind": "url", + "content": preview.preview, + "truncated": preview.truncated + ] + } + + if pasteboard.hasStrings, + let raw = pasteboard.string, + let preview = textPreview(from: raw, maxCharacters: 500) { + return [ + "kind": "text", + "content": preview.preview, + "truncated": preview.truncated + ] + } + + return [ + "kind": "unsupported", + "item_count": pasteboard.numberOfItems + ] + } + + switch snapshot["kind"] as? String { + case "text": + let preview = snapshot["content"] as? String ?? "" + let truncated = snapshot["truncated"] as? Bool ?? false + let suffix = truncated ? "（内容较长，已截断显示）" : "" + return successPayload( + result: "剪贴板当前文本内容是：\(preview)\(suffix)", + extras: [ + "type": "text", + "content": preview, + "truncated": truncated + ] + ) + + case "url": + let preview = snapshot["content"] as? String ?? "" + let truncated = snapshot["truncated"] as? Bool ?? false + let suffix = truncated ? "（内容较长，已截断显示）" : "" + return successPayload( + result: "剪贴板当前是链接：\(preview)\(suffix)", + extras: [ + "type": "url", + "content": preview, + "truncated": truncated + ] + ) + + case "image": + let itemCount = snapshot["item_count"] as? Int ?? 1 + return successPayload( + result: "剪贴板当前是图片内容。为避免额外内存占用，暂不直接解码图片。", + extras: [ + "type": "image", + "item_count": itemCount + ] + ) + + case "unsupported": + let itemCount = snapshot["item_count"] as? Int ?? 1 + return successPayload( + result: "剪贴板当前包含 \(itemCount) 项非文本内容，暂不直接读取。", + extras: [ + "type": "unsupported", + "item_count": itemCount + ] + ) + + default: + return failurePayload(error: "剪贴板为空") + } + }) + + // ── clipboard-write ── + registry.register(RegisteredTool( + name: "clipboard-write", + description: "将文本写入剪贴板", + parameters: "text: 要复制的文本内容", + requiredParameters: ["text"], + skipFollowUp: true + ) { args in + guard let text = args["text"] as? String else { + return failurePayload(error: "缺少 text 参数") + } + await MainActor.run { UIPasteboard.general.string = text } + return successPayload( + result: "已写入剪贴板，共 \(text.count) 个字符。", + extras: ["copied_length": text.count] + ) + }) + } + + // MARK: - Private Helpers + + private static func textPreview( + from text: String, + maxCharacters: Int = 500 + ) -> (preview: String, truncated: Bool)? { + let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { return nil } + + let endIndex = trimmed.index( + trimmed.startIndex, + offsetBy: maxCharacters, + limitedBy: trimmed.endIndex + ) ?? trimmed.endIndex + + return ( + preview: String(trimmed[.. 1 { + let previews = matches.prefix(5).map(contactSummaryText).joined(separator: "；") + return failurePayload(error: "匹配到多个联系人，请提供更具体的信息：\(previews)") + } + + let contact = matches[0] + let mutableContact = contact.mutableCopy() as! CNMutableContact + let saveRequest = CNSaveRequest() + saveRequest.delete(mutableContact) + try SystemStores.contacts.execute(saveRequest) + + return successPayload( + result: "已删除联系人\u{201C}\(formattedContactName(contact))\u{201D}。", + extras: [ + "identifier": contact.identifier, + "name": formattedContactName(contact), + "phone": primaryPhone(contact) ?? "", + "email": primaryEmail(contact) ?? "" + ] + ) + } catch { + return failurePayload(error: "删除联系人失败：\(error.localizedDescription)") + } + }) + } + + // MARK: - Private Helpers + + private static func contactKeysToFetch() -> [CNKeyDescriptor] { + [ + CNContactIdentifierKey as CNKeyDescriptor, + CNContactGivenNameKey as CNKeyDescriptor, + CNContactFamilyNameKey as CNKeyDescriptor, + CNContactMiddleNameKey as CNKeyDescriptor, + CNContactNicknameKey as CNKeyDescriptor, + CNContactJobTitleKey as CNKeyDescriptor, + CNContactPhoneNumbersKey as CNKeyDescriptor, + CNContactOrganizationNameKey as CNKeyDescriptor, + CNContactEmailAddressesKey as CNKeyDescriptor + ] + } + + private static func findExistingContact(phone: String) throws -> CNContact? { + let trimmed = phone.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { return nil } + + let predicate = CNContact.predicateForContacts( + matching: CNPhoneNumber(stringValue: trimmed) + ) + return try SystemStores.contacts.unifiedContacts( + matching: predicate, + keysToFetch: contactKeysToFetch() + ).first + } + + private static func allContacts() throws -> [CNContact] { + var contacts: [CNContact] = [] + let request = CNContactFetchRequest(keysToFetch: contactKeysToFetch()) + request.sortOrder = .userDefault + try SystemStores.contacts.enumerateContacts(with: request) { contact, _ in + contacts.append(contact) + } + return contacts + } + + private static func formattedContactName(_ contact: CNContact) -> String { + let manual = [contact.familyName, contact.middleName, contact.givenName] + .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) } + .filter { !$0.isEmpty } + .joined() + if !manual.isEmpty { + return manual + } + + let nickname = contact.nickname.trimmingCharacters(in: .whitespacesAndNewlines) + if !nickname.isEmpty { + return nickname + } + + let organization = contact.organizationName.trimmingCharacters(in: .whitespacesAndNewlines) + if !organization.isEmpty { + return organization + } + + return "未命名联系人" + } + + private static func contactSearchTexts(_ contact: CNContact) -> [String] { + [ + formattedContactName(contact), + contact.familyName, + contact.middleName, + contact.givenName, + contact.nickname, + contact.organizationName, + contact.jobTitle + ] + .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) } + .filter { !$0.isEmpty } + } + + private static func relaxedSearchAliases(for raw: String) -> [String] { + let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { return [] } + + var aliases = [trimmed] + let suffixes = ["总经理", "经理", "总监", "老板", "老师", "医生", "主任", "总", "哥", "姐"] + for suffix in suffixes where trimmed.hasSuffix(suffix) { + let candidate = String(trimmed.dropLast(suffix.count)) + .trimmingCharacters(in: .whitespacesAndNewlines) + if candidate.count >= 2 { + aliases.append(candidate) + } + } + + let prefixes = ["老", "小", "阿"] + for prefix in prefixes where trimmed.hasPrefix(prefix) { + let candidate = String(trimmed.dropFirst(prefix.count)) + .trimmingCharacters(in: .whitespacesAndNewlines) + if candidate.count >= 2 { + aliases.append(candidate) + } + } + + return Array(NSOrderedSet(array: aliases)) as? [String] ?? aliases + } + + private static func primaryPhone(_ contact: CNContact) -> String? { + contact.phoneNumbers + .map { $0.value.stringValue.trimmingCharacters(in: .whitespacesAndNewlines) } + .first(where: { !$0.isEmpty }) + } + + private static func primaryEmail(_ contact: CNContact) -> String? { + contact.emailAddresses + .map { String($0.value).trimmingCharacters(in: .whitespacesAndNewlines) } + .first(where: { !$0.isEmpty }) + } + + private static func contactSummaryDictionary(_ contact: CNContact) -> [String: Any] { + [ + "identifier": contact.identifier, + "name": formattedContactName(contact), + "phone": primaryPhone(contact) ?? "", + "company": contact.organizationName, + "email": primaryEmail(contact) ?? "" + ] + } + + private static func contactSummaryText(_ contact: CNContact) -> String { + var parts = [formattedContactName(contact)] + if let phone = primaryPhone(contact) { + parts.append("电话 \(phone)") + } + if let email = primaryEmail(contact) { + parts.append("邮箱 \(email)") + } + let company = contact.organizationName.trimmingCharacters(in: .whitespacesAndNewlines) + if !company.isEmpty { + parts.append("公司 \(company)") + } + return parts.joined(separator: "，") + } + + private static func searchContacts( + identifier: String? = nil, + name: String? = nil, + phone: String? = nil, + email: String? = nil, + query: String? = nil + ) throws -> [CNContact] { + let identifier = identifier?.trimmingCharacters(in: .whitespacesAndNewlines) + let name = name?.trimmingCharacters(in: .whitespacesAndNewlines) + let phone = phone?.trimmingCharacters(in: .whitespacesAndNewlines) + let email = email?.trimmingCharacters(in: .whitespacesAndNewlines) + let query = query?.trimmingCharacters(in: .whitespacesAndNewlines) + + let candidates: [CNContact] + if let identifier, !identifier.isEmpty { + candidates = try SystemStores.contacts.unifiedContacts( + matching: CNContact.predicateForContacts(withIdentifiers: [identifier]), + keysToFetch: contactKeysToFetch() + ) + } else { + candidates = try allContacts() + } + + let matches = candidates.filter { contact in + if let identifier, !identifier.isEmpty, contact.identifier != identifier { + return false + } + + if let name, !name.isEmpty { + let aliases = relaxedSearchAliases(for: name) + let searchTexts = contactSearchTexts(contact) + let matched = aliases.contains { alias in + searchTexts.contains { $0.localizedCaseInsensitiveContains(alias) } + } + if !matched { + return false + } + } + + if let phone, !phone.isEmpty, + !contact.phoneNumbers.contains(where: { + $0.value.stringValue.localizedCaseInsensitiveContains(phone) + }) { + return false + } + + if let email, !email.isEmpty, + !contact.emailAddresses.contains(where: { + String($0.value).localizedCaseInsensitiveContains(email) + }) { + return false + } + + if let query, !query.isEmpty { + let aliases = relaxedSearchAliases(for: query) + let textMatch = aliases.contains { alias in + contactSearchTexts(contact).contains { + $0.localizedCaseInsensitiveContains(alias) + } + } + let phoneMatch = contact.phoneNumbers.contains { + $0.value.stringValue.localizedCaseInsensitiveContains(query) + } + let emailMatch = contact.emailAddresses.contains { + String($0.value).localizedCaseInsensitiveContains(query) + } + if !(textMatch || phoneMatch || emailMatch) { + return false + } + } + + return true + } + + return matches.sorted { + formattedContactName($0).localizedCaseInsensitiveCompare(formattedContactName($1)) == .orderedAscending + } + } +} diff --git a/Tools/Handlers/Device.swift b/Tools/Handlers/Device.swift new file mode 100644 index 0000000..2a8eef0 --- /dev/null +++ b/Tools/Handlers/Device.swift @@ -0,0 +1,179 @@ +import Foundation +import UIKit + +enum DeviceTools { + + static func register(into registry: ToolRegistry) { + + // ── device-info ── + registry.register(RegisteredTool( + name: "device-info", + description: "使用 iOS 官方公开 API 汇总获取当前设备名称、设备类型、系统版本、内存和处理器数量", + parameters: "无", + isParameterless: true + ) { _ in + let payload = await devicePayload() + let name = payload["name"] as? String ?? "" + let localizedModel = (payload["localized_model"] as? String)?.isEmpty == false + ? (payload["localized_model"] as? String ?? "") + : (payload["model"] as? String ?? "") + let systemName = payload["system_name"] as? String ?? "" + let systemVersion = payload["system_version"] as? String ?? "" + let memoryGB = payload["memory_gb"] as? Double ?? 0 + let processorCount = payload["processor_count"] as? Int ?? 0 + + let summary = [ + name.isEmpty ? nil : "设备名称：\(name)", + localizedModel.isEmpty ? nil : "设备类型：\(localizedModel)", + systemVersion.isEmpty ? nil : "系统版本：\(systemName.isEmpty ? "" : systemName + " ")\(systemVersion)", + memoryGB > 0 ? String(format: "物理内存：%.1f GB", memoryGB) : nil, + processorCount > 0 ? "处理器核心数：\(processorCount)" : nil + ].compactMap { $0 }.joined(separator: "\n") + + var enriched = payload + enriched["result"] = summary + enriched["status"] = "succeeded" + return jsonString(enriched) + }) + + // ── device-name ── + registry.register(RegisteredTool( + name: "device-name", + description: "使用 UIDevice.current.name 获取当前设备名称", + parameters: "无", + isParameterless: true + ) { _ in + let payload = await devicePayload() + let name = payload["name"] as? String ?? "" + return successPayload( + result: "这台设备的名称是 \(name)。", + extras: ["name": name] + ) + }) + + // ── device-model ── + registry.register(RegisteredTool( + name: "device-model", + description: "使用 UIDevice.current.model 和 localizedModel 获取当前设备类型", + parameters: "无", + isParameterless: true + ) { _ in + let payload = await devicePayload() + let model = payload["model"] as? String ?? "" + let localizedModel = payload["localized_model"] as? String ?? "" + return jsonString([ + "success": true, + "status": "succeeded", + "result": "这台设备的官方设备类型是 \((localizedModel.isEmpty ? model : localizedModel))。", + "model": model, + "localized_model": localizedModel + ]) + }) + + // ── device-system-version ── + registry.register(RegisteredTool( + name: "device-system-version", + description: "使用 UIDevice.current.systemName 和 systemVersion 获取系统版本", + parameters: "无", + isParameterless: true + ) { _ in + let payload = await devicePayload() + let systemName = payload["system_name"] as? String ?? "" + let systemVersion = payload["system_version"] as? String ?? "" + return jsonString([ + "success": true, + "status": "succeeded", + "result": "当前系统版本是 \(systemName) \(systemVersion)。", + "system_name": systemName, + "system_version": systemVersion + ]) + }) + + // ── device-memory ── + registry.register(RegisteredTool( + name: "device-memory", + description: "使用 ProcessInfo.processInfo.physicalMemory 获取设备物理内存", + parameters: "无", + isParameterless: true + ) { _ in + let payload = await devicePayload() + let memoryBytes = payload["memory_bytes"] as? Double ?? 0 + let memoryGB = payload["memory_gb"] as? Double ?? 0 + return jsonString([ + "success": true, + "status": "succeeded", + "result": String(format: "这台设备的物理内存约为 %.1f GB。", memoryGB), + "memory_bytes": memoryBytes, + "memory_gb": memoryGB + ]) + }) + + // ── device-processor-count ── + registry.register(RegisteredTool( + name: "device-processor-count", + description: "使用 ProcessInfo.processInfo.processorCount 获取处理器核心数", + parameters: "无", + isParameterless: true + ) { _ in + let payload = await devicePayload() + let processorCount = payload["processor_count"] as? Int ?? 0 + return jsonString([ + "success": true, + "status": "succeeded", + "result": "这台设备的处理器核心数是 \(processorCount)。", + "processor_count": processorCount + ]) + }) + + // ── device-identifier-for-vendor ── + registry.register(RegisteredTool( + name: "device-identifier-for-vendor", + description: "使用 UIDevice.current.identifierForVendor 获取当前 App 在该设备上的 vendor 标识", + parameters: "无", + isParameterless: true + ) { _ in + let payload = await devicePayload() + let identifier = payload["identifier_for_vendor"] as? String ?? "" + return jsonString([ + "success": true, + "status": "succeeded", + "result": "当前 App 在这台设备上的 identifierForVendor 是 \(identifier)。", + "identifier_for_vendor": identifier + ]) + }) + } + + // MARK: - Private Helpers + + private static func devicePayload() async -> [String: Any] { + let info = ProcessInfo.processInfo + let device = await MainActor.run { + ( + UIDevice.current.name, + UIDevice.current.model, + UIDevice.current.localizedModel, + UIDevice.current.systemName, + UIDevice.current.systemVersion, + UIDevice.current.identifierForVendor?.uuidString + ) + } + + var payload: [String: Any] = [ + "success": true, + "name": device.0, + "model": device.1, + "localized_model": device.2, + "system_name": device.3, + "system_version": device.4, + "memory_bytes": Double(info.physicalMemory), + "memory_gb": Double(info.physicalMemory) / 1_073_741_824.0, + "processor_count": info.processorCount + ] + + if let identifierForVendor = device.5, !identifierForVendor.isEmpty { + payload["identifier_for_vendor"] = identifierForVendor + } + + return payload + } +} diff --git a/Tools/Handlers/Reminders.swift b/Tools/Handlers/Reminders.swift new file mode 100644 index 0000000..2e19c47 --- /dev/null +++ b/Tools/Handlers/Reminders.swift @@ -0,0 +1,163 @@ +import EventKit +import Foundation + +enum RemindersTools { + + static func register(into registry: ToolRegistry) { + + // ── reminders-create ── + registry.register(RegisteredTool( + name: "reminders-create", + description: "创建新的提醒事项，可写入标题、到期时间和备注", + parameters: "title: 提醒标题, due: ISO 8601 到期时间（可选）, notes: 备注（可选）", + requiredParameters: ["title"] + ) { args in + guard let rawTitle = args["title"] as? String else { + return failurePayload(error: "缺少 title 参数") + } + let title = rawTitle.trimmingCharacters(in: .whitespacesAndNewlines) + guard !title.isEmpty else { + return failurePayload(error: "缺少 title 参数") + } + + let dueRaw = (args["due"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) + if let dueRaw, !dueRaw.isEmpty, parseISO8601Date(dueRaw) == nil { + return failurePayload(error: "due 必须是有效的 ISO 8601 时间字符串") + } + + let dueDate = dueRaw.flatMap(parseISO8601Date) + let notes = (args["notes"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) + + do { + guard try await ToolRegistry.shared.requestAccess(for: .reminders) else { + return failurePayload(error: "未获得提醒事项权限") + } + + guard let calendar = try ensureWritableReminderCalendar() else { + return failurePayload(error: "没有可用于新建提醒事项的可写列表，且无法自动创建提醒列表，请先在系统提醒事项 App 中启用或创建一个列表") + } + + let reminder = EKReminder(eventStore: SystemStores.event) + reminder.calendar = calendar + reminder.title = title + if let dueDate { + reminder.dueDateComponents = reminderDateComponents(from: dueDate) + reminder.addAlarm(EKAlarm(absoluteDate: dueDate)) + } + if let notes, !notes.isEmpty { + reminder.notes = notes + } + + try SystemStores.event.save(reminder, commit: true) + + return successPayload( + result: dueDate != nil + ? "已创建提醒事项\u{201C}\(title)\u{201D}，提醒时间为 \(iso8601String(from: dueDate!))。" + : "已创建提醒事项\u{201C}\(title)\u{201D}。", + extras: [ + "calendarItemId": reminder.calendarItemIdentifier, + "title": title, + "due": dueDate.map { iso8601String(from: $0) } ?? "", + "notes": notes ?? "" + ] + ) + } catch { + return failurePayload(error: "创建提醒事项失败：\(error.localizedDescription)") + } + }) + } + + // MARK: - Private Helpers + + private static func writableReminderCalendar() -> EKCalendar? { + if let calendar = SystemStores.event.defaultCalendarForNewReminders(), + calendar.allowsContentModifications { + return calendar + } + + return SystemStores.event.calendars(for: .reminder) + .first(where: \.allowsContentModifications) + } + + private static func newReminderListTitle() -> String { + let prefersChinese = Locale.preferredLanguages.contains { $0.hasPrefix("zh") } + return prefersChinese ? "PhoneClaw 提醒事项" : "PhoneClaw Reminders" + } + + private static func reminderCalendarCreationSources() -> [EKSource] { + let existingReminderSources = Set( + SystemStores.event.calendars(for: .reminder) + .map(\.source.sourceIdentifier) + ) + + func priority(for source: EKSource) -> Int? { + switch source.sourceType { + case .local: + return existingReminderSources.contains(source.sourceIdentifier) ? 0 : 1 + case .mobileMe: + return existingReminderSources.contains(source.sourceIdentifier) ? 2 : 3 + case .calDAV: + return existingReminderSources.contains(source.sourceIdentifier) ? 4 : 5 + case .exchange: + return existingReminderSources.contains(source.sourceIdentifier) ? 6 : 7 + case .subscribed, .birthdays: + return nil + @unknown default: + return existingReminderSources.contains(source.sourceIdentifier) ? 8 : 9 + } + } + + let prioritizedSources: [(priority: Int, source: EKSource)] = SystemStores.event.sources.compactMap { source -> (priority: Int, source: EKSource)? in + guard let priority = priority(for: source) else { return nil } + return (priority, source) + } + + return prioritizedSources + .sorted { lhs, rhs in + if lhs.priority == rhs.priority { + return lhs.source.title.localizedCaseInsensitiveCompare(rhs.source.title) == .orderedAscending + } + return lhs.priority < rhs.priority + } + .map(\.source) + } + + private static func ensureWritableReminderCalendar() throws -> EKCalendar? { + if let calendar = writableReminderCalendar() { + return calendar + } + + var lastError: Error? + for source in reminderCalendarCreationSources() { + let reminderList = EKCalendar(for: .reminder, eventStore: SystemStores.event) + reminderList.title = newReminderListTitle() + reminderList.source = source + + do { + try SystemStores.event.saveCalendar(reminderList, commit: true) + if reminderList.allowsContentModifications { + return reminderList + } + if let saved = SystemStores.event.calendar(withIdentifier: reminderList.calendarIdentifier), + saved.allowsContentModifications { + return saved + } + } catch { + lastError = error + } + } + + if let lastError { + throw lastError + } + + return nil + } + + private static func reminderDateComponents(from date: Date) -> DateComponents { + Calendar.current.dateComponents( + in: .current, + from: date + ) + } +} diff --git a/Tools/Handlers/Text.swift b/Tools/Handlers/Text.swift new file mode 100644 index 0000000..93319da --- /dev/null +++ b/Tools/Handlers/Text.swift @@ -0,0 +1,47 @@ +import Foundation + +enum TextTools { + + static func register(into registry: ToolRegistry) { + + // ── calculate-hash ── + registry.register(RegisteredTool( + name: "calculate-hash", + description: "计算文本的哈希值", + parameters: "text: 要计算哈希的文本", + requiredParameters: ["text"] + ) { args in + guard let text = args["text"] as? String else { + return failurePayload(error: "缺少 text 参数") + } + let hash = text.hashValue + return successPayload( + result: "文本\u{201C}\(text)\u{201D}的哈希值是 \(hash)。", + extras: [ + "input": text, + "hash": hash + ] + ) + }) + + // ── text-reverse ── + registry.register(RegisteredTool( + name: "text-reverse", + description: "翻转文本", + parameters: "text: 要翻转的文本", + requiredParameters: ["text"] + ) { args in + guard let text = args["text"] as? String else { + return failurePayload(error: "缺少 text 参数") + } + let reversed = String(text.reversed()) + return successPayload( + result: "翻转结果：\(reversed)", + extras: [ + "original": text, + "reversed": reversed + ] + ) + }) + } +} diff --git a/Tools/Helpers.swift b/Tools/Helpers.swift new file mode 100644 index 0000000..284eb36 --- /dev/null +++ b/Tools/Helpers.swift @@ -0,0 +1,94 @@ +import Foundation + +// MARK: - JSON Utilities + +func jsonEscape(_ str: String) -> String { + str.replacingOccurrences(of: "\\", with: "\\\\") + .replacingOccurrences(of: "\"", with: "\\\"") + .replacingOccurrences(of: "\n", with: "\\n") + .replacingOccurrences(of: "\t", with: "\\t") +} + +func jsonString(_ object: Any) -> String { + guard JSONSerialization.isValidJSONObject(object), + let data = try? JSONSerialization.data(withJSONObject: object), + let string = String(data: data, encoding: .utf8) else { + return "{\"success\": false, \"error\": \"JSON 编码失败\"}" + } + return string +} + +// MARK: - Tool Result Payloads + +func successPayload( + result: String, + extras: [String: Any] = [:] +) -> String { + var payload = extras + payload["success"] = true + payload["status"] = "succeeded" + payload["result"] = result + return jsonString(payload) +} + +func failurePayload(error: String, extras: [String: Any] = [:]) -> String { + var payload = extras + payload["success"] = false + payload["status"] = "failed" + payload["error"] = error + return jsonString(payload) +} + +// MARK: - Date Helpers + +func parseISO8601Date(_ raw: String) -> Date? { + let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { return nil } + + let isoFormatters: [ISO8601DateFormatter] = [ + { + let formatter = ISO8601DateFormatter() + formatter.timeZone = .current + formatter.formatOptions = [.withInternetDateTime, .withFractionalSeconds] + return formatter + }(), + { + let formatter = ISO8601DateFormatter() + formatter.timeZone = .current + formatter.formatOptions = [.withInternetDateTime] + return formatter + }() + ] + + for formatter in isoFormatters { + if let date = formatter.date(from: trimmed) { + return date + } + } + + let formats = [ + "yyyy-MM-dd'T'HH:mm:ss", + "yyyy-MM-dd'T'HH:mm", + "yyyy-MM-dd HH:mm:ss", + "yyyy-MM-dd HH:mm" + ] + + for format in formats { + let formatter = DateFormatter() + formatter.locale = Locale(identifier: "en_US_POSIX") + formatter.timeZone = .current + formatter.dateFormat = format + if let date = formatter.date(from: trimmed) { + return date + } + } + + return nil +} + +func iso8601String(from date: Date) -> String { + let formatter = ISO8601DateFormatter() + formatter.timeZone = .current + formatter.formatOptions = [.withInternetDateTime] + return formatter.string(from: date) +} diff --git a/Tools/Stores.swift b/Tools/Stores.swift new file mode 100644 index 0000000..a707424 --- /dev/null +++ b/Tools/Stores.swift @@ -0,0 +1,9 @@ +import Contacts +import EventKit + +/// 共享系统 store 单例，供所有 ToolHandler 和 AppPermissions 使用。 +/// 解耦 ToolRegistry，避免重复实例。 +enum SystemStores { + static let event = EKEventStore() + static let contacts = CNContactStore() +} diff --git a/Tools/ToolRegistry.swift b/Tools/ToolRegistry.swift new file mode 100644 index 0000000..d004d8a --- /dev/null +++ b/Tools/ToolRegistry.swift @@ -0,0 +1,189 @@ +import Foundation + +// MARK: - 原生工具注册表 +// +// 所有原生 API 封装集中注册在这里。 +// SKILL.md 通过 allowed-tools 字段引用工具名。 + +struct RegisteredTool { + let name: String + let description: String + let parameters: String + + /// 必填参数名列表，用于泛化参数校验（替代 per-tool switch）。 + /// 空数组表示无参数或无需强制校验。 + let requiredParameters: [String] + + /// 至少要提供其中一个参数名；用于表达 search/delete 这类"多选一"契约。 + let requiredAnyOfParameters: [String] + + /// 该工具名的别名（纯格式层面），用于泛化 canonicalToolName。 + /// 例如 ["contacts_delete", "contacts-delete-contact"] + let aliases: [String] + + /// 该工具是否完全无参，可直接执行。 + let isParameterless: Bool + + /// 如果为 true，工具执行后直接使用 result 字段作为最终回答，不再走 LLM follow-up。 + let skipFollowUp: Bool + + let execute: ([String: Any]) async throws -> String + + init( + name: String, + description: String, + parameters: String, + requiredParameters: [String] = [], + requiredAnyOfParameters: [String] = [], + aliases: [String] = [], + isParameterless: Bool = false, + skipFollowUp: Bool = false, + execute: @escaping ([String: Any]) async throws -> String + ) { + self.name = name + self.description = description + self.parameters = parameters + self.requiredParameters = requiredParameters + self.requiredAnyOfParameters = requiredAnyOfParameters + self.aliases = aliases + self.isParameterless = isParameterless + self.skipFollowUp = skipFollowUp + self.execute = execute + } + + func validates(arguments: [String: Any]) -> Bool { + if isParameterless { + return arguments.isEmpty + } + + for parameter in requiredParameters { + guard Self.hasMeaningfulValue(arguments[parameter]) else { + return false + } + } + + if !requiredAnyOfParameters.isEmpty, + !requiredAnyOfParameters.contains(where: { Self.hasMeaningfulValue(arguments[$0]) }) { + return false + } + + return true + } + + private static func hasMeaningfulValue(_ value: Any?) -> Bool { + switch value { + case let string as String: + return !string.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty + case let array as [Any]: + return !array.isEmpty + case let dictionary as [String: Any]: + return !dictionary.isEmpty + case nil: + return false + default: + return true + } + } +} + +class ToolRegistry { + static let shared = ToolRegistry() + + private var tools: [String: RegisteredTool] = [:] + /// alias → canonical tool name + private var aliasIndex: [String: String] = [:] + + private init() { + registerBuiltInTools() + } + + // MARK: - 公开接口 + + func register(_ tool: RegisteredTool) { + tools[tool.name] = tool + for alias in tool.aliases { + aliasIndex[alias] = tool.name + } + } + + func find(name: String) -> RegisteredTool? { + if let tool = tools[name] { + return tool + } + if let resolved = canonicalName(for: name) { + return tools[resolved] + } + return nil + } + + func execute(name: String, args: [String: Any]) async throws -> String { + guard let tool = find(name: name) else { + return "{\"success\": false, \"error\": \"未知工具: \(name)\"}" + } + return try await tool.execute(args) + } + + /// 根据名称列表获取工具（用于 SKILL.md 的 allowed-tools） + func toolsFor(names: [String]) -> [RegisteredTool] { + names.compactMap { find(name: $0) } + } + + /// 返回 true 如果该工具已注册 + func hasToolNamed(_ name: String) -> Bool { + find(name: name) != nil + } + + /// 所有已注册的工具名 + var allToolNames: [String] { + Array(tools.keys).sorted() + } + + // MARK: - 泛化查询（替代 AgentEngine 中的 per-tool switch） + + /// 将别名或格式变体解析为注册的 canonical tool name。 + func canonicalName(for rawName: String) -> String? { + let normalized = rawName + .trimmingCharacters(in: .whitespacesAndNewlines) + .replacingOccurrences(of: "_", with: "-") + .lowercased() + if tools[normalized] != nil { return normalized } + if let mapped = aliasIndex[normalized] { return mapped } + return nil + } + + /// 某个工具的必填参数列表。 + func requiredParams(for toolName: String) -> [String] { + find(name: toolName)?.requiredParameters ?? [] + } + + /// 某个工具的"至少一个参数"候选列表。 + func requiredAnyOfParams(for toolName: String) -> [String] { + find(name: toolName)?.requiredAnyOfParameters ?? [] + } + + /// 某个工具是否为无参工具。 + func isParameterlessTool(named toolName: String) -> Bool { + find(name: toolName)?.isParameterless ?? false + } + + /// 统一的参数契约校验。 + func validatesArguments(_ arguments: [String: Any], for toolName: String) -> Bool { + find(name: toolName)?.validates(arguments: arguments) ?? false + } + + /// 某个工具是否声明了跳过 LLM follow-up。 + func shouldSkipFollowUp(for toolName: String) -> Bool { + find(name: toolName)?.skipFollowUp ?? false + } + + // MARK: - 内置工具注册 + + private func registerBuiltInTools() { + ClipboardTools.register(into: self) + DeviceTools.register(into: self) + TextTools.register(into: self) + CalendarTools.register(into: self) + RemindersTools.register(into: self) + ContactsTools.register(into: self) + } +} diff --git a/UI/ContentView.swift b/UI/ContentView.swift index e03f83f..91647ba 100644 --- a/UI/ContentView.swift +++ b/UI/ContentView.swift @@ -290,17 +290,33 @@ struct ContentView: View { // MARK: - Skill 快捷标签 + /// chip 显示用的简写标签 (UI 表现层选择, 不影响核心数据)。 + /// key = SkillEntry.id, value = 在 chip pill 上显示的短动作短语。 + /// 没在表里的 skill 会 fallback 到 skill.displayName。 + /// chip 实际发送的 prompt 仍然是 skill.samplePrompt (完整、自包含)。 + private static let chipShortLabels: [String: String] = [ + "calendar": "帮我创建日程", + "contacts": "帮我保存联系人", + "reminders": "帮我创建提醒事项", + "device": "查询我手机信息", + ] + + /// chip 隐藏名单 (UI 表现层选择): 这些 skill 不会出现在欢迎页 chip 列表里, + /// 但仍然在引擎里启用、可被用户手动输入触发。 + private static let chipHiddenSkillIds: Set = ["text"] + private var skillChips: some View { ScrollView(.horizontal, showsIndicators: false) { HStack(spacing: 8) { - ForEach(engine.enabledSkillInfos, id: \.name) { skill in + ForEach(engine.enabledSkillInfos.filter { !Self.chipHiddenSkillIds.contains($0.name) }, id: \.name) { skill in + let label = Self.chipShortLabels[skill.name] ?? skill.displayName Button { inputText = skill.samplePrompt Task { await send() } } label: { HStack(spacing: 5) { Image(systemName: skill.icon).font(.system(size: 11)) - Text(skill.displayName).font(.system(size: 12, weight: .medium)) + Text(label).font(.system(size: 12, weight: .medium)) } .foregroundStyle(Theme.textSecondary) .padding(.horizontal, 12) @@ -682,9 +698,17 @@ struct UserBubble: View { Text(text) .font(.system(size: 15)) .foregroundStyle(Theme.userText) + .textSelection(.enabled) .padding(.horizontal, 16) .padding(.vertical, 10) .background(Theme.userBubble, in: UserBubbleShape()) + .contextMenu { + Button { + UIPasteboard.general.string = text + } label: { + Label("复制", systemImage: "doc.on.doc") + } + } } } } @@ -1254,6 +1278,13 @@ struct AIResponseView: View { .fixedSize(horizontal: false, vertical: true) .padding(.leading, 4) .animation(nil, value: text) + .contextMenu { + Button { + UIPasteboard.general.string = text + } label: { + Label("复制", systemImage: "doc.on.doc") + } + } } } .animation(.easeInOut(duration: 0.3), value: block.skills.count) @@ -1460,6 +1491,16 @@ struct SpinnerIcon: View { // MARK: - 思考动画 +/// 演示 chip 按钮样式: press 时 spring 缩放 + 轻微透明度变化, 让卡片有"活气"。 +struct SkillCardButtonStyle: ButtonStyle { + func makeBody(configuration: Configuration) -> some View { + configuration.label + .scaleEffect(configuration.isPressed ? 0.96 : 1.0) + .opacity(configuration.isPressed ? 0.85 : 1.0) + .animation(.spring(response: 0.32, dampingFraction: 0.72), value: configuration.isPressed) + } +} + struct ThinkingIndicator: View { @State private var active = 0 let timer = Timer.publish(every: 0.45, on: .main, in: .common).autoconnect() diff --git a/UI/SkillsManagerView.swift b/UI/SkillsManagerView.swift index 2df696f..2fbc0fd 100644 --- a/UI/SkillsManagerView.swift +++ b/UI/SkillsManagerView.swift @@ -43,7 +43,7 @@ struct SkillsManagerView: View { isExpanded: expandedSkills.contains(engine.skillEntries[i].id), onToggleExpand: { toggleExpand(engine.skillEntries[i].id) }, onSave: { content in - try? engine.skillLoader.saveSkill(skillId: engine.skillEntries[i].id, content: content) + try? engine.skillRegistry.saveSkill(skillId: engine.skillEntries[i].id, content: content) engine.reloadSkills() } ) diff --git a/doc/architecture-decisions.md b/doc/architecture-decisions.md new file mode 100644 index 0000000..e77b419 --- /dev/null +++ b/doc/architecture-decisions.md @@ -0,0 +1,116 @@ +# PhoneClaw Architecture Decision Records (ADR) + +> 日期：2026-04-07 +> 上下文：五轮交叉评审（对比 OpenClaw / Claude Skills / Vera），确认端侧小模型推理的架构边界。 + +--- + +## ADR-001：拒绝统一 ReAct 循环取代多路分叉 + +**决策**：保留 processInput 的 5 路分流（VLM / Planner / Preflight / Agent / Light）。 + +**理由**： +- 端侧每次 LLM 调用延迟 3-8 秒，统一 ReAct 会将所有请求（包括纯聊天）暴露在 Skill 注入开销下 +- Light 路径不注入任何 Skill 信息，节省 context window +- Preflight 路径跳过 LLM，<100ms 响应高频操作 +- 2B 模型无法在 ReAct 循环中可靠地自主链式调用多个工具 +- Vera 的统一 ReAct 依赖云端 100B+ 模型（200-500ms/次），前提不同 + +**重评条件**：端侧模型能力达到可靠自主链式调用（预计需要 8B+ 量级）且推理延迟降至 <1s。 + +--- + +## ADR-002：拒绝 [STATUS:COMPLETED] 终止标记 + +**决策**：继续使用 `parseToolCall() == nil` 作为工具链终止条件。 + +**理由**： +- Gemma 4 E2B/E4B 对受约束格式的遵从度不足（空格/大小写/位置不稳定） +- 强制要求输出 `[STATUS:COMPLETED]` 引入新的故障面（不输出 / 位置错 / 格式变体） +- 现有终止逻辑（无 tool_call + maxRounds=10 + validation failure + shouldSkipToolFollowUp）已足够健壮 + +**重评条件**：Gemma 格式遵从度显著提升，或切换到支持结构化输出的模型。 + +--- + +## ADR-003：拒绝合并 Planner Selection + Planning 为单一 Prompt + +**决策**：保留 Selection → Planning 两步 LLM 调用。 + +**理由**： +- 分两步是"外置 CoT"——让小模型一次只做一件事，更可靠 +- ≤3 个候选时 Selection 已走本地快捷路径（L732），不调 LLM +- 合并后单个 prompt 更长，JSON 遵从度下降 +- Selection 有独立的 `validateSkillSelection()` 验证，失败可以便宜重试；合并后重试成本翻倍 + +**重评条件**：埋点证实 `matchedSkills > 3` 触发频率 > 5%（此时改为 Selection 短路跳过，不合并 prompt）。 + +--- + +## ADR-004：拒绝 E2B Keyword Sequential Fallback + +**决策**：E2B 遇到多 Skill 请求时不尝试自动串行执行。 + +**理由**： +- 步骤间参数传递需要 intent 信息（"把 step1 的什么传给 step2"），仅靠 keyword 无法确定 +- 让 2B 模型用 LLM 来补全参数传递 → 回到了"2B 能否自主链式调用"的问题，形成死循环 +- "降级执行第一个 + 明确告知"是更诚实的 UX 选择 + +**重评条件**：2B 模型 CoT 能力提升到能可靠理解步骤间依赖关系。 + +--- + +## ADR-005：拒绝硬编码模型 ID 做路径分岔 + +**决策**：使用 `supportsStructuredPlanning` 等能力标志替代 `!= e2bModelID`。 + +**理由**： +- 硬编码模型 ID 是技术债，加新模型需要改逻辑代码 +- 能力标志只需修改模型元数据表，未来可扩展 + +**状态**：P1-a，已纳入 v4 实施计划。 + +--- + +## ADR-006：拒绝 YAML 声明式正则快速路径（parameter-hints） + +**决策**：`heuristicArgumentsForTool` 的 per-tool 正则保持 Swift 硬编码，不迁移到 SKILL.md。 + +**理由**： +- 中文正则在 YAML 里不可维护（无 IDE 补全、无类型安全、无测试覆盖） +- 当前 Swift 正则是反复调参的产物（如 contacts-search 有两层 fallback），YAML 声明式无法复现 +- ReDoS 风险需要额外防护层 +- 内置 vs 第三方的体验分层是合理的产品取舍（参考 Anthropic Claude） + +**推迟到 N≥15 的改动**： +1. **top-K skill 注入**：system prompt 改为只注入 top-K 匹配项 +2. **`quick-paths` frontmatter**（仅 keyword，无 regex）：第三方 Skill 声明关键词→工具名映射 + +**永久拒绝**：YAML regex `parameter-hints`。 + +**不重评条件**：若 Skill 数量长期保持 <15，推迟项永不重评。 + +--- + +## ADR-007：无 Prompt Prefix Caching + +**决策**：不针对 KV cache 前缀命中率调整 prompt 结构。 + +**理由**： +- `MLXLocalLLMService.generateStream` 每次推理前调用 `MLX.GPU.clearCache()` +- 每次推理从头构建完整 KV cache，prompt 前缀稳定性不影响延迟 +- 任何"缓存友好 prompt 重构"需先实现实际的 prefix caching + +**重评条件**：PhoneClaw 实现跨请求 KV cache 持久化 + 前缀匹配。 + +--- + +## 三级 Skill 体验模型 + +| 级别 | 来源 | LLM 次数 | 响应 | +|------|------|---------|------| +| 一等 | 内置 Swift heuristic | 0 | <1s | +| 二等（未来） | 第三方 + `quick-paths` keyword | 1 | 3-8s | +| 三等 | 第三方裸装 | 2 | 6-16s | + +这不是缺陷，是取舍。内置 Skill 经过手工调优，不可迁移到 YAML。第三方 Skill 走 LLM 路径功能完整，只是慢一些。