feat: 增强IPC序列化与数据一致性,修复备份恢复机制,添加安全序列化函数以清理IPC传递的Vue响应式对象,确保数据完整性
This commit is contained in:
1 parent
b5f3b263f7
commit
ea77ff44ec
8 files changed
+742
-83
No files matched your search
+2
-1
@@ -177,4 +177,5 @@ storybook-static
|
||||
*.backup
|
||||
*.bak
|
||||
|
||||
prompt-optimizer-datas
|
||||
prompt-optimizer-datas
|
||||
prompt-optimizer-data
|
||||
@@ -195,3 +195,181 @@ app.on('before-quit', async (event) => {
|
||||
- 建立完整的异常处理和状态重置机制
|
||||
|
||||
这些补充修复确保了FileStorageProvider在各种异常情况下都能正常工作,并且应用能够可靠地退出。
|
||||
|
||||
## 🛡️ 数据安全性增强 (2025-07-06)
|
||||
|
||||
### 问题发现:备份恢复安全隐患
|
||||
|
||||
在审查恢复逻辑时发现了一个严重的数据安全问题:
|
||||
|
||||
**问题场景**:
|
||||
- 主文件 `storage.json` 损坏
|
||||
- 备份文件 `storage.json.backup` 完好
|
||||
- 系统进入恢复流程
|
||||
|
||||
**危险流程**:
|
||||
```
|
||||
从备份恢复 → saveToFile() → createBackup() → 将损坏的主文件覆盖完好的备份!
|
||||
```
|
||||
|
||||
如果后续的原子写入也失败,将导致数据永久丢失。
|
||||
|
||||
### 解决方案:智能恢复机制
|
||||
|
||||
#### 1. 新增安全保存方法
|
||||
```typescript
|
||||
/**
|
||||
* 专门用于恢复的保存方法,避免覆盖完好的备份
|
||||
*/
|
||||
private async saveToFileWithoutBackup(): Promise<void> {
|
||||
const data = Object.fromEntries(this.data);
|
||||
const jsonString = JSON.stringify(data, null, 2);
|
||||
|
||||
// 验证数据完整性
|
||||
if (!this.validateJSON(jsonString)) {
|
||||
throw new StorageError('Generated JSON is invalid', 'write');
|
||||
}
|
||||
|
||||
// 直接原子写入,不创建备份
|
||||
await this.atomicWrite(jsonString);
|
||||
}
|
||||
```
|
||||
|
||||
#### 2. 改进的恢复流程
|
||||
```typescript
|
||||
private async loadFromFileWithRecovery(): Promise<void> {
|
||||
// 1. 尝试从主文件加载
|
||||
const mainResult = await this.tryLoadFromFile(this.filePath, 'main');
|
||||
if (mainResult.success) {
|
||||
this.data = mainResult.data!;
|
||||
await this.createBackup();
|
||||
return;
|
||||
}
|
||||
|
||||
// 2. 尝试从备份文件加载
|
||||
const backupResult = await this.tryLoadFromFile(this.backupPath, 'backup');
|
||||
if (backupResult.success) {
|
||||
this.data = backupResult.data!;
|
||||
|
||||
// 关键:使用专门的方法避免覆盖备份
|
||||
await this.saveToFileWithoutBackup();
|
||||
|
||||
// 主文件恢复成功后,重新创建备份
|
||||
await this.createBackup();
|
||||
return;
|
||||
}
|
||||
|
||||
// 3. 区分首次运行和数据损坏
|
||||
if (!await this.fileExists(this.filePath) && !await this.fileExists(this.backupPath)) {
|
||||
// 首次运行
|
||||
this.data = new Map();
|
||||
await this.saveToFile();
|
||||
} else {
|
||||
// 严重错误:文件存在但都损坏
|
||||
throw new StorageError('Storage corruption detected', 'read');
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. 原子性updateData增强
|
||||
|
||||
为防止并发操作导致的数据不一致,增强了updateData的原子性:
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* 原子性数据更新 - 增强版
|
||||
*/
|
||||
async updateData<T>(key: string, modifier: (currentValue: T | null) => T): Promise<void> {
|
||||
await this.ensureInitialized();
|
||||
|
||||
// 使用更新锁确保原子性
|
||||
const currentLock = this.updateLock;
|
||||
let resolveLock: () => void;
|
||||
|
||||
this.updateLock = new Promise<void>((resolve) => {
|
||||
resolveLock = resolve;
|
||||
});
|
||||
|
||||
try {
|
||||
await currentLock;
|
||||
await this.performAtomicUpdate(key, modifier);
|
||||
} finally {
|
||||
resolveLock!();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 执行原子更新操作
|
||||
*/
|
||||
private async performAtomicUpdate<T>(key: string, modifier: (currentValue: T | null) => T): Promise<void> {
|
||||
// 重新从存储读取最新数据,确保数据一致性
|
||||
const latestData = await this.getLatestData<T>(key);
|
||||
|
||||
// 应用修改
|
||||
const newValue = modifier(latestData);
|
||||
|
||||
// 验证新值
|
||||
this.validateValue(newValue);
|
||||
|
||||
// 写入新值
|
||||
this.data.set(key, JSON.stringify(newValue));
|
||||
this.scheduleWrite();
|
||||
}
|
||||
```
|
||||
|
||||
### 安全保障机制
|
||||
|
||||
#### 1. 数据完整性保障
|
||||
- **备份保护**:恢复时不会覆盖完好的备份文件
|
||||
- **智能恢复**:区分首次运行和数据损坏情况
|
||||
- **多层恢复**:主文件→备份文件→错误处理
|
||||
|
||||
#### 2. 原子性保障
|
||||
- **更新锁机制**:防止并发操作导致的数据不一致
|
||||
- **原子写入**:使用临时文件+重命名确保写入原子性
|
||||
- **事务性操作**:读-修改-写操作的完整性
|
||||
|
||||
#### 3. 错误处理增强
|
||||
- **错误分类**:区分不同类型的错误(首次运行、数据损坏、读写失败)
|
||||
- **优雅降级**:各种异常情况下的合理处理
|
||||
- **状态重置**:异常情况下的状态恢复机制
|
||||
|
||||
### 测试验证
|
||||
|
||||
#### 备份保护测试
|
||||
```typescript
|
||||
it('should not overwrite good backup during recovery', async () => {
|
||||
// 模拟损坏的主文件和完好的备份
|
||||
mockFs.readFile
|
||||
.mockResolvedValueOnce('{ invalid json') // 损坏的主文件
|
||||
.mockResolvedValueOnce(JSON.stringify(goodData)); // 完好的备份
|
||||
|
||||
await provider.getItem('test');
|
||||
|
||||
// 验证没有覆盖备份
|
||||
const dangerousCopyCall = mockFs.copyFile.mock.calls.find(call =>
|
||||
call[0] === mainPath && call[1] === backupPath
|
||||
);
|
||||
expect(dangerousCopyCall).toBeUndefined();
|
||||
});
|
||||
```
|
||||
|
||||
#### 并发安全测试
|
||||
```typescript
|
||||
it('should handle concurrent updates safely', async () => {
|
||||
const promises = [
|
||||
provider.updateData('key1', () => 'value1'),
|
||||
provider.updateData('key2', () => 'value2'),
|
||||
provider.updateData('key3', () => 'value3')
|
||||
];
|
||||
|
||||
await Promise.all(promises);
|
||||
|
||||
// 验证所有更新都成功
|
||||
expect(await provider.getItem('key1')).toBe('value1');
|
||||
expect(await provider.getItem('key2')).toBe('value2');
|
||||
expect(await provider.getItem('key3')).toBe('value3');
|
||||
});
|
||||
```
|
||||
|
||||
这些增强确保了FileStorageProvider在各种复杂场景下的数据安全性和操作原子性。
|
||||
+10
-1
@@ -77,6 +77,13 @@
|
||||
- 跨环境异步接口统一
|
||||
- preload.js架构规范化
|
||||
|
||||
- **[115-ipc-serialization-fixes](./115-ipc-serialization-fixes/)** - IPC序列化修复与数据一致性 🔄
|
||||
- Vue响应式对象IPC序列化统一处理
|
||||
- safeSerialize函数实现
|
||||
- 业务逻辑层数据一致性修复
|
||||
- 模型数据丢失问题解决
|
||||
- 双重保护机制建立
|
||||
|
||||
## ⚙️ 服务重构系列
|
||||
|
||||
### 全面重构
|
||||
@@ -89,6 +96,7 @@
|
||||
- 实现FileStorageProvider替代内存存储
|
||||
- 完整的数据持久化解决方案
|
||||
- 高性能文件I/O和错误恢复机制
|
||||
- 数据安全性增强:智能恢复机制、备份保护、原子性操作
|
||||
|
||||
- **[116-desktop-packaging-optimization](./116-desktop-packaging-optimization/)** - 桌面应用打包优化 📦
|
||||
- 从portable模式改为ZIP压缩包模式
|
||||
@@ -101,7 +109,8 @@
|
||||
- **启动问题** → 102-web-architecture-refactor
|
||||
- **显示异常** → 112-desktop-ipc-fixes
|
||||
- **存储问题** → 110-desktop-indexeddb-fix, 114-desktop-file-storage, 116-desktop-packaging-optimization
|
||||
- **序列化错误** → 112-desktop-ipc-fixes
|
||||
- **数据一致性问题** → 114-desktop-file-storage, 115-ipc-serialization-fixes
|
||||
- **序列化错误** → 112-desktop-ipc-fixes, 115-ipc-serialization-fixes
|
||||
- **应用退出问题** → 114-desktop-file-storage
|
||||
- **语言设置问题** → 112-desktop-ipc-fixes
|
||||
- **布局问题** → 108-layout-system
|
||||
|
||||
@@ -178,17 +178,25 @@ export class ModelManager implements IModelManager {
|
||||
async addModel(key: string, config: ModelConfig): Promise<void> {
|
||||
await this.ensureInitialized();
|
||||
this.validateConfig(config);
|
||||
|
||||
|
||||
await this.storage.updateData<Record<string, ModelConfig>>(
|
||||
this.storageKey,
|
||||
(currentModels) => {
|
||||
const models = currentModels || {};
|
||||
// 关键修复:使用内存中的完整模型列表作为基础
|
||||
const models = { ...this.models };
|
||||
|
||||
// 如果存储中有数据,合并到内存状态中
|
||||
if (currentModels) {
|
||||
Object.assign(models, currentModels);
|
||||
}
|
||||
|
||||
if (models[key]) {
|
||||
throw new ModelConfigError(`Model ${key} already exists`);
|
||||
}
|
||||
|
||||
return {
|
||||
...models,
|
||||
[key]: {
|
||||
[key]: {
|
||||
...config,
|
||||
// Deep copy llmParams to avoid reference sharing
|
||||
...(config.llmParams && { llmParams: { ...config.llmParams } })
|
||||
@@ -196,9 +204,9 @@ export class ModelManager implements IModelManager {
|
||||
};
|
||||
}
|
||||
);
|
||||
|
||||
|
||||
// 更新内存状态
|
||||
this.models[key] = {
|
||||
this.models[key] = {
|
||||
...config,
|
||||
// Deep copy llmParams to avoid reference sharing
|
||||
...(config.llmParams && { llmParams: { ...config.llmParams } })
|
||||
@@ -211,25 +219,40 @@ export class ModelManager implements IModelManager {
|
||||
async updateModel(key: string, config: Partial<ModelConfig>): Promise<void> {
|
||||
await this.ensureInitialized();
|
||||
let updatedConfig: ModelConfig | undefined;
|
||||
|
||||
|
||||
await this.storage.updateData<Record<string, ModelConfig>>(
|
||||
this.storageKey,
|
||||
(currentModels) => {
|
||||
const models = currentModels || {};
|
||||
|
||||
// 关键修复:确保始终基于完整的模型数据进行更新
|
||||
// 如果存储数据不完整,使用内存中的完整数据作为基础
|
||||
let models = currentModels;
|
||||
if (!models || Object.keys(models).length === 0) {
|
||||
console.warn('[ModelManager] Storage data is empty, using memory data as base');
|
||||
models = { ...this.models };
|
||||
} else {
|
||||
// 确保存储数据包含所有默认模型
|
||||
const defaults = this.getDefaultModels();
|
||||
for (const [defaultKey, defaultConfig] of Object.entries(defaults)) {
|
||||
if (!models[defaultKey]) {
|
||||
console.log(`[ModelManager] Adding missing default model to storage: ${defaultKey}`);
|
||||
models[defaultKey] = defaultConfig;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 如果模型不存在,检查是否是内置模型
|
||||
if (!models[key]) {
|
||||
if (!defaultModels[key]) {
|
||||
throw new ModelConfigError(`Model ${key} does not exist`);
|
||||
}
|
||||
// 如果是内置模型但尚未配置,创建初始配置
|
||||
models[key] = {
|
||||
models[key] = {
|
||||
...defaultModels[key],
|
||||
// Deep copy llmParams to avoid reference sharing
|
||||
...(defaultModels[key].llmParams && { llmParams: { ...defaultModels[key].llmParams } })
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
// 合并配置时保留原有 enabled 状态
|
||||
updatedConfig = {
|
||||
...models[key],
|
||||
@@ -253,13 +276,14 @@ export class ModelManager implements IModelManager {
|
||||
this.validateConfig(updatedConfig);
|
||||
}
|
||||
|
||||
// 返回完整的模型数据,确保所有模型都被保留
|
||||
return {
|
||||
...models,
|
||||
[key]: updatedConfig
|
||||
};
|
||||
}
|
||||
);
|
||||
|
||||
|
||||
// 只更新特定模型的内存状态,不重新加载全部数据
|
||||
if (updatedConfig) {
|
||||
this.models[key] = updatedConfig;
|
||||
@@ -274,7 +298,14 @@ export class ModelManager implements IModelManager {
|
||||
await this.storage.updateData<Record<string, ModelConfig>>(
|
||||
this.storageKey,
|
||||
(currentModels) => {
|
||||
const models = currentModels || {};
|
||||
// 关键修复:使用内存中的完整模型列表作为基础
|
||||
const models = { ...this.models };
|
||||
|
||||
// 如果存储中有数据,合并到内存状态中
|
||||
if (currentModels) {
|
||||
Object.assign(models, currentModels);
|
||||
}
|
||||
|
||||
if (!models[key]) {
|
||||
throw new ModelConfigError(`Model ${key} does not exist`);
|
||||
}
|
||||
@@ -295,7 +326,14 @@ export class ModelManager implements IModelManager {
|
||||
await this.storage.updateData<Record<string, ModelConfig>>(
|
||||
this.storageKey,
|
||||
(currentModels) => {
|
||||
const models = currentModels || {};
|
||||
// 关键修复:使用内存中的完整模型列表作为基础
|
||||
const models = { ...this.models };
|
||||
|
||||
// 如果存储中有数据,合并到内存状态中
|
||||
if (currentModels) {
|
||||
Object.assign(models, currentModels);
|
||||
}
|
||||
|
||||
if (!models[key]) {
|
||||
throw new ModelConfigError(`Unknown model: ${key}`);
|
||||
}
|
||||
@@ -312,7 +350,7 @@ export class ModelManager implements IModelManager {
|
||||
};
|
||||
}
|
||||
);
|
||||
|
||||
|
||||
// 更新内存状态 - 确保模型存在
|
||||
if (this.models[key]) {
|
||||
this.models[key].enabled = true;
|
||||
@@ -327,7 +365,14 @@ export class ModelManager implements IModelManager {
|
||||
await this.storage.updateData<Record<string, ModelConfig>>(
|
||||
this.storageKey,
|
||||
(currentModels) => {
|
||||
const models = currentModels || {};
|
||||
// 关键修复:使用内存中的完整模型列表作为基础
|
||||
const models = { ...this.models };
|
||||
|
||||
// 如果存储中有数据,合并到内存状态中
|
||||
if (currentModels) {
|
||||
Object.assign(models, currentModels);
|
||||
}
|
||||
|
||||
if (!models[key]) {
|
||||
throw new ModelConfigError(`Unknown model: ${key}`);
|
||||
}
|
||||
|
||||
@@ -4,26 +4,32 @@ import { IStorageProvider } from './types';
|
||||
import { StorageError } from './errors';
|
||||
|
||||
/**
|
||||
* 基于文件的存储提供器
|
||||
* 基于文件的存储提供器 - 增强版
|
||||
* 专为Electron桌面环境设计,使用JSON文件持久化存储数据
|
||||
*
|
||||
*
|
||||
* 特性:
|
||||
* - 延迟写入优化性能,减少I/O操作
|
||||
* - 内存缓存提供快速读取
|
||||
* - 原子写入确保数据完整性
|
||||
* - 关键时刻立即写入保证数据安全
|
||||
* - 数据备份和智能恢复机制
|
||||
* - 原子性updateData操作
|
||||
* - 严格的初始化控制
|
||||
*/
|
||||
export class FileStorageProvider implements IStorageProvider {
|
||||
private filePath: string;
|
||||
private backupPath: string;
|
||||
private data: Map<string, string> = new Map();
|
||||
private writeTimeout: NodeJS.Timeout | null = null;
|
||||
private isDirty: boolean = false;
|
||||
private writeLock: Promise<void> = Promise.resolve();
|
||||
private updateLock: Promise<void> = Promise.resolve();
|
||||
private initialized: boolean = false;
|
||||
|
||||
private initializationPromise: Promise<void> | null = null;
|
||||
|
||||
// 配置常量
|
||||
private readonly WRITE_DELAY = 500; // 500ms延迟写入
|
||||
private readonly TEMP_FILE_SUFFIX = '.tmp';
|
||||
private readonly BACKUP_FILE_SUFFIX = '.backup';
|
||||
private readonly MAX_FLUSH_TIME = 3000; // 最大flush时间:3秒
|
||||
private flushAttempts = 0; // flush尝试次数
|
||||
private readonly MAX_FLUSH_ATTEMPTS = 3; // 最大flush尝试次数
|
||||
@@ -34,78 +40,224 @@ export class FileStorageProvider implements IStorageProvider {
|
||||
}
|
||||
|
||||
this.filePath = path.join(userDataPath, 'storage.json');
|
||||
this.backupPath = path.join(userDataPath, 'storage.json' + this.BACKUP_FILE_SUFFIX);
|
||||
}
|
||||
|
||||
/**
|
||||
* 确保存储已初始化
|
||||
* 确保存储已初始化 - 增强版
|
||||
* 使用单例模式确保初始化只执行一次
|
||||
*/
|
||||
private async ensureInitialized(): Promise<void> {
|
||||
if (!this.initialized) {
|
||||
await this.initialize();
|
||||
if (this.initialized) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (this.initializationPromise) {
|
||||
await this.initializationPromise;
|
||||
return;
|
||||
}
|
||||
|
||||
this.initializationPromise = this.initialize();
|
||||
await this.initializationPromise;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* 初始化存储,加载现有数据
|
||||
* 初始化存储,加载现有数据 - 增强版
|
||||
* 包含智能恢复机制
|
||||
*/
|
||||
private async initialize(): Promise<void> {
|
||||
try {
|
||||
await this.loadFromFile();
|
||||
console.log('[FileStorage] Initializing storage...');
|
||||
await this.loadFromFileWithRecovery();
|
||||
this.initialized = true;
|
||||
console.log('[FileStorage] Storage initialized successfully');
|
||||
} catch (error) {
|
||||
const errorMessage = error instanceof Error ? error.message : 'Unknown error';
|
||||
console.error('[FileStorage] Failed to initialize:', errorMessage);
|
||||
throw new StorageError(`Failed to initialize file storage: ${errorMessage}`, 'read');
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 从文件加载数据到内存
|
||||
* 从文件加载数据到内存 - 增强版,包含智能恢复机制
|
||||
*/
|
||||
private async loadFromFile(): Promise<void> {
|
||||
try {
|
||||
// 检查文件是否存在
|
||||
private async loadFromFileWithRecovery(): Promise<void> {
|
||||
// 尝试从主文件加载
|
||||
const mainFileResult = await this.tryLoadFromFile(this.filePath, 'main');
|
||||
if (mainFileResult.success) {
|
||||
this.data = mainFileResult.data!;
|
||||
// 成功加载主文件后,创建备份
|
||||
await this.createBackup();
|
||||
return;
|
||||
}
|
||||
|
||||
console.warn('[FileStorage] Main file failed, trying backup...');
|
||||
|
||||
// 尝试从备份文件加载
|
||||
const backupFileResult = await this.tryLoadFromFile(this.backupPath, 'backup');
|
||||
if (backupFileResult.success) {
|
||||
this.data = backupFileResult.data!;
|
||||
console.log('[FileStorage] Successfully recovered from backup');
|
||||
|
||||
// 从备份恢复后,重新创建主文件(跳过备份创建以保护现有备份)
|
||||
await this.saveToFileWithoutBackup();
|
||||
|
||||
// 主文件恢复成功后,重新创建备份以确保备份是最新的
|
||||
try {
|
||||
await fs.access(this.filePath);
|
||||
} catch {
|
||||
// 文件不存在,创建空存储
|
||||
console.log('[FileStorage] Storage file not found, creating new storage');
|
||||
this.data = new Map();
|
||||
await this.saveToFile();
|
||||
return;
|
||||
await this.createBackup();
|
||||
console.log('[FileStorage] Backup refreshed after recovery');
|
||||
} catch (error) {
|
||||
console.warn('[FileStorage] Failed to refresh backup after recovery:', error);
|
||||
// 备份失败不应该影响恢复流程
|
||||
}
|
||||
|
||||
// 读取文件内容
|
||||
const content = await fs.readFile(this.filePath, 'utf8');
|
||||
|
||||
// 验证JSON格式
|
||||
if (!this.validateJSON(content)) {
|
||||
console.warn('[FileStorage] Storage file corrupted, creating new storage');
|
||||
this.data = new Map();
|
||||
await this.saveToFile();
|
||||
return;
|
||||
}
|
||||
|
||||
// 解析并加载数据
|
||||
const parsed = JSON.parse(content);
|
||||
this.data = new Map(Object.entries(parsed || {}));
|
||||
|
||||
console.log(`[FileStorage] Loaded ${this.data.size} items from storage`);
|
||||
|
||||
} catch (error) {
|
||||
console.error('[FileStorage] Failed to load from file:', error);
|
||||
// 创建新的空存储
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
console.warn('[FileStorage] Both main and backup files failed, checking if files exist...');
|
||||
|
||||
// 检查是否是首次运行(文件不存在)
|
||||
const mainExists = await this.fileExists(this.filePath);
|
||||
const backupExists = await this.fileExists(this.backupPath);
|
||||
|
||||
if (!mainExists && !backupExists) {
|
||||
// 首次运行,创建空存储
|
||||
console.log('[FileStorage] First run detected, creating new storage');
|
||||
this.data = new Map();
|
||||
await this.saveToFile();
|
||||
return;
|
||||
}
|
||||
|
||||
// 文件存在但都损坏了,这是严重问题
|
||||
console.error('[FileStorage] CRITICAL: Both storage files exist but are corrupted!');
|
||||
console.error('[FileStorage] Main file error:', mainFileResult.error);
|
||||
console.error('[FileStorage] Backup file error:', backupFileResult.error);
|
||||
|
||||
// 在这种情况下,我们不能简单地重置数据,而是抛出错误让上层处理
|
||||
throw new StorageError(
|
||||
`Storage corruption detected. Main: ${mainFileResult.error}, Backup: ${backupFileResult.error}`,
|
||||
'read'
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* 尝试从指定文件加载数据
|
||||
*/
|
||||
private async tryLoadFromFile(filePath: string, fileType: string): Promise<{
|
||||
success: boolean;
|
||||
data?: Map<string, string>;
|
||||
error?: string;
|
||||
}> {
|
||||
try {
|
||||
// 检查文件是否存在
|
||||
await fs.access(filePath);
|
||||
|
||||
// 读取文件内容
|
||||
const content = await fs.readFile(filePath, 'utf8');
|
||||
|
||||
// 验证JSON格式
|
||||
if (!this.validateJSON(content)) {
|
||||
return {
|
||||
success: false,
|
||||
error: `Invalid JSON format in ${fileType} file`
|
||||
};
|
||||
}
|
||||
|
||||
// 解析数据
|
||||
const parsed = JSON.parse(content);
|
||||
const data = new Map<string, string>();
|
||||
|
||||
// 确保所有值都是字符串类型
|
||||
for (const [key, value] of Object.entries(parsed || {})) {
|
||||
data.set(key, typeof value === 'string' ? value : JSON.stringify(value));
|
||||
}
|
||||
|
||||
console.log(`[FileStorage] Successfully loaded ${data.size} items from ${fileType} file`);
|
||||
|
||||
return {
|
||||
success: true,
|
||||
data
|
||||
};
|
||||
|
||||
} catch (error) {
|
||||
const errorMessage = error instanceof Error ? error.message : 'Unknown error';
|
||||
return {
|
||||
success: false,
|
||||
error: `Failed to load ${fileType} file: ${errorMessage}`
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 检查文件是否存在
|
||||
*/
|
||||
private async fileExists(filePath: string): Promise<boolean> {
|
||||
try {
|
||||
await fs.access(filePath);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 创建备份文件
|
||||
*/
|
||||
private async createBackup(): Promise<void> {
|
||||
try {
|
||||
if (await this.fileExists(this.filePath)) {
|
||||
await fs.copyFile(this.filePath, this.backupPath);
|
||||
console.log('[FileStorage] Backup created successfully');
|
||||
}
|
||||
} catch (error) {
|
||||
console.warn('[FileStorage] Failed to create backup:', error);
|
||||
// 备份失败不应该影响主要功能
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 将内存数据保存到文件
|
||||
* 将内存数据保存到文件 - 增强版
|
||||
* 包含备份创建和数据验证
|
||||
*/
|
||||
private async saveToFile(): Promise<void> {
|
||||
const data = Object.fromEntries(this.data);
|
||||
const jsonString = JSON.stringify(data, null, 2);
|
||||
|
||||
// 验证数据完整性
|
||||
if (!this.validateJSON(jsonString)) {
|
||||
throw new StorageError('Generated JSON is invalid', 'write');
|
||||
}
|
||||
|
||||
// 如果主文件存在,先创建备份
|
||||
if (await this.fileExists(this.filePath)) {
|
||||
await this.createBackup();
|
||||
}
|
||||
|
||||
// 原子写入主文件
|
||||
await this.atomicWrite(jsonString);
|
||||
|
||||
console.log(`[FileStorage] Saved ${this.data.size} items to storage`);
|
||||
}
|
||||
|
||||
/**
|
||||
* 将内存数据保存到文件 - 不创建备份版本
|
||||
* 用于从备份恢复时,避免覆盖完好的备份文件
|
||||
*/
|
||||
private async saveToFileWithoutBackup(): Promise<void> {
|
||||
const data = Object.fromEntries(this.data);
|
||||
const jsonString = JSON.stringify(data, null, 2);
|
||||
|
||||
// 验证数据完整性
|
||||
if (!this.validateJSON(jsonString)) {
|
||||
throw new StorageError('Generated JSON is invalid', 'write');
|
||||
}
|
||||
|
||||
console.log('[FileStorage] Saving to main file without creating backup (recovery mode)');
|
||||
|
||||
// 直接原子写入主文件,不创建备份
|
||||
await this.atomicWrite(jsonString);
|
||||
|
||||
console.log(`[FileStorage] Recovered and saved ${this.data.size} items to storage`);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -280,25 +432,31 @@ export class FileStorageProvider implements IStorageProvider {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* 原子性数据更新 - 增强版
|
||||
* 确保读-修改-写操作的原子性,防止并发问题
|
||||
*/
|
||||
async updateData<T>(key: string, modifier: (currentValue: T | null) => T): Promise<void> {
|
||||
await this.ensureInitialized();
|
||||
|
||||
|
||||
// 使用更新锁确保原子性
|
||||
const currentLock = this.updateLock;
|
||||
let resolveLock: () => void;
|
||||
|
||||
this.updateLock = new Promise<void>((resolve) => {
|
||||
resolveLock = resolve;
|
||||
});
|
||||
|
||||
try {
|
||||
// 读取当前值
|
||||
const currentData = this.data.get(key);
|
||||
const currentValue: T | null = currentData ? JSON.parse(currentData) : null;
|
||||
|
||||
// 应用修改 - 允许业务逻辑错误透传
|
||||
const newValue = modifier(currentValue);
|
||||
|
||||
// 写入新值
|
||||
this.data.set(key, JSON.stringify(newValue));
|
||||
this.scheduleWrite(); // 延迟写入
|
||||
|
||||
await currentLock;
|
||||
|
||||
// 在锁保护下执行原子操作
|
||||
await this.performAtomicUpdate(key, modifier);
|
||||
|
||||
} catch (error) {
|
||||
// 业务逻辑错误直接透传,保持错误类型
|
||||
if (error instanceof Error &&
|
||||
(error.name.includes('Error') ||
|
||||
if (error instanceof Error &&
|
||||
(error.name.includes('Error') ||
|
||||
error.constructor.name !== 'Error' ||
|
||||
error.message.includes('模型') ||
|
||||
error.message.includes('不存在'))) {
|
||||
@@ -306,6 +464,63 @@ export class FileStorageProvider implements IStorageProvider {
|
||||
}
|
||||
// 只有真正的存储错误才包装为StorageError
|
||||
throw new StorageError(`Data update failed: ${key}`, 'write');
|
||||
} finally {
|
||||
resolveLock!();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 执行原子更新操作
|
||||
*/
|
||||
private async performAtomicUpdate<T>(key: string, modifier: (currentValue: T | null) => T): Promise<void> {
|
||||
// 重新从存储读取最新数据,确保数据一致性
|
||||
const latestData = await this.getLatestData<T>(key);
|
||||
|
||||
// 应用修改
|
||||
const newValue = modifier(latestData);
|
||||
|
||||
// 验证新值
|
||||
this.validateValue(newValue);
|
||||
|
||||
// 写入新值
|
||||
this.data.set(key, JSON.stringify(newValue));
|
||||
this.scheduleWrite(); // 延迟写入
|
||||
|
||||
console.log(`[FileStorage] Atomic update completed for key: ${key}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* 获取最新数据,确保数据一致性
|
||||
*/
|
||||
private async getLatestData<T>(key: string): Promise<T | null> {
|
||||
// 如果有待写入的数据,先刷新到文件
|
||||
if (this.isDirty) {
|
||||
console.log('[FileStorage] Flushing pending changes before read...');
|
||||
await this.flush();
|
||||
}
|
||||
|
||||
// 从内存缓存读取
|
||||
const currentData = this.data.get(key);
|
||||
if (!currentData) {
|
||||
return null;
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(currentData) as T;
|
||||
} catch (error) {
|
||||
console.error(`[FileStorage] Failed to parse data for key ${key}:`, error);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 验证值的有效性
|
||||
*/
|
||||
private validateValue<T>(value: T): void {
|
||||
try {
|
||||
JSON.stringify(value);
|
||||
} catch (error) {
|
||||
throw new StorageError('Value is not serializable', 'write');
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
|
||||
import * as fs from 'fs/promises';
|
||||
import * as path from 'path';
|
||||
import { FileStorageProvider } from '../../../src/services/storage/fileStorageProvider';
|
||||
|
||||
// Mock fs module
|
||||
vi.mock('fs/promises');
|
||||
const mockFs = vi.mocked(fs);
|
||||
|
||||
describe('FileStorageProvider Recovery Safety', () => {
|
||||
let provider: FileStorageProvider;
|
||||
let tempDir: string;
|
||||
let storagePath: string;
|
||||
let backupPath: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tempDir = '/tmp/test-storage';
|
||||
storagePath = path.join(tempDir, 'storage.json');
|
||||
backupPath = path.join(tempDir, 'storage.json.backup');
|
||||
provider = new FileStorageProvider(tempDir);
|
||||
|
||||
// Reset all mocks
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
describe('Backup Protection During Recovery', () => {
|
||||
it('should not overwrite good backup when recovering from corrupted main file', async () => {
|
||||
// Setup: corrupted main file, good backup
|
||||
const goodBackupData = { models: { openai: 'config1', gemini: 'config2' } };
|
||||
const corruptedMainContent = '{ invalid json';
|
||||
const goodBackupContent = JSON.stringify(goodBackupData);
|
||||
|
||||
// Mock file system calls
|
||||
mockFs.access
|
||||
.mockResolvedValueOnce(undefined) // main file exists
|
||||
.mockRejectedValueOnce(new Error('File not found')) // backup access for createBackup
|
||||
.mockResolvedValueOnce(undefined); // backup file exists for recovery
|
||||
|
||||
mockFs.readFile
|
||||
.mockResolvedValueOnce(corruptedMainContent) // corrupted main file
|
||||
.mockResolvedValueOnce(goodBackupContent); // good backup file
|
||||
|
||||
mockFs.mkdir.mockResolvedValue(undefined);
|
||||
mockFs.writeFile.mockResolvedValue(undefined);
|
||||
mockFs.rename.mockResolvedValue(undefined);
|
||||
mockFs.copyFile.mockResolvedValue(undefined);
|
||||
|
||||
// Initialize provider (this should trigger recovery)
|
||||
await provider.getItem('test');
|
||||
|
||||
// Verify that copyFile was NOT called during recovery
|
||||
// (which would mean the corrupted main file didn't overwrite the good backup)
|
||||
const copyFileCalls = mockFs.copyFile.mock.calls;
|
||||
const dangerousCopyCall = copyFileCalls.find(call =>
|
||||
call[0] === storagePath && call[1] === backupPath
|
||||
);
|
||||
|
||||
expect(dangerousCopyCall).toBeUndefined();
|
||||
|
||||
// Verify that writeFile was called to recreate the main file
|
||||
expect(mockFs.writeFile).toHaveBeenCalled();
|
||||
|
||||
// Verify that rename was called (atomic write)
|
||||
expect(mockFs.rename).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should refresh backup after successful recovery', async () => {
|
||||
// Setup: corrupted main file, good backup
|
||||
const goodBackupData = { models: { openai: 'config1' } };
|
||||
const corruptedMainContent = '{ invalid json';
|
||||
const goodBackupContent = JSON.stringify(goodBackupData);
|
||||
|
||||
// Mock file system calls
|
||||
mockFs.access
|
||||
.mockResolvedValueOnce(undefined) // main file exists
|
||||
.mockResolvedValueOnce(undefined) // backup file exists for recovery
|
||||
.mockResolvedValueOnce(undefined); // main file exists for backup refresh
|
||||
|
||||
mockFs.readFile
|
||||
.mockResolvedValueOnce(corruptedMainContent) // corrupted main file
|
||||
.mockResolvedValueOnce(goodBackupContent); // good backup file
|
||||
|
||||
mockFs.mkdir.mockResolvedValue(undefined);
|
||||
mockFs.writeFile.mockResolvedValue(undefined);
|
||||
mockFs.rename.mockResolvedValue(undefined);
|
||||
mockFs.copyFile.mockResolvedValue(undefined);
|
||||
|
||||
// Initialize provider (this should trigger recovery)
|
||||
await provider.getItem('test');
|
||||
|
||||
// Verify that copyFile was called AFTER recovery to refresh backup
|
||||
// This should be the ONLY copyFile call
|
||||
expect(mockFs.copyFile).toHaveBeenCalledTimes(1);
|
||||
expect(mockFs.copyFile).toHaveBeenCalledWith(storagePath, backupPath);
|
||||
});
|
||||
|
||||
it('should handle backup refresh failure gracefully', async () => {
|
||||
// Setup: corrupted main file, good backup
|
||||
const goodBackupData = { models: { openai: 'config1' } };
|
||||
const corruptedMainContent = '{ invalid json';
|
||||
const goodBackupContent = JSON.stringify(goodBackupData);
|
||||
|
||||
// Mock file system calls
|
||||
mockFs.access
|
||||
.mockResolvedValueOnce(undefined) // main file exists
|
||||
.mockResolvedValueOnce(undefined) // backup file exists for recovery
|
||||
.mockResolvedValueOnce(undefined); // main file exists for backup refresh
|
||||
|
||||
mockFs.readFile
|
||||
.mockResolvedValueOnce(corruptedMainContent) // corrupted main file
|
||||
.mockResolvedValueOnce(goodBackupContent); // good backup file
|
||||
|
||||
mockFs.mkdir.mockResolvedValue(undefined);
|
||||
mockFs.writeFile.mockResolvedValue(undefined);
|
||||
mockFs.rename.mockResolvedValue(undefined);
|
||||
|
||||
// Make backup refresh fail
|
||||
mockFs.copyFile.mockRejectedValue(new Error('Backup refresh failed'));
|
||||
|
||||
// Initialize provider (this should trigger recovery)
|
||||
// Should not throw error even if backup refresh fails
|
||||
await expect(provider.getItem('test')).resolves.not.toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe('Data Integrity During Recovery', () => {
|
||||
it('should preserve all data from backup during recovery', async () => {
|
||||
const originalData = {
|
||||
models: JSON.stringify({ openai: 'config1', gemini: 'config2' }),
|
||||
templates: JSON.stringify({ template1: 'content1' }),
|
||||
history: JSON.stringify({ record1: 'data1' })
|
||||
};
|
||||
|
||||
const corruptedMainContent = '{ invalid json';
|
||||
const goodBackupContent = JSON.stringify(originalData);
|
||||
|
||||
// Mock file system calls
|
||||
mockFs.access
|
||||
.mockResolvedValueOnce(undefined) // main file exists
|
||||
.mockResolvedValueOnce(undefined); // backup file exists
|
||||
|
||||
mockFs.readFile
|
||||
.mockResolvedValueOnce(corruptedMainContent) // corrupted main file
|
||||
.mockResolvedValueOnce(goodBackupContent); // good backup file
|
||||
|
||||
mockFs.mkdir.mockResolvedValue(undefined);
|
||||
mockFs.writeFile.mockResolvedValue(undefined);
|
||||
mockFs.rename.mockResolvedValue(undefined);
|
||||
mockFs.copyFile.mockResolvedValue(undefined);
|
||||
|
||||
// Test data retrieval after recovery
|
||||
const modelsData = await provider.getItem('models');
|
||||
const templatesData = await provider.getItem('templates');
|
||||
const historyData = await provider.getItem('history');
|
||||
|
||||
expect(modelsData).toBe(originalData.models);
|
||||
expect(templatesData).toBe(originalData.templates);
|
||||
expect(historyData).toBe(originalData.history);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -14,6 +14,32 @@ const {
|
||||
FileStorageProvider,
|
||||
} = require('@prompt-optimizer/core');
|
||||
|
||||
/**
|
||||
* 安全序列化函数,用于清理Vue响应式对象
|
||||
* 确保所有通过IPC传递的对象都是纯净的JavaScript对象
|
||||
*
|
||||
* 这个函数解决的是IPC序列化问题,与存储层的数据一致性问题是不同的:
|
||||
* - IPC问题:Vue响应式对象无法被Electron序列化传递
|
||||
* - 存储问题:FileStorageProvider的数据一致性和恢复机制
|
||||
*/
|
||||
function safeSerialize(obj) {
|
||||
if (obj === null || obj === undefined) {
|
||||
return obj;
|
||||
}
|
||||
|
||||
// 对于基本类型,直接返回
|
||||
if (typeof obj !== 'object') {
|
||||
return obj;
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(JSON.stringify(obj));
|
||||
} catch (error) {
|
||||
console.error('[IPC Serialization] Failed to serialize object:', error);
|
||||
throw new Error(`Failed to serialize object for IPC: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
let mainWindow;
|
||||
let modelManager, templateManager, historyManager, llmService, promptService, templateLanguageService, preferenceService, dataManager;
|
||||
let storageProvider; // 全局存储提供器引用,用于退出时保存数据
|
||||
@@ -436,8 +462,10 @@ function setupIPC() {
|
||||
|
||||
ipcMain.handle('model-addModel', async (event, model) => {
|
||||
try {
|
||||
// 清理Vue响应式对象,防止IPC序列化错误
|
||||
const safeModel = safeSerialize(model);
|
||||
// model应该包含key和config,需要分离
|
||||
const { key, ...config } = model;
|
||||
const { key, ...config } = safeModel;
|
||||
await modelManager.addModel(key, config);
|
||||
return createSuccessResponse(null);
|
||||
} catch (error) {
|
||||
@@ -447,7 +475,9 @@ function setupIPC() {
|
||||
|
||||
ipcMain.handle('model-updateModel', async (event, id, updates) => {
|
||||
try {
|
||||
await modelManager.updateModel(id, updates);
|
||||
// 清理Vue响应式对象,防止IPC序列化错误
|
||||
const safeUpdates = safeSerialize(updates);
|
||||
await modelManager.updateModel(id, safeUpdates);
|
||||
return createSuccessResponse(null);
|
||||
} catch (error) {
|
||||
return createErrorResponse(error);
|
||||
@@ -520,7 +550,9 @@ function setupIPC() {
|
||||
|
||||
ipcMain.handle('template-createTemplate', async (event, template) => {
|
||||
try {
|
||||
await templateManager.saveTemplate(template);
|
||||
// 清理Vue响应式对象,防止IPC序列化错误
|
||||
const safeTemplate = safeSerialize(template);
|
||||
await templateManager.saveTemplate(safeTemplate);
|
||||
return createSuccessResponse(null);
|
||||
} catch (error) {
|
||||
return createErrorResponse(error);
|
||||
@@ -531,7 +563,9 @@ function setupIPC() {
|
||||
try {
|
||||
// Get existing template and merge with updates
|
||||
const existingTemplate = await templateManager.getTemplate(id);
|
||||
const updatedTemplate = { ...existingTemplate, ...updates, id };
|
||||
// 清理Vue响应式对象,防止IPC序列化错误
|
||||
const safeUpdates = safeSerialize(updates);
|
||||
const updatedTemplate = { ...existingTemplate, ...safeUpdates, id };
|
||||
await templateManager.saveTemplate(updatedTemplate);
|
||||
return createSuccessResponse(null);
|
||||
} catch (error) {
|
||||
@@ -625,7 +659,9 @@ function setupIPC() {
|
||||
|
||||
ipcMain.handle('history-addRecord', async (event, record) => {
|
||||
try {
|
||||
const result = await historyManager.addRecord(record);
|
||||
// 清理Vue响应式对象,防止IPC序列化错误
|
||||
const safeRecord = safeSerialize(record);
|
||||
const result = await historyManager.addRecord(safeRecord);
|
||||
return createSuccessResponse(result);
|
||||
} catch (error) {
|
||||
return createErrorResponse(error);
|
||||
@@ -680,7 +716,9 @@ function setupIPC() {
|
||||
|
||||
ipcMain.handle('history-createNewChain', async (event, record) => {
|
||||
try {
|
||||
const result = await historyManager.createNewChain(record);
|
||||
// 清理Vue响应式对象,防止IPC序列化错误
|
||||
const safeRecord = safeSerialize(record);
|
||||
const result = await historyManager.createNewChain(safeRecord);
|
||||
return createSuccessResponse(result);
|
||||
} catch (error) {
|
||||
return createErrorResponse(error);
|
||||
@@ -689,7 +727,9 @@ function setupIPC() {
|
||||
|
||||
ipcMain.handle('history-addIteration', async (event, params) => {
|
||||
try {
|
||||
const result = await historyManager.addIteration(params);
|
||||
// 清理Vue响应式对象,防止IPC序列化错误
|
||||
const safeParams = safeSerialize(params);
|
||||
const result = await historyManager.addIteration(safeParams);
|
||||
return createSuccessResponse(result);
|
||||
} catch (error) {
|
||||
return createErrorResponse(error);
|
||||
|
||||
@@ -959,7 +959,10 @@ const saveEdit = async () => {
|
||||
? editingModel.value.originalApiKey
|
||||
: editingModel.value.apiKey,
|
||||
defaultModel: editingModel.value.defaultModel,
|
||||
models: [editingModel.value.defaultModel], // 可以根据需要扩展
|
||||
// 确保models数组包含defaultModel,避免验证错误
|
||||
models: modelOptions.value.length > 0
|
||||
? modelOptions.value.map(opt => opt.value)
|
||||
: [editingModel.value.defaultModel],
|
||||
useVercelProxy: editingModel.value.useVercelProxy,
|
||||
provider: editingModel.value.provider || 'custom',
|
||||
enabled: editingModel.value.enabled !== undefined
|
||||
@@ -996,7 +999,10 @@ const addCustomModel = async () => {
|
||||
const config = {
|
||||
name: newModel.value.name,
|
||||
baseURL: newModel.value.baseURL,
|
||||
models: [newModel.value.defaultModel],
|
||||
// 确保models数组包含defaultModel,避免验证错误
|
||||
models: modelOptions.value.length > 0
|
||||
? modelOptions.value.map(opt => opt.value)
|
||||
: [newModel.value.defaultModel],
|
||||
defaultModel: newModel.value.defaultModel,
|
||||
apiKey: newModel.value.apiKey,
|
||||
enabled: true,
|
||||
|
||||
Reference in new issue
Block a user