feat: enhance web crawler sidecar build process and add package content policy

- Updated the build_web_crawler_sidecar.py script to remove full Chromium from bundled browsers and added a function to prune non-runtime files.
- Enhanced create-release-fragment.mjs to validate package security audit data.
- Introduced package-content-policy.mjs to enforce restrictions on forbidden file types and directories in packages.
- Added prepare-package-stage.mjs for staging package preparation, including runtime dependency installation and resource pruning.
- Implemented print-package-audit-summary.mjs to generate a summary of package security audits.
- Created runtime-bundle-graph.mjs to analyze runtime dependencies and collect package names.
- Refactored verify-runtime-bundles.mjs to utilize the new runtime-bundle-graph module for better dependency management.
- Updated release-fragment-contract.test.ts to include tests for package audit integration and validation.
This commit is contained in:
hy committed 2026-07-15 09:39:19 +08:00
1 parent 03d6151d78
commit a6bf082e0a
20 files changed
+1470 -141

No files matched your search

+6 -1
View File
@@ -64,7 +64,12 @@ jobs:
echo "RELEASE_TAG=$tag" >> "$GITHUB_ENV"
- run: pnpm install --frozen-lockfile
- run: pnpm run build:win
- name: Build and security-audit Windows package
run: pnpm run build:win
- name: Add package audit to workflow summary
shell: bash
run: node scripts/print-package-audit-summary.mjs dist/package-audit.json >> "$GITHUB_STEP_SUMMARY"
- name: Upload Windows release directly to Cloudflare R2
shell: bash
+1
View File
@@ -15,4 +15,5 @@ __pycache__
.pyinstaller
resources/web_crawler_sidecar/bin
resources/native
release-stage
design/
+14
View File
@@ -0,0 +1,14 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.cs.allow-jit</key>
<true/>
<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
<true/>
<key>com.apple.security.cs.allow-dyld-environment-variables</key>
<true/>
<key>com.apple.security.cs.disable-library-validation</key>
<true/>
</dict>
</plist>
+242 -40
View File
@@ -1,38 +1,173 @@
# 桌面端发布说明
# 桌面端构建与发布
桌面端安装包和自动更新文件统一由 Cloudflare R2 提供。GitHub Release 只保存版本说明和 R2 下载链接,不保存二进制文件。
本文是桌面端打包、签名、审计、上传和自动更新的唯一发布说明。
## 发布结构
发布流程遵守两个原则:
1. 安装包只包含应用运行必须的文件。
2. 没有通过最终产物安全审计的文件不能上传到 R2。
向量模型由应用首次使用时单独下载,不属于安装包。本发布流程不修改模型名称、精度、下载文件或缓存逻辑。
## 打包安全边界
electron-builder 不再从项目根目录打包。`pnpm run build` 会重新创建:
```text
release-stage/app/
├── package.json
├── out/
│ ├── main/
│ ├── preload/
│ └── renderer/
├── node_modules/
└── resources/
├── native/
└── web_crawler_sidecar/bin/
```
staging 目录只包含:
- 压缩后的 main、preload 和 renderer;
- 从编译结果自动计算出的运行时依赖;
- 当前系统、当前架构的原生二进制;
- 只包含 headless Chromium 的网页抓取 sidecar;
- 最小化生产 `package.json`;
- 法律要求保留的第三方 License 和 Notice。
以下内容禁止进入 staging 和最终安装目录:
- 项目 `src`、`tests`、`scripts`、`doc`;
- `.claude`、`.codex`、`.github`;
- `.env`、锁文件、工作区配置、构建配置;
- TypeScript、Python、C/C++ 原始源码;
- Source Map;
- 测试日志、截图、Playwright trace、fixture;
- 完整 Chromium 和 Patchright;
- 错误系统或错误架构的原生二进制;
- 私钥、签名证书和高风险 Token。
`app.asar` 只是归档,不是加密。编译后的 JavaScript 是 Electron 运行必需内容,仍然可以被逆向分析。生产构建会压缩 JavaScript,但不能把客户端代码当作秘密存储位置。
## Source Map 和密钥
最终安装包内禁止 `.map`。如果以后接入错误监控:
1. 构建时生成 Source Map;
2. 上传到受访问控制的错误监控平台;
3. 从 staging 删除;
4. 再执行打包和最终审计。
Source Map 不能上传到公开 CDN,因为 `sourcesContent` 可能包含完整原始源码。
所有 `VITE_*` 环境变量都按公开信息处理。发布 Token、R2 凭据和签名证书只能保存在本机安全环境或 GitHub Actions Secret 中,不能写入 renderer、源码或打包配置。
## 构建流程
`pnpm run build` 依次执行:
1. TypeScript 类型检查;
2. 构建只含 Chromium headless shell 的 Python sidecar;
3. 导出 Electron 使用的 better-sqlite3 原生模块;
4. 构建并压缩 main、preload、renderer,禁止 Source Map;
5. 验证 main、worker 和 embedding 运行时边界;
6. 删除旧 `dist` 和旧 staging;
7. 根据编译产物生成最小生产 `package.json`;
8. 使用主锁文件离线安装运行时依赖;
9. 删除测试、文档、源码、Source Map 和错误平台二进制;
10. 执行 staging 安全审计。
electron-builder 完成后,`pnpm run package:audit` 会再次检查真实 `.app` 或 `win-unpacked`,而不是只检查构建配置。
最终审计会验证:
- `app.asar` 根目录只能有 `out`、`node_modules`、`package.json`;
- `package.json` 只能包含最小生产字段;
- `app.asar` 和外部 resources 都没有禁止文件;
- 没有绝对路径或逃出安装目录的符号链接;
- 没有私钥、AWS Access Key、GitHub Token 等高风险内容;
- sidecar 没有完整 Chromium、Patchright 和 Python 源码;
- ONNX Runtime 只包含当前平台和架构;
- 生成 `dist/package-audit.json`,记录安装目录、`app.asar` 和安装包体积。
体积硬性预算暂不启用。等 macOS arm64、macOS x64、Windows x64 都完成优化和回归后,再根据实测结果设置。
## 常用本地命令
只构建编译产物和 staging:
```bash
pnpm run build
```
构建未压缩安装目录并审计:
```bash
pnpm run build:unpack
```
构建当前 Mac 架构:
```bash
pnpm run build:mac:arm64
pnpm run build:mac:x64
```
重新执行最终安装目录审计:
```bash
pnpm run package:audit
pnpm run package:audit:summary
```
所有正式打包命令已经包含最终审计。审计失败时不能手工跳过并上传。
## 发布平台
- macOS arm64:在 Apple 芯片 Mac 本地构建。
- macOS x64:在同一台 Mac 的 Rosetta x64 Node/Python 环境构建。
- Windows x64:由 `.github/workflows/release.yml` 使用 GitHub Windows Runner 构建并直接上传 R2。
- GitHub Actions 不保存 Artifact,也不再运行 macOS 或 Linux 发布任务。
- Windows x64:由 `.github/workflows/release.yml` 使用 GitHub Windows Runner 构建。
- GitHub Actions 不保存安装包 Artifact,也不执行 macOS 或 Linux 发布。
R2 目录:
Windows 工作流顺序固定为:
1. 从 Tag 检出干净源码;
2. 安装锁定依赖;
3. 构建 Windows x64;
4. 审计 staging 和最终 `win-unpacked`;
5. 把体积报告写入 Workflow Summary;
6. 生成包含审计摘要的发布片段;
7. 上传 R2。
生成发布片段时必须存在匹配平台和架构的 `dist/package-audit.json`。因此即使有人绕过正常构建命令,也不能直接上传未审计安装包。
## R2 发布结构
桌面端安装包和自动更新文件统一由 Cloudflare R2 提供。GitHub Release 只保存版本说明和 R2 下载链接,不保存二进制文件。
```text
artifacts/vX.Y.Z/* 当前版本的全部安装包和更新文件
release-status/vX.Y.Z/*.json 三个平台的构建完成标记
updates/latest/latest.yml Windows 最新版自动更新元数据
updates/latest/latest-mac.yml macOS 最新版自动更新元数据
artifacts/vX.Y.Z/* 当前版本安装包和更新文件
release-status/vX.Y.Z/*.json 平台构建完成标记和审计摘要
updates/latest/latest.yml Windows 自动更新元数据
updates/latest/latest-mac.yml macOS 自动更新元数据
releases/latest.json 官网最新版下载清单
```
`latest.yml`、`latest-mac.yml` 和 `latest.json` 最后发布。它们更新之前,新版本不会被应用或官网发现。
`latest.yml`、`latest-mac.yml` 和 `latest.json` 最后发布。它们更新前,新版本不会被应用或官网发现。
版本文件使用一年不可变缓存,最新版元数据使用 `no-store`。
版本文件使用一年不可变缓存;最新版元数据使用 `no-store`。
## R2 保留规则
## R2 保留和回滚
新版本发布并通过公网校验后,发布脚本会删除:
- 旧的 `artifacts/vX.Y.Z`;
- 旧的 `release-status/vX.Y.Z`;
- 旧逻辑产生的 `updates/vX.Y.Z`;
- 旧逻辑产生的 `releases/vX.Y.Z` 和 `releases/latest/*` 安装包。
- 旧逻辑产生的 `releases/vX.Y.Z`;
- 旧逻辑产生的 `releases/latest/*` 安装包。
R2 稳态只保存最新版本。需要回滚时,从旧 Tag 重新构建并发布。
R2 稳态只保存最新版本。需要回滚时,从旧 annotated Tag 重新构建、审计并发布,不能复用未审计的本地旧产物。
## 一次性本地配置
@@ -45,9 +180,31 @@ Mac 需要:
- 配置 R2 环境变量;
- 如需自动创建 GitHub Release,配置 `NCURATOR_RELEASE_TOKEN`。
发布配置统一读取项目根目录 `.env`。R2 可以使用 `R2_ACCESS_KEY_ID`、`R2_SECRET_ACCESS_KEY`,脚本会自动映射成 AWS CLI 所需变量。`.env` 已加入 `.gitignore`,不得提交。
发布配置读取项目根目录 `.env`。R2 可以使用:
默认本地工具目录是 `~/.ncurator/release-tools`。也可以用 `NCURATOR_X64_NODE`、`NCURATOR_X64_PYTHON` 和 `NCURATOR_AWS_CLI` 指定其他路径。
```text
R2_ACCESS_KEY_ID
R2_SECRET_ACCESS_KEY
R2_ACCOUNT_ID
R2_BUCKET_NAME
R2_PUBLIC_BASE_URL
```
脚本会把 R2 Key 映射为 AWS CLI 所需变量。`.env` 已被 Git 忽略,也被打包安全策略禁止,不能提交或复制到 staging。
默认本地工具目录是:
```text
~/.ncurator/release-tools
```
可以使用以下变量指定其他路径:
```text
NCURATOR_X64_NODE
NCURATOR_X64_PYTHON
NCURATOR_AWS_CLI
```
首次导入签名证书:
@@ -55,39 +212,64 @@ Mac 需要:
pnpm release:signing:setup
```
敏感值不要写入仓库。
当前 macOS 使用项目自建的 `Ncurator Update Signing` 证书,并在
`electron-builder.yml` 中设置 `timestamp: none`,因为 Apple 时间戳服务不接受该证书。
这套签名用于保证两个架构的更新包来自同一发布方,当前不等同于 Apple Developer ID 公证。
自建证书没有 Apple Team ID。sidecar 需要加载内置 Python Framework,因此子进程使用
`build/entitlements.mac.inherit.plist` 关闭 library validation;顶层 App 不使用这个例外。
修改签名配置后必须实际运行打包后的 sidecar,不能只看 `codesign --verify`。
如果以后切换到 Apple Developer ID,必须一次性完成以下调整:
1. 替换签名证书和 CI Secret;
2. 删除 `timestamp: none`,恢复 Apple 可信时间戳;
3. 评估并移除子进程的 `disable-library-validation`;
4. 启用并配置 notarize;
5. 重新验证 arm64、x64、安装、Gatekeeper、sidecar 和自动更新。
## 正式发布
1. 修改 `package.json` 版本并提交。
2. 创建带发布说明的 annotated Tag 并推送,例如 `v0.1.3`。
3. Tag 自动触发 Windows 构建并直传 R2。
4. 在 Mac 执行:
推荐使用发布准备命令创建提交和 annotated Tag:
```bash
pnpm release:mac -- --tag v0.1.3
pnpm release:prepare -- --version 0.1.4 --notes "版本说明"
```
也可以手工完成,但必须满足:
1. 在 `main` 修改 `package.json` 版本并提交;
2. 创建 annotated Tag,例如 `v0.1.4`;
3. Tag 必须与 `package.json` 版本一致;
4. 推送提交和 Tag。
Tag 会触发 Windows 构建、审计并上传 R2。然后在 Apple 芯片 Mac 执行:
```bash
pnpm release:mac -- --tag v0.1.4
```
本地命令会:
1. 从 Tag 导出两份干净源码;
2. 分别构建 arm64 和 Rosetta x64;
3. 验证两个 App 使用相同的稳定签名;
4. 上传 macOS 文件;
5. 等待 Windows 文件;
6. 生成并发布最新元数据;
7. 创建只含 R2 链接的 GitHub Release;
8. 清理 R2 旧版本。
2. 分别构建并审计 macOS arm64 和 x64;
3. 验证两个 App 使用相同稳定签名;
4. 上传 macOS 文件和审计摘要;
5. 等待 Windows 发布片段;
6. 校验 R2 文件大小和 sha512;
7. 生成并发布最新版元数据;
8. 创建只含 R2 链接的 GitHub Release;
9. 清理 R2 旧版本。
如果只想重新执行最后发布步骤:
只重新执行最终汇总:
```bash
pnpm release:r2:finalize -- --tag v0.1.3
pnpm release:r2:finalize -- --tag v0.1.4
```
调试时可以添加 `--keep-old` 暂时不删除旧版本。
调试时可以添加 `--keep-old`,暂时不删除旧版本。它不会跳过安装包审计。
## 下载地址
## 公网地址
官网读取:
@@ -102,26 +284,46 @@ https://download.ncurator.com/updates/latest/latest.yml
https://download.ncurator.com/updates/latest/latest-mac.yml
```
实际文件始终来自:
实际安装文件来自:
```text
https://download.ncurator.com/artifacts/vX.Y.Z/*
```
## 测试
## 发布前回归
基础测试:
基础检查:
```bash
pnpm lint
pnpm typecheck
pnpm test:update:unit
pnpm test:update:e2e
```
真实自动更新只在明确需要时执行:
还必须验证:
- 应用启动、preload 和 IPC;
- SQLite、LanceDB 和 embedding;
- PDF/DOCX 读取与预览;
- 网页搜索和动态网页抓取;
- 安装、覆盖升级、卸载;
- 无网络启动;
- 自动更新下载、sha512、签名替换和用户数据保留。
真实自动更新测试只在明确需要时执行:
```bash
pnpm test:update:real -- --source-version 0.1.2 --expected-version 0.1.3
pnpm test:update:real -- --source-version 0.1.3 --expected-version 0.1.4
```
真实测试必须确认发现新版、R2 下载、sha512 校验、签名替换、重启版本和用户数据保留。
## 失败处理
- staging 审计失败:不要添加排除例外,先确认文件是否真的属于运行时必需内容。
- 最终审计失败:检查 `app.asar` 和外部 resources,不能只改 electron-builder glob。
- sidecar 缺浏览器:重新运行 `pnpm run build:sidecar`,确认只下载 headless shell。
- 运行时模块缺失:检查编译产物的外部 import 和生成的 staging `package.json`。
- 平台二进制缺失:检查构建所用 Node/Python 架构是否与目标一致。
- R2 汇总失败:保留现有 latest 元数据,修复后重新运行 finalize,不能发布部分平台。
任何失败都不能通过复制旧 `dist`、关闭审计或手工上传来绕过。
+17 -13
View File
@@ -2,28 +2,25 @@ appId: com.yoan.ncurator
productName: Ncurator
artifactName: ${productName}-${os}-${arch}.${ext}
directories:
app: release-stage/app
buildResources: build
files:
- '!**/.vscode/*'
- '!src/*'
- '!electron.vite.config.{js,ts,mjs,cjs}'
- '!{.eslintcache,eslint.config.mjs,.prettierignore,.prettierrc.yaml,dev-app-update.yml,CHANGELOG.md,README.md}'
- '!{.env,.env.*,.npmrc,pnpm-lock.yaml}'
- '!{tsconfig.json,tsconfig.node.json,tsconfig.web.json}'
- '!{.codex-tmp,.pip-cache,.pyinstaller,.runvenv}/**'
- '!resources/web_crawler_sidecar/**'
- '!resources/native/**'
- 'out/**/*'
- 'package.json'
- '!**/*.map'
- '!**/*.{ts,tsx,mts,cts,py,pyc,pyo,c,cc,cpp,h,hpp,pdb}'
- '!**/{test,tests,__tests__,doc,docs,example,examples,benchmark,benchmarks,scripts}/**/*'
extraResources:
- from: resources/web_crawler_sidecar/bin
- from: release-stage/app/resources/web_crawler_sidecar/bin
to: web_crawler_sidecar/bin
filter:
- '**/*'
- from: resources/native
- from: release-stage/app/resources/native
to: native
filter:
- '**/*'
asarUnpack:
- '**/node_modules/onnxruntime-node/**/*'
- '**/node_modules/onnxruntime-node/bin/napi-v3/**/*'
win:
icon: build/icon.ico
executableName: Ncurator
@@ -42,10 +39,17 @@ nsis:
mac:
icon: build/icon.icns
artifactName: ${productName}-mac-${arch}.${ext}
# @electron/osx-sign 会把部分二进制格式的数据文件误判成可执行文件。
# 这些资源由外层 bundle 签名保护;Mach-O、.node、.dylib 和 .so 不在忽略范围内。
signIgnore:
- '\.(?:asar|bin|dat|icns|nib|pak|png|ttf|zip)$'
- '/\.jekyll-metadata$'
# 当前使用项目自建的更新签名证书,不使用 Apple 时间戳服务。
timestamp: none
target:
- dmg
- zip
entitlementsInherit: build/entitlements.mac.plist
entitlementsInherit: build/entitlements.mac.inherit.plist
extendInfo:
- NSCameraUsageDescription: Application requests access to the device's camera.
- NSMicrophoneUsageDescription: Application requests access to the device's microphone.
+12 -1
View File
@@ -6,6 +6,8 @@ import react from '@vitejs/plugin-react'
export default defineConfig({
main: {
build: {
minify: 'esbuild',
sourcemap: false,
rollupOptions: {
input: {
index: resolve('src/main/index.ts'),
@@ -19,7 +21,12 @@ export default defineConfig({
}
}
},
preload: {},
preload: {
build: {
minify: 'esbuild',
sourcemap: false
}
},
renderer: {
resolve: {
alias: {
@@ -27,6 +34,10 @@ export default defineConfig({
}
},
plugins: [react(), tailwindcss()],
build: {
minify: 'esbuild',
sourcemap: false
},
css: {
preprocessorOptions: {
less: {
+12 -1
View File
@@ -6,7 +6,18 @@ import eslintPluginReactHooks from 'eslint-plugin-react-hooks'
import eslintPluginReactRefresh from 'eslint-plugin-react-refresh'
export default defineConfig(
{ ignores: ['**/node_modules', '**/dist', '**/out'] },
{
ignores: [
'**/node_modules',
'**/dist',
'**/out',
'.codex-tmp',
'.pyinstaller',
'.runvenv',
'release-stage',
'resources/web_crawler_sidecar/bin'
]
},
tseslint.configs.recommended,
eslintPluginReact.configs.flat.recommended,
eslintPluginReact.configs.flat['jsx-runtime'],
+15 -11
View File
@@ -36,16 +36,19 @@
"start": "electron-vite preview",
"dev": "node scripts/prepare-better-sqlite3-native.mjs ensure electron && node scripts/ensure-web-crawler-sidecar.mjs && electron-vite dev",
"dev:no-sidecar": "node scripts/prepare-better-sqlite3-native.mjs ensure electron && electron-vite dev",
"build": "node scripts/prepare-better-sqlite3-native.mjs ensure node && tsc --noEmit -p tsconfig.node.json --composite false && tsc --noEmit -p tsconfig.web.json --composite false && node scripts/build-web-crawler-sidecar.mjs && node scripts/prepare-better-sqlite3-native.mjs export-electron-resource && electron-vite build && node scripts/verify-runtime-bundles.mjs",
"build": "node scripts/prepare-better-sqlite3-native.mjs ensure node && tsc --noEmit -p tsconfig.node.json --composite false && tsc --noEmit -p tsconfig.web.json --composite false && node scripts/build-web-crawler-sidecar.mjs && node scripts/prepare-better-sqlite3-native.mjs export-electron-resource && electron-vite build && node scripts/verify-runtime-bundles.mjs && node scripts/prepare-package-stage.mjs",
"postinstall": "node scripts/postinstall.mjs",
"build:unpack": "pnpm run build && electron-builder --dir",
"build:win": "pnpm run build && electron-builder --win --x64 --publish never",
"build:mac": "pnpm run build && electron-builder --mac --publish never",
"build:mac:x64": "pnpm run build && electron-builder --mac --x64 --publish never",
"build:mac:arm64": "pnpm run build && electron-builder --mac --arm64 --publish never",
"build:mac:release:x64": "pnpm run build && electron-builder --mac --x64 --publish never --config.forceCodeSigning=true",
"build:mac:release:arm64": "pnpm run build && electron-builder --mac --arm64 --publish never --config.forceCodeSigning=true",
"build:linux": "pnpm run build && electron-builder --linux",
"package:audit:stage": "node scripts/audit-package-stage.mjs",
"package:audit": "node scripts/audit-packaged-app.mjs dist",
"package:audit:summary": "node scripts/print-package-audit-summary.mjs dist/package-audit.json",
"build:unpack": "pnpm run build && electron-builder --dir && pnpm run package:audit",
"build:win": "pnpm run build && electron-builder --win --x64 --publish never && pnpm run package:audit",
"build:mac": "pnpm run build && electron-builder --mac --publish never && pnpm run package:audit",
"build:mac:x64": "pnpm run build && electron-builder --mac --x64 --publish never && pnpm run package:audit",
"build:mac:arm64": "pnpm run build && electron-builder --mac --arm64 --publish never && pnpm run package:audit",
"build:mac:release:x64": "pnpm run build && electron-builder --mac --x64 --publish never --config.forceCodeSigning=true && pnpm run package:audit",
"build:mac:release:arm64": "pnpm run build && electron-builder --mac --arm64 --publish never --config.forceCodeSigning=true && pnpm run package:audit",
"build:linux": "pnpm run build && electron-builder --linux && pnpm run package:audit",
"native:node": "node scripts/prepare-better-sqlite3-native.mjs ensure node",
"native:electron": "node scripts/prepare-better-sqlite3-native.mjs ensure electron",
"native:status": "node scripts/prepare-better-sqlite3-native.mjs status",
@@ -60,7 +63,6 @@
"@huggingface/transformers": "^3.8.1",
"@js-preview/docx": "^1.6.4",
"@lancedb/lancedb": "^0.23.0",
"@lancedb/lancedb-darwin-arm64": "^0.22.3",
"@langchain/community": "^1.1.1",
"@langchain/core": "^1.1.44",
"@langchain/deepseek": "^1.0.25",
@@ -100,6 +102,7 @@
"@electron-toolkit/eslint-config-prettier": "^3.0.0",
"@electron-toolkit/eslint-config-ts": "^3.1.0",
"@electron-toolkit/tsconfig": "^2.0.0",
"@electron/asar": "3.4.1",
"@tailwindcss/vite": "^4.1.18",
"@testing-library/react": "^16.3.2",
"@testing-library/user-event": "^14.6.1",
@@ -124,7 +127,8 @@
"tailwindcss": "^4.1.18",
"typescript": "^5.9.3",
"vite": "^7.2.6",
"vitest": "^4.0.18"
"vitest": "^4.0.18",
"yaml": "2.8.2"
},
"packageManager": "pnpm@9.12.2+sha512.22721b3a11f81661ae1ec68ce1a7b879425a1ca5b991c975b074ac220b187ce56c708fe5db69f4c962c989452eee76c82877f4ee80f474cebd61ee13461b6228"
}
+14 -22
View File
@@ -29,9 +29,6 @@ importers:
'@lancedb/lancedb':
specifier: ^0.23.0
version: 0.23.0(apache-arrow@17.0.0)
'@lancedb/lancedb-darwin-arm64':
specifier: ^0.22.3
version: 0.22.3
'@langchain/community':
specifier: ^1.1.1
version: 1.1.2(@browserbasehq/sdk@2.14.0(encoding@0.1.13))(@browserbasehq/stagehand@3.5.0(@cfworker/json-schema@4.1.1)(encoding@0.1.13)(playwright-core@1.58.2)(zod@4.3.6))(@huggingface/transformers@3.8.1)(@ibm-cloud/watsonx-ai@1.7.5)(@lancedb/lancedb@0.23.0(apache-arrow@17.0.0))(@langchain/core@1.1.44(@opentelemetry/api@1.9.0)(openai@6.36.0(ws@8.21.0(bufferutil@4.1.0))(zod@4.3.6)))(@opentelemetry/api@1.9.0)(@smithy/util-utf8@2.3.0)(better-sqlite3@12.11.1)(cheerio@1.2.0)(crypto-js@4.2.0)(google-auth-library@10.7.0)(ibm-cloud-sdk-core@5.4.5)(ignore@5.3.2)(jsdom@26.1.0(bufferutil@4.1.0))(jsonwebtoken@9.0.3)(lodash@4.17.21)(mammoth@1.11.0)(openai@6.36.0(ws@8.21.0(bufferutil@4.1.0))(zod@4.3.6))(pdf-parse@1.1.1)(playwright@1.58.2)(ws@8.21.0(bufferutil@4.1.0))
@@ -144,6 +141,9 @@ importers:
'@electron-toolkit/tsconfig':
specifier: ^2.0.0
version: 2.0.0(@types/node@22.19.3)
'@electron/asar':
specifier: 3.4.1
version: 3.4.1
'@tailwindcss/vite':
specifier: ^4.1.18
version: 4.1.18(vite@7.3.0(@types/node@22.19.3)(jiti@2.6.1)(less@4.5.1)(lightningcss@1.30.2)(yaml@2.8.2))
@@ -219,6 +219,9 @@ importers:
vitest:
specifier: ^4.0.18
version: 4.0.18(@opentelemetry/api@1.9.0)(@types/node@22.19.3)(jiti@2.6.1)(jsdom@26.1.0(bufferutil@4.1.0))(less@4.5.1)(lightningcss@1.30.2)(yaml@2.8.2)
yaml:
specifier: 2.8.2
version: 2.8.2
packages:
@@ -599,8 +602,8 @@ packages:
resolution: {integrity: sha512-Qkzpg2s9GnVV2I2BjRksUi43U5e6+zaQMcjoJy0C+C5oxaKl+fmckGDQFtRpZpZV0NQekuZZ+tGz7EA9TVnQtQ==}
engines: {node: '>=12'}
'@electron/node-gyp@git+https://git@github.com:electron/node-gyp.git#06b29aafb7708acef8b3669835c8a7857ebc92d2':
resolution: {commit: 06b29aafb7708acef8b3669835c8a7857ebc92d2, repo: git@github.com:electron/node-gyp.git, type: git}
'@electron/node-gyp@https://codeload.github.com/electron/node-gyp/tar.gz/06b29aafb7708acef8b3669835c8a7857ebc92d2':
resolution: {tarball: https://codeload.github.com/electron/node-gyp/tar.gz/06b29aafb7708acef8b3669835c8a7857ebc92d2}
version: 10.2.0-electron.1
engines: {node: '>=12.13.0'}
hasBin: true
@@ -1237,11 +1240,6 @@ packages:
peerDependencies:
core-js: ^3.0.0
'@lancedb/lancedb-darwin-arm64@0.22.3':
resolution: {integrity: sha512-oP2Kic51nLqs27Xo+AzSVlcMgmmfZbU/PseQ3KBtU92rczO5DYU2St1Y7qDUWcjw+RF3H2v/DKzYed16h1wCBQ==}
engines: {node: '>= 18'}
os: [darwin]
'@lancedb/lancedb-darwin-arm64@0.23.0':
resolution: {integrity: sha512-8w0sMCNMwBv2kv5+fczGeSVlNOL+BOKChSsO4usM0hMw3PmxasONPctQBsESDuPS8lQ6/AKAQc2HT/ddd5Mg5w==}
engines: {node: '>= 18'}
@@ -3912,6 +3910,7 @@ packages:
glob@10.5.0:
resolution: {integrity: sha512-DfXN8DfhJ7NH3Oe7cFmu3NCu1wKbkReJ8TorzSAFbSKrlNaQSKfIzqYqVY8zlbs2NLBbWpRiU52GX2PbaBVNkg==}
deprecated: Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me
hasBin: true
glob@7.2.3:
@@ -4176,10 +4175,6 @@ packages:
resolution: {integrity: sha512-4gd7VpWNQNB4UKKCFFVcp1AVv+FMOgs9NKzjHKusc8jTMhd5eL1NqQqOpE0KzMds804/yHlglp3uxgluOqAPLw==}
engines: {node: '>= 0.4'}
ip-address@10.1.0:
resolution: {integrity: sha512-XXADHxXmvT9+CRxhXg56LJovE+bmWnEWB78LB83VZTprKTmaC5QfruXocxzTZ2Kl0DNwKuBdlIhjL8LeY8Sf8Q==}
engines: {node: '>= 12'}
ip-address@10.2.0:
resolution: {integrity: sha512-/+S6j4E9AHvW9SWMSEY9Xfy66O5PWvVEJ08O0y5JGyEKQpojb0K0GKpz/v5HJ/G0vi3D2sjGK78119oXZeE0qA==}
engines: {node: '>= 12'}
@@ -6775,6 +6770,7 @@ packages:
whatwg-encoding@3.1.1:
resolution: {integrity: sha512-6qN4hJdMwfYBtE3YBTTHhoeuUrDBPZmbQaxWAqSALV/MeEnR5z1xd8UKud2RAkFoPkmB+hli1TZSnyi84xz1vQ==}
engines: {node: '>=18'}
deprecated: Use @exodus/bytes instead for a more spec-conformant and faster implementation
whatwg-mimetype@4.0.0:
resolution: {integrity: sha512-QaKxh0eNIi2mE9p2vEdzfagOKHCcj1pJ56EEHGQOVxp8r9/iszLUUV7v89x9O1p/T+NlTM5W7jW6+cz4Fq1YVg==}
@@ -7457,7 +7453,7 @@ snapshots:
transitivePeerDependencies:
- supports-color
'@electron/node-gyp@git+https://git@github.com:electron/node-gyp.git#06b29aafb7708acef8b3669835c8a7857ebc92d2':
'@electron/node-gyp@https://codeload.github.com/electron/node-gyp/tar.gz/06b29aafb7708acef8b3669835c8a7857ebc92d2':
dependencies:
env-paths: 2.2.1
exponential-backoff: 3.1.3
@@ -7494,7 +7490,7 @@ snapshots:
'@electron/rebuild@3.7.0':
dependencies:
'@electron/node-gyp': git+https://git@github.com:electron/node-gyp.git#06b29aafb7708acef8b3669835c8a7857ebc92d2
'@electron/node-gyp': https://codeload.github.com/electron/node-gyp/tar.gz/06b29aafb7708acef8b3669835c8a7857ebc92d2
'@malept/cross-spawn-promise': 2.0.0
chalk: 4.1.2
debug: 4.4.3(supports-color@5.5.0)
@@ -7514,7 +7510,7 @@ snapshots:
'@electron/universal@2.0.1':
dependencies:
'@electron/asar': 3.2.18
'@electron/asar': 3.4.1
'@malept/cross-spawn-promise': 2.0.0
debug: 4.4.3(supports-color@5.5.0)
dir-compare: 4.2.0
@@ -7951,8 +7947,6 @@ snapshots:
dependencies:
core-js: 3.48.0
'@lancedb/lancedb-darwin-arm64@0.22.3': {}
'@lancedb/lancedb-darwin-arm64@0.23.0':
optional: true
@@ -11097,8 +11091,6 @@ snapshots:
hasown: 2.0.2
side-channel: 1.1.0
ip-address@10.1.0: {}
ip-address@10.2.0: {}
ipaddr.js@1.9.1: {}
@@ -13585,7 +13577,7 @@ snapshots:
socks@2.8.7:
dependencies:
ip-address: 10.1.0
ip-address: 10.2.0
smart-buffer: 4.2.0
sonic-boom@4.2.1:
+15
View File
@@ -0,0 +1,15 @@
import path from 'node:path'
import { fileURLToPath } from 'node:url'
import { auditPackageDirectory } from './package-content-policy.mjs'
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..')
const stageDirectory = path.resolve(process.argv[2] || path.join(root, 'release-stage', 'app'))
const result = auditPackageDirectory(stageDirectory, {
allowedTopLevel: ['node_modules', 'out', 'package.json', 'resources'],
requireMinimalPackageJson: true
})
const bytes = result.files.reduce((sum, item) => sum + item.size, 0)
process.stdout.write(
`[package] staging audit passed: ${result.files.length} files, ${(bytes / 1024 / 1024).toFixed(1)} MiB\n`
)
+130
View File
@@ -0,0 +1,130 @@
/* eslint-disable @typescript-eslint/explicit-function-return-type */
import fs from 'node:fs'
import path from 'node:path'
import process from 'node:process'
import { listPackage, extractFile, getRawHeader } from '@electron/asar'
import {
assertMinimalPackageJson,
assertSafePackagePaths,
auditPackageDirectory,
scanTextForSecrets
} from './package-content-policy.mjs'
const inputDirectory = path.resolve(process.argv[2] || 'dist')
const reportPath = path.join(inputDirectory, 'package-audit.json')
const findFiles = (directory, predicate) => {
const result = []
const walk = (current) => {
for (const entry of fs.readdirSync(current, { withFileTypes: true })) {
const entryPath = path.join(current, entry.name)
if (entry.isDirectory()) walk(entryPath)
else if (entry.isFile() && predicate(entryPath)) result.push(entryPath)
}
}
walk(directory)
return result
}
const findApplicationRoot = (archivePath) => {
let current = path.dirname(archivePath)
while (current !== path.dirname(current)) {
if (current.endsWith('.app')) return current
current = path.dirname(current)
}
return path.dirname(path.dirname(archivePath))
}
const listArchiveFiles = (archivePath) => {
const files = []
const walk = (node, parent = '') => {
for (const [name, entry] of Object.entries(node.files || {})) {
const relativePath = parent ? `${parent}/${name}` : name
if (entry.files) walk(entry, relativePath)
else
files.push({
relativePath,
size: Number(entry.size || 0),
unpacked: Boolean(entry.unpacked)
})
}
}
walk(getRawHeader(archivePath).header)
return files
}
const formatBytes = (bytes) => `${(bytes / 1024 / 1024).toFixed(1)} MiB`
if (!fs.existsSync(inputDirectory)) throw new Error(`package output not found: ${inputDirectory}`)
const archives = findFiles(inputDirectory, (filePath) => path.basename(filePath) === 'app.asar')
if (archives.length === 0) throw new Error(`no packaged app.asar found under ${inputDirectory}`)
const applications = []
for (const archivePath of archives) {
const archiveEntries = listPackage(archivePath).map((entry) => entry.replace(/^\//u, ''))
assertSafePackagePaths(archiveEntries)
const archiveRoots = [...new Set(archiveEntries.map((entry) => entry.split('/')[0]))].sort()
const unexpectedRoots = archiveRoots.filter(
(name) => !['node_modules', 'out', 'package.json'].includes(name)
)
if (unexpectedRoots.length > 0) {
throw new Error(`unexpected app.asar root entries: ${unexpectedRoots.join(', ')}`)
}
const packagedPackageJson = JSON.parse(extractFile(archivePath, 'package.json').toString('utf8'))
assertMinimalPackageJson(packagedPackageJson, `${archivePath}:package.json`)
const archiveFiles = listArchiveFiles(archivePath)
for (const item of archiveFiles) {
if (!item.relativePath.startsWith('out/') || item.size > 25 * 1024 * 1024) continue
if (!/\.(?:js|cjs|mjs|json|html|css)$/u.test(item.relativePath)) continue
const source = extractFile(archivePath, item.relativePath).toString('utf8')
scanTextForSecrets(source, `${archivePath}:${item.relativePath}`)
}
const appRoot = findApplicationRoot(archivePath)
const directoryAudit = auditPackageDirectory(appRoot)
const totalBytes = directoryAudit.files.reduce((sum, item) => sum + item.size, 0)
const topFiles = [...directoryAudit.files]
.sort((left, right) => right.size - left.size)
.slice(0, 30)
.map((item) => ({ path: item.relativePath, bytes: item.size }))
applications.push({
root: path.relative(inputDirectory, appRoot) || '.',
archive: path.relative(inputDirectory, archivePath),
totalBytes,
archiveBytes: fs.statSync(archivePath).size,
archiveFileCount: archiveFiles.length,
unpackedPayloadBytes: archiveFiles
.filter((item) => item.unpacked)
.reduce((sum, item) => sum + item.size, 0),
topFiles
})
process.stdout.write(
`[package] audit passed: ${path.basename(appRoot)} ${formatBytes(totalBytes)}, app.asar ${formatBytes(fs.statSync(archivePath).size)}\n`
)
}
const installerExtensions = new Set(['.dmg', '.zip', '.exe', '.appimage', '.deb', '.snap'])
const installers = fs
.readdirSync(inputDirectory, { withFileTypes: true })
.filter(
(entry) => entry.isFile() && installerExtensions.has(path.extname(entry.name).toLowerCase())
)
.map((entry) => ({
name: entry.name,
bytes: fs.statSync(path.join(inputDirectory, entry.name)).size
}))
.sort((left, right) => right.bytes - left.bytes)
const report = {
schemaVersion: 1,
generatedAt: new Date().toISOString(),
platform: process.platform,
arch: process.arch,
applications,
installers
}
fs.writeFileSync(reportPath, `${JSON.stringify(report, null, 2)}\n`, 'utf8')
process.stdout.write(`[package] audit report: ${reportPath}\n`)
+54 -5
View File
@@ -11,6 +11,7 @@ const SIDECAR_DIR = path.join(ROOT, 'resources', 'web_crawler_sidecar')
const SIDECAR_SCRIPT_PATH = path.join(SIDECAR_DIR, 'web_crawler_sidecar.py')
const REQUIREMENTS_PATH = path.join(SIDECAR_DIR, 'requirements.txt')
const BUILD_SCRIPT_PATH = path.join(ROOT, 'scripts', 'build_web_crawler_sidecar.py')
const THIS_SCRIPT_PATH = fileURLToPath(import.meta.url)
const VENV_DIR = path.join(
ROOT,
'.runvenv',
@@ -21,6 +22,7 @@ const PLAYWRIGHT_BROWSER_STAMP_PATH = path.join(VENV_DIR, '.playwright-browser.s
const PIP_CACHE_DIR = path.join(ROOT, '.pip-cache', 'web_crawler_sidecar')
const BIN_DIR = path.join(SIDECAR_DIR, 'bin')
const SOURCE_STAMP_PATH = path.join(BIN_DIR, '.source.sha256')
const SIDECAR_RUNTIME_LAYOUT_VERSION = 'headless-shell-v1'
const pathExists = (targetPath) => fs.existsSync(targetPath)
@@ -110,16 +112,50 @@ const ensureVenv = (bootstrapPython) => {
return venvPythonPath
}
const computeRequirementsHash = () =>
crypto.createHash('sha256').update(fs.readFileSync(REQUIREMENTS_PATH)).digest('hex')
const computeRequirementsHash = () => {
const hash = crypto.createHash('sha256')
hash.update(fs.readFileSync(REQUIREMENTS_PATH))
hash.update(SIDECAR_RUNTIME_LAYOUT_VERSION)
return hash.digest('hex')
}
const computeSourceHash = () => {
const hash = crypto.createHash('sha256')
hash.update(fs.readFileSync(SIDECAR_SCRIPT_PATH))
hash.update(fs.readFileSync(REQUIREMENTS_PATH))
hash.update(fs.readFileSync(BUILD_SCRIPT_PATH))
hash.update(fs.readFileSync(THIS_SCRIPT_PATH))
hash.update(SIDECAR_RUNTIME_LAYOUT_VERSION)
return hash.digest('hex')
}
const hasCurrentBuiltSidecar = () => {
if (!pathExists(SOURCE_STAMP_PATH)) return false
if (fs.readFileSync(SOURCE_STAMP_PATH, 'utf8').trim() !== computeSourceHash()) return false
const executableName =
process.platform === 'win32' ? 'web_crawler_sidecar.exe' : 'web_crawler_sidecar'
if (!pathExists(path.join(BIN_DIR, 'web_crawler_sidecar', executableName))) return false
const browserRoots = []
const walk = (directory) => {
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
const entryPath = path.join(directory, entry.name)
if (entry.isDirectory()) {
if (entry.name === '.local-browsers') browserRoots.push(entryPath)
else walk(entryPath)
}
}
}
walk(BIN_DIR)
if (browserRoots.length !== 1) return false
const browserNames = fs.readdirSync(browserRoots[0])
return (
!browserNames.some((name) => name.startsWith('chromium-')) &&
browserNames.some((name) => name.startsWith('chromium_headless_shell-')) &&
browserNames.some((name) => name.startsWith('ffmpeg-'))
)
}
const readInstalledRequirementsHash = () =>
pathExists(STAMP_PATH) ? fs.readFileSync(STAMP_PATH, 'utf8').trim() : ''
@@ -150,9 +186,10 @@ const PLAYWRIGHT_BROWSER_CHECK_SNIPPET = [
'import pathlib',
'import playwright',
"base = pathlib.Path(playwright.__file__).resolve().parent / 'driver' / 'package' / '.local-browsers'",
"prefixes = ('chromium-', 'chromium_headless_shell-', 'ffmpeg-')",
"prefixes = ('chromium_headless_shell-', 'ffmpeg-')",
'names = {item.name for item in base.iterdir()} if base.exists() else set()',
'ok = all(any(name.startswith(prefix) for name in names) for prefix in prefixes)',
"has_full_chromium = any(name.startswith('chromium-') for name in names)",
'ok = not has_full_chromium and all(any(name.startswith(prefix) for name in names) for prefix in prefixes)',
'raise SystemExit(0 if ok else 1)'
].join('\n')
@@ -181,6 +218,11 @@ const restoreBundledPlaywrightBrowserCache = (pythonPath) => {
const target = resolveBundledPlaywrightBrowserDirectory(pythonPath)
fs.rmSync(target, { recursive: true, force: true })
fs.cpSync(source, target, { recursive: true, force: true })
for (const entry of fs.readdirSync(target)) {
if (entry.startsWith('chromium-')) {
fs.rmSync(path.join(target, entry), { recursive: true, force: true })
}
}
return hasBundledPlaywrightBrowser(pythonPath)
}
@@ -203,7 +245,9 @@ const ensureBundledPlaywrightBrowser = (pythonPath) => {
PLAYWRIGHT_BROWSERS_PATH: '0',
PLAYWRIGHT_DOWNLOAD_CONNECTION_TIMEOUT: '300000'
}
spawnChecked(pythonPath, ['-m', 'playwright', 'install', 'chromium'], { env: browserEnv })
spawnChecked(pythonPath, ['-m', 'playwright', 'install', '--only-shell', 'chromium'], {
env: browserEnv
})
fs.writeFileSync(PLAYWRIGHT_BROWSER_STAMP_PATH, `${expectedHash}\n`, 'utf8')
}
@@ -265,6 +309,11 @@ const main = () => {
throw new Error(`Missing sidecar requirements: ${REQUIREMENTS_PATH}`)
}
if (hasCurrentBuiltSidecar()) {
process.stdout.write(`[sidecar] current ${process.platform}-${process.arch} bundle reused\n`)
return
}
const bootstrapPython = resolveBootstrapPython()
const venvPythonPath = ensureVenv(bootstrapPython)
ensureSidecarDependencies(venvPythonPath)
+35 -3
View File
@@ -11,7 +11,7 @@ SIDECAR_SCRIPT = ROOT / "resources" / "web_crawler_sidecar" / "web_crawler_sidec
OUTPUT_DIR = ROOT / "resources" / "web_crawler_sidecar" / "bin"
SIDECAR_NAME = "web_crawler_sidecar"
PYINSTALLER_CONFIG_DIR = ROOT / ".pyinstaller" / "web_crawler_sidecar"
BUNDLED_PLAYWRIGHT_BROWSER_PREFIXES = ("chromium-", "chromium_headless_shell-", "ffmpeg-")
BUNDLED_PLAYWRIGHT_BROWSER_PREFIXES = ("chromium_headless_shell-", "ffmpeg-")
def platform_name() -> str:
@@ -45,6 +45,8 @@ def resolve_playwright_driver_path() -> Path:
def assert_playwright_browser_bundle(local_browsers_path: Path) -> None:
existing_names = {item.name for item in local_browsers_path.iterdir()} if local_browsers_path.exists() else set()
if any(name.startswith("chromium-") for name in existing_names):
raise RuntimeError("Full Chromium must not be bundled; install Chromium with --only-shell")
missing = [
prefix
for prefix in BUNDLED_PLAYWRIGHT_BROWSER_PREFIXES
@@ -69,6 +71,29 @@ def copy_playwright_driver_bundle(sidecar_dist_dir: Path) -> None:
shutil.copytree(long_path(source), long_path(target), symlinks=True)
def prune_non_runtime_files(sidecar_dist_dir: Path) -> None:
removable_extensions = {".map", ".ts", ".tsx", ".mts", ".cts", ".py", ".pyc", ".pyo"}
removable_directories = {"__pycache__", "tests", "test", "docs", "doc", "examples", "example", "types"}
patchright_dir = sidecar_dist_dir / "_internal" / "patchright"
if patchright_dir.exists():
shutil.rmtree(long_path(patchright_dir))
for current, directories, files in os.walk(sidecar_dist_dir, topdown=True):
current_path = Path(current)
kept_directories = []
for name in directories:
candidate = current_path / name
if name.lower() in removable_directories:
shutil.rmtree(long_path(candidate))
else:
kept_directories.append(name)
directories[:] = kept_directories
for name in files:
candidate = current_path / name
if candidate.suffix.lower() in removable_extensions:
candidate.unlink()
def stash_playwright_browser_bundle(build_path: Path) -> tuple[Path, Path]:
source = resolve_playwright_local_browsers_path()
assert_playwright_browser_bundle(source)
@@ -139,10 +164,16 @@ def main() -> int:
str(build_path / "work"),
"--specpath",
str(build_path / "spec"),
"--collect-all",
"--collect-submodules",
"crawl4ai",
"--collect-all",
"--collect-data",
"crawl4ai",
"--collect-submodules",
"ddgs",
"--collect-data",
"ddgs",
"--exclude-module",
"patchright",
str(SIDECAR_SCRIPT),
]
env = {
@@ -159,6 +190,7 @@ def main() -> int:
print(f"Missing built sidecar executable: {executable_path}", file=sys.stderr)
return 1
copy_playwright_driver_bundle(sidecar_dist_dir)
prune_non_runtime_files(sidecar_dist_dir)
return 0
+25
View File
@@ -46,6 +46,20 @@ if (!inputDirectory || !outputPath || !/^v\d+\.\d+\.\d+$/u.test(releaseTag)) {
const definition = definitions[platform]
if (!definition) throw new Error(`unsupported release platform: ${platform}`)
const expectedAuditPlatform = platform.startsWith('macos-') ? 'darwin' : 'win32'
const expectedAuditArch = platform.endsWith('-arm64') ? 'arm64' : 'x64'
const auditPath = path.join(inputDirectory, 'package-audit.json')
const audit = JSON.parse(await readFile(auditPath, 'utf8'))
if (
audit.schemaVersion !== 1 ||
audit.platform !== expectedAuditPlatform ||
audit.arch !== expectedAuditArch ||
!Array.isArray(audit.applications) ||
audit.applications.length === 0
) {
throw new Error(`package security audit does not match ${platform}: ${auditPath}`)
}
const version = releaseTag.slice(1)
const sha512 = async (filePath) =>
createHash('sha512')
@@ -79,6 +93,17 @@ await writeFile(
createdAt: new Date().toISOString(),
files,
metadata,
audit: {
generatedAt: audit.generatedAt,
platform: audit.platform,
arch: audit.arch,
applications: audit.applications.map((application) => ({
root: application.root,
totalBytes: application.totalBytes,
archiveBytes: application.archiveBytes,
archiveFileCount: application.archiveFileCount
}))
},
...(signature ? { signature } : {})
},
null,
+307
View File
@@ -0,0 +1,307 @@
/* eslint-disable @typescript-eslint/explicit-function-return-type */
import fs from 'node:fs'
import path from 'node:path'
const FORBIDDEN_EXTENSIONS = new Set([
'.adoc',
'.bat',
'.map',
'.ts',
'.tsx',
'.mts',
'.cts',
'.tsbuildinfo',
'.jsx',
'.vue',
'.svelte',
'.py',
'.pyc',
'.pyo',
'.c',
'.cc',
'.cpp',
'.h',
'.hpp',
'.cs',
'.fs',
'.fsx',
'.go',
'.java',
'.kt',
'.kts',
'.m',
'.mm',
'.php',
'.rb',
'.rs',
'.scala',
'.swift',
'.vb',
'.gyp',
'.gypi',
'.gradle',
'.sln',
'.vcxproj',
'.sh',
'.ps1',
'.cmd',
'.flow',
'.jsdoc',
'.patch',
'.orig',
'.md',
'.markdown',
'.rst',
'.key',
'.p12',
'.pfx',
'.pdb'
])
const FORBIDDEN_SEGMENTS = new Set([
'.changeset',
'.circleci',
'.claude',
'.codex',
'.github',
'.git',
'.history',
'.husky',
'.idea',
'.vscode',
'__tests__',
'docs',
'doc',
'coverage',
'examples',
'example',
'patchright',
'tests',
'test',
'scripts'
])
const FORBIDDEN_ROOT_SEGMENTS = new Set(['src', 'tests', 'scripts', 'doc'])
const FORBIDDEN_NAMES = new Set([
'.ds_store',
'.editorconfig',
'.eslintignore',
'.flowconfig',
'.gitattributes',
'.gitignore',
'.gitkeep',
'.keep',
'.nojekyll',
'.npmignore',
'.nycrc',
'.prettierignore',
'.taprc',
'pnpm-lock.yaml',
'pnpm-workspace.yaml',
'package-lock.json',
'yarn.lock',
'tsconfig.json',
'tsconfig.node.json',
'tsconfig.web.json'
])
const SECRET_PATTERNS = [
{
label: 'private key',
pattern:
/-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----\r?\n[A-Za-z0-9+/=\r\n]{80,}-----END (?:RSA |EC |OPENSSH )?PRIVATE KEY-----/u
},
{ label: 'AWS access key', pattern: /\bAKIA[0-9A-Z]{16}\b/u },
{
label: 'GitHub token',
pattern: /\b(?:github_pat_[A-Za-z0-9_]{20,}|gh[opusr]_[A-Za-z0-9]{30,})\b/u
},
{ label: 'Stripe live secret', pattern: /\bsk_live_[A-Za-z0-9]{20,}\b/u }
]
const normalizeRelativePath = (value) => value.replaceAll('\\', '/').replace(/^\/+|\/+$/gu, '')
const isLegalNotice = (name) =>
/^(?:license|licence|notice|copying)(?:[._-].*)?$/u.test(name) ||
/^(?:apache2?|bsd|mit)$/u.test(name)
export const assertSafePackagePaths = (paths) => {
const violations = []
const targetPrebuildDirectory = `${process.platform}-${process.arch}`
for (const rawPath of paths) {
const relativePath = normalizeRelativePath(rawPath)
if (!relativePath) continue
const segments = relativePath.split('/')
const lowerSegments = segments.map((segment) => segment.toLowerCase())
const name = lowerSegments.at(-1) || ''
const extension = path.extname(name)
const legalNotice = isLegalNotice(name)
const prebuildsIndex = lowerSegments.indexOf('prebuilds')
const prebuildDirectory = prebuildsIndex >= 0 ? lowerSegments[prebuildsIndex + 1] : undefined
if (
prebuildDirectory &&
/^(?:darwin|linux|win32)-(?:arm64|ia32|x64)$/u.test(prebuildDirectory) &&
prebuildDirectory !== targetPrebuildDirectory
) {
violations.push(`${relativePath}: foreign native prebuild ${prebuildDirectory}`)
continue
}
if (lowerSegments.some((segment) => segment.startsWith('chromium-'))) {
violations.push(`${relativePath}: full Chromium is forbidden; only headless shell is allowed`)
continue
}
if (FORBIDDEN_ROOT_SEGMENTS.has(lowerSegments[0])) {
violations.push(`${relativePath}: forbidden project directory ${lowerSegments[0]}`)
continue
}
if (lowerSegments.some((segment) => segment.startsWith('.env'))) {
violations.push(`${relativePath}: environment file`)
continue
}
const forbiddenSegment = lowerSegments.find((segment) => FORBIDDEN_SEGMENTS.has(segment))
if (forbiddenSegment) {
violations.push(`${relativePath}: forbidden directory ${forbiddenSegment}`)
continue
}
if (
FORBIDDEN_NAMES.has(name) ||
name.startsWith('.eslintrc') ||
name.startsWith('.prettierrc')
) {
violations.push(`${relativePath}: forbidden build file`)
continue
}
if (!legalNotice && FORBIDDEN_EXTENSIONS.has(extension)) {
violations.push(`${relativePath}: forbidden extension ${extension}`)
continue
}
if (/\.(?:spec|test)\.[^.]+$/u.test(name)) {
violations.push(`${relativePath}: test file`)
continue
}
if (name === 'trace.zip' || name.endsWith('.trace')) {
violations.push(`${relativePath}: test trace`)
}
}
if (violations.length > 0) {
throw new Error(`package content policy failed:\n${violations.slice(0, 100).join('\n')}`)
}
}
export const listDirectoryFiles = (rootDirectory) => {
const files = []
const symlinks = []
const walk = (directory) => {
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
const absolutePath = path.join(directory, entry.name)
const relativePath = normalizeRelativePath(path.relative(rootDirectory, absolutePath))
if (entry.isSymbolicLink()) {
symlinks.push({ absolutePath, relativePath, target: fs.readlinkSync(absolutePath) })
} else if (entry.isDirectory()) {
walk(absolutePath)
} else if (entry.isFile()) {
files.push({ absolutePath, relativePath, size: fs.statSync(absolutePath).size })
}
}
}
walk(rootDirectory)
return { files, symlinks }
}
export const assertSafeSymlinks = (rootDirectory, symlinks) => {
const root = path.resolve(rootDirectory)
const violations = []
for (const item of symlinks) {
if (path.isAbsolute(item.target)) {
violations.push(`${item.relativePath}: absolute symlink`)
continue
}
const resolved = path.resolve(path.dirname(item.absolutePath), item.target)
if (resolved !== root && !resolved.startsWith(`${root}${path.sep}`)) {
violations.push(`${item.relativePath}: symlink escapes package root`)
}
}
if (violations.length > 0) {
throw new Error(`unsafe package symlinks:\n${violations.join('\n')}`)
}
}
export const assertMinimalPackageJson = (packageJson, sourceLabel) => {
const allowedKeys = new Set([
'name',
'version',
'description',
'main',
'author',
'homepage',
'private',
'dependencies'
])
const extraKeys = Object.keys(packageJson).filter((key) => !allowedKeys.has(key))
if (extraKeys.length > 0) {
throw new Error(`${sourceLabel} contains development metadata: ${extraKeys.join(', ')}`)
}
if (packageJson.main !== './out/main/index.js') {
throw new Error(`${sourceLabel} has an unexpected main entry`)
}
if (!packageJson.private || !packageJson.name || !packageJson.version) {
throw new Error(`${sourceLabel} is missing required production metadata`)
}
}
export const scanTextForSecrets = (source, sourceLabel) => {
for (const item of SECRET_PATTERNS) {
if (item.pattern.test(source)) {
throw new Error(`${sourceLabel} contains a possible ${item.label}`)
}
}
if (/(?:\/\/[#@]\s*sourceMappingURL\s*=|\/\*[#@]\s*sourceMappingURL\s*=)/u.test(source)) {
throw new Error(`${sourceLabel} references a source map`)
}
}
export const auditPackageDirectory = (rootDirectory, options = {}) => {
const root = path.resolve(rootDirectory)
if (!fs.existsSync(root) || !fs.statSync(root).isDirectory()) {
throw new Error(`package directory not found: ${root}`)
}
const { files, symlinks } = listDirectoryFiles(root)
assertSafePackagePaths([
...files.map((item) => item.relativePath),
...symlinks.map((item) => item.relativePath)
])
assertSafeSymlinks(root, symlinks)
if (options.allowedTopLevel) {
const allowed = new Set(options.allowedTopLevel)
const unexpected = fs.readdirSync(root).filter((name) => !allowed.has(name))
if (unexpected.length > 0) {
throw new Error(`unexpected package root entries: ${unexpected.join(', ')}`)
}
}
const packageJsonPath = path.join(root, 'package.json')
if (options.requireMinimalPackageJson) {
const packageJson = JSON.parse(fs.readFileSync(packageJsonPath, 'utf8'))
assertMinimalPackageJson(packageJson, packageJsonPath)
}
for (const item of files) {
if (
item.size <= 5 * 1024 * 1024 &&
/\.(?:c?js|mjs|json|html|css|ya?ml|toml|xml|pem|txt)$/u.test(item.relativePath)
) {
scanTextForSecrets(fs.readFileSync(item.absolutePath, 'utf8'), item.relativePath)
}
if (!item.relativePath.startsWith('out/') || item.size > 25 * 1024 * 1024) continue
if (!/\.(?:js|cjs|mjs|json|html|css)$/u.test(item.relativePath)) continue
scanTextForSecrets(fs.readFileSync(item.absolutePath, 'utf8'), item.relativePath)
}
return { files, symlinks }
}
+423
View File
@@ -0,0 +1,423 @@
/* eslint-disable @typescript-eslint/explicit-function-return-type */
import fs from 'node:fs'
import path from 'node:path'
import process from 'node:process'
import { spawnSync } from 'node:child_process'
import { fileURLToPath } from 'node:url'
import { parse, stringify } from 'yaml'
import { collectRuntimePackageNames } from './runtime-bundle-graph.mjs'
const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..')
const STAGE_DIR = path.join(ROOT, 'release-stage', 'app')
const OUT_DIR = path.join(ROOT, 'out')
const DIST_DIR = path.join(ROOT, 'dist')
const ROOT_PACKAGE_PATH = path.join(ROOT, 'package.json')
const ROOT_LOCKFILE_PATH = path.join(ROOT, 'pnpm-lock.yaml')
const SIDECAR_BIN_DIR = path.join(ROOT, 'resources', 'web_crawler_sidecar', 'bin')
const NATIVE_RESOURCE_DIR = path.join(ROOT, 'resources', 'native')
const assertDirectory = (directory, label) => {
if (!fs.existsSync(directory) || !fs.statSync(directory).isDirectory()) {
throw new Error(`${label} not found: ${directory}`)
}
}
const listJavaScriptFiles = (directory) => {
const result = []
const walk = (current) => {
for (const entry of fs.readdirSync(current, { withFileTypes: true })) {
const entryPath = path.join(current, entry.name)
if (entry.isDirectory()) walk(entryPath)
else if (entry.isFile() && /\.(?:js|cjs|mjs)$/u.test(entry.name)) result.push(entryPath)
}
}
walk(directory)
return result
}
const resolveInstalledVersion = (packageName) => {
const packageJsonPath = path.join(ROOT, 'node_modules', ...packageName.split('/'), 'package.json')
if (!fs.existsSync(packageJsonPath)) {
throw new Error(`runtime dependency is not installed at the workspace root: ${packageName}`)
}
const packageJson = JSON.parse(fs.readFileSync(packageJsonPath, 'utf8'))
const version = String(packageJson.version || '').trim()
if (!version) throw new Error(`runtime dependency has no version: ${packageName}`)
return version
}
const resolvePnpmCommand = () => {
const npmExecPath = String(process.env.npm_execpath || '').trim()
if (npmExecPath && /\.(?:c?js|mjs)$/u.test(npmExecPath)) {
return { command: process.execPath, prefixArgs: [npmExecPath] }
}
return { command: process.platform === 'win32' ? 'pnpm.cmd' : 'pnpm', prefixArgs: [] }
}
const installRuntimeDependencies = () => {
const pnpm = resolvePnpmCommand()
const args = [
...pnpm.prefixArgs,
'install',
'--prod',
'--ignore-scripts',
'--ignore-workspace',
'--frozen-lockfile',
'--offline',
'--package-import-method',
'copy',
'--config.node-linker=hoisted',
'--config.auto-install-peers=false'
]
const result = spawnSync(pnpm.command, args, {
cwd: STAGE_DIR,
env: { ...process.env, NODE_ENV: 'production' },
stdio: 'inherit'
})
if (result.error) throw result.error
if (result.status !== 0) {
throw new Error(`staging dependency install failed with code ${String(result.status)}`)
}
}
const REMOVE_DIRECTORY_NAMES = new Set([
'.changeset',
'.circleci',
'.github',
'.claude',
'.history',
'.husky',
'.idea',
'.vscode',
'__tests__',
'benchmarks',
'benchmark',
'coverage',
'docs',
'doc',
'examples',
'example',
'scripts',
'tests',
'test'
])
const REMOVE_EXTENSIONS = new Set([
'.adoc',
'.bat',
'.map',
'.ts',
'.tsx',
'.mts',
'.cts',
'.tsbuildinfo',
'.jsx',
'.vue',
'.svelte',
'.py',
'.pyc',
'.pyo',
'.c',
'.cc',
'.cpp',
'.h',
'.hpp',
'.cs',
'.fs',
'.fsx',
'.go',
'.java',
'.kt',
'.kts',
'.m',
'.mm',
'.php',
'.rb',
'.rs',
'.scala',
'.swift',
'.vb',
'.gyp',
'.gypi',
'.gradle',
'.sln',
'.vcxproj',
'.sh',
'.ps1',
'.cmd',
'.flow',
'.jsdoc',
'.patch',
'.orig',
'.md',
'.markdown',
'.rst',
'.pdb'
])
const isLegalNotice = (name) =>
/^(?:license|licence|notice|copying)(?:[._-].*)?$/u.test(name) ||
/^(?:apache2?|bsd|mit)$/u.test(name)
const shouldRemoveFile = (filePath) => {
const name = path.basename(filePath).toLowerCase()
if (isLegalNotice(name)) return false
if (REMOVE_EXTENSIONS.has(path.extname(name))) return true
return (
/\.(?:spec|test)\.[^.]+$/u.test(name) ||
/^tsconfig(?:\.[^.]+)?\.json$/u.test(name) ||
name === 'readme' ||
name.startsWith('readme.') ||
name.startsWith('changelog.') ||
name.startsWith('history.') ||
name === '.npmignore' ||
name === '.ds_store' ||
name === '.editorconfig' ||
name === '.eslintignore' ||
name === '.flowconfig' ||
name === '.gitattributes' ||
name === '.gitignore' ||
name === '.gitkeep' ||
name === '.keep' ||
name === '.nojekyll' ||
name === '.nycrc' ||
name === '.prettierignore' ||
name === '.taprc' ||
name.startsWith('.eslintrc') ||
name.startsWith('.prettierrc') ||
name === 'yarn.lock' ||
name === 'package-lock.json' ||
name === 'pnpm-lock.yaml'
)
}
const pruneCompiledOutput = () => {
const output = path.join(STAGE_DIR, 'out')
const walk = (directory) => {
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
const entryPath = path.join(directory, entry.name)
if (entry.isDirectory()) walk(entryPath)
else if (entry.isFile() && shouldRemoveFile(entryPath)) {
fs.rmSync(entryPath, { force: true })
}
}
}
walk(output)
}
const stripSourceMapReferences = () => {
const textExtensions = new Set(['.js', '.cjs', '.mjs', '.css'])
const walk = (directory) => {
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
const entryPath = path.join(directory, entry.name)
if (entry.isDirectory()) {
walk(entryPath)
continue
}
if (!entry.isFile() || !textExtensions.has(path.extname(entry.name).toLowerCase())) continue
const source = fs.readFileSync(entryPath, 'utf8')
const stripped = source
.replace(/\/\/[#@]\s*sourceMappingURL\s*=.*?(?=\r?\n|$)/gu, '')
.replace(/\/\*[#@]\s*sourceMappingURL\s*=.*?\*\//gsu, '')
if (stripped !== source) fs.writeFileSync(entryPath, stripped, 'utf8')
}
}
walk(STAGE_DIR)
}
const pruneNodeModules = () => {
const nodeModules = path.join(STAGE_DIR, 'node_modules')
const targetPrebuildDirectory = `${process.platform}-${process.arch}`
const prunePlatformPrebuilds = (directory) => {
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
if (!entry.isDirectory()) continue
if (!/^(?:darwin|linux|win32)-(?:arm64|ia32|x64)$/u.test(entry.name)) continue
if (entry.name !== targetPrebuildDirectory) {
fs.rmSync(path.join(directory, entry.name), { recursive: true, force: true })
}
}
}
const reducePackageToManifest = (directory) => {
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
if (entry.isFile() && ['package.json', 'LICENSE', 'NOTICE'].includes(entry.name)) continue
fs.rmSync(path.join(directory, entry.name), { recursive: true, force: true })
}
}
const walk = (directory) => {
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
const entryPath = path.join(directory, entry.name)
const lowerName = entry.name.toLowerCase()
if (entry.isSymbolicLink()) continue
if (entry.isDirectory()) {
if (lowerName === 'prebuilds') {
prunePlatformPrebuilds(entryPath)
}
if (lowerName === 'onnxruntime-web') {
reducePackageToManifest(entryPath)
continue
}
if (
REMOVE_DIRECTORY_NAMES.has(lowerName) ||
(directory === nodeModules && lowerName === '.bin')
) {
fs.rmSync(entryPath, { recursive: true, force: true })
continue
}
walk(entryPath)
if (fs.existsSync(entryPath) && fs.readdirSync(entryPath).length === 0) {
fs.rmdirSync(entryPath)
}
} else if (entry.isFile() && shouldRemoveFile(entryPath)) {
fs.rmSync(entryPath, { force: true })
}
}
}
walk(nodeModules)
const onnxRoot = path.join(nodeModules, 'onnxruntime-node', 'bin', 'napi-v3')
if (fs.existsSync(onnxRoot)) {
const expectedPlatform = process.platform
const expectedArch = process.arch
for (const platformEntry of fs.readdirSync(onnxRoot, { withFileTypes: true })) {
const platformPath = path.join(onnxRoot, platformEntry.name)
if (!platformEntry.isDirectory()) continue
if (platformEntry.name !== expectedPlatform) {
fs.rmSync(platformPath, { recursive: true, force: true })
continue
}
for (const archEntry of fs.readdirSync(platformPath, { withFileTypes: true })) {
if (archEntry.isDirectory() && archEntry.name !== expectedArch) {
fs.rmSync(path.join(platformPath, archEntry.name), { recursive: true, force: true })
}
}
}
const expectedBinaryDirectory = path.join(onnxRoot, expectedPlatform, expectedArch)
assertDirectory(expectedBinaryDirectory, 'target ONNX Runtime binary directory')
}
const transformersDist = path.join(nodeModules, '@huggingface', 'transformers', 'dist')
if (fs.existsSync(transformersDist)) {
for (const entry of fs.readdirSync(transformersDist, { withFileTypes: true })) {
if (entry.isFile() && entry.name.endsWith('.wasm')) {
fs.rmSync(path.join(transformersDist, entry.name), { force: true })
}
}
}
}
const pruneStagedResources = () => {
const resources = path.join(STAGE_DIR, 'resources')
const walk = (directory) => {
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
const entryPath = path.join(directory, entry.name)
const lowerName = entry.name.toLowerCase()
if (entry.isSymbolicLink()) continue
if (entry.isDirectory()) {
if (
REMOVE_DIRECTORY_NAMES.has(lowerName) ||
lowerName === 'patchright' ||
lowerName === 'types' ||
lowerName.startsWith('chromium-')
) {
fs.rmSync(entryPath, { recursive: true, force: true })
continue
}
walk(entryPath)
} else if (entry.isFile() && shouldRemoveFile(entryPath)) {
fs.rmSync(entryPath, { force: true })
}
}
}
walk(resources)
}
const writeStageLockfile = (runtimeDependencies) => {
const lockfile = parse(fs.readFileSync(ROOT_LOCKFILE_PATH, 'utf8'))
const rootImporter = lockfile.importers?.['.']
if (!rootImporter?.dependencies) throw new Error('workspace lockfile has no root dependencies')
const dependencies = {}
for (const [name, version] of Object.entries(runtimeDependencies)) {
const locked = rootImporter.dependencies[name]
if (!locked) throw new Error(`runtime dependency is missing from pnpm-lock.yaml: ${name}`)
dependencies[name] = { ...locked, specifier: version }
}
lockfile.settings = { ...lockfile.settings, autoInstallPeers: false }
lockfile.importers = { '.': { dependencies } }
fs.writeFileSync(path.join(STAGE_DIR, 'pnpm-lock.yaml'), stringify(lockfile), 'utf8')
}
const main = () => {
assertDirectory(OUT_DIR, 'compiled application output')
assertDirectory(SIDECAR_BIN_DIR, 'web crawler sidecar binary')
assertDirectory(NATIVE_RESOURCE_DIR, 'native resource directory')
const rootPackage = JSON.parse(fs.readFileSync(ROOT_PACKAGE_PATH, 'utf8'))
const runtimeEntries = [
...listJavaScriptFiles(path.join(OUT_DIR, 'main')),
...listJavaScriptFiles(path.join(OUT_DIR, 'preload'))
]
const runtimePackageNames = collectRuntimePackageNames(runtimeEntries)
const undeclared = runtimePackageNames.filter((name) => !rootPackage.dependencies?.[name])
if (undeclared.length > 0) {
throw new Error(`runtime bundles import undeclared dependencies: ${undeclared.join(', ')}`)
}
const runtimeDependencies = Object.fromEntries(
runtimePackageNames.map((name) => [name, resolveInstalledVersion(name)])
)
const stagePackage = {
name: rootPackage.name,
version: rootPackage.version,
description: rootPackage.description,
main: './out/main/index.js',
author: rootPackage.author,
homepage: rootPackage.homepage,
private: true,
dependencies: runtimeDependencies
}
fs.rmSync(DIST_DIR, { recursive: true, force: true })
fs.rmSync(STAGE_DIR, { recursive: true, force: true })
fs.mkdirSync(STAGE_DIR, { recursive: true })
fs.cpSync(OUT_DIR, path.join(STAGE_DIR, 'out'), { recursive: true, force: true })
fs.mkdirSync(path.join(STAGE_DIR, 'resources', 'web_crawler_sidecar'), { recursive: true })
fs.cpSync(SIDECAR_BIN_DIR, path.join(STAGE_DIR, 'resources', 'web_crawler_sidecar', 'bin'), {
recursive: true,
force: true,
verbatimSymlinks: true
})
fs.cpSync(NATIVE_RESOURCE_DIR, path.join(STAGE_DIR, 'resources', 'native'), {
recursive: true,
force: true,
verbatimSymlinks: true
})
fs.rmSync(path.join(STAGE_DIR, 'resources', 'web_crawler_sidecar', 'bin', '.source.sha256'), {
force: true
})
fs.writeFileSync(
path.join(STAGE_DIR, 'package.json'),
`${JSON.stringify(stagePackage, null, 2)}\n`
)
writeStageLockfile(runtimeDependencies)
installRuntimeDependencies()
fs.rmSync(path.join(STAGE_DIR, 'pnpm-lock.yaml'), { force: true })
pruneCompiledOutput()
pruneNodeModules()
pruneStagedResources()
stripSourceMapReferences()
const audit = spawnSync(
process.execPath,
[path.join(ROOT, 'scripts', 'audit-package-stage.mjs'), STAGE_DIR],
{ cwd: ROOT, stdio: 'inherit' }
)
if (audit.error) throw audit.error
if (audit.status !== 0) throw new Error('staging package security audit failed')
process.stdout.write(
`[package] staging ready for ${process.platform}-${process.arch}; runtime roots: ${runtimePackageNames.join(', ')}\n`
)
}
main()
+27
View File
@@ -0,0 +1,27 @@
/* eslint-disable @typescript-eslint/explicit-function-return-type */
import fs from 'node:fs'
import path from 'node:path'
const reportPath = path.resolve(process.argv[2] || 'dist/package-audit.json')
const report = JSON.parse(fs.readFileSync(reportPath, 'utf8'))
const formatSize = (bytes) => `${(Number(bytes || 0) / 1024 / 1024).toFixed(1)} MiB`
process.stdout.write('## Package security audit\n\n')
process.stdout.write(`Platform: ${report.platform}-${report.arch}\n\n`)
process.stdout.write('| Type | Name | Size |\n| --- | --- | ---: |\n')
for (const application of report.applications || []) {
process.stdout.write(
`| Installed app | ${application.root} | ${formatSize(application.totalBytes)} |\n`
)
process.stdout.write(
`| app.asar | ${application.archive} | ${formatSize(application.archiveBytes)} |\n`
)
}
for (const installer of report.installers || []) {
process.stdout.write(
`| Installer/update | ${installer.name} | ${formatSize(installer.bytes)} |\n`
)
}
process.stdout.write(
'\nThe source, secret and forbidden-file audit passed. Numeric size budgets are not enabled yet.\n'
)
+68
View File
@@ -0,0 +1,68 @@
/* eslint-disable @typescript-eslint/explicit-function-return-type */
import fs from 'node:fs'
import { builtinModules } from 'node:module'
import path from 'node:path'
const BUILTIN_MODULES = new Set([
...builtinModules,
...builtinModules.map((name) => `node:${name}`),
'electron'
])
const resolveLocalModule = (importerPath, specifier) => {
if (!specifier.startsWith('.')) return null
const candidate = path.resolve(path.dirname(importerPath), specifier)
const candidates = [candidate, `${candidate}.js`, path.join(candidate, 'index.js')]
return candidates.find((item) => fs.existsSync(item)) || null
}
export const readBundleGraph = (entryPath) => {
if (!fs.existsSync(entryPath)) {
throw new Error(`runtime bundle not found: ${entryPath}`)
}
const pending = [entryPath]
const visited = new Set()
const externalImports = new Set()
while (pending.length > 0) {
const currentPath = pending.pop()
if (!currentPath || visited.has(currentPath)) continue
visited.add(currentPath)
const source = fs.readFileSync(currentPath, 'utf8')
const importPatterns = [
/require\(["']([^"']+)["']\)/gu,
/require\.resolve\(["']([^"']+)["']\)/gu,
/from\s+["']([^"']+)["']/gu,
/import\(["']([^"']+)["']\)/gu
]
for (const pattern of importPatterns) {
for (const match of source.matchAll(pattern)) {
const specifier = match[1]
const localPath = resolveLocalModule(currentPath, specifier)
if (localPath) pending.push(localPath)
else externalImports.add(specifier)
}
}
}
return { visited, externalImports }
}
export const packageNameFromSpecifier = (specifier) => {
if (!specifier || BUILTIN_MODULES.has(specifier) || specifier.startsWith('node:')) return null
if (specifier.startsWith('.') || path.isAbsolute(specifier)) return null
const parts = specifier.split('/')
return specifier.startsWith('@') ? parts.slice(0, 2).join('/') : parts[0]
}
export const collectRuntimePackageNames = (entryPaths) => {
const packageNames = new Set()
for (const entryPath of entryPaths) {
const graph = readBundleGraph(entryPath)
for (const specifier of graph.externalImports) {
const packageName = packageNameFromSpecifier(specifier)
if (packageName) packageNames.add(packageName)
}
}
return [...packageNames].sort()
}
+5 -44
View File
@@ -1,50 +1,11 @@
import fs from 'node:fs'
/* eslint-disable @typescript-eslint/explicit-function-return-type */
import path from 'node:path'
import { fileURLToPath } from 'node:url'
import { readBundleGraph } from './runtime-bundle-graph.mjs'
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..')
const mainOutDir = path.join(root, 'out', 'main')
const resolveLocalModule = (importerPath, specifier) => {
if (!specifier.startsWith('.')) return null
const candidate = path.resolve(path.dirname(importerPath), specifier)
if (fs.existsSync(candidate)) return candidate
if (fs.existsSync(`${candidate}.js`)) return `${candidate}.js`
return null
}
const readBundleGraph = (entryName) => {
const entryPath = path.join(mainOutDir, entryName)
if (!fs.existsSync(entryPath)) {
throw new Error(`runtime bundle not found: ${entryPath}`)
}
const pending = [entryPath]
const visited = new Set()
const externalImports = new Set()
while (pending.length > 0) {
const currentPath = pending.pop()
if (!currentPath || visited.has(currentPath)) continue
visited.add(currentPath)
const source = fs.readFileSync(currentPath, 'utf8')
const importPatterns = [
/require\(["']([^"']+)["']\)/gu,
/from\s+["']([^"']+)["']/gu,
/import\(["']([^"']+)["']\)/gu
]
for (const pattern of importPatterns) {
for (const match of source.matchAll(pattern)) {
const specifier = match[1]
const localPath = resolveLocalModule(currentPath, specifier)
if (localPath) pending.push(localPath)
else externalImports.add(specifier)
}
}
}
return { visited, externalImports }
}
const assertNoImports = (label, graph, forbidden) => {
const found = forbidden.filter((name) => graph.externalImports.has(name))
if (found.length > 0) {
@@ -52,9 +13,9 @@ const assertNoImports = (label, graph, forbidden) => {
}
}
const mainGraph = readBundleGraph('index.js')
const prepareWorkerGraph = readBundleGraph('workspacePrepareWorker.js')
const embeddingRuntimeGraph = readBundleGraph('embeddingRuntimeProcess.js')
const mainGraph = readBundleGraph(path.join(mainOutDir, 'index.js'))
const prepareWorkerGraph = readBundleGraph(path.join(mainOutDir, 'workspacePrepareWorker.js'))
const embeddingRuntimeGraph = readBundleGraph(path.join(mainOutDir, 'embeddingRuntimeProcess.js'))
assertNoImports('Electron main process', mainGraph, [
'@huggingface/transformers',
@@ -28,6 +28,24 @@ describe('release fragment', () => {
'version: 0.1.3\npath: Ncurator-windows-x64.exe\n',
'utf8'
)
await writeFile(
path.join(input, 'package-audit.json'),
`${JSON.stringify({
schemaVersion: 1,
generatedAt: '2026-07-15T00:00:00.000Z',
platform: 'win32',
arch: 'x64',
applications: [
{
root: 'win-unpacked',
totalBytes: 1_000,
archiveBytes: 500,
archiveFileCount: 20
}
]
})}\n`,
'utf8'
)
await execFileAsync(process.execPath, [
path.resolve('scripts/create-release-fragment.mjs'),
@@ -41,6 +59,19 @@ describe('release fragment', () => {
expect(fragment.tag).toBe('v0.1.3')
expect(fragment.platform).toBe('windows-x64')
expect(fragment.metadata).toContain('version: 0.1.3')
expect(fragment.audit).toEqual({
generatedAt: '2026-07-15T00:00:00.000Z',
platform: 'win32',
arch: 'x64',
applications: [
{
root: 'win-unpacked',
totalBytes: 1_000,
archiveBytes: 500,
archiveFileCount: 20
}
]
})
expect(fragment.files).toEqual(
expect.arrayContaining([
expect.objectContaining({
@@ -52,4 +83,21 @@ describe('release fragment', () => {
])
)
})
it('rejects release assets without a package security audit', async () => {
temporaryDirectory = await mkdtemp(path.join(os.tmpdir(), 'ncurator-release-fragment-'))
const input = path.join(temporaryDirectory, 'input')
const output = path.join(temporaryDirectory, 'windows.json')
await mkdir(input, { recursive: true })
await expect(
execFileAsync(process.execPath, [
path.resolve('scripts/create-release-fragment.mjs'),
input,
output,
'v0.1.3',
'windows-x64'
])
).rejects.toThrow(/package-audit\.json/u)
})
})