docs: add VitePress documentation site (#112)

- Add VitePress with full navigation, sidebar, i18n (en/zh), local search
- Create 50+ doc pages: guide, adapters (browser + desktop), developer, advanced
- Migrate content from README.md, CONTRIBUTING.md, TESTING.md, CDP.md, CLI-ELECTRON.md
- Migrate all 11 adapter READMEs to structured documentation
- Add new pages: architecture, yaml-adapter guide, ts-adapter guide, ai-workflow
- Add GitHub Actions workflow for deploying to GitHub Pages
- Add Chinese locale pages (getting-started, installation, browser-bridge, etc.)
- Add docs:dev, docs:build, docs:preview npm scripts
This commit is contained in:
AlexYue authored and GitHub committed 2026-03-20 14:11:56 +08:00
1 parent 812db29ed8
commit 31f58ae699
59 files changed
+5138 -5

No files matched your search

+66
View File
@@ -0,0 +1,66 @@
# AI Workflow
OpenCLI is designed with AI agents in mind. This guide covers the AI-native discovery and code generation tools.
## Quick Mode (One-Shot)
Generate a single command for a specific page URL — just a URL + one-line goal, 4 steps done:
```bash
opencli generate https://example.com --goal "trending"
```
This runs: explore → synthesize → register in one shot.
For the complete one-shot workflow details, see [CLI-ONESHOT.md](https://github.com/jackwener/opencli/blob/main/CLI-ONESHOT.md).
## Full Mode (Explorer Workflow)
### Step 1: Deep Explore
Discover APIs, infer capabilities, and detect framework:
```bash
opencli explore https://example.com --site mysite
```
Outputs to `.opencli/explore/<site>/`:
- `manifest.json` — Site metadata
- `endpoints.json` — Discovered API endpoints
- `capabilities.json` — Inferred capabilities
- `auth.json` — Authentication strategy details
### Step 2: Synthesize
Generate YAML adapters from explore artifacts:
```bash
opencli synthesize mysite
```
### Step 3: Strategy Cascade
Auto-probe authentication strategies: `PUBLIC → COOKIE → HEADER`:
```bash
opencli cascade https://api.example.com/data
```
### Step 4: Validate & Test
```bash
opencli validate # Validate generated YAML
opencli <site> <command> --limit 3 -f json # Test the command
```
## 5-Tier Authentication Strategy
The explorer uses a decision tree to determine the best authentication approach:
1. **PUBLIC** — No auth, direct API call
2. **COOKIE** — Reuse Chrome session cookies
3. **HEADER** — Custom auth headers
4. **BROWSER** — Full browser automation
5. **CDP** — Chrome DevTools Protocol for Electron apps
For the complete browser exploration workflow and debugging guide, see [CLI-EXPLORER.md](https://github.com/jackwener/opencli/blob/main/CLI-EXPLORER.md).
+90
View File
@@ -0,0 +1,90 @@
# Architecture
OpenCLI is built on a **Dual-Engine Architecture** that supports both declarative YAML pipelines and programmatic TypeScript adapters.
## High-Level Architecture
```
┌─────────────────────────────────────────────────────┐
│ opencli CLI │
│ (Commander.js entry point) │
├─────────────────────────────────────────────────────┤
│ Engine Layer │
│ ┌──────────────┐ ┌──────────────┐ ┌────────────┐ │
│ │ Registry │ │ Dynamic │ │ Output │ │
│ │ (commands) │ │ Loader │ │ Formatter │ │
│ └──────────────┘ └──────────────┘ └────────────┘ │
├─────────────────────────────────────────────────────┤
│ Adapter Layer │
│ ┌─────────────────┐ ┌──────────────────────────┐ │
│ │ YAML Pipeline │ │ TypeScript Adapters │ │
│ │ (declarative) │ │ (browser/desktop/AI) │ │
│ └─────────────────┘ └──────────────────────────┘ │
├─────────────────────────────────────────────────────┤
│ Connection Layer │
│ ┌─────────────────┐ ┌──────────────────────────┐ │
│ │ Browser Bridge │ │ CDP (Chrome DevTools) │ │
│ │ (Extension+WS) │ │ (Electron apps) │ │
│ └─────────────────┘ └──────────────────────────┘ │
└─────────────────────────────────────────────────────┘
```
## Core Modules
### Registry (`src/registry.ts`)
Central command registry. All adapters register their commands via the `cli()` function with metadata: site, name, description, domain, strategy, args, columns.
### Engine (`src/engine.ts`)
Command discovery and execution engine. Discovers commands from the registry, parses arguments, executes the appropriate adapter, and routes output through the formatter.
### Browser (`src/browser.ts`)
Manages connections to Chrome via the Browser Bridge WebSocket daemon. Handles JSON-RPC messaging, tab management, and extension/standalone mode switching.
### Pipeline (`src/pipeline/`)
The YAML pipeline engine. Processes declarative steps:
- **fetch** — HTTP requests with cookie/header strategies
- **map** — Data transformation with template expressions
- **limit** — Result truncation
- **filter** — Conditional filtering
- **download** — Media download support
### Output (`src/output.ts`)
Unified output formatting: `table`, `json`, `yaml`, `md`, `csv`.
## Authentication Strategies
OpenCLI uses a 3-tier authentication strategy:
| Strategy | How It Works | When to Use |
|----------|-------------|-------------|
| `public` | Direct HTTP fetch, no auth | Public APIs (HackerNews, BBC) |
| `cookie` | Reuse Chrome cookies via Browser Bridge | Logged-in sites (Bilibili, Zhihu) |
| `header` | Custom auth headers | API-key based services |
## Directory Structure
```
src/
├── main.ts # Entry point
├── engine.ts # Command execution engine
├── registry.ts # Command registry
├── browser.ts # Browser Bridge connection
├── output.ts # Output formatting
├── doctor.ts # Diagnostic tool
├── pipeline/ # YAML pipeline engine
│ ├── runner.ts
│ ├── template.ts
│ ├── transform.ts
│ └── steps/
│ ├── fetch.ts
│ ├── map.ts
│ ├── limit.ts
│ ├── filter.ts
│ └── download.ts
└── clis/ # Site adapters
├── twitter/
├── reddit/
├── bilibili/
├── cursor/
└── ...
```
+136
View File
@@ -0,0 +1,136 @@
# Contributing
Thanks for your interest in contributing to OpenCLI.
## Quick Start
```bash
# 1. Fork & clone
git clone git@github.com:<your-username>/opencli.git
cd opencli
# 2. Install dependencies
npm install
# 3. Build
npm run build
# 4. Run a few checks
npx tsc --noEmit
npx vitest run src/
# 5. Link globally (optional, for testing `opencli` command)
npm link
```
## Adding a New Site Adapter
This is the most common type of contribution. Start with YAML when possible, and use TypeScript only when you need browser-side logic or multi-step flows.
### YAML Adapter (Recommended for data-fetching commands)
Create a file like `src/clis/<site>/<command>.yaml`:
::: v-pre
```yaml
site: mysite
name: trending
description: Trending posts on MySite
domain: www.mysite.com
strategy: public # public | cookie | header
browser: false # true if browser session is needed
args:
limit:
type: int
default: 20
description: Number of items
pipeline:
- fetch:
url: https://api.mysite.com/trending
- map:
rank: ${{ index + 1 }}
title: ${{ item.title }}
score: ${{ item.score }}
url: ${{ item.url }}
- limit: ${{ args.limit }}
columns: [rank, title, score, url]
```
:::
See [`hackernews/top.yaml`](https://github.com/jackwener/opencli/blob/main/src/clis/hackernews/top.yaml) for a real example.
### TypeScript Adapter (For complex browser interactions)
Create a file like `src/clis/<site>/<command>.ts`:
```typescript
import { cli, Strategy } from '../../registry.js';
cli({
site: 'mysite',
name: 'search',
description: 'Search MySite',
domain: 'www.mysite.com',
strategy: Strategy.COOKIE,
args: [
{ name: 'query', required: true, help: 'Search query' },
{ name: 'limit', type: 'int', default: 10, help: 'Max results' },
],
columns: ['title', 'url', 'date'],
func: async (page, kwargs) => {
const { query, limit = 10 } = kwargs;
// ... browser automation logic
return data.slice(0, Number(limit)).map((item: any) => ({
title: item.title,
url: item.url,
date: item.created_at,
}));
},
});
```
### Validate Your Adapter
```bash
opencli validate # Validate YAML syntax and schema
opencli <site> <command> --limit 3 -f json # Test your command
opencli <site> <command> -v # Verbose mode for debugging
```
## Code Style
- **TypeScript strict mode** — avoid `any` where possible.
- **ES Modules** — use `.js` extensions in imports (TypeScript output).
- **Naming**: `kebab-case` for files, `camelCase` for variables/functions, `PascalCase` for types/classes.
- **No default exports** — use named exports.
## Commit Convention
We use [Conventional Commits](https://www.conventionalcommits.org/):
```
feat(twitter): add thread command
fix(browser): handle CDP timeout gracefully
docs: update CONTRIBUTING.md
test(reddit): add e2e test for save command
chore: bump vitest to v4
```
## Submitting a Pull Request
1. Create a feature branch: `git checkout -b feat/mysite-trending`
2. Make your changes and add tests when relevant
3. Run the checks:
```bash
npx tsc --noEmit # Type check
npx vitest run src/ # Unit tests
opencli validate # YAML validation (if applicable)
```
4. Commit using conventional commit format
5. Push and open a PR
+237
View File
@@ -0,0 +1,237 @@
# Testing Guide
> 面向开发者和 AI Agent 的测试参考手册。
## 目录
- [测试架构](#测试架构)
- [当前覆盖范围](#当前覆盖范围)
- [本地运行测试](#本地运行测试)
- [如何添加新测试](#如何添加新测试)
- [CI/CD 流水线](#cicd-流水线)
- [浏览器模式](#浏览器模式)
- [站点兼容性](#站点兼容性)
---
## 测试架构
测试分为三层,全部使用 **vitest** 运行:
```
tests/
├── e2e/ # E2E 集成测试(子进程运行真实 CLI)
│ ├── helpers.ts # runCli() 共享工具
│ ├── public-commands.test.ts # 公开 API 命令(无需浏览器)
│ ├── browser-public.test.ts # 浏览器命令(公开数据)
│ ├── browser-auth.test.ts # 需登录命令(graceful failure 测试)
│ ├── management.test.ts # 管理命令(list, validate, verify, help)
│ └── output-formats.test.ts # 输出格式(json/yaml/csv/md)
├── smoke/ # 烟雾测试(仅定时 / 手动触发)
│ └── api-health.test.ts # 外部 API 可用性检测
src/
├── *.test.ts # 单元测试(已有 8 个)
```
| 层 | 位置 | 运行方式 | 用途 |
|---|---|---|---|
| 单元测试 | `src/**/*.test.ts` | `npx vitest run src/` | 内部模块逻辑 |
| E2E 测试 | `tests/e2e/*.test.ts` | `npx vitest run tests/e2e/` | 真实 CLI 命令执行 |
| 烟雾测试 | `tests/smoke/*.test.ts` | `npx vitest run tests/smoke/` | 外部 API 健康 |
---
## 当前覆盖范围
### 单元测试(8 个文件)
| 文件 | 覆盖内容 |
|---|---|
| `browser.test.ts` | JSON-RPC、tab 管理、extension/standalone 模式切换 |
| `engine.test.ts` | 命令发现与执行 |
| `registry.test.ts` | 命令注册与策略分配 |
| `output.test.ts` | 输出格式渲染 |
| `doctor.test.ts` | Token 诊断 |
| `coupang.test.ts` | 数据归一化 |
| `pipeline/template.test.ts` | 模板表达式求值 |
| `pipeline/transform.test.ts` | 数据变换步骤 |
### E2E 测试(~52 个用例)
| 文件 | 覆盖站点/功能 | 测试数 |
|---|---|---|
| `public-commands.test.ts` | hackernews/top, v2ex/hot, v2ex/latest, v2ex/topic | 5 |
| `browser-public.test.ts` | bbc, bilibili×3, weibo, zhihu×2, reddit×2, twitter, xueqiu×2, reuters, youtube, smzdm, boss, ctrip, coupang, xiaohongshu, yahoo-finance, v2ex/daily | 21 |
| `browser-auth.test.ts` | bilibili/me,dynamic,favorite,history,following + twitter/bookmarks,timeline,notifications + v2ex/me,notifications + xueqiu/feed,watchlist + xiaohongshu/feed,notifications | 14 |
| `management.test.ts` | list×5 格式, validate×3 级别, verify, --version, --help, unknown cmd | 12 |
| `output-formats.test.ts` | json, yaml, csv, md 格式验证 | 5 |
### 烟雾测试
公开 API 可用性(hackernews, v2ex×2, v2ex/topic)+ 全站点注册完整性检查。
---
## 本地运行测试
### 前置条件
```bash
npm ci # 安装依赖
npm run build # 编译(E2E 测试需要 dist/main.js)
```
### 运行命令
```bash
# 全部单元测试
npx vitest run src/
# 全部 E2E 测试(会真实调用外部 API)
npx vitest run tests/e2e/
# 单个测试文件
npx vitest run tests/e2e/management.test.ts
# 全部测试(单元 + E2E)
npx vitest run
# 烟雾测试
npx vitest run tests/smoke/
# watch 模式(开发时推荐)
npx vitest src/
```
### 浏览器命令本地测试须知
- opencli 通过 Browser Bridge 扩展连接已运行的 Chrome 浏览器
- `browser-public.test.ts` 使用 `tryBrowserCommand()`,站点反爬导致空数据时 warn + pass
- `browser-auth.test.ts` 验证 **graceful failure**(不 crash 不 hang 即通过)
- 如需测试完整登录态,保持 Chrome 登录态并安装 Browser Bridge 扩展,手动跑对应测试
---
## 如何添加新测试
### 新增 YAML Adapter(如 `src/clis/producthunt/trending.yaml`)
1. **无需额外操作**:`validate` 测试会自动覆盖 YAML 结构验证
2. 根据 adapter 类型,在对应文件加一个 `it()` block:
```typescript
// 如果 browser: false(公开 API)→ tests/e2e/public-commands.test.ts
it('producthunt trending returns data', async () => {
const { stdout, code } = await runCli(['producthunt', 'trending', '--limit', '3', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(Array.isArray(data)).toBe(true);
expect(data.length).toBeGreaterThanOrEqual(1);
expect(data[0]).toHaveProperty('title');
}, 30_000);
```
```typescript
// 如果 browser: true 但可公开访问 → tests/e2e/browser-public.test.ts
it('producthunt trending returns data', async () => {
const data = await tryBrowserCommand(['producthunt', 'trending', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'producthunt trending');
}, 60_000);
```
```typescript
// 如果 browser: true 且需登录 → tests/e2e/browser-auth.test.ts
it('producthunt me fails gracefully without login', async () => {
await expectGracefulAuthFailure(['producthunt', 'me', '-f', 'json'], 'producthunt me');
}, 60_000);
```
### 新增管理命令(如 `opencli export`)
在 `tests/e2e/management.test.ts` 添加测试。
### 新增内部模块
在 `src/` 下对应位置创建 `*.test.ts`。
### 决策流程图
```
新增功能 → 是内部模块? → 是 → src/ 下加 *.test.ts
↓ 否
是 CLI 命令? → browser: false? → tests/e2e/public-commands.test.ts
↓ true
公开数据? → tests/e2e/browser-public.test.ts
↓ 需登录
tests/e2e/browser-auth.test.ts
```
---
## CI/CD 流水线
### ci.yml(主流水线)
| Job | 触发条件 | 内容 |
|---|---|---|
| **build** | push/PR to main,dev | typecheck + build |
| **unit-test** | push/PR to main,dev | 单元测试,2 shard 并行 |
| **smoke-test** | 每周一 08:00 UTC / 手动 | xvfb + real Chrome,外部 API 健康检查 |
### e2e-headed.yml(E2E 测试)
| Job | 触发条件 | 内容 |
|---|---|---|
| **e2e-headed** | push/PR to main,dev | xvfb + real Chrome,全部 E2E 测试 |
E2E 使用 `browser-actions/setup-chrome` 安装真实 Chrome,配合 `xvfb-run` 提供虚拟显示器,以 headed 模式运行浏览器。
### Sharding
单元测试使用 vitest 内置 shard:
::: v-pre
```yaml
strategy:
matrix:
shard: [1, 2]
steps:
- run: npx vitest run src/ --shard=${{ matrix.shard }}/2
```
:::
---
## 浏览器模式
opencli 通过 Browser Bridge 扩展连接浏览器:
| 条件 | 模式 | 使用场景 |
|---|---|---|
| 扩展已安装 | Extension 模式 | 本地用户,连接已登录的 Chrome |
| 扩展未安装 | CLI 报错提示安装 | 需要安装 Browser Bridge 扩展 |
CI 中使用 `OPENCLI_BROWSER_EXECUTABLE_PATH` 指定真实 Chrome 路径:
::: v-pre
```yaml
env:
OPENCLI_BROWSER_EXECUTABLE_PATH: ${{ steps.setup-chrome.outputs.chrome-path }}
```
:::
---
## 站点兼容性
在 GitHub Actions 美国 runner 上,部分站点因地域限制或登录要求返回空数据。E2E 测试对这些站点使用 warn + pass 策略,不影响 CI 绿灯。
| 站点 | CI 状态 | 限制原因 |
|---|---|---|
| hackernews, bbc, v2ex | ✅ 返回数据 | 无限制 |
| yahoo-finance | ✅ 返回数据 | 无限制 |
| bilibili, zhihu, weibo, xiaohongshu | ⚠️ 空数据 | 地域限制(中国站点) |
| reddit, twitter, youtube | ⚠️ 空数据 | 需登录或 cookie |
| smzdm, boss, ctrip, coupang, xueqiu | ⚠️ 空数据 | 地域限制 / 需登录 |
> 使用 self-hosted runner(国内服务器)可解决地域限制问题。
+87
View File
@@ -0,0 +1,87 @@
# TypeScript Adapter Guide
Use TypeScript adapters when you need browser-side logic, multi-step flows, DOM manipulation, or complex data extraction that goes beyond simple API fetching.
## Basic Structure
```typescript
import { cli, Strategy } from '../../registry.js';
cli({
site: 'mysite',
name: 'search',
description: 'Search MySite',
domain: 'www.mysite.com',
strategy: Strategy.COOKIE, // PUBLIC | COOKIE | HEADER
args: [
{ name: 'query', required: true, help: 'Search query' },
{ name: 'limit', type: 'int', default: 10, help: 'Max results' },
],
columns: ['title', 'url', 'date'],
func: async (page, kwargs) => {
const { query, limit = 10 } = kwargs;
// Navigate and extract data
await page.goto('https://www.mysite.com');
const data = await page.evaluate(`
(async () => {
const res = await fetch('/api/search?q=${encodeURIComponent(String(query))}', {
credentials: 'include'
});
return (await res.json()).results;
})()
`);
return data.slice(0, Number(limit)).map((item: any) => ({
title: item.title,
url: item.url,
date: item.created_at,
}));
},
});
```
## Strategy Types
| Strategy | Constant | Use Case |
|----------|----------|----------|
| Public | `Strategy.PUBLIC` | No auth needed |
| Cookie | `Strategy.COOKIE` | Browser session cookies |
| Header | `Strategy.HEADER` | Custom headers/tokens |
## The `page` Object
The `page` parameter provides browser interaction methods:
- `page.goto(url)` — Navigate to a URL
- `page.evaluate(script)` — Execute JavaScript in the page context
- `page.waitForSelector(selector)` — Wait for an element
- `page.click(selector)` — Click an element
- `page.type(selector, text)` — Type text into an input
## The `kwargs` Object
Contains parsed CLI arguments as key-value pairs. Always destructure with defaults:
```typescript
const { query, limit = 10, format = 'json' } = kwargs;
```
## AI-Assisted Development
Use the AI workflow tools to accelerate adapter creation:
```bash
# Discover APIs and page structure
opencli explore https://example.com --site mysite
# Auto-generate adapter from explore artifacts
opencli synthesize mysite
# One-shot: explore → synthesize → register
opencli generate https://example.com --goal "trending"
```
See [AI Workflow](/developer/ai-workflow) for the complete guide.
+108
View File
@@ -0,0 +1,108 @@
# YAML Adapter Guide
YAML adapters are the recommended way to add new commands when the site offers a straightforward API. They use a declarative pipeline approach — no TypeScript required.
## Basic Structure
::: v-pre
```yaml
site: mysite # Site identifier
name: trending # Command name (opencli mysite trending)
description: ... # Help text
domain: www.mysite.com
strategy: public # public | cookie | header
browser: false # true if browser session is needed
args: # CLI arguments
limit:
type: int
default: 20
description: Number of items
pipeline: # Data processing steps
- fetch:
url: https://api.mysite.com/trending
- map:
rank: ${{ index + 1 }}
title: ${{ item.title }}
- limit: ${{ args.limit }}
columns: [rank, title, score, url]
```
:::
## Pipeline Steps
### `fetch`
Fetch data from a URL. Supports template expressions for dynamic URLs.
::: v-pre
```yaml
- fetch:
url: https://api.example.com/search?q=${{ args.query }}
headers:
Accept: application/json
```
:::
### `map`
::: v-pre
Transform each item in the result array. Use `${{ item.xxx }}` for field access and `${{ index }}` for position.
```yaml
- map:
rank: ${{ index + 1 }}
title: ${{ item.title }}
url: https://example.com${{ item.path }}
```
:::
### `limit`
Truncate results to N items.
::: v-pre
```yaml
- limit: ${{ args.limit }}
```
:::
### `filter`
Filter items by condition.
::: v-pre
```yaml
- filter: ${{ item.score > 100 }}
```
:::
### `download`
Download media files.
::: v-pre
```yaml
- download:
url: ${{ item.imageUrl }}
dir: ./downloads
filename: ${{ item.title | sanitize }}.jpg
```
:::
## Template Expressions
::: v-pre
Use `${{ ... }}` for dynamic values:
| Expression | Description |
|-----------|-------------|
| `${{ args.limit }}` | CLI argument |
| `${{ item.title }}` | Current item field |
| `${{ index }}` | Current index (0-based) |
| `${{ item.x \| sanitize }}` | Pipe filters |
:::
## Real Example
See [`src/clis/hackernews/top.yaml`](https://github.com/jackwener/opencli/blob/main/src/clis/hackernews/top.yaml).