docs: add VitePress documentation site (#112)
- Add VitePress with full navigation, sidebar, i18n (en/zh), local search - Create 50+ doc pages: guide, adapters (browser + desktop), developer, advanced - Migrate content from README.md, CONTRIBUTING.md, TESTING.md, CDP.md, CLI-ELECTRON.md - Migrate all 11 adapter READMEs to structured documentation - Add new pages: architecture, yaml-adapter guide, ts-adapter guide, ai-workflow - Add GitHub Actions workflow for deploying to GitHub Pages - Add Chinese locale pages (getting-started, installation, browser-bridge, etc.) - Add docs:dev, docs:build, docs:preview npm scripts
This commit is contained in:
1 parent
812db29ed8
commit
31f58ae699
59 files changed
+5138
-5
No files matched your search
@@ -0,0 +1,66 @@
|
||||
# AI Workflow
|
||||
|
||||
OpenCLI is designed with AI agents in mind. This guide covers the AI-native discovery and code generation tools.
|
||||
|
||||
## Quick Mode (One-Shot)
|
||||
|
||||
Generate a single command for a specific page URL — just a URL + one-line goal, 4 steps done:
|
||||
|
||||
```bash
|
||||
opencli generate https://example.com --goal "trending"
|
||||
```
|
||||
|
||||
This runs: explore → synthesize → register in one shot.
|
||||
|
||||
For the complete one-shot workflow details, see [CLI-ONESHOT.md](https://github.com/jackwener/opencli/blob/main/CLI-ONESHOT.md).
|
||||
|
||||
## Full Mode (Explorer Workflow)
|
||||
|
||||
### Step 1: Deep Explore
|
||||
|
||||
Discover APIs, infer capabilities, and detect framework:
|
||||
|
||||
```bash
|
||||
opencli explore https://example.com --site mysite
|
||||
```
|
||||
|
||||
Outputs to `.opencli/explore/<site>/`:
|
||||
- `manifest.json` — Site metadata
|
||||
- `endpoints.json` — Discovered API endpoints
|
||||
- `capabilities.json` — Inferred capabilities
|
||||
- `auth.json` — Authentication strategy details
|
||||
|
||||
### Step 2: Synthesize
|
||||
|
||||
Generate YAML adapters from explore artifacts:
|
||||
|
||||
```bash
|
||||
opencli synthesize mysite
|
||||
```
|
||||
|
||||
### Step 3: Strategy Cascade
|
||||
|
||||
Auto-probe authentication strategies: `PUBLIC → COOKIE → HEADER`:
|
||||
|
||||
```bash
|
||||
opencli cascade https://api.example.com/data
|
||||
```
|
||||
|
||||
### Step 4: Validate & Test
|
||||
|
||||
```bash
|
||||
opencli validate # Validate generated YAML
|
||||
opencli <site> <command> --limit 3 -f json # Test the command
|
||||
```
|
||||
|
||||
## 5-Tier Authentication Strategy
|
||||
|
||||
The explorer uses a decision tree to determine the best authentication approach:
|
||||
|
||||
1. **PUBLIC** — No auth, direct API call
|
||||
2. **COOKIE** — Reuse Chrome session cookies
|
||||
3. **HEADER** — Custom auth headers
|
||||
4. **BROWSER** — Full browser automation
|
||||
5. **CDP** — Chrome DevTools Protocol for Electron apps
|
||||
|
||||
For the complete browser exploration workflow and debugging guide, see [CLI-EXPLORER.md](https://github.com/jackwener/opencli/blob/main/CLI-EXPLORER.md).
|
||||
@@ -0,0 +1,90 @@
|
||||
# Architecture
|
||||
|
||||
OpenCLI is built on a **Dual-Engine Architecture** that supports both declarative YAML pipelines and programmatic TypeScript adapters.
|
||||
|
||||
## High-Level Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ opencli CLI │
|
||||
│ (Commander.js entry point) │
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Engine Layer │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌────────────┐ │
|
||||
│ │ Registry │ │ Dynamic │ │ Output │ │
|
||||
│ │ (commands) │ │ Loader │ │ Formatter │ │
|
||||
│ └──────────────┘ └──────────────┘ └────────────┘ │
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Adapter Layer │
|
||||
│ ┌─────────────────┐ ┌──────────────────────────┐ │
|
||||
│ │ YAML Pipeline │ │ TypeScript Adapters │ │
|
||||
│ │ (declarative) │ │ (browser/desktop/AI) │ │
|
||||
│ └─────────────────┘ └──────────────────────────┘ │
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Connection Layer │
|
||||
│ ┌─────────────────┐ ┌──────────────────────────┐ │
|
||||
│ │ Browser Bridge │ │ CDP (Chrome DevTools) │ │
|
||||
│ │ (Extension+WS) │ │ (Electron apps) │ │
|
||||
│ └─────────────────┘ └──────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Core Modules
|
||||
|
||||
### Registry (`src/registry.ts`)
|
||||
Central command registry. All adapters register their commands via the `cli()` function with metadata: site, name, description, domain, strategy, args, columns.
|
||||
|
||||
### Engine (`src/engine.ts`)
|
||||
Command discovery and execution engine. Discovers commands from the registry, parses arguments, executes the appropriate adapter, and routes output through the formatter.
|
||||
|
||||
### Browser (`src/browser.ts`)
|
||||
Manages connections to Chrome via the Browser Bridge WebSocket daemon. Handles JSON-RPC messaging, tab management, and extension/standalone mode switching.
|
||||
|
||||
### Pipeline (`src/pipeline/`)
|
||||
The YAML pipeline engine. Processes declarative steps:
|
||||
- **fetch** — HTTP requests with cookie/header strategies
|
||||
- **map** — Data transformation with template expressions
|
||||
- **limit** — Result truncation
|
||||
- **filter** — Conditional filtering
|
||||
- **download** — Media download support
|
||||
|
||||
### Output (`src/output.ts`)
|
||||
Unified output formatting: `table`, `json`, `yaml`, `md`, `csv`.
|
||||
|
||||
## Authentication Strategies
|
||||
|
||||
OpenCLI uses a 3-tier authentication strategy:
|
||||
|
||||
| Strategy | How It Works | When to Use |
|
||||
|----------|-------------|-------------|
|
||||
| `public` | Direct HTTP fetch, no auth | Public APIs (HackerNews, BBC) |
|
||||
| `cookie` | Reuse Chrome cookies via Browser Bridge | Logged-in sites (Bilibili, Zhihu) |
|
||||
| `header` | Custom auth headers | API-key based services |
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
src/
|
||||
├── main.ts # Entry point
|
||||
├── engine.ts # Command execution engine
|
||||
├── registry.ts # Command registry
|
||||
├── browser.ts # Browser Bridge connection
|
||||
├── output.ts # Output formatting
|
||||
├── doctor.ts # Diagnostic tool
|
||||
├── pipeline/ # YAML pipeline engine
|
||||
│ ├── runner.ts
|
||||
│ ├── template.ts
|
||||
│ ├── transform.ts
|
||||
│ └── steps/
|
||||
│ ├── fetch.ts
|
||||
│ ├── map.ts
|
||||
│ ├── limit.ts
|
||||
│ ├── filter.ts
|
||||
│ └── download.ts
|
||||
└── clis/ # Site adapters
|
||||
├── twitter/
|
||||
├── reddit/
|
||||
├── bilibili/
|
||||
├── cursor/
|
||||
└── ...
|
||||
```
|
||||
@@ -0,0 +1,136 @@
|
||||
# Contributing
|
||||
|
||||
Thanks for your interest in contributing to OpenCLI.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# 1. Fork & clone
|
||||
git clone git@github.com:<your-username>/opencli.git
|
||||
cd opencli
|
||||
|
||||
# 2. Install dependencies
|
||||
npm install
|
||||
|
||||
# 3. Build
|
||||
npm run build
|
||||
|
||||
# 4. Run a few checks
|
||||
npx tsc --noEmit
|
||||
npx vitest run src/
|
||||
|
||||
# 5. Link globally (optional, for testing `opencli` command)
|
||||
npm link
|
||||
```
|
||||
|
||||
## Adding a New Site Adapter
|
||||
|
||||
This is the most common type of contribution. Start with YAML when possible, and use TypeScript only when you need browser-side logic or multi-step flows.
|
||||
|
||||
### YAML Adapter (Recommended for data-fetching commands)
|
||||
|
||||
Create a file like `src/clis/<site>/<command>.yaml`:
|
||||
|
||||
::: v-pre
|
||||
```yaml
|
||||
site: mysite
|
||||
name: trending
|
||||
description: Trending posts on MySite
|
||||
domain: www.mysite.com
|
||||
strategy: public # public | cookie | header
|
||||
browser: false # true if browser session is needed
|
||||
|
||||
args:
|
||||
limit:
|
||||
type: int
|
||||
default: 20
|
||||
description: Number of items
|
||||
|
||||
pipeline:
|
||||
- fetch:
|
||||
url: https://api.mysite.com/trending
|
||||
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
title: ${{ item.title }}
|
||||
score: ${{ item.score }}
|
||||
url: ${{ item.url }}
|
||||
|
||||
- limit: ${{ args.limit }}
|
||||
|
||||
columns: [rank, title, score, url]
|
||||
```
|
||||
:::
|
||||
|
||||
See [`hackernews/top.yaml`](https://github.com/jackwener/opencli/blob/main/src/clis/hackernews/top.yaml) for a real example.
|
||||
|
||||
### TypeScript Adapter (For complex browser interactions)
|
||||
|
||||
Create a file like `src/clis/<site>/<command>.ts`:
|
||||
|
||||
```typescript
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'mysite',
|
||||
name: 'search',
|
||||
description: 'Search MySite',
|
||||
domain: 'www.mysite.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
args: [
|
||||
{ name: 'query', required: true, help: 'Search query' },
|
||||
{ name: 'limit', type: 'int', default: 10, help: 'Max results' },
|
||||
],
|
||||
columns: ['title', 'url', 'date'],
|
||||
|
||||
func: async (page, kwargs) => {
|
||||
const { query, limit = 10 } = kwargs;
|
||||
// ... browser automation logic
|
||||
return data.slice(0, Number(limit)).map((item: any) => ({
|
||||
title: item.title,
|
||||
url: item.url,
|
||||
date: item.created_at,
|
||||
}));
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
### Validate Your Adapter
|
||||
|
||||
```bash
|
||||
opencli validate # Validate YAML syntax and schema
|
||||
opencli <site> <command> --limit 3 -f json # Test your command
|
||||
opencli <site> <command> -v # Verbose mode for debugging
|
||||
```
|
||||
|
||||
## Code Style
|
||||
|
||||
- **TypeScript strict mode** — avoid `any` where possible.
|
||||
- **ES Modules** — use `.js` extensions in imports (TypeScript output).
|
||||
- **Naming**: `kebab-case` for files, `camelCase` for variables/functions, `PascalCase` for types/classes.
|
||||
- **No default exports** — use named exports.
|
||||
|
||||
## Commit Convention
|
||||
|
||||
We use [Conventional Commits](https://www.conventionalcommits.org/):
|
||||
|
||||
```
|
||||
feat(twitter): add thread command
|
||||
fix(browser): handle CDP timeout gracefully
|
||||
docs: update CONTRIBUTING.md
|
||||
test(reddit): add e2e test for save command
|
||||
chore: bump vitest to v4
|
||||
```
|
||||
|
||||
## Submitting a Pull Request
|
||||
|
||||
1. Create a feature branch: `git checkout -b feat/mysite-trending`
|
||||
2. Make your changes and add tests when relevant
|
||||
3. Run the checks:
|
||||
```bash
|
||||
npx tsc --noEmit # Type check
|
||||
npx vitest run src/ # Unit tests
|
||||
opencli validate # YAML validation (if applicable)
|
||||
```
|
||||
4. Commit using conventional commit format
|
||||
5. Push and open a PR
|
||||
@@ -0,0 +1,237 @@
|
||||
# Testing Guide
|
||||
|
||||
> 面向开发者和 AI Agent 的测试参考手册。
|
||||
|
||||
## 目录
|
||||
|
||||
- [测试架构](#测试架构)
|
||||
- [当前覆盖范围](#当前覆盖范围)
|
||||
- [本地运行测试](#本地运行测试)
|
||||
- [如何添加新测试](#如何添加新测试)
|
||||
- [CI/CD 流水线](#cicd-流水线)
|
||||
- [浏览器模式](#浏览器模式)
|
||||
- [站点兼容性](#站点兼容性)
|
||||
|
||||
---
|
||||
|
||||
## 测试架构
|
||||
|
||||
测试分为三层,全部使用 **vitest** 运行:
|
||||
|
||||
```
|
||||
tests/
|
||||
├── e2e/ # E2E 集成测试(子进程运行真实 CLI)
|
||||
│ ├── helpers.ts # runCli() 共享工具
|
||||
│ ├── public-commands.test.ts # 公开 API 命令(无需浏览器)
|
||||
│ ├── browser-public.test.ts # 浏览器命令(公开数据)
|
||||
│ ├── browser-auth.test.ts # 需登录命令(graceful failure 测试)
|
||||
│ ├── management.test.ts # 管理命令(list, validate, verify, help)
|
||||
│ └── output-formats.test.ts # 输出格式(json/yaml/csv/md)
|
||||
├── smoke/ # 烟雾测试(仅定时 / 手动触发)
|
||||
│ └── api-health.test.ts # 外部 API 可用性检测
|
||||
src/
|
||||
├── *.test.ts # 单元测试(已有 8 个)
|
||||
```
|
||||
|
||||
| 层 | 位置 | 运行方式 | 用途 |
|
||||
|---|---|---|---|
|
||||
| 单元测试 | `src/**/*.test.ts` | `npx vitest run src/` | 内部模块逻辑 |
|
||||
| E2E 测试 | `tests/e2e/*.test.ts` | `npx vitest run tests/e2e/` | 真实 CLI 命令执行 |
|
||||
| 烟雾测试 | `tests/smoke/*.test.ts` | `npx vitest run tests/smoke/` | 外部 API 健康 |
|
||||
|
||||
---
|
||||
|
||||
## 当前覆盖范围
|
||||
|
||||
### 单元测试(8 个文件)
|
||||
|
||||
| 文件 | 覆盖内容 |
|
||||
|---|---|
|
||||
| `browser.test.ts` | JSON-RPC、tab 管理、extension/standalone 模式切换 |
|
||||
| `engine.test.ts` | 命令发现与执行 |
|
||||
| `registry.test.ts` | 命令注册与策略分配 |
|
||||
| `output.test.ts` | 输出格式渲染 |
|
||||
| `doctor.test.ts` | Token 诊断 |
|
||||
| `coupang.test.ts` | 数据归一化 |
|
||||
| `pipeline/template.test.ts` | 模板表达式求值 |
|
||||
| `pipeline/transform.test.ts` | 数据变换步骤 |
|
||||
|
||||
### E2E 测试(~52 个用例)
|
||||
|
||||
| 文件 | 覆盖站点/功能 | 测试数 |
|
||||
|---|---|---|
|
||||
| `public-commands.test.ts` | hackernews/top, v2ex/hot, v2ex/latest, v2ex/topic | 5 |
|
||||
| `browser-public.test.ts` | bbc, bilibili×3, weibo, zhihu×2, reddit×2, twitter, xueqiu×2, reuters, youtube, smzdm, boss, ctrip, coupang, xiaohongshu, yahoo-finance, v2ex/daily | 21 |
|
||||
| `browser-auth.test.ts` | bilibili/me,dynamic,favorite,history,following + twitter/bookmarks,timeline,notifications + v2ex/me,notifications + xueqiu/feed,watchlist + xiaohongshu/feed,notifications | 14 |
|
||||
| `management.test.ts` | list×5 格式, validate×3 级别, verify, --version, --help, unknown cmd | 12 |
|
||||
| `output-formats.test.ts` | json, yaml, csv, md 格式验证 | 5 |
|
||||
|
||||
### 烟雾测试
|
||||
|
||||
公开 API 可用性(hackernews, v2ex×2, v2ex/topic)+ 全站点注册完整性检查。
|
||||
|
||||
---
|
||||
|
||||
## 本地运行测试
|
||||
|
||||
### 前置条件
|
||||
|
||||
```bash
|
||||
npm ci # 安装依赖
|
||||
npm run build # 编译(E2E 测试需要 dist/main.js)
|
||||
```
|
||||
|
||||
### 运行命令
|
||||
|
||||
```bash
|
||||
# 全部单元测试
|
||||
npx vitest run src/
|
||||
|
||||
# 全部 E2E 测试(会真实调用外部 API)
|
||||
npx vitest run tests/e2e/
|
||||
|
||||
# 单个测试文件
|
||||
npx vitest run tests/e2e/management.test.ts
|
||||
|
||||
# 全部测试(单元 + E2E)
|
||||
npx vitest run
|
||||
|
||||
# 烟雾测试
|
||||
npx vitest run tests/smoke/
|
||||
|
||||
# watch 模式(开发时推荐)
|
||||
npx vitest src/
|
||||
```
|
||||
|
||||
### 浏览器命令本地测试须知
|
||||
|
||||
- opencli 通过 Browser Bridge 扩展连接已运行的 Chrome 浏览器
|
||||
- `browser-public.test.ts` 使用 `tryBrowserCommand()`,站点反爬导致空数据时 warn + pass
|
||||
- `browser-auth.test.ts` 验证 **graceful failure**(不 crash 不 hang 即通过)
|
||||
- 如需测试完整登录态,保持 Chrome 登录态并安装 Browser Bridge 扩展,手动跑对应测试
|
||||
|
||||
---
|
||||
|
||||
## 如何添加新测试
|
||||
|
||||
### 新增 YAML Adapter(如 `src/clis/producthunt/trending.yaml`)
|
||||
|
||||
1. **无需额外操作**:`validate` 测试会自动覆盖 YAML 结构验证
|
||||
2. 根据 adapter 类型,在对应文件加一个 `it()` block:
|
||||
|
||||
```typescript
|
||||
// 如果 browser: false(公开 API)→ tests/e2e/public-commands.test.ts
|
||||
it('producthunt trending returns data', async () => {
|
||||
const { stdout, code } = await runCli(['producthunt', 'trending', '--limit', '3', '-f', 'json']);
|
||||
expect(code).toBe(0);
|
||||
const data = parseJsonOutput(stdout);
|
||||
expect(Array.isArray(data)).toBe(true);
|
||||
expect(data.length).toBeGreaterThanOrEqual(1);
|
||||
expect(data[0]).toHaveProperty('title');
|
||||
}, 30_000);
|
||||
```
|
||||
|
||||
```typescript
|
||||
// 如果 browser: true 但可公开访问 → tests/e2e/browser-public.test.ts
|
||||
it('producthunt trending returns data', async () => {
|
||||
const data = await tryBrowserCommand(['producthunt', 'trending', '--limit', '3', '-f', 'json']);
|
||||
expectDataOrSkip(data, 'producthunt trending');
|
||||
}, 60_000);
|
||||
```
|
||||
|
||||
```typescript
|
||||
// 如果 browser: true 且需登录 → tests/e2e/browser-auth.test.ts
|
||||
it('producthunt me fails gracefully without login', async () => {
|
||||
await expectGracefulAuthFailure(['producthunt', 'me', '-f', 'json'], 'producthunt me');
|
||||
}, 60_000);
|
||||
```
|
||||
|
||||
### 新增管理命令(如 `opencli export`)
|
||||
|
||||
在 `tests/e2e/management.test.ts` 添加测试。
|
||||
|
||||
### 新增内部模块
|
||||
|
||||
在 `src/` 下对应位置创建 `*.test.ts`。
|
||||
|
||||
### 决策流程图
|
||||
|
||||
```
|
||||
新增功能 → 是内部模块? → 是 → src/ 下加 *.test.ts
|
||||
↓ 否
|
||||
是 CLI 命令? → browser: false? → tests/e2e/public-commands.test.ts
|
||||
↓ true
|
||||
公开数据? → tests/e2e/browser-public.test.ts
|
||||
↓ 需登录
|
||||
tests/e2e/browser-auth.test.ts
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI/CD 流水线
|
||||
|
||||
### ci.yml(主流水线)
|
||||
|
||||
| Job | 触发条件 | 内容 |
|
||||
|---|---|---|
|
||||
| **build** | push/PR to main,dev | typecheck + build |
|
||||
| **unit-test** | push/PR to main,dev | 单元测试,2 shard 并行 |
|
||||
| **smoke-test** | 每周一 08:00 UTC / 手动 | xvfb + real Chrome,外部 API 健康检查 |
|
||||
|
||||
### e2e-headed.yml(E2E 测试)
|
||||
|
||||
| Job | 触发条件 | 内容 |
|
||||
|---|---|---|
|
||||
| **e2e-headed** | push/PR to main,dev | xvfb + real Chrome,全部 E2E 测试 |
|
||||
|
||||
E2E 使用 `browser-actions/setup-chrome` 安装真实 Chrome,配合 `xvfb-run` 提供虚拟显示器,以 headed 模式运行浏览器。
|
||||
|
||||
### Sharding
|
||||
|
||||
单元测试使用 vitest 内置 shard:
|
||||
|
||||
::: v-pre
|
||||
```yaml
|
||||
strategy:
|
||||
matrix:
|
||||
shard: [1, 2]
|
||||
steps:
|
||||
- run: npx vitest run src/ --shard=${{ matrix.shard }}/2
|
||||
```
|
||||
:::
|
||||
|
||||
---
|
||||
|
||||
## 浏览器模式
|
||||
|
||||
opencli 通过 Browser Bridge 扩展连接浏览器:
|
||||
|
||||
| 条件 | 模式 | 使用场景 |
|
||||
|---|---|---|
|
||||
| 扩展已安装 | Extension 模式 | 本地用户,连接已登录的 Chrome |
|
||||
| 扩展未安装 | CLI 报错提示安装 | 需要安装 Browser Bridge 扩展 |
|
||||
|
||||
CI 中使用 `OPENCLI_BROWSER_EXECUTABLE_PATH` 指定真实 Chrome 路径:
|
||||
|
||||
::: v-pre
|
||||
```yaml
|
||||
env:
|
||||
OPENCLI_BROWSER_EXECUTABLE_PATH: ${{ steps.setup-chrome.outputs.chrome-path }}
|
||||
```
|
||||
:::
|
||||
|
||||
---
|
||||
|
||||
## 站点兼容性
|
||||
|
||||
在 GitHub Actions 美国 runner 上,部分站点因地域限制或登录要求返回空数据。E2E 测试对这些站点使用 warn + pass 策略,不影响 CI 绿灯。
|
||||
|
||||
| 站点 | CI 状态 | 限制原因 |
|
||||
|---|---|---|
|
||||
| hackernews, bbc, v2ex | ✅ 返回数据 | 无限制 |
|
||||
| yahoo-finance | ✅ 返回数据 | 无限制 |
|
||||
| bilibili, zhihu, weibo, xiaohongshu | ⚠️ 空数据 | 地域限制(中国站点) |
|
||||
| reddit, twitter, youtube | ⚠️ 空数据 | 需登录或 cookie |
|
||||
| smzdm, boss, ctrip, coupang, xueqiu | ⚠️ 空数据 | 地域限制 / 需登录 |
|
||||
|
||||
> 使用 self-hosted runner(国内服务器)可解决地域限制问题。
|
||||
@@ -0,0 +1,87 @@
|
||||
# TypeScript Adapter Guide
|
||||
|
||||
Use TypeScript adapters when you need browser-side logic, multi-step flows, DOM manipulation, or complex data extraction that goes beyond simple API fetching.
|
||||
|
||||
## Basic Structure
|
||||
|
||||
```typescript
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'mysite',
|
||||
name: 'search',
|
||||
description: 'Search MySite',
|
||||
domain: 'www.mysite.com',
|
||||
strategy: Strategy.COOKIE, // PUBLIC | COOKIE | HEADER
|
||||
args: [
|
||||
{ name: 'query', required: true, help: 'Search query' },
|
||||
{ name: 'limit', type: 'int', default: 10, help: 'Max results' },
|
||||
],
|
||||
columns: ['title', 'url', 'date'],
|
||||
|
||||
func: async (page, kwargs) => {
|
||||
const { query, limit = 10 } = kwargs;
|
||||
|
||||
// Navigate and extract data
|
||||
await page.goto('https://www.mysite.com');
|
||||
|
||||
const data = await page.evaluate(`
|
||||
(async () => {
|
||||
const res = await fetch('/api/search?q=${encodeURIComponent(String(query))}', {
|
||||
credentials: 'include'
|
||||
});
|
||||
return (await res.json()).results;
|
||||
})()
|
||||
`);
|
||||
|
||||
return data.slice(0, Number(limit)).map((item: any) => ({
|
||||
title: item.title,
|
||||
url: item.url,
|
||||
date: item.created_at,
|
||||
}));
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Strategy Types
|
||||
|
||||
| Strategy | Constant | Use Case |
|
||||
|----------|----------|----------|
|
||||
| Public | `Strategy.PUBLIC` | No auth needed |
|
||||
| Cookie | `Strategy.COOKIE` | Browser session cookies |
|
||||
| Header | `Strategy.HEADER` | Custom headers/tokens |
|
||||
|
||||
## The `page` Object
|
||||
|
||||
The `page` parameter provides browser interaction methods:
|
||||
|
||||
- `page.goto(url)` — Navigate to a URL
|
||||
- `page.evaluate(script)` — Execute JavaScript in the page context
|
||||
- `page.waitForSelector(selector)` — Wait for an element
|
||||
- `page.click(selector)` — Click an element
|
||||
- `page.type(selector, text)` — Type text into an input
|
||||
|
||||
## The `kwargs` Object
|
||||
|
||||
Contains parsed CLI arguments as key-value pairs. Always destructure with defaults:
|
||||
|
||||
```typescript
|
||||
const { query, limit = 10, format = 'json' } = kwargs;
|
||||
```
|
||||
|
||||
## AI-Assisted Development
|
||||
|
||||
Use the AI workflow tools to accelerate adapter creation:
|
||||
|
||||
```bash
|
||||
# Discover APIs and page structure
|
||||
opencli explore https://example.com --site mysite
|
||||
|
||||
# Auto-generate adapter from explore artifacts
|
||||
opencli synthesize mysite
|
||||
|
||||
# One-shot: explore → synthesize → register
|
||||
opencli generate https://example.com --goal "trending"
|
||||
```
|
||||
|
||||
See [AI Workflow](/developer/ai-workflow) for the complete guide.
|
||||
@@ -0,0 +1,108 @@
|
||||
# YAML Adapter Guide
|
||||
|
||||
YAML adapters are the recommended way to add new commands when the site offers a straightforward API. They use a declarative pipeline approach — no TypeScript required.
|
||||
|
||||
## Basic Structure
|
||||
|
||||
::: v-pre
|
||||
```yaml
|
||||
site: mysite # Site identifier
|
||||
name: trending # Command name (opencli mysite trending)
|
||||
description: ... # Help text
|
||||
domain: www.mysite.com
|
||||
strategy: public # public | cookie | header
|
||||
browser: false # true if browser session is needed
|
||||
|
||||
args: # CLI arguments
|
||||
limit:
|
||||
type: int
|
||||
default: 20
|
||||
description: Number of items
|
||||
|
||||
pipeline: # Data processing steps
|
||||
- fetch:
|
||||
url: https://api.mysite.com/trending
|
||||
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
title: ${{ item.title }}
|
||||
|
||||
- limit: ${{ args.limit }}
|
||||
|
||||
columns: [rank, title, score, url]
|
||||
```
|
||||
:::
|
||||
|
||||
## Pipeline Steps
|
||||
|
||||
### `fetch`
|
||||
Fetch data from a URL. Supports template expressions for dynamic URLs.
|
||||
|
||||
::: v-pre
|
||||
```yaml
|
||||
- fetch:
|
||||
url: https://api.example.com/search?q=${{ args.query }}
|
||||
headers:
|
||||
Accept: application/json
|
||||
```
|
||||
:::
|
||||
|
||||
### `map`
|
||||
|
||||
::: v-pre
|
||||
Transform each item in the result array. Use `${{ item.xxx }}` for field access and `${{ index }}` for position.
|
||||
|
||||
```yaml
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
title: ${{ item.title }}
|
||||
url: https://example.com${{ item.path }}
|
||||
```
|
||||
:::
|
||||
|
||||
### `limit`
|
||||
Truncate results to N items.
|
||||
|
||||
::: v-pre
|
||||
```yaml
|
||||
- limit: ${{ args.limit }}
|
||||
```
|
||||
:::
|
||||
|
||||
### `filter`
|
||||
Filter items by condition.
|
||||
|
||||
::: v-pre
|
||||
```yaml
|
||||
- filter: ${{ item.score > 100 }}
|
||||
```
|
||||
:::
|
||||
|
||||
### `download`
|
||||
Download media files.
|
||||
|
||||
::: v-pre
|
||||
```yaml
|
||||
- download:
|
||||
url: ${{ item.imageUrl }}
|
||||
dir: ./downloads
|
||||
filename: ${{ item.title | sanitize }}.jpg
|
||||
```
|
||||
:::
|
||||
|
||||
## Template Expressions
|
||||
|
||||
::: v-pre
|
||||
Use `${{ ... }}` for dynamic values:
|
||||
|
||||
| Expression | Description |
|
||||
|-----------|-------------|
|
||||
| `${{ args.limit }}` | CLI argument |
|
||||
| `${{ item.title }}` | Current item field |
|
||||
| `${{ index }}` | Current index (0-based) |
|
||||
| `${{ item.x \| sanitize }}` | Pipe filters |
|
||||
:::
|
||||
|
||||
## Real Example
|
||||
|
||||
See [`src/clis/hackernews/top.yaml`](https://github.com/jackwener/opencli/blob/main/src/clis/hackernews/top.yaml).
|
||||
Reference in new issue
Block a user