feat: add 8 new sites — bbc, weibo, boss, yahoo-finance, reuters, smzdm, ctrip, youtube
New adapters ported from bb-sites: - bbc/news: BBC News RSS headlines (public API) - weibo/hot: 微博热搜 (browser) - boss/search: BOSS直聘搜索职位 (browser) - yahoo-finance/quote: 股票行情 (browser) - reuters/search: 路透社新闻搜索 (browser) - smzdm/search: 什么值得买搜索好价 (browser) - ctrip/search: 携程旅行搜索 (browser) - youtube/search: YouTube视频搜索 (browser) Also: build script adds clean-yaml step, docs updated to 28 commands / 16 sites
This commit is contained in:
jackwener committed
1 parent
862be02732
commit
0fd9adf0f5
42 files changed
+2637
-645
No files matched your search
@@ -7,11 +7,10 @@
|
||||
|
||||
[](https://www.npmjs.com/package/@jackwener/opencli)
|
||||
|
||||
OpenCLI turns any website into a command-line tool by bridging your Chrome browser through [Playwright MCP](https://github.com/nichochar/playwright-mcp). No passwords stored, no tokens leaked — it just rides your existing browser session.
|
||||
A CLI tool that turns **any website** into a command-line interface. **28+ commands** across **16 sites** — bilibili, zhihu, xiaohongshu, twitter, reddit, github, v2ex, hackernews, bbc, weibo, boss, yahoo-finance, reuters, smzdm, ctrip, youtube — powered by browser session reuse and AI-native discovery.
|
||||
|
||||
## ✨ Highlights
|
||||
|
||||
- 🌐 **25+ commands, 8 sites** — Bilibili, Zhihu, GitHub, Twitter/X, Reddit, V2EX, Xiaohongshu, Hacker News
|
||||
- 🔐 **Account-safe** — Reuses Chrome's logged-in state; your credentials never leave the browser
|
||||
- 🤖 **AI Agent ready** — `explore` discovers APIs, `synthesize` generates adapters, `cascade` finds auth strategies
|
||||
- 📝 **Declarative YAML** — Most adapters are ~30 lines of YAML pipeline
|
||||
@@ -86,9 +85,17 @@ Public API commands (`hackernews`, `github search`, `v2ex`) need no browser at a
|
||||
| **xiaohongshu** | `search` `notifications` `feed` | 🔐 Browser |
|
||||
| **twitter** | `trending` | 🔐 Browser |
|
||||
| **reddit** | `hot` | 🔐 Browser |
|
||||
| **weibo** | `hot` | 🔐 Browser |
|
||||
| **boss** | `search` | 🔐 Browser |
|
||||
| **youtube** | `search` | 🔐 Browser |
|
||||
| **yahoo-finance** | `quote` | 🔐 Browser |
|
||||
| **reuters** | `search` | 🔐 Browser |
|
||||
| **smzdm** | `search` | 🔐 Browser |
|
||||
| **ctrip** | `search` | 🔐 Browser |
|
||||
| **github** | `trending` `search` | 🔐 / 🌐 |
|
||||
| **v2ex** | `hot` `latest` `topic` | 🌐 Public |
|
||||
| **hackernews** | `top` | 🌐 Public |
|
||||
| **bbc** | `news` | 🌐 Public |
|
||||
|
||||
## 🎨 Output Formats
|
||||
|
||||
|
||||
+9
-1
@@ -11,7 +11,7 @@ OpenCLI 通过 Chrome 浏览器 + [Playwright MCP Bridge](https://github.com/nic
|
||||
|
||||
## ✨ 亮点
|
||||
|
||||
- 🌐 **25+ 命令，8 个站点** — B站、知乎、GitHub、Twitter/X、Reddit、V2EX、小红书、Hacker News
|
||||
- 🌐 **28+ 命令，16 个站点** — B站、知乎、小红书、Twitter、Reddit、GitHub、V2EX、Hacker News、BBC、微博、BOSS直聘、Yahoo Finance、路透社、什么值得买、携程、YouTube
|
||||
- 🔐 **零风控** — 复用 Chrome 登录态，无需存储任何凭证
|
||||
- 🤖 **AI 原生** — `explore` 自动发现 API，`synthesize` 生成适配器，`cascade` 探测认证策略
|
||||
- 📝 **声明式 YAML** — 大部分适配器只需 ~30 行 YAML
|
||||
@@ -86,9 +86,17 @@ npm install -g @jackwener/opencli@latest
|
||||
| **xiaohongshu** | `search` `notifications` `feed` | 🔐 浏览器 |
|
||||
| **twitter** | `trending` | 🔐 浏览器 |
|
||||
| **reddit** | `hot` | 🔐 浏览器 |
|
||||
| **weibo** | `hot` | 🔐 浏览器 |
|
||||
| **boss** | `search` | 🔐 浏览器 |
|
||||
| **youtube** | `search` | 🔐 浏览器 |
|
||||
| **yahoo-finance** | `quote` | 🔐 浏览器 |
|
||||
| **reuters** | `search` | 🔐 浏览器 |
|
||||
| **smzdm** | `search` | 🔐 浏览器 |
|
||||
| **ctrip** | `search` | 🔐 浏览器 |
|
||||
| **github** | `trending` `search` | 🔐 / 🌐 |
|
||||
| **v2ex** | `hot` `latest` `topic` | 🌐 公共 API |
|
||||
| **hackernews** | `top` | 🌐 公共 API |
|
||||
| **bbc** | `news` | 🌐 公共 API |
|
||||
|
||||
## 🎨 输出格式
|
||||
|
||||
|
||||
@@ -78,6 +78,30 @@ opencli v2ex topic --id 1024 # 主题详情
|
||||
|
||||
# Hacker News (public)
|
||||
opencli hackernews top --limit 10 # Top stories
|
||||
|
||||
# BBC (public)
|
||||
opencli bbc news --limit 10 # BBC News RSS headlines
|
||||
|
||||
# 微博 (browser)
|
||||
opencli weibo hot --limit 10 # 微博热搜
|
||||
|
||||
# BOSS直聘 (browser)
|
||||
opencli boss search --query "AI agent" # 搜索职位
|
||||
|
||||
# YouTube (browser)
|
||||
opencli youtube search --query "rust" # 搜索视频
|
||||
|
||||
# Yahoo Finance (browser)
|
||||
opencli yahoo-finance quote --symbol AAPL # 股票行情
|
||||
|
||||
# Reuters (browser)
|
||||
opencli reuters search --query "AI" # 路透社搜索
|
||||
|
||||
# 什么值得买 (browser)
|
||||
opencli smzdm search --keyword "耳机" # 搜索好价
|
||||
|
||||
# 携程 (browser)
|
||||
opencli ctrip search --query "三亚" # 搜索目的地
|
||||
```
|
||||
|
||||
### Management Commands
|
||||
|
||||
Generated
+1236
-1
File diff suppressed because it is too large.
Load diff
+5
-2
@@ -18,7 +18,9 @@
|
||||
"start": "node dist/main.js",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"lint": "tsc --noEmit",
|
||||
"prepublishOnly": "npm run build"
|
||||
"prepublishOnly": "npm run build",
|
||||
"test": "vitest run",
|
||||
"test:watch": "vitest"
|
||||
},
|
||||
"keywords": [
|
||||
"cli",
|
||||
@@ -44,6 +46,7 @@
|
||||
"@types/js-yaml": "^4.0.9",
|
||||
"@types/node": "^22.13.10",
|
||||
"tsx": "^4.19.3",
|
||||
"typescript": "^5.8.2"
|
||||
"typescript": "^5.8.2",
|
||||
"vitest": "^4.1.0"
|
||||
}
|
||||
}
|
||||
+9
-7
@@ -2,6 +2,8 @@
|
||||
* Bilibili shared helpers: WBI signing, authenticated fetch, nav data, UID resolution.
|
||||
*/
|
||||
|
||||
import type { IPage } from './types.js';
|
||||
|
||||
const MIXIN_KEY_ENC_TAB = [
|
||||
46,47,18,2,53,8,23,32,15,50,10,31,58,3,45,35,27,43,5,49,
|
||||
33,9,42,19,29,28,14,39,12,38,41,13,37,48,7,16,24,55,40,
|
||||
@@ -17,7 +19,7 @@ export function payloadData(payload: any): any {
|
||||
return payload?.data ?? payload;
|
||||
}
|
||||
|
||||
async function getNavData(page: any): Promise<any> {
|
||||
async function getNavData(page: IPage): Promise<any> {
|
||||
return page.evaluate(`
|
||||
async () => {
|
||||
const res = await fetch('https://api.bilibili.com/x/web-interface/nav', { credentials: 'include' });
|
||||
@@ -26,7 +28,7 @@ async function getNavData(page: any): Promise<any> {
|
||||
`);
|
||||
}
|
||||
|
||||
async function getWbiKeys(page: any): Promise<{ imgKey: string; subKey: string }> {
|
||||
async function getWbiKeys(page: IPage): Promise<{ imgKey: string; subKey: string }> {
|
||||
const nav = await getNavData(page);
|
||||
const wbiImg = nav?.data?.wbi_img ?? {};
|
||||
const imgUrl = wbiImg.img_url ?? '';
|
||||
@@ -47,7 +49,7 @@ async function md5(text: string): Promise<string> {
|
||||
}
|
||||
|
||||
export async function wbiSign(
|
||||
page: any,
|
||||
page: IPage,
|
||||
params: Record<string, any>,
|
||||
): Promise<Record<string, string>> {
|
||||
const { imgKey, subKey } = await getWbiKeys(page);
|
||||
@@ -65,7 +67,7 @@ export async function wbiSign(
|
||||
}
|
||||
|
||||
export async function apiGet(
|
||||
page: any,
|
||||
page: IPage,
|
||||
path: string,
|
||||
opts: { params?: Record<string, any>; signed?: boolean } = {},
|
||||
): Promise<any> {
|
||||
@@ -81,7 +83,7 @@ export async function apiGet(
|
||||
return fetchJson(page, url);
|
||||
}
|
||||
|
||||
export async function fetchJson(page: any, url: string): Promise<any> {
|
||||
export async function fetchJson(page: IPage, url: string): Promise<any> {
|
||||
const escapedUrl = url.replace(/"/g, '\\"');
|
||||
return page.evaluate(`
|
||||
async () => {
|
||||
@@ -91,14 +93,14 @@ export async function fetchJson(page: any, url: string): Promise<any> {
|
||||
`);
|
||||
}
|
||||
|
||||
export async function getSelfUid(page: any): Promise<string> {
|
||||
export async function getSelfUid(page: IPage): Promise<string> {
|
||||
const nav = await getNavData(page);
|
||||
const mid = nav?.data?.mid;
|
||||
if (!mid) throw new Error('Not logged in to Bilibili');
|
||||
return String(mid);
|
||||
}
|
||||
|
||||
export async function resolveUid(page: any, input: string): Promise<string> {
|
||||
export async function resolveUid(page: IPage, input: string): Promise<string> {
|
||||
if (/^\d+$/.test(input)) return input;
|
||||
// Search for user by name
|
||||
const payload = await apiGet(page, '/x/web-interface/wbi/search/type', {
|
||||
|
||||
+8
-2
@@ -10,6 +10,10 @@ import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { formatSnapshot } from './snapshotFormatter.js';
|
||||
|
||||
// Read version from package.json (single source of truth)
|
||||
const __browser_dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
const PKG_VERSION = (() => { try { return JSON.parse(fs.readFileSync(path.resolve(__browser_dirname, '..', 'package.json'), 'utf-8')).version; } catch { return '0.0.0'; } })();
|
||||
|
||||
const EXTENSION_LOCK_TIMEOUT = parseInt(process.env.OPENCLI_EXTENSION_LOCK_TIMEOUT ?? '120', 10);
|
||||
const EXTENSION_LOCK_POLL = parseInt(process.env.OPENCLI_EXTENSION_LOCK_POLL_INTERVAL ?? '1', 10);
|
||||
const CONNECT_TIMEOUT = parseInt(process.env.OPENCLI_BROWSER_CONNECT_TIMEOUT ?? '30', 10);
|
||||
@@ -21,10 +25,12 @@ function jsonRpcRequest(method: string, params: Record<string, any> = {}): strin
|
||||
return JSON.stringify({ jsonrpc: '2.0', id: _nextId++, method, params }) + '\n';
|
||||
}
|
||||
|
||||
import type { IPage } from './types.js';
|
||||
|
||||
/**
|
||||
* Page abstraction wrapping JSON-RPC calls to Playwright MCP.
|
||||
*/
|
||||
export class Page {
|
||||
export class Page implements IPage {
|
||||
constructor(private _send: (msg: string) => void, private _recv: () => Promise<any>) {}
|
||||
|
||||
async call(method: string, params: Record<string, any> = {}): Promise<any> {
|
||||
@@ -173,7 +179,7 @@ export class PlaywrightMCP {
|
||||
const initMsg = jsonRpcRequest('initialize', {
|
||||
protocolVersion: '2024-11-05',
|
||||
capabilities: {},
|
||||
clientInfo: { name: 'opencli', version: '0.1.0' },
|
||||
clientInfo: { name: 'opencli', version: PKG_VERSION },
|
||||
});
|
||||
this._proc.stdin?.write(initMsg);
|
||||
|
||||
|
||||
+3
-2
@@ -11,6 +11,7 @@
|
||||
*/
|
||||
|
||||
import { Strategy } from './registry.js';
|
||||
import type { IPage } from './types.js';
|
||||
|
||||
/** Strategy cascade order (simplest → most complex) */
|
||||
const CASCADE_ORDER: Strategy[] = [
|
||||
@@ -41,7 +42,7 @@ interface CascadeResult {
|
||||
* Returns whether the probe succeeded and basic response info.
|
||||
*/
|
||||
export async function probeEndpoint(
|
||||
page: any,
|
||||
page: IPage,
|
||||
url: string,
|
||||
strategy: Strategy,
|
||||
opts: { timeout?: number } = {},
|
||||
@@ -168,7 +169,7 @@ export async function probeEndpoint(
|
||||
* Returns the simplest working strategy.
|
||||
*/
|
||||
export async function cascadeProbe(
|
||||
page: any,
|
||||
page: IPage,
|
||||
url: string,
|
||||
opts: { maxStrategy?: Strategy; timeout?: number } = {},
|
||||
): Promise<CascadeResult> {
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
/**
|
||||
* BBC News headlines — public RSS feed, no browser needed.
|
||||
* Source: bb-sites/bbc/news.js
|
||||
*/
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'bbc',
|
||||
name: 'news',
|
||||
description: 'BBC News headlines (RSS)',
|
||||
domain: 'www.bbc.com',
|
||||
strategy: Strategy.PUBLIC,
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 20, help: 'Number of headlines (max 50)' },
|
||||
],
|
||||
columns: ['rank', 'title', 'description', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
const count = Math.min(kwargs.limit || 20, 50);
|
||||
const resp = await fetch('https://feeds.bbci.co.uk/news/rss.xml');
|
||||
if (!resp.ok) return [];
|
||||
const xml = await resp.text();
|
||||
// Simple XML parsing without DOMParser (works in Node)
|
||||
const items: any[] = [];
|
||||
const itemRegex = /<item>([\s\S]*?)<\/item>/g;
|
||||
let match;
|
||||
while ((match = itemRegex.exec(xml)) && items.length < count) {
|
||||
const block = match[1];
|
||||
const title = block.match(/<title><!\[CDATA\[(.*?)\]\]>|<title>(.*?)<\/title>/)?.[1] || block.match(/<title>(.*?)<\/title>/)?.[1] || '';
|
||||
const desc = block.match(/<description><!\[CDATA\[(.*?)\]\]>|<description>(.*?)<\/description>/)?.[1] || block.match(/<description>(.*?)<\/description>/)?.[1] || '';
|
||||
const link = block.match(/<link>(.*?)<\/link>/)?.[1] || block.match(/<guid[^>]*>(.*?)<\/guid>/)?.[1] || '';
|
||||
if (title) {
|
||||
items.push({
|
||||
rank: items.length + 1,
|
||||
title: title.trim(),
|
||||
description: desc.trim().substring(0, 200),
|
||||
url: link.trim(),
|
||||
});
|
||||
}
|
||||
}
|
||||
return items;
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,47 @@
|
||||
/**
|
||||
* BOSS直聘 job search — browser cookie API.
|
||||
* Source: bb-sites/boss/search.js
|
||||
*/
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'boss',
|
||||
name: 'search',
|
||||
description: 'BOSS直聘搜索职位',
|
||||
domain: 'www.zhipin.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
args: [
|
||||
{ name: 'query', required: true, help: 'Search keyword (e.g. AI agent, 前端)' },
|
||||
{ name: 'city', default: '101010100', help: 'City code (101010100=北京, 101020100=上海, 101210100=杭州, 101280100=广州)' },
|
||||
{ name: 'limit', type: 'int', default: 15, help: 'Number of results' },
|
||||
],
|
||||
columns: ['name', 'salary', 'company', 'city', 'experience', 'degree', 'boss', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
await page.goto('https://www.zhipin.com');
|
||||
await page.wait(2);
|
||||
const data = await page.evaluate(`
|
||||
(async () => {
|
||||
const params = new URLSearchParams({
|
||||
scene: '1', query: '${kwargs.query.replace(/'/g, "\\'")}',
|
||||
city: '${kwargs.city || '101010100'}', page: '1', pageSize: '15',
|
||||
experience: '', degree: '', payType: '', partTime: '',
|
||||
industry: '', scale: '', stage: '', position: '',
|
||||
jobType: '', salary: '', multiBusinessDistrict: '', multiSubway: ''
|
||||
});
|
||||
const resp = await fetch('/wapi/zpgeek/search/joblist.json?' + params.toString(), {credentials: 'include'});
|
||||
if (!resp.ok) return {error: 'HTTP ' + resp.status};
|
||||
const d = await resp.json();
|
||||
if (d.code !== 0) return {error: d.message || 'API error'};
|
||||
const zpData = d.zpData || {};
|
||||
return (zpData.jobList || []).map(j => ({
|
||||
name: j.jobName, salary: j.salaryDesc, company: j.brandName,
|
||||
city: j.cityName, experience: j.jobExperience, degree: j.jobDegree,
|
||||
boss: j.bossName + ' · ' + j.bossTitle,
|
||||
url: j.encryptJobId ? 'https://www.zhipin.com/job_detail/' + j.encryptJobId + '.html' : ''
|
||||
}));
|
||||
})()
|
||||
`);
|
||||
if (!Array.isArray(data)) return [];
|
||||
return data.slice(0, kwargs.limit || 15);
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,62 @@
|
||||
/**
|
||||
* 携程旅行搜索 — browser cookie, multi-strategy.
|
||||
* Source: bb-sites/ctrip/search.js (simplified to suggestion API)
|
||||
*/
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'ctrip',
|
||||
name: 'search',
|
||||
description: '携程旅行搜索',
|
||||
domain: 'www.ctrip.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
args: [
|
||||
{ name: 'query', required: true, help: 'Search keyword (city or attraction)' },
|
||||
{ name: 'limit', type: 'int', default: 15, help: 'Number of results' },
|
||||
],
|
||||
columns: ['rank', 'name', 'type', 'score', 'price', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
const limit = kwargs.limit || 15;
|
||||
await page.goto('https://www.ctrip.com');
|
||||
await page.wait(2);
|
||||
const data = await page.evaluate(`
|
||||
(async () => {
|
||||
const query = '${kwargs.query.replace(/'/g, "\\'")}';
|
||||
const limit = ${limit};
|
||||
|
||||
// Strategy 1: Suggestion API
|
||||
try {
|
||||
const suggestUrl = 'https://m.ctrip.com/restapi/h5api/searchapp/search?action=onekeyali&keyword=' + encodeURIComponent(query);
|
||||
const resp = await fetch(suggestUrl, {credentials: 'include'});
|
||||
if (resp.ok) {
|
||||
const d = await resp.json();
|
||||
const raw = d.data || d.result || d;
|
||||
if (raw && typeof raw === 'object') {
|
||||
// Flatten all result categories
|
||||
const items = [];
|
||||
for (const key of Object.keys(raw)) {
|
||||
const list = Array.isArray(raw[key]) ? raw[key] : [];
|
||||
for (const item of list) {
|
||||
if (items.length >= limit) break;
|
||||
items.push({
|
||||
rank: items.length + 1,
|
||||
name: item.word || item.name || item.title || '',
|
||||
type: item.type || item.tpName || key,
|
||||
score: item.score || '',
|
||||
price: item.price || item.minPrice || '',
|
||||
url: item.url || item.surl || '',
|
||||
});
|
||||
}
|
||||
}
|
||||
if (items.length > 0) return items;
|
||||
}
|
||||
}
|
||||
} catch(e) {}
|
||||
|
||||
return {error: 'No results for: ' + query};
|
||||
})()
|
||||
`);
|
||||
if (!Array.isArray(data)) return [];
|
||||
return data;
|
||||
},
|
||||
});
|
||||
@@ -1,21 +0,0 @@
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'github', name: 'search', description: 'Search GitHub repositories', domain: 'github.com', strategy: Strategy.PUBLIC, browser: false,
|
||||
args: [
|
||||
{ name: 'keyword', required: true, help: 'Search keyword' },
|
||||
{ name: 'sort', default: 'stars', help: 'Sort by: stars, forks, updated' },
|
||||
{ name: 'limit', type: 'int', default: 20, help: 'Number of results' },
|
||||
],
|
||||
columns: ['rank', 'name', 'stars', 'language', 'description'],
|
||||
func: async (_page, kwargs) => {
|
||||
const { keyword, sort = 'stars', limit = 20 } = kwargs;
|
||||
const resp = await fetch(`https://api.github.com/search/repositories?${new URLSearchParams({ q: keyword, sort, order: 'desc', per_page: String(Math.min(Number(limit), 100)) })}`, {
|
||||
headers: { Accept: 'application/vnd.github.v3+json', 'User-Agent': 'opencli/0.1' },
|
||||
});
|
||||
const data = await resp.json() as any;
|
||||
return (data.items ?? []).slice(0, Number(limit)).map((item: any, i: number) => ({
|
||||
rank: i + 1, name: item.full_name, stars: item.stargazers_count, language: item.language ?? '', description: (item.description ?? '').slice(0, 80),
|
||||
}));
|
||||
},
|
||||
});
|
||||
@@ -1,58 +0,0 @@
|
||||
site: github
|
||||
name: trending
|
||||
description: GitHub trending repositories
|
||||
domain: github.com
|
||||
|
||||
args:
|
||||
language:
|
||||
type: str
|
||||
default: ""
|
||||
description: "Programming language filter (e.g. python, rust)"
|
||||
since:
|
||||
type: str
|
||||
default: daily
|
||||
description: "Time range: daily, weekly, monthly"
|
||||
limit:
|
||||
type: int
|
||||
default: 20
|
||||
description: Number of repos
|
||||
|
||||
pipeline:
|
||||
- evaluate: |
|
||||
(async () => {
|
||||
const lang = '${{ args.language }}' ? '/${{ args.language }}' : '';
|
||||
const res = await fetch(`https://github.com/trending${lang}?since=${{ args.since }}`, {
|
||||
headers: { 'Accept': 'text/html' }
|
||||
});
|
||||
const html = await res.text();
|
||||
const parser = new DOMParser();
|
||||
const doc = parser.parseFromString(html, 'text/html');
|
||||
const rows = doc.querySelectorAll('article.Box-row');
|
||||
return Array.from(rows).map(row => {
|
||||
const nameEl = row.querySelector('h2 a');
|
||||
const descEl = row.querySelector('p');
|
||||
const langEl = row.querySelector('[itemprop="programmingLanguage"]');
|
||||
const starsEl = row.querySelectorAll('a.Link--muted');
|
||||
const todayEl = row.querySelector('span.d-inline-block.float-sm-right');
|
||||
return {
|
||||
repo: nameEl?.textContent?.trim()?.replace(/\s+/g, '') || '',
|
||||
description: descEl?.textContent?.trim() || '',
|
||||
language: langEl?.textContent?.trim() || '',
|
||||
stars: starsEl[0]?.textContent?.trim() || '',
|
||||
forks: starsEl[1]?.textContent?.trim() || '',
|
||||
today: todayEl?.textContent?.trim() || ''
|
||||
};
|
||||
});
|
||||
})()
|
||||
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
repo: ${{ item.repo }}
|
||||
description: ${{ item.description }}
|
||||
language: ${{ item.language }}
|
||||
stars: ${{ item.stars }}
|
||||
today: ${{ item.today }}
|
||||
|
||||
- limit: ${{ args.limit }}
|
||||
|
||||
columns: [rank, repo, language, stars, today]
|
||||
@@ -20,3 +20,27 @@ import './zhihu/question.js';
|
||||
|
||||
// xiaohongshu
|
||||
import './xiaohongshu/search.js';
|
||||
|
||||
// bbc
|
||||
import './bbc/news.js';
|
||||
|
||||
// weibo
|
||||
import './weibo/hot.js';
|
||||
|
||||
// boss
|
||||
import './boss/search.js';
|
||||
|
||||
// yahoo-finance
|
||||
import './yahoo-finance/quote.js';
|
||||
|
||||
// reuters
|
||||
import './reuters/search.js';
|
||||
|
||||
// smzdm
|
||||
import './smzdm/search.js';
|
||||
|
||||
// ctrip
|
||||
import './ctrip/search.js';
|
||||
|
||||
// youtube
|
||||
import './youtube/search.js';
|
||||
@@ -0,0 +1,52 @@
|
||||
/**
|
||||
* Reuters news search — API with HTML fallback.
|
||||
* Source: bb-sites/reuters/search.js
|
||||
*/
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'reuters',
|
||||
name: 'search',
|
||||
description: 'Reuters 路透社新闻搜索',
|
||||
domain: 'www.reuters.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
args: [
|
||||
{ name: 'query', required: true, help: 'Search query' },
|
||||
{ name: 'limit', type: 'int', default: 10, help: 'Number of results (max 40)' },
|
||||
],
|
||||
columns: ['rank', 'title', 'date', 'section', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
const count = Math.min(kwargs.limit || 10, 40);
|
||||
await page.goto('https://www.reuters.com');
|
||||
await page.wait(2);
|
||||
const data = await page.evaluate(`
|
||||
(async () => {
|
||||
const count = ${count};
|
||||
const apiQuery = JSON.stringify({
|
||||
keyword: '${kwargs.query.replace(/'/g, "\\'")}',
|
||||
offset: 0, orderby: 'display_date:desc', size: count, website: 'reuters'
|
||||
});
|
||||
const apiUrl = 'https://www.reuters.com/pf/api/v3/content/fetch/articles-by-search-v2?query=' + encodeURIComponent(apiQuery);
|
||||
try {
|
||||
const resp = await fetch(apiUrl, {credentials: 'include'});
|
||||
if (resp.ok) {
|
||||
const data = await resp.json();
|
||||
const articles = data.result?.articles || data.articles || [];
|
||||
if (articles.length > 0) {
|
||||
return articles.slice(0, count).map((a, i) => ({
|
||||
rank: i + 1,
|
||||
title: a.title || a.headlines?.basic || '',
|
||||
date: (a.display_date || a.published_time || '').split('T')[0],
|
||||
section: a.taxonomy?.section?.name || '',
|
||||
url: a.canonical_url ? 'https://www.reuters.com' + a.canonical_url : '',
|
||||
}));
|
||||
}
|
||||
}
|
||||
} catch(e) {}
|
||||
return {error: 'Reuters API unavailable'};
|
||||
})()
|
||||
`);
|
||||
if (!Array.isArray(data)) return [];
|
||||
return data;
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,66 @@
|
||||
/**
|
||||
* 什么值得买搜索好价 — browser cookie, HTML parse.
|
||||
* Source: bb-sites/smzdm/search.js
|
||||
*/
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'smzdm',
|
||||
name: 'search',
|
||||
description: '什么值得买搜索好价',
|
||||
domain: 'www.smzdm.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
args: [
|
||||
{ name: 'keyword', required: true, help: 'Search keyword' },
|
||||
{ name: 'limit', type: 'int', default: 20, help: 'Number of results' },
|
||||
],
|
||||
columns: ['rank', 'title', 'price', 'mall', 'comments', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
const q = encodeURIComponent(kwargs.keyword);
|
||||
const limit = kwargs.limit || 20;
|
||||
await page.goto('https://www.smzdm.com');
|
||||
await page.wait(2);
|
||||
const data = await page.evaluate(`
|
||||
(async () => {
|
||||
const q = '${q}';
|
||||
const limit = ${limit};
|
||||
// Try youhui channel first, then home
|
||||
for (const channel of ['youhui', 'home']) {
|
||||
try {
|
||||
const resp = await fetch('https://search.smzdm.com/ajax/?c=' + channel + '&s=' + q + '&p=1&v=b', {
|
||||
credentials: 'include',
|
||||
headers: {'X-Requested-With': 'XMLHttpRequest'}
|
||||
});
|
||||
if (!resp.ok) continue;
|
||||
const html = await resp.text();
|
||||
if (html.indexOf('feed-row-wide') === -1) continue;
|
||||
const parser = new DOMParser();
|
||||
const doc = parser.parseFromString(html, 'text/html');
|
||||
const items = doc.querySelectorAll('li.feed-row-wide');
|
||||
const results = [];
|
||||
items.forEach((li, i) => {
|
||||
if (results.length >= limit) return;
|
||||
const titleEl = li.querySelector('h5.feed-block-title > a')
|
||||
|| li.querySelector('h5 > a');
|
||||
if (!titleEl) return;
|
||||
const title = (titleEl.getAttribute('title') || titleEl.textContent || '').trim();
|
||||
const url = titleEl.getAttribute('href') || '';
|
||||
const priceEl = li.querySelector('.z-highlight');
|
||||
const price = priceEl ? priceEl.textContent.trim() : '';
|
||||
let mall = '';
|
||||
const extrasSpan = li.querySelector('.z-feed-foot-r .feed-block-extras span');
|
||||
if (extrasSpan) mall = extrasSpan.textContent.trim();
|
||||
const commentEl = li.querySelector('.feed-btn-comment');
|
||||
const comments = commentEl ? parseInt(commentEl.textContent.trim()) || 0 : 0;
|
||||
results.push({rank: results.length + 1, title, price, mall, comments, url});
|
||||
});
|
||||
if (results.length > 0) return results;
|
||||
} catch(e) { continue; }
|
||||
}
|
||||
return {error: 'No results'};
|
||||
})()
|
||||
`);
|
||||
if (!Array.isArray(data)) return [];
|
||||
return data;
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,41 @@
|
||||
/**
|
||||
* Weibo hot search — browser cookie API.
|
||||
* Source: bb-sites/weibo/hot.js
|
||||
*/
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'weibo',
|
||||
name: 'hot',
|
||||
description: '微博热搜',
|
||||
domain: 'weibo.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 30, help: 'Number of items (max 50)' },
|
||||
],
|
||||
columns: ['rank', 'word', 'hot_value', 'category', 'label', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
const count = Math.min(kwargs.limit || 30, 50);
|
||||
await page.goto('https://weibo.com');
|
||||
await page.wait(2);
|
||||
const data = await page.evaluate(`
|
||||
(async () => {
|
||||
const resp = await fetch('/ajax/statuses/hot_band', {credentials: 'include'});
|
||||
if (!resp.ok) return {error: 'HTTP ' + resp.status};
|
||||
const data = await resp.json();
|
||||
if (!data.ok) return {error: 'API error'};
|
||||
const bandList = data.data?.band_list || [];
|
||||
return bandList.map((item, i) => ({
|
||||
rank: item.realpos || (i + 1),
|
||||
word: item.word,
|
||||
hot_value: item.num || 0,
|
||||
category: item.category || '',
|
||||
label: item.label_name || '',
|
||||
url: 'https://s.weibo.com/weibo?q=' + encodeURIComponent('#' + item.word + '#')
|
||||
}));
|
||||
})()
|
||||
`);
|
||||
if (!Array.isArray(data)) return [];
|
||||
return data.slice(0, count);
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,74 @@
|
||||
/**
|
||||
* Yahoo Finance stock quote — multi-strategy API fallback.
|
||||
* Source: bb-sites/yahoo-finance/quote.js
|
||||
*/
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'yahoo-finance',
|
||||
name: 'quote',
|
||||
description: 'Yahoo Finance 股票行情',
|
||||
domain: 'finance.yahoo.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
args: [
|
||||
{ name: 'symbol', required: true, help: 'Stock ticker (e.g. AAPL, MSFT, TSLA)' },
|
||||
],
|
||||
columns: ['symbol', 'name', 'price', 'change', 'changePercent', 'open', 'high', 'low', 'volume', 'marketCap'],
|
||||
func: async (page, kwargs) => {
|
||||
const symbol = kwargs.symbol.toUpperCase().trim();
|
||||
await page.goto(`https://finance.yahoo.com/quote/${encodeURIComponent(symbol)}/`);
|
||||
await page.wait(3);
|
||||
const data = await page.evaluate(`
|
||||
(async () => {
|
||||
const sym = '${symbol}';
|
||||
|
||||
// Strategy 1: v8 chart API
|
||||
try {
|
||||
const chartUrl = 'https://query1.finance.yahoo.com/v8/finance/chart/' + encodeURIComponent(sym) + '?interval=1d&range=1d';
|
||||
const resp = await fetch(chartUrl);
|
||||
if (resp.ok) {
|
||||
const d = await resp.json();
|
||||
const chart = d?.chart?.result?.[0];
|
||||
if (chart) {
|
||||
const meta = chart.meta || {};
|
||||
const prevClose = meta.previousClose || meta.chartPreviousClose;
|
||||
const price = meta.regularMarketPrice;
|
||||
const change = price != null && prevClose != null ? (price - prevClose) : null;
|
||||
const changePct = change != null && prevClose ? ((change / prevClose) * 100) : null;
|
||||
return {
|
||||
symbol: meta.symbol || sym, name: meta.shortName || meta.longName || sym,
|
||||
price: price != null ? Number(price.toFixed(2)) : null,
|
||||
change: change != null ? change.toFixed(2) : null,
|
||||
changePercent: changePct != null ? changePct.toFixed(2) + '%' : null,
|
||||
open: chart.indicators?.quote?.[0]?.open?.[0] || null,
|
||||
high: meta.regularMarketDayHigh || null,
|
||||
low: meta.regularMarketDayLow || null,
|
||||
volume: meta.regularMarketVolume || null,
|
||||
marketCap: null, currency: meta.currency, exchange: meta.exchangeName,
|
||||
};
|
||||
}
|
||||
}
|
||||
} catch(e) {}
|
||||
|
||||
// Strategy 2: Parse from page
|
||||
const titleEl = document.querySelector('title');
|
||||
const priceEl = document.querySelector('[data-testid="qsp-price"]');
|
||||
const changeEl = document.querySelector('[data-testid="qsp-price-change"]');
|
||||
const changePctEl = document.querySelector('[data-testid="qsp-price-change-percent"]');
|
||||
if (priceEl) {
|
||||
return {
|
||||
symbol: sym,
|
||||
name: titleEl ? titleEl.textContent.split('(')[0].trim() : sym,
|
||||
price: priceEl.textContent.replace(/,/g, ''),
|
||||
change: changeEl ? changeEl.textContent : null,
|
||||
changePercent: changePctEl ? changePctEl.textContent : null,
|
||||
open: null, high: null, low: null, volume: null, marketCap: null,
|
||||
};
|
||||
}
|
||||
return {error: 'Could not fetch quote for ' + sym};
|
||||
})()
|
||||
`);
|
||||
if (!data || data.error) return [];
|
||||
return [data];
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,60 @@
|
||||
/**
|
||||
* YouTube search — innertube API via browser session.
|
||||
* Source: bb-sites/youtube/search.js
|
||||
*/
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'youtube',
|
||||
name: 'search',
|
||||
description: 'Search YouTube videos',
|
||||
domain: 'www.youtube.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
args: [
|
||||
{ name: 'query', required: true, help: 'Search query' },
|
||||
{ name: 'limit', type: 'int', default: 20, help: 'Max results (max 50)' },
|
||||
],
|
||||
columns: ['rank', 'title', 'channel', 'views', 'duration', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
const limit = Math.min(kwargs.limit || 20, 50);
|
||||
await page.goto('https://www.youtube.com');
|
||||
await page.wait(2);
|
||||
const data = await page.evaluate(`
|
||||
(async () => {
|
||||
const cfg = window.ytcfg?.data_ || {};
|
||||
const apiKey = cfg.INNERTUBE_API_KEY;
|
||||
const context = cfg.INNERTUBE_CONTEXT;
|
||||
if (!apiKey || !context) return {error: 'YouTube config not found'};
|
||||
|
||||
const resp = await fetch('/youtubei/v1/search?key=' + apiKey + '&prettyPrint=false', {
|
||||
method: 'POST', credentials: 'include',
|
||||
headers: {'Content-Type': 'application/json'},
|
||||
body: JSON.stringify({context, query: '${kwargs.query.replace(/'/g, "\\'")}'})
|
||||
});
|
||||
if (!resp.ok) return {error: 'HTTP ' + resp.status};
|
||||
|
||||
const data = await resp.json();
|
||||
const contents = data.contents?.twoColumnSearchResultsRenderer?.primaryContents?.sectionListRenderer?.contents || [];
|
||||
const videos = [];
|
||||
for (const section of contents) {
|
||||
for (const item of (section.itemSectionRenderer?.contents || [])) {
|
||||
if (item.videoRenderer && videos.length < ${limit}) {
|
||||
const v = item.videoRenderer;
|
||||
videos.push({
|
||||
rank: videos.length + 1,
|
||||
title: v.title?.runs?.[0]?.text || '',
|
||||
channel: v.ownerText?.runs?.[0]?.text || '',
|
||||
views: v.viewCountText?.simpleText || v.shortViewCountText?.simpleText || '',
|
||||
duration: v.lengthText?.simpleText || 'LIVE',
|
||||
url: 'https://www.youtube.com/watch?v=' + v.videoId
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
return videos;
|
||||
})()
|
||||
`);
|
||||
if (!Array.isArray(data)) return [];
|
||||
return data;
|
||||
},
|
||||
});
|
||||
+2
-1
@@ -6,6 +6,7 @@ import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import yaml from 'js-yaml';
|
||||
import { type CliCommand, type Arg, Strategy, registerCommand } from './registry.js';
|
||||
import type { IPage } from './types.js';
|
||||
import { executePipeline } from './pipeline.js';
|
||||
|
||||
export function discoverClis(...dirs: string[]): void {
|
||||
@@ -72,7 +73,7 @@ function registerYamlCli(filePath: string, defaultSite: string): void {
|
||||
|
||||
export async function executeCommand(
|
||||
cmd: CliCommand,
|
||||
page: any,
|
||||
page: IPage | null,
|
||||
kwargs: Record<string, any>,
|
||||
debug: boolean = false,
|
||||
): Promise<any> {
|
||||
|
||||
+1
-1
@@ -477,7 +477,7 @@ export function renderExploreSummary(result: Record<string, any>): string {
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
async function readPageMetadata(page: any): Promise<{ url: string; title: string }> {
|
||||
async function readPageMetadata(page: any /* IPage */): Promise<{ url: string; title: string }> {
|
||||
try {
|
||||
const result = await page.evaluate(`() => ({ url: window.location.href, title: document.title || '' })`);
|
||||
if (result && typeof result === 'object') return { url: String(result.url ?? ''), title: String(result.title ?? '') };
|
||||
|
||||
+3
-1
@@ -10,7 +10,9 @@
|
||||
|
||||
import { exploreUrl } from './explore.js';
|
||||
import { synthesizeFromExplore } from './synthesize.js';
|
||||
import { registerCandidates } from './register.js';
|
||||
|
||||
// TODO: implement real CLI registration (copy candidate YAML to user clis dir)
|
||||
function registerCandidates(_opts: any): any { return { ok: true, count: 0 }; }
|
||||
|
||||
const CAPABILITY_ALIASES: Record<string, string[]> = {
|
||||
search: ['search', '搜索', '查找', 'query', 'keyword'],
|
||||
|
||||
+7
-5
@@ -3,6 +3,7 @@
|
||||
* opencli — Make any website your CLI. AI-powered.
|
||||
*/
|
||||
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
@@ -20,10 +21,14 @@ const __dirname = path.dirname(__filename);
|
||||
const BUILTIN_CLIS = path.resolve(__dirname, 'clis');
|
||||
const USER_CLIS = path.join(os.homedir(), '.opencli', 'clis');
|
||||
|
||||
// Read version from package.json (single source of truth)
|
||||
const pkgJsonPath = path.resolve(__dirname, '..', 'package.json');
|
||||
const PKG_VERSION = JSON.parse(fs.readFileSync(pkgJsonPath, 'utf-8')).version ?? '0.0.0';
|
||||
|
||||
discoverClis(BUILTIN_CLIS, USER_CLIS);
|
||||
|
||||
const program = new Command();
|
||||
program.name('opencli').description('Make any website your CLI. Zero setup. AI-powered.').version('0.1.0');
|
||||
program.name('opencli').description('Make any website your CLI. Zero setup. AI-powered.').version(PKG_VERSION);
|
||||
|
||||
// ── Built-in commands ──────────────────────────────────────────────────────
|
||||
|
||||
@@ -49,10 +54,7 @@ program.command('validate').description('Validate CLI definitions').argument('[t
|
||||
program.command('verify').description('Validate + smoke test').argument('[target]').option('--smoke', 'Run smoke tests', false)
|
||||
.action(async (target, opts) => { const { verifyClis, renderVerifyReport } = await import('./verify.js'); const r = await verifyClis({ builtinClis: BUILTIN_CLIS, userClis: USER_CLIS, target, smoke: opts.smoke }); console.log(renderVerifyReport(r)); process.exitCode = r.ok ? 0 : 1; });
|
||||
|
||||
program.command('explore').description('Explore a website').argument('<url>').option('--site <name>').option('--goal <text>').option('--wait <s>', '', '3')
|
||||
.action(async (url, opts) => { const { exploreUrl, renderExploreSummary } = await import('./explore.js'); console.log(renderExploreSummary(await exploreUrl(url, { BrowserFactory: PlaywrightMCP, site: opts.site, goal: opts.goal, waitSeconds: parseFloat(opts.wait) }))); });
|
||||
|
||||
program.command('probe').description('Probe a website: discover APIs, stores, and recommend strategies').argument('<url>').option('--site <name>').option('--goal <text>').option('--wait <s>', '', '3')
|
||||
program.command('explore').alias('probe').description('Explore a website: discover APIs, stores, and recommend strategies').argument('<url>').option('--site <name>').option('--goal <text>').option('--wait <s>', '', '3')
|
||||
.action(async (url, opts) => { const { exploreUrl, renderExploreSummary } = await import('./explore.js'); console.log(renderExploreSummary(await exploreUrl(url, { BrowserFactory: PlaywrightMCP, site: opts.site, goal: opts.goal, waitSeconds: parseFloat(opts.wait) }))); });
|
||||
|
||||
program.command('synthesize').description('Synthesize CLIs from explore').argument('<target>').option('--top <n>', '', '3')
|
||||
|
||||
+5
-529
@@ -1,532 +1,8 @@
|
||||
/**
|
||||
* YAML pipeline executor.
|
||||
* Steps: fetch, navigate, evaluate, map, filter, sort, limit, select, snapshot, click, type, wait, press, intercept.
|
||||
* YAML pipeline executor — re-exports from modular pipeline system.
|
||||
*
|
||||
* This file exists for backward compatibility. All logic has been
|
||||
* refactored into src/pipeline/ with modular step handlers.
|
||||
*/
|
||||
|
||||
import chalk from 'chalk';
|
||||
|
||||
export interface PipelineContext {
|
||||
args?: Record<string, any>;
|
||||
debug?: boolean;
|
||||
}
|
||||
|
||||
export async function executePipeline(
|
||||
page: any,
|
||||
pipeline: any[],
|
||||
ctx: PipelineContext = {},
|
||||
): Promise<any> {
|
||||
const args = ctx.args ?? {};
|
||||
const debug = ctx.debug ?? false;
|
||||
let data: any = null;
|
||||
const total = pipeline.length;
|
||||
|
||||
for (let i = 0; i < pipeline.length; i++) {
|
||||
const step = pipeline[i];
|
||||
if (!step || typeof step !== 'object') continue;
|
||||
for (const [op, params] of Object.entries(step)) {
|
||||
if (debug) debugStepStart(i + 1, total, op, params);
|
||||
data = await executeStep(page, op, params, data, args);
|
||||
// Detect error objects returned by steps (e.g. tap store not found)
|
||||
if (data && typeof data === 'object' && !Array.isArray(data) && data.error) {
|
||||
process.stderr.write(` ${chalk.yellow('⚠')} ${chalk.yellow(op)}: ${data.error}\n`);
|
||||
if (data.hint) process.stderr.write(` ${chalk.dim('💡')} ${chalk.dim(data.hint)}\n`);
|
||||
}
|
||||
if (debug) debugStepResult(op, data);
|
||||
}
|
||||
}
|
||||
return data;
|
||||
}
|
||||
|
||||
function normalizeEvaluateSource(source: string): string {
|
||||
const stripped = source.trim();
|
||||
if (!stripped) return '() => undefined';
|
||||
if (stripped.startsWith('(') && stripped.endsWith(')()')) return `() => (${stripped})`;
|
||||
if (/^(async\s+)?\([^)]*\)\s*=>/.test(stripped)) return stripped;
|
||||
if (/^(async\s+)?[A-Za-z_][A-Za-z0-9_]*\s*=>/.test(stripped)) return stripped;
|
||||
if (stripped.startsWith('function ') || stripped.startsWith('async function ')) return stripped;
|
||||
return `() => (${stripped})`;
|
||||
}
|
||||
|
||||
function debugStepStart(stepNum: number, total: number, op: string, params: any): void {
|
||||
let preview = '';
|
||||
if (typeof params === 'string') {
|
||||
preview = params.length <= 80 ? ` → ${params}` : ` → ${params.slice(0, 77)}...`;
|
||||
} else if (params && typeof params === 'object' && !Array.isArray(params)) {
|
||||
preview = ` (${Object.keys(params).join(', ')})`;
|
||||
}
|
||||
process.stderr.write(` ${chalk.dim(`[${stepNum}/${total}]`)} ${chalk.bold.cyan(op)}${preview}\n`);
|
||||
}
|
||||
|
||||
function debugStepResult(op: string, data: any): void {
|
||||
if (data === null || data === undefined) {
|
||||
process.stderr.write(` ${chalk.dim('→ (no data)')}\n`);
|
||||
} else if (Array.isArray(data)) {
|
||||
process.stderr.write(` ${chalk.dim(`→ ${data.length} items`)}\n`);
|
||||
} else if (typeof data === 'object') {
|
||||
const keys = Object.keys(data).slice(0, 5);
|
||||
process.stderr.write(` ${chalk.dim(`→ dict (${keys.join(', ')}${Object.keys(data).length > 5 ? '...' : ''})`)}\n`);
|
||||
} else if (typeof data === 'string') {
|
||||
const p = data.slice(0, 60).replace(/\n/g, '\\n');
|
||||
process.stderr.write(` ${chalk.dim(`→ "${p}${data.length > 60 ? '...' : ''}"`)}\n`);
|
||||
} else {
|
||||
process.stderr.write(` ${chalk.dim(`→ ${typeof data}`)}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
// Single URL fetch helper
|
||||
async function fetchSingle(
|
||||
page: any, url: string, method: string,
|
||||
queryParams: Record<string, any>, headers: Record<string, any>,
|
||||
args: Record<string, any>, data: any,
|
||||
): Promise<any> {
|
||||
const renderedParams: Record<string, string> = {};
|
||||
for (const [k, v] of Object.entries(queryParams)) renderedParams[k] = String(render(v, { args, data }));
|
||||
const renderedHeaders: Record<string, string> = {};
|
||||
for (const [k, v] of Object.entries(headers)) renderedHeaders[k] = String(render(v, { args, data }));
|
||||
|
||||
let finalUrl = url;
|
||||
if (Object.keys(renderedParams).length > 0) {
|
||||
const qs = new URLSearchParams(renderedParams).toString();
|
||||
finalUrl = `${finalUrl}${finalUrl.includes('?') ? '&' : '?'}${qs}`;
|
||||
}
|
||||
|
||||
if (page === null) {
|
||||
const resp = await fetch(finalUrl, { method: method.toUpperCase(), headers: renderedHeaders });
|
||||
return resp.json();
|
||||
}
|
||||
|
||||
const headersJs = JSON.stringify(renderedHeaders);
|
||||
const escapedUrl = finalUrl.replace(/"/g, '\\"');
|
||||
return page.evaluate(`
|
||||
async () => {
|
||||
const resp = await fetch("${escapedUrl}", {
|
||||
method: "${method}", headers: ${headersJs}, credentials: "include"
|
||||
});
|
||||
return await resp.json();
|
||||
}
|
||||
`);
|
||||
}
|
||||
|
||||
async function executeStep(page: any, op: string, params: any, data: any, args: Record<string, any>): Promise<any> {
|
||||
switch (op) {
|
||||
case 'navigate': {
|
||||
const url = render(params, { args, data });
|
||||
await page.goto(String(url));
|
||||
return data;
|
||||
}
|
||||
case 'fetch': {
|
||||
const urlOrObj = typeof params === 'string' ? params : (params?.url ?? '');
|
||||
const method = params?.method ?? 'GET';
|
||||
const queryParams: Record<string, any> = params?.params ?? {};
|
||||
const headers: Record<string, any> = params?.headers ?? {};
|
||||
const urlTemplate = String(urlOrObj);
|
||||
|
||||
// Per-item fetch when data is array and URL references item
|
||||
if (Array.isArray(data) && urlTemplate.includes('item')) {
|
||||
const results: any[] = [];
|
||||
for (let i = 0; i < data.length; i++) {
|
||||
const itemUrl = String(render(urlTemplate, { args, data, item: data[i], index: i }));
|
||||
results.push(await fetchSingle(page, itemUrl, method, queryParams, headers, args, data));
|
||||
}
|
||||
return results;
|
||||
}
|
||||
const url = render(urlOrObj, { args, data });
|
||||
return fetchSingle(page, String(url), method, queryParams, headers, args, data);
|
||||
}
|
||||
case 'select': {
|
||||
const pathStr = String(render(params, { args, data }));
|
||||
if (data && typeof data === 'object') {
|
||||
let current = data;
|
||||
for (const part of pathStr.split('.')) {
|
||||
if (current && typeof current === 'object' && !Array.isArray(current)) current = (current as any)[part];
|
||||
else if (Array.isArray(current) && /^\d+$/.test(part)) current = current[parseInt(part, 10)];
|
||||
else return null;
|
||||
}
|
||||
return current;
|
||||
}
|
||||
return data;
|
||||
}
|
||||
case 'evaluate': {
|
||||
const js = String(render(params, { args, data }));
|
||||
let result = await page.evaluate(normalizeEvaluateSource(js));
|
||||
// MCP may return JSON as a string — auto-parse it
|
||||
if (typeof result === 'string') {
|
||||
const trimmed = result.trim();
|
||||
if ((trimmed.startsWith('[') && trimmed.endsWith(']')) || (trimmed.startsWith('{') && trimmed.endsWith('}'))) {
|
||||
try { result = JSON.parse(trimmed); } catch {}
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
case 'snapshot': {
|
||||
const opts = (typeof params === 'object' && params) ? params : {};
|
||||
return page.snapshot({ interactive: opts.interactive ?? false, compact: opts.compact ?? false, maxDepth: opts.max_depth, raw: opts.raw ?? false });
|
||||
}
|
||||
case 'click': {
|
||||
await page.click(String(render(params, { args, data })).replace(/^@/, ''));
|
||||
return data;
|
||||
}
|
||||
case 'type': {
|
||||
if (typeof params === 'object' && params) {
|
||||
const ref = String(render(params.ref ?? '', { args, data })).replace(/^@/, '');
|
||||
const text = String(render(params.text ?? '', { args, data }));
|
||||
await page.typeText(ref, text);
|
||||
if (params.submit) await page.pressKey('Enter');
|
||||
}
|
||||
return data;
|
||||
}
|
||||
case 'wait': {
|
||||
if (typeof params === 'number') await page.wait(params);
|
||||
else if (typeof params === 'object' && params) {
|
||||
if ('text' in params) {
|
||||
const timeout = params.timeout ?? 10;
|
||||
const start = Date.now();
|
||||
while ((Date.now() - start) / 1000 < timeout) {
|
||||
const snap = await page.snapshot({ raw: true });
|
||||
if (typeof snap === 'string' && snap.includes(params.text)) break;
|
||||
await page.wait(0.5);
|
||||
}
|
||||
} else if ('time' in params) await page.wait(Number(params.time));
|
||||
} else if (typeof params === 'string') await page.wait(Number(render(params, { args, data })));
|
||||
return data;
|
||||
}
|
||||
case 'press': {
|
||||
await page.pressKey(String(render(params, { args, data })));
|
||||
return data;
|
||||
}
|
||||
case 'map': {
|
||||
if (!data || typeof data !== 'object') return data;
|
||||
let items: any[] = Array.isArray(data) ? data : [data];
|
||||
if (!Array.isArray(data) && typeof data === 'object' && 'data' in data) items = data.data;
|
||||
const result: any[] = [];
|
||||
for (let i = 0; i < items.length; i++) {
|
||||
const item = items[i];
|
||||
const row: Record<string, any> = {};
|
||||
for (const [key, template] of Object.entries(params)) row[key] = render(template, { args, data, item, index: i });
|
||||
result.push(row);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
case 'filter': {
|
||||
if (!Array.isArray(data)) return data;
|
||||
return data.filter((item, i) => evalExpr(String(params), { args, item, index: i }));
|
||||
}
|
||||
case 'sort': {
|
||||
if (!Array.isArray(data)) return data;
|
||||
const key = typeof params === 'object' ? (params.by ?? '') : String(params);
|
||||
const reverse = typeof params === 'object' ? params.order === 'desc' : false;
|
||||
return [...data].sort((a, b) => { const va = a[key] ?? ''; const vb = b[key] ?? ''; const cmp = va < vb ? -1 : va > vb ? 1 : 0; return reverse ? -cmp : cmp; });
|
||||
}
|
||||
case 'limit': {
|
||||
if (!Array.isArray(data)) return data;
|
||||
return data.slice(0, Number(render(params, { args, data })));
|
||||
}
|
||||
case 'intercept': {
|
||||
// Declarative XHR interception step
|
||||
// Usage:
|
||||
// intercept:
|
||||
// trigger: "navigate:https://..." | "evaluate:store.note.fetch()" | "click:ref"
|
||||
// capture: "api/pattern" # URL substring to match
|
||||
// timeout: 5 # seconds to wait for matching request
|
||||
// select: "data.items" # optional: extract sub-path from response
|
||||
const cfg = typeof params === 'object' ? params : {};
|
||||
const trigger = cfg.trigger ?? '';
|
||||
const capturePattern = cfg.capture ?? '';
|
||||
const timeout = cfg.timeout ?? 8;
|
||||
const selectPath = cfg.select ?? null;
|
||||
|
||||
if (!capturePattern) return data;
|
||||
|
||||
// Step 1: Execute the trigger action
|
||||
if (trigger.startsWith('navigate:')) {
|
||||
const url = render(trigger.slice('navigate:'.length), { args, data });
|
||||
await page.goto(String(url));
|
||||
} else if (trigger.startsWith('evaluate:')) {
|
||||
const js = trigger.slice('evaluate:'.length);
|
||||
await page.evaluate(normalizeEvaluateSource(render(js, { args, data }) as string));
|
||||
} else if (trigger.startsWith('click:')) {
|
||||
const ref = render(trigger.slice('click:'.length), { args, data });
|
||||
await page.click(String(ref).replace(/^@/, ''));
|
||||
} else if (trigger === 'scroll') {
|
||||
await page.scroll('down');
|
||||
}
|
||||
|
||||
// Step 2: Wait a bit for network requests to fire
|
||||
await page.wait(Math.min(timeout, 3));
|
||||
|
||||
// Step 3: Get network requests and find matching ones
|
||||
const rawNetwork = await page.networkRequests(false);
|
||||
const matchingResponses: any[] = [];
|
||||
|
||||
if (typeof rawNetwork === 'string') {
|
||||
// Parse the network output to find matching URLs
|
||||
const lines = rawNetwork.split('\n');
|
||||
for (const line of lines) {
|
||||
const match = line.match(/\[?(GET|POST)\]?\s+(\S+)\s*(?:=>|→)\s*\[?(\d+)\]?/i);
|
||||
if (match) {
|
||||
const [, method, url, status] = match;
|
||||
if (url.includes(capturePattern) && status === '200') {
|
||||
// Re-fetch the matching URL to get the response body
|
||||
try {
|
||||
const body = await page.evaluate(`
|
||||
async () => {
|
||||
try {
|
||||
const resp = await fetch(${JSON.stringify(url)}, { credentials: 'include' });
|
||||
if (!resp.ok) return null;
|
||||
return await resp.json();
|
||||
} catch { return null; }
|
||||
}
|
||||
`);
|
||||
if (body) matchingResponses.push(body);
|
||||
} catch {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Step 4: Select from response if specified
|
||||
let result = matchingResponses.length === 1 ? matchingResponses[0] :
|
||||
matchingResponses.length > 1 ? matchingResponses : data;
|
||||
|
||||
if (selectPath && result) {
|
||||
let current = result;
|
||||
for (const part of String(selectPath).split('.')) {
|
||||
if (current && typeof current === 'object' && !Array.isArray(current)) {
|
||||
current = current[part];
|
||||
} else break;
|
||||
}
|
||||
result = current ?? result;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
case 'tap': {
|
||||
// ── Declarative Store Action Bridge ──────────────────────────────────
|
||||
// Usage:
|
||||
// tap:
|
||||
// store: feed # Pinia/Vuex store name
|
||||
// action: fetchFeeds # Store action to call
|
||||
// args: [] # Optional args to pass to action
|
||||
// capture: homefeed # URL pattern to capture response
|
||||
// timeout: 5 # Seconds to wait for network (default: 5)
|
||||
// select: data.items # Optional: extract sub-path from response
|
||||
// framework: pinia # Optional: pinia | vuex (auto-detected if omitted)
|
||||
//
|
||||
// Generates a self-contained IIFE that:
|
||||
// 1. Injects fetch + XHR dual interception proxy
|
||||
// 2. Finds the Pinia/Vuex store and calls the action
|
||||
// 3. Captures the response matching the URL pattern
|
||||
// 4. Auto-cleans up interception in finally block
|
||||
// 5. Returns the captured data (optionally sub-selected)
|
||||
|
||||
const cfg = typeof params === 'object' ? params : {};
|
||||
const storeName = String(render(cfg.store ?? '', { args, data }));
|
||||
const actionName = String(render(cfg.action ?? '', { args, data }));
|
||||
const capturePattern = String(render(cfg.capture ?? '', { args, data }));
|
||||
const timeout = cfg.timeout ?? 5;
|
||||
const selectPath = cfg.select ? String(render(cfg.select, { args, data })) : null;
|
||||
const framework = cfg.framework ?? null; // auto-detect if null
|
||||
const actionArgs = cfg.args ?? [];
|
||||
|
||||
if (!storeName || !actionName) throw new Error('tap: store and action are required');
|
||||
|
||||
// Build select chain for the captured response
|
||||
const selectChain = selectPath
|
||||
? selectPath.split('.').map((p: string) => `?.[${JSON.stringify(p)}]`).join('')
|
||||
: '';
|
||||
|
||||
// Serialize action arguments
|
||||
const actionArgsRendered = actionArgs.map((a: any) => {
|
||||
const rendered = render(a, { args, data });
|
||||
return JSON.stringify(rendered);
|
||||
});
|
||||
const actionCall = actionArgsRendered.length
|
||||
? `store[${JSON.stringify(actionName)}](${actionArgsRendered.join(', ')})`
|
||||
: `store[${JSON.stringify(actionName)}]()`;
|
||||
|
||||
const js = `
|
||||
async () => {
|
||||
// ── 1. Setup capture proxy (fetch + XHR dual interception) ──
|
||||
let captured = null;
|
||||
const capturePattern = ${JSON.stringify(capturePattern)};
|
||||
|
||||
// Intercept fetch API
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function(...fetchArgs) {
|
||||
const resp = await origFetch.apply(this, fetchArgs);
|
||||
try {
|
||||
const url = typeof fetchArgs[0] === 'string' ? fetchArgs[0]
|
||||
: fetchArgs[0] instanceof Request ? fetchArgs[0].url : String(fetchArgs[0]);
|
||||
if (capturePattern && url.includes(capturePattern) && !captured) {
|
||||
try { captured = await resp.clone().json(); } catch {}
|
||||
}
|
||||
} catch {}
|
||||
return resp;
|
||||
};
|
||||
|
||||
// Intercept XMLHttpRequest
|
||||
const origXhrOpen = XMLHttpRequest.prototype.open;
|
||||
const origXhrSend = XMLHttpRequest.prototype.send;
|
||||
XMLHttpRequest.prototype.open = function(method, url) {
|
||||
this.__tapUrl = String(url);
|
||||
return origXhrOpen.apply(this, arguments);
|
||||
};
|
||||
XMLHttpRequest.prototype.send = function(body) {
|
||||
if (capturePattern && this.__tapUrl?.includes(capturePattern)) {
|
||||
const xhr = this;
|
||||
const origHandler = xhr.onreadystatechange;
|
||||
xhr.onreadystatechange = function() {
|
||||
if (xhr.readyState === 4 && !captured) {
|
||||
try { captured = JSON.parse(xhr.responseText); } catch {}
|
||||
}
|
||||
if (origHandler) origHandler.apply(this, arguments);
|
||||
};
|
||||
// Also handle onload
|
||||
const origOnload = xhr.onload;
|
||||
xhr.onload = function() {
|
||||
if (!captured) { try { captured = JSON.parse(xhr.responseText); } catch {} }
|
||||
if (origOnload) origOnload.apply(this, arguments);
|
||||
};
|
||||
}
|
||||
return origXhrSend.apply(this, arguments);
|
||||
};
|
||||
|
||||
try {
|
||||
// ── 2. Find store ──
|
||||
let store = null;
|
||||
const storeName = ${JSON.stringify(storeName)};
|
||||
const fw = ${JSON.stringify(framework)};
|
||||
|
||||
// Auto-detect framework if not specified
|
||||
const app = document.querySelector('#app');
|
||||
if (!fw || fw === 'pinia') {
|
||||
// Try Pinia (Vue 3)
|
||||
try {
|
||||
const pinia = app?.__vue_app__?.config?.globalProperties?.$pinia;
|
||||
if (pinia?._s) store = pinia._s.get(storeName);
|
||||
} catch {}
|
||||
}
|
||||
if (!store && (!fw || fw === 'vuex')) {
|
||||
// Try Vuex (Vue 2/3)
|
||||
try {
|
||||
const vuexStore = app?.__vue_app__?.config?.globalProperties?.$store
|
||||
?? app?.__vue__?.$store;
|
||||
if (vuexStore) {
|
||||
// Vuex doesn't have named stores like Pinia, dispatch action
|
||||
store = { [${JSON.stringify(actionName)}]: (...a) => vuexStore.dispatch(storeName + '/' + ${JSON.stringify(actionName)}, ...a) };
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
|
||||
if (!store) return { error: 'Store not found: ' + storeName, hint: 'Page may not be fully loaded or store name may be incorrect' };
|
||||
if (typeof store[${JSON.stringify(actionName)}] !== 'function') {
|
||||
return { error: 'Action not found: ' + ${JSON.stringify(actionName)} + ' on store ' + storeName,
|
||||
hint: 'Available: ' + Object.keys(store).filter(k => typeof store[k] === 'function' && !k.startsWith('$') && !k.startsWith('_')).join(', ') };
|
||||
}
|
||||
|
||||
// ── 3. Call store action ──
|
||||
await ${actionCall};
|
||||
|
||||
// ── 4. Wait for network response ──
|
||||
const deadline = Date.now() + ${timeout} * 1000;
|
||||
while (!captured && Date.now() < deadline) {
|
||||
await new Promise(r => setTimeout(r, 200));
|
||||
}
|
||||
} finally {
|
||||
// ── 5. Always restore originals ──
|
||||
window.fetch = origFetch;
|
||||
XMLHttpRequest.prototype.open = origXhrOpen;
|
||||
XMLHttpRequest.prototype.send = origXhrSend;
|
||||
}
|
||||
|
||||
if (!captured) return { error: 'No matching response captured for pattern: ' + capturePattern };
|
||||
return captured${selectChain} ?? captured;
|
||||
}
|
||||
`;
|
||||
|
||||
return page.evaluate(js);
|
||||
}
|
||||
default: return data;
|
||||
}
|
||||
}
|
||||
|
||||
// Template engine: ${{ ... }}
|
||||
interface RenderContext { args?: Record<string, any>; data?: any; item?: any; index?: number; }
|
||||
|
||||
function render(template: any, ctx: RenderContext): any {
|
||||
if (typeof template !== 'string') return template;
|
||||
const fullMatch = template.match(/^\$\{\{\s*(.*?)\s*\}\}$/);
|
||||
if (fullMatch) return evalExpr(fullMatch[1].trim(), ctx);
|
||||
return template.replace(/\$\{\{\s*(.*?)\s*\}\}/g, (_m, expr) => String(evalExpr(expr.trim(), ctx)));
|
||||
}
|
||||
|
||||
function evalExpr(expr: string, ctx: RenderContext): any {
|
||||
const args = ctx.args ?? {};
|
||||
const item = ctx.item ?? {};
|
||||
const data = ctx.data;
|
||||
const index = ctx.index ?? 0;
|
||||
|
||||
// Default filter: args.limit | default(20)
|
||||
if (expr.includes('|') && expr.includes('default(')) {
|
||||
const [mainExpr, rest] = expr.split('|', 2);
|
||||
const defaultMatch = rest.match(/default\((.+?)\)/);
|
||||
const defaultVal = defaultMatch ? defaultMatch[1] : null;
|
||||
const result = resolvePath(mainExpr.trim(), { args, item, data, index });
|
||||
if (result === null || result === undefined) {
|
||||
if (defaultVal !== null) {
|
||||
const intVal = parseInt(defaultVal!, 10);
|
||||
if (!isNaN(intVal) && String(intVal) === defaultVal!.trim()) return intVal;
|
||||
return defaultVal!.replace(/^['"]|['"]$/g, '');
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
// Arithmetic: index + 1
|
||||
const arithMatch = expr.match(/^([\w][\w.]*)\s*([+\-*/])\s*(\d+)$/);
|
||||
if (arithMatch) {
|
||||
const [, varName, op, numStr] = arithMatch;
|
||||
const val = resolvePath(varName, { args, item, data, index });
|
||||
if (val !== null && val !== undefined) {
|
||||
const numVal = Number(val); const num = Number(numStr);
|
||||
if (!isNaN(numVal)) {
|
||||
switch (op) {
|
||||
case '+': return numVal + num; case '-': return numVal - num;
|
||||
case '*': return numVal * num; case '/': return num !== 0 ? numVal / num : 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// JS-like fallback expression: item.tweetCount || 'N/A'
|
||||
const orMatch = expr.match(/^(.+?)\s*\|\|\s*(.+)$/);
|
||||
if (orMatch) {
|
||||
const left = evalExpr(orMatch[1].trim(), ctx);
|
||||
if (left) return left;
|
||||
const right = orMatch[2].trim();
|
||||
return right.replace(/^['"]|['"]$/g, '');
|
||||
}
|
||||
|
||||
return resolvePath(expr, { args, item, data, index });
|
||||
}
|
||||
|
||||
function resolvePath(pathStr: string, ctx: RenderContext): any {
|
||||
const args = ctx.args ?? {};
|
||||
const item = ctx.item ?? {};
|
||||
const data = ctx.data;
|
||||
const index = ctx.index ?? 0;
|
||||
const parts = pathStr.split('.');
|
||||
const rootName = parts[0];
|
||||
let obj: any; let rest: string[];
|
||||
if (rootName === 'args') { obj = args; rest = parts.slice(1); }
|
||||
else if (rootName === 'item') { obj = item; rest = parts.slice(1); }
|
||||
else if (rootName === 'data') { obj = data; rest = parts.slice(1); }
|
||||
else if (rootName === 'index') return index;
|
||||
else { obj = item; rest = parts; }
|
||||
for (const part of rest) {
|
||||
if (obj && typeof obj === 'object' && !Array.isArray(obj)) obj = obj[part];
|
||||
else if (Array.isArray(obj) && /^\d+$/.test(part)) obj = obj[parseInt(part, 10)];
|
||||
else return null;
|
||||
}
|
||||
return obj;
|
||||
}
|
||||
export { executePipeline, type PipelineContext } from './pipeline/index.js';
|
||||
@@ -0,0 +1,98 @@
|
||||
/**
|
||||
* Pipeline executor: runs YAML pipeline steps sequentially.
|
||||
*/
|
||||
|
||||
import chalk from 'chalk';
|
||||
import type { IPage } from '../types.js';
|
||||
import { stepNavigate, stepClick, stepType, stepWait, stepPress, stepSnapshot, stepEvaluate } from './steps/browser.js';
|
||||
import { stepFetch } from './steps/fetch.js';
|
||||
import { stepSelect, stepMap, stepFilter, stepSort, stepLimit } from './steps/transform.js';
|
||||
import { stepIntercept } from './steps/intercept.js';
|
||||
import { stepTap } from './steps/tap.js';
|
||||
|
||||
export interface PipelineContext {
|
||||
args?: Record<string, any>;
|
||||
debug?: boolean;
|
||||
}
|
||||
|
||||
/** Step handler signature */
|
||||
type StepHandler = (page: IPage | null, params: any, data: any, args: Record<string, any>) => Promise<any>;
|
||||
|
||||
/** Registry of all available step handlers */
|
||||
const STEP_HANDLERS: Record<string, StepHandler> = {
|
||||
navigate: stepNavigate as StepHandler,
|
||||
fetch: stepFetch,
|
||||
select: stepSelect as StepHandler,
|
||||
evaluate: stepEvaluate as StepHandler,
|
||||
snapshot: stepSnapshot as StepHandler,
|
||||
click: stepClick as StepHandler,
|
||||
type: stepType as StepHandler,
|
||||
wait: stepWait as StepHandler,
|
||||
press: stepPress as StepHandler,
|
||||
map: stepMap as StepHandler,
|
||||
filter: stepFilter as StepHandler,
|
||||
sort: stepSort as StepHandler,
|
||||
limit: stepLimit as StepHandler,
|
||||
intercept: stepIntercept as StepHandler,
|
||||
tap: stepTap as StepHandler,
|
||||
};
|
||||
|
||||
export async function executePipeline(
|
||||
page: IPage | null,
|
||||
pipeline: any[],
|
||||
ctx: PipelineContext = {},
|
||||
): Promise<any> {
|
||||
const args = ctx.args ?? {};
|
||||
const debug = ctx.debug ?? false;
|
||||
let data: any = null;
|
||||
const total = pipeline.length;
|
||||
|
||||
for (let i = 0; i < pipeline.length; i++) {
|
||||
const step = pipeline[i];
|
||||
if (!step || typeof step !== 'object') continue;
|
||||
for (const [op, params] of Object.entries(step)) {
|
||||
if (debug) debugStepStart(i + 1, total, op, params);
|
||||
|
||||
const handler = STEP_HANDLERS[op];
|
||||
if (handler) {
|
||||
data = await handler(page, params, data, args);
|
||||
} else {
|
||||
if (debug) process.stderr.write(` ${chalk.yellow('⚠')} Unknown step: ${op}\n`);
|
||||
}
|
||||
|
||||
// Detect error objects returned by steps (e.g. tap store not found)
|
||||
if (data && typeof data === 'object' && !Array.isArray(data) && data.error) {
|
||||
process.stderr.write(` ${chalk.yellow('⚠')} ${chalk.yellow(op)}: ${data.error}\n`);
|
||||
if (data.hint) process.stderr.write(` ${chalk.dim('💡')} ${chalk.dim(data.hint)}\n`);
|
||||
}
|
||||
if (debug) debugStepResult(op, data);
|
||||
}
|
||||
}
|
||||
return data;
|
||||
}
|
||||
|
||||
function debugStepStart(stepNum: number, total: number, op: string, params: any): void {
|
||||
let preview = '';
|
||||
if (typeof params === 'string') {
|
||||
preview = params.length <= 80 ? ` → ${params}` : ` → ${params.slice(0, 77)}...`;
|
||||
} else if (params && typeof params === 'object' && !Array.isArray(params)) {
|
||||
preview = ` (${Object.keys(params).join(', ')})`;
|
||||
}
|
||||
process.stderr.write(` ${chalk.dim(`[${stepNum}/${total}]`)} ${chalk.bold.cyan(op)}${preview}\n`);
|
||||
}
|
||||
|
||||
function debugStepResult(op: string, data: any): void {
|
||||
if (data === null || data === undefined) {
|
||||
process.stderr.write(` ${chalk.dim('→ (no data)')}\n`);
|
||||
} else if (Array.isArray(data)) {
|
||||
process.stderr.write(` ${chalk.dim(`→ ${data.length} items`)}\n`);
|
||||
} else if (typeof data === 'object') {
|
||||
const keys = Object.keys(data).slice(0, 5);
|
||||
process.stderr.write(` ${chalk.dim(`→ dict (${keys.join(', ')}${Object.keys(data).length > 5 ? '...' : ''})`)}\n`);
|
||||
} else if (typeof data === 'string') {
|
||||
const p = data.slice(0, 60).replace(/\n/g, '\\n');
|
||||
process.stderr.write(` ${chalk.dim(`→ "${p}${data.length > 60 ? '...' : ''}"`)}\n`);
|
||||
} else {
|
||||
process.stderr.write(` ${chalk.dim(`→ ${typeof data}`)}\n`);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
/**
|
||||
* Pipeline module: re-exports for backward compatibility.
|
||||
*/
|
||||
|
||||
export { executePipeline, type PipelineContext } from './executor.js';
|
||||
export { render, evalExpr, resolvePath, normalizeEvaluateSource, type RenderContext } from './template.js';
|
||||
@@ -0,0 +1,67 @@
|
||||
/**
|
||||
* Pipeline step: navigate, click, type, wait, press, snapshot.
|
||||
* Browser interaction primitives.
|
||||
*/
|
||||
|
||||
import type { IPage } from '../../types.js';
|
||||
import { render, normalizeEvaluateSource } from '../template.js';
|
||||
|
||||
export async function stepNavigate(page: IPage, params: any, data: any, args: Record<string, any>): Promise<any> {
|
||||
const url = render(params, { args, data });
|
||||
await page.goto(String(url));
|
||||
return data;
|
||||
}
|
||||
|
||||
export async function stepClick(page: IPage, params: any, data: any, args: Record<string, any>): Promise<any> {
|
||||
await page.click(String(render(params, { args, data })).replace(/^@/, ''));
|
||||
return data;
|
||||
}
|
||||
|
||||
export async function stepType(page: IPage, params: any, data: any, args: Record<string, any>): Promise<any> {
|
||||
if (typeof params === 'object' && params) {
|
||||
const ref = String(render(params.ref ?? '', { args, data })).replace(/^@/, '');
|
||||
const text = String(render(params.text ?? '', { args, data }));
|
||||
await page.typeText(ref, text);
|
||||
if (params.submit) await page.pressKey('Enter');
|
||||
}
|
||||
return data;
|
||||
}
|
||||
|
||||
export async function stepWait(page: IPage, params: any, data: any, args: Record<string, any>): Promise<any> {
|
||||
if (typeof params === 'number') await page.wait(params);
|
||||
else if (typeof params === 'object' && params) {
|
||||
if ('text' in params) {
|
||||
const timeout = params.timeout ?? 10;
|
||||
const start = Date.now();
|
||||
while ((Date.now() - start) / 1000 < timeout) {
|
||||
const snap = await page.snapshot({ raw: true });
|
||||
if (typeof snap === 'string' && snap.includes(params.text)) break;
|
||||
await page.wait(0.5);
|
||||
}
|
||||
} else if ('time' in params) await page.wait(Number(params.time));
|
||||
} else if (typeof params === 'string') await page.wait(Number(render(params, { args, data })));
|
||||
return data;
|
||||
}
|
||||
|
||||
export async function stepPress(page: IPage, params: any, data: any, args: Record<string, any>): Promise<any> {
|
||||
await page.pressKey(String(render(params, { args, data })));
|
||||
return data;
|
||||
}
|
||||
|
||||
export async function stepSnapshot(page: IPage, params: any, _data: any, _args: Record<string, any>): Promise<any> {
|
||||
const opts = (typeof params === 'object' && params) ? params : {};
|
||||
return page.snapshot({ interactive: opts.interactive ?? false, compact: opts.compact ?? false, maxDepth: opts.max_depth, raw: opts.raw ?? false });
|
||||
}
|
||||
|
||||
export async function stepEvaluate(page: IPage, params: any, data: any, args: Record<string, any>): Promise<any> {
|
||||
const js = String(render(params, { args, data }));
|
||||
let result = await page.evaluate(normalizeEvaluateSource(js));
|
||||
// MCP may return JSON as a string — auto-parse it
|
||||
if (typeof result === 'string') {
|
||||
const trimmed = result.trim();
|
||||
if ((trimmed.startsWith('[') && trimmed.endsWith(']')) || (trimmed.startsWith('{') && trimmed.endsWith('}'))) {
|
||||
try { result = JSON.parse(trimmed); } catch {}
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
/**
|
||||
* Pipeline step: fetch — HTTP API requests.
|
||||
*/
|
||||
|
||||
import type { IPage } from '../../types.js';
|
||||
import { render } from '../template.js';
|
||||
|
||||
/** Single URL fetch helper */
|
||||
async function fetchSingle(
|
||||
page: IPage | null, url: string, method: string,
|
||||
queryParams: Record<string, any>, headers: Record<string, any>,
|
||||
args: Record<string, any>, data: any,
|
||||
): Promise<any> {
|
||||
const renderedParams: Record<string, string> = {};
|
||||
for (const [k, v] of Object.entries(queryParams)) renderedParams[k] = String(render(v, { args, data }));
|
||||
const renderedHeaders: Record<string, string> = {};
|
||||
for (const [k, v] of Object.entries(headers)) renderedHeaders[k] = String(render(v, { args, data }));
|
||||
|
||||
let finalUrl = url;
|
||||
if (Object.keys(renderedParams).length > 0) {
|
||||
const qs = new URLSearchParams(renderedParams).toString();
|
||||
finalUrl = `${finalUrl}${finalUrl.includes('?') ? '&' : '?'}${qs}`;
|
||||
}
|
||||
|
||||
if (page === null) {
|
||||
const resp = await fetch(finalUrl, { method: method.toUpperCase(), headers: renderedHeaders });
|
||||
return resp.json();
|
||||
}
|
||||
|
||||
const headersJs = JSON.stringify(renderedHeaders);
|
||||
const escapedUrl = finalUrl.replace(/"/g, '\\"');
|
||||
return page.evaluate(`
|
||||
async () => {
|
||||
const resp = await fetch("${escapedUrl}", {
|
||||
method: "${method}", headers: ${headersJs}, credentials: "include"
|
||||
});
|
||||
return await resp.json();
|
||||
}
|
||||
`);
|
||||
}
|
||||
|
||||
export async function stepFetch(page: IPage | null, params: any, data: any, args: Record<string, any>): Promise<any> {
|
||||
const urlOrObj = typeof params === 'string' ? params : (params?.url ?? '');
|
||||
const method = params?.method ?? 'GET';
|
||||
const queryParams: Record<string, any> = params?.params ?? {};
|
||||
const headers: Record<string, any> = params?.headers ?? {};
|
||||
const urlTemplate = String(urlOrObj);
|
||||
|
||||
// Per-item fetch when data is array and URL references item
|
||||
if (Array.isArray(data) && urlTemplate.includes('item')) {
|
||||
const results: any[] = [];
|
||||
for (let i = 0; i < data.length; i++) {
|
||||
const itemUrl = String(render(urlTemplate, { args, data, item: data[i], index: i }));
|
||||
results.push(await fetchSingle(page, itemUrl, method, queryParams, headers, args, data));
|
||||
}
|
||||
return results;
|
||||
}
|
||||
const url = render(urlOrObj, { args, data });
|
||||
return fetchSingle(page, String(url), method, queryParams, headers, args, data);
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
/**
|
||||
* Pipeline step: intercept — declarative XHR interception.
|
||||
*/
|
||||
|
||||
import type { IPage } from '../../types.js';
|
||||
import { render } from '../template.js';
|
||||
|
||||
export async function stepIntercept(page: IPage, params: any, data: any, args: Record<string, any>): Promise<any> {
|
||||
const cfg = typeof params === 'object' ? params : {};
|
||||
const trigger = cfg.trigger ?? '';
|
||||
const capturePattern = cfg.capture ?? '';
|
||||
const timeout = cfg.timeout ?? 8;
|
||||
const selectPath = cfg.select ?? null;
|
||||
|
||||
if (!capturePattern) return data;
|
||||
|
||||
// Step 1: Execute the trigger action
|
||||
if (trigger.startsWith('navigate:')) {
|
||||
const url = render(trigger.slice('navigate:'.length), { args, data });
|
||||
await page.goto(String(url));
|
||||
} else if (trigger.startsWith('evaluate:')) {
|
||||
const js = trigger.slice('evaluate:'.length);
|
||||
const { normalizeEvaluateSource } = await import('../template.js');
|
||||
await page.evaluate(normalizeEvaluateSource(render(js, { args, data }) as string));
|
||||
} else if (trigger.startsWith('click:')) {
|
||||
const ref = render(trigger.slice('click:'.length), { args, data });
|
||||
await page.click(String(ref).replace(/^@/, ''));
|
||||
} else if (trigger === 'scroll') {
|
||||
await page.scroll('down');
|
||||
}
|
||||
|
||||
// Step 2: Wait a bit for network requests to fire
|
||||
await page.wait(Math.min(timeout, 3));
|
||||
|
||||
// Step 3: Get network requests and find matching ones
|
||||
const rawNetwork = await page.networkRequests(false);
|
||||
const matchingResponses: any[] = [];
|
||||
|
||||
if (typeof rawNetwork === 'string') {
|
||||
const lines = rawNetwork.split('\n');
|
||||
for (const line of lines) {
|
||||
const match = line.match(/\[?(GET|POST)\]?\s+(\S+)\s*(?:=>|→)\s*\[?(\d+)\]?/i);
|
||||
if (match) {
|
||||
const [, , url, status] = match;
|
||||
if (url.includes(capturePattern) && status === '200') {
|
||||
try {
|
||||
const body = await page.evaluate(`
|
||||
async () => {
|
||||
try {
|
||||
const resp = await fetch(${JSON.stringify(url)}, { credentials: 'include' });
|
||||
if (!resp.ok) return null;
|
||||
return await resp.json();
|
||||
} catch { return null; }
|
||||
}
|
||||
`);
|
||||
if (body) matchingResponses.push(body);
|
||||
} catch {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Step 4: Select from response if specified
|
||||
let result = matchingResponses.length === 1 ? matchingResponses[0] :
|
||||
matchingResponses.length > 1 ? matchingResponses : data;
|
||||
|
||||
if (selectPath && result) {
|
||||
let current = result;
|
||||
for (const part of String(selectPath).split('.')) {
|
||||
if (current && typeof current === 'object' && !Array.isArray(current)) {
|
||||
current = current[part];
|
||||
} else break;
|
||||
}
|
||||
result = current ?? result;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
/**
|
||||
* Pipeline step: tap — declarative Store Action Bridge.
|
||||
*
|
||||
* Generates a self-contained IIFE that:
|
||||
* 1. Injects fetch + XHR dual interception proxy
|
||||
* 2. Finds the Pinia/Vuex store and calls the action
|
||||
* 3. Captures the response matching the URL pattern
|
||||
* 4. Auto-cleans up interception in finally block
|
||||
* 5. Returns the captured data (optionally sub-selected)
|
||||
*/
|
||||
|
||||
import type { IPage } from '../../types.js';
|
||||
import { render } from '../template.js';
|
||||
|
||||
export async function stepTap(page: IPage, params: any, data: any, args: Record<string, any>): Promise<any> {
|
||||
const cfg = typeof params === 'object' ? params : {};
|
||||
const storeName = String(render(cfg.store ?? '', { args, data }));
|
||||
const actionName = String(render(cfg.action ?? '', { args, data }));
|
||||
const capturePattern = String(render(cfg.capture ?? '', { args, data }));
|
||||
const timeout = cfg.timeout ?? 5;
|
||||
const selectPath = cfg.select ? String(render(cfg.select, { args, data })) : null;
|
||||
const framework = cfg.framework ?? null;
|
||||
const actionArgs = cfg.args ?? [];
|
||||
|
||||
if (!storeName || !actionName) throw new Error('tap: store and action are required');
|
||||
|
||||
// Build select chain for the captured response
|
||||
const selectChain = selectPath
|
||||
? selectPath.split('.').map((p: string) => `?.[${JSON.stringify(p)}]`).join('')
|
||||
: '';
|
||||
|
||||
// Serialize action arguments
|
||||
const actionArgsRendered = actionArgs.map((a: any) => {
|
||||
const rendered = render(a, { args, data });
|
||||
return JSON.stringify(rendered);
|
||||
});
|
||||
const actionCall = actionArgsRendered.length
|
||||
? `store[${JSON.stringify(actionName)}](${actionArgsRendered.join(', ')})`
|
||||
: `store[${JSON.stringify(actionName)}]()`;
|
||||
|
||||
const js = `
|
||||
async () => {
|
||||
// ── 1. Setup capture proxy (fetch + XHR dual interception) ──
|
||||
let captured = null;
|
||||
const capturePattern = ${JSON.stringify(capturePattern)};
|
||||
|
||||
// Intercept fetch API
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function(...fetchArgs) {
|
||||
const resp = await origFetch.apply(this, fetchArgs);
|
||||
try {
|
||||
const url = typeof fetchArgs[0] === 'string' ? fetchArgs[0]
|
||||
: fetchArgs[0] instanceof Request ? fetchArgs[0].url : String(fetchArgs[0]);
|
||||
if (capturePattern && url.includes(capturePattern) && !captured) {
|
||||
try { captured = await resp.clone().json(); } catch {}
|
||||
}
|
||||
} catch {}
|
||||
return resp;
|
||||
};
|
||||
|
||||
// Intercept XMLHttpRequest
|
||||
const origXhrOpen = XMLHttpRequest.prototype.open;
|
||||
const origXhrSend = XMLHttpRequest.prototype.send;
|
||||
XMLHttpRequest.prototype.open = function(method, url) {
|
||||
this.__tapUrl = String(url);
|
||||
return origXhrOpen.apply(this, arguments);
|
||||
};
|
||||
XMLHttpRequest.prototype.send = function(body) {
|
||||
if (capturePattern && this.__tapUrl?.includes(capturePattern)) {
|
||||
const xhr = this;
|
||||
const origHandler = xhr.onreadystatechange;
|
||||
xhr.onreadystatechange = function() {
|
||||
if (xhr.readyState === 4 && !captured) {
|
||||
try { captured = JSON.parse(xhr.responseText); } catch {}
|
||||
}
|
||||
if (origHandler) origHandler.apply(this, arguments);
|
||||
};
|
||||
const origOnload = xhr.onload;
|
||||
xhr.onload = function() {
|
||||
if (!captured) { try { captured = JSON.parse(xhr.responseText); } catch {} }
|
||||
if (origOnload) origOnload.apply(this, arguments);
|
||||
};
|
||||
}
|
||||
return origXhrSend.apply(this, arguments);
|
||||
};
|
||||
|
||||
try {
|
||||
// ── 2. Find store ──
|
||||
let store = null;
|
||||
const storeName = ${JSON.stringify(storeName)};
|
||||
const fw = ${JSON.stringify(framework)};
|
||||
|
||||
const app = document.querySelector('#app');
|
||||
if (!fw || fw === 'pinia') {
|
||||
try {
|
||||
const pinia = app?.__vue_app__?.config?.globalProperties?.$pinia;
|
||||
if (pinia?._s) store = pinia._s.get(storeName);
|
||||
} catch {}
|
||||
}
|
||||
if (!store && (!fw || fw === 'vuex')) {
|
||||
try {
|
||||
const vuexStore = app?.__vue_app__?.config?.globalProperties?.$store
|
||||
?? app?.__vue__?.$store;
|
||||
if (vuexStore) {
|
||||
store = { [${JSON.stringify(actionName)}]: (...a) => vuexStore.dispatch(storeName + '/' + ${JSON.stringify(actionName)}, ...a) };
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
|
||||
if (!store) return { error: 'Store not found: ' + storeName, hint: 'Page may not be fully loaded or store name may be incorrect' };
|
||||
if (typeof store[${JSON.stringify(actionName)}] !== 'function') {
|
||||
return { error: 'Action not found: ' + ${JSON.stringify(actionName)} + ' on store ' + storeName,
|
||||
hint: 'Available: ' + Object.keys(store).filter(k => typeof store[k] === 'function' && !k.startsWith('$') && !k.startsWith('_')).join(', ') };
|
||||
}
|
||||
|
||||
// ── 3. Call store action ──
|
||||
await ${actionCall};
|
||||
|
||||
// ── 4. Wait for network response ──
|
||||
const deadline = Date.now() + ${timeout} * 1000;
|
||||
while (!captured && Date.now() < deadline) {
|
||||
await new Promise(r => setTimeout(r, 200));
|
||||
}
|
||||
} finally {
|
||||
// ── 5. Always restore originals ──
|
||||
window.fetch = origFetch;
|
||||
XMLHttpRequest.prototype.open = origXhrOpen;
|
||||
XMLHttpRequest.prototype.send = origXhrSend;
|
||||
}
|
||||
|
||||
if (!captured) return { error: 'No matching response captured for pattern: ' + capturePattern };
|
||||
return captured${selectChain} ?? captured;
|
||||
}
|
||||
`;
|
||||
|
||||
return page.evaluate(js);
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
/**
|
||||
* Pipeline steps: data transforms — select, map, filter, sort, limit.
|
||||
*/
|
||||
|
||||
import { render, evalExpr } from '../template.js';
|
||||
|
||||
export async function stepSelect(_page: any, params: any, data: any, args: Record<string, any>): Promise<any> {
|
||||
const pathStr = String(render(params, { args, data }));
|
||||
if (data && typeof data === 'object') {
|
||||
let current = data;
|
||||
for (const part of pathStr.split('.')) {
|
||||
if (current && typeof current === 'object' && !Array.isArray(current)) current = (current as any)[part];
|
||||
else if (Array.isArray(current) && /^\d+$/.test(part)) current = current[parseInt(part, 10)];
|
||||
else return null;
|
||||
}
|
||||
return current;
|
||||
}
|
||||
return data;
|
||||
}
|
||||
|
||||
export async function stepMap(_page: any, params: any, data: any, args: Record<string, any>): Promise<any> {
|
||||
if (!data || typeof data !== 'object') return data;
|
||||
let items: any[] = Array.isArray(data) ? data : [data];
|
||||
if (!Array.isArray(data) && typeof data === 'object' && 'data' in data) items = data.data;
|
||||
const result: any[] = [];
|
||||
for (let i = 0; i < items.length; i++) {
|
||||
const item = items[i];
|
||||
const row: Record<string, any> = {};
|
||||
for (const [key, template] of Object.entries(params)) row[key] = render(template, { args, data, item, index: i });
|
||||
result.push(row);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
export async function stepFilter(_page: any, params: any, data: any, args: Record<string, any>): Promise<any> {
|
||||
if (!Array.isArray(data)) return data;
|
||||
return data.filter((item, i) => evalExpr(String(params), { args, item, index: i }));
|
||||
}
|
||||
|
||||
export async function stepSort(_page: any, params: any, data: any, _args: Record<string, any>): Promise<any> {
|
||||
if (!Array.isArray(data)) return data;
|
||||
const key = typeof params === 'object' ? (params.by ?? '') : String(params);
|
||||
const reverse = typeof params === 'object' ? params.order === 'desc' : false;
|
||||
return [...data].sort((a, b) => { const va = a[key] ?? ''; const vb = b[key] ?? ''; const cmp = va < vb ? -1 : va > vb ? 1 : 0; return reverse ? -cmp : cmp; });
|
||||
}
|
||||
|
||||
export async function stepLimit(_page: any, params: any, data: any, args: Record<string, any>): Promise<any> {
|
||||
if (!Array.isArray(data)) return data;
|
||||
return data.slice(0, Number(render(params, { args, data })));
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
/**
|
||||
* Tests for the pipeline template engine: render, evalExpr, resolvePath.
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { render, evalExpr, resolvePath, normalizeEvaluateSource } from './template.js';
|
||||
|
||||
describe('resolvePath', () => {
|
||||
it('resolves args path', () => {
|
||||
expect(resolvePath('args.limit', { args: { limit: 20 } })).toBe(20);
|
||||
});
|
||||
it('resolves nested args path', () => {
|
||||
expect(resolvePath('args.query.keyword', { args: { query: { keyword: 'test' } } })).toBe('test');
|
||||
});
|
||||
it('resolves item path', () => {
|
||||
expect(resolvePath('item.title', { item: { title: 'Hello' } })).toBe('Hello');
|
||||
});
|
||||
it('resolves implicit item path (no prefix)', () => {
|
||||
expect(resolvePath('title', { item: { title: 'World' } })).toBe('World');
|
||||
});
|
||||
it('resolves index', () => {
|
||||
expect(resolvePath('index', { index: 5 })).toBe(5);
|
||||
});
|
||||
it('resolves data path', () => {
|
||||
expect(resolvePath('data.items', { data: { items: [1, 2, 3] } })).toEqual([1, 2, 3]);
|
||||
});
|
||||
it('returns null for missing path', () => {
|
||||
expect(resolvePath('args.missing', { args: {} })).toBeNull();
|
||||
});
|
||||
it('resolves array index', () => {
|
||||
expect(resolvePath('data.0', { data: ['a', 'b'] })).toBe('a');
|
||||
});
|
||||
});
|
||||
|
||||
describe('evalExpr', () => {
|
||||
it('evaluates default filter', () => {
|
||||
expect(evalExpr('args.limit | default(20)', { args: {} })).toBe(20);
|
||||
});
|
||||
it('uses actual value over default', () => {
|
||||
expect(evalExpr('args.limit | default(20)', { args: { limit: 10 } })).toBe(10);
|
||||
});
|
||||
it('evaluates string default', () => {
|
||||
expect(evalExpr("args.name | default('unknown')", { args: {} })).toBe('unknown');
|
||||
});
|
||||
it('evaluates arithmetic: index + 1', () => {
|
||||
expect(evalExpr('index + 1', { index: 0 })).toBe(1);
|
||||
});
|
||||
it('evaluates arithmetic: index * 2', () => {
|
||||
expect(evalExpr('index * 2', { index: 5 })).toBe(10);
|
||||
});
|
||||
it('evaluates || fallback', () => {
|
||||
expect(evalExpr("item.name || 'N/A'", { item: {} })).toBe('N/A');
|
||||
});
|
||||
it('evaluates || with truthy left', () => {
|
||||
expect(evalExpr("item.name || 'N/A'", { item: { name: 'Alice' } })).toBe('Alice');
|
||||
});
|
||||
it('resolves simple path', () => {
|
||||
expect(evalExpr('item.title', { item: { title: 'Test' } })).toBe('Test');
|
||||
});
|
||||
});
|
||||
|
||||
describe('render', () => {
|
||||
it('renders full expression', () => {
|
||||
expect(render('${{ args.limit }}', { args: { limit: 30 } })).toBe(30);
|
||||
});
|
||||
it('renders inline expression in string', () => {
|
||||
expect(render('Hello ${{ item.name }}!', { item: { name: 'World' } })).toBe('Hello World!');
|
||||
});
|
||||
it('renders multiple inline expressions', () => {
|
||||
expect(render('${{ item.a }}-${{ item.b }}', { item: { a: 1, b: 2 } })).toBe('1-2');
|
||||
});
|
||||
it('returns non-string values as-is', () => {
|
||||
expect(render(42, {})).toBe(42);
|
||||
expect(render(null, {})).toBeNull();
|
||||
expect(render(undefined, {})).toBeUndefined();
|
||||
});
|
||||
it('returns full expression result as native type', () => {
|
||||
expect(render('${{ args.list }}', { args: { list: [1, 2, 3] } })).toEqual([1, 2, 3]);
|
||||
});
|
||||
it('renders URL template', () => {
|
||||
expect(render('https://api.example.com/search?q=${{ args.keyword }}', { args: { keyword: 'test' } })).toBe('https://api.example.com/search?q=test');
|
||||
});
|
||||
});
|
||||
|
||||
describe('normalizeEvaluateSource', () => {
|
||||
it('wraps bare expression', () => {
|
||||
expect(normalizeEvaluateSource('document.title')).toBe('() => (document.title)');
|
||||
});
|
||||
it('passes through arrow function', () => {
|
||||
expect(normalizeEvaluateSource('() => 42')).toBe('() => 42');
|
||||
});
|
||||
it('passes through async arrow function', () => {
|
||||
const src = 'async () => { return 1; }';
|
||||
expect(normalizeEvaluateSource(src)).toBe(src);
|
||||
});
|
||||
it('passes through named function', () => {
|
||||
const src = 'function foo() { return 1; }';
|
||||
expect(normalizeEvaluateSource(src)).toBe(src);
|
||||
});
|
||||
it('wraps IIFE pattern', () => {
|
||||
const src = '(async () => { return 1; })()';
|
||||
expect(normalizeEvaluateSource(src)).toBe(`() => (${src})`);
|
||||
});
|
||||
it('handles empty string', () => {
|
||||
expect(normalizeEvaluateSource('')).toBe('() => undefined');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,101 @@
|
||||
/**
|
||||
* Pipeline template engine: ${{ ... }} expression rendering.
|
||||
*/
|
||||
|
||||
export interface RenderContext {
|
||||
args?: Record<string, any>;
|
||||
data?: any;
|
||||
item?: any;
|
||||
index?: number;
|
||||
}
|
||||
|
||||
export function render(template: any, ctx: RenderContext): any {
|
||||
if (typeof template !== 'string') return template;
|
||||
const fullMatch = template.match(/^\$\{\{\s*(.*?)\s*\}\}$/);
|
||||
if (fullMatch) return evalExpr(fullMatch[1].trim(), ctx);
|
||||
return template.replace(/\$\{\{\s*(.*?)\s*\}\}/g, (_m, expr) => String(evalExpr(expr.trim(), ctx)));
|
||||
}
|
||||
|
||||
export function evalExpr(expr: string, ctx: RenderContext): any {
|
||||
const args = ctx.args ?? {};
|
||||
const item = ctx.item ?? {};
|
||||
const data = ctx.data;
|
||||
const index = ctx.index ?? 0;
|
||||
|
||||
// Default filter: args.limit | default(20)
|
||||
if (expr.includes('|') && expr.includes('default(')) {
|
||||
const [mainExpr, rest] = expr.split('|', 2);
|
||||
const defaultMatch = rest.match(/default\((.+?)\)/);
|
||||
const defaultVal = defaultMatch ? defaultMatch[1] : null;
|
||||
const result = resolvePath(mainExpr.trim(), { args, item, data, index });
|
||||
if (result === null || result === undefined) {
|
||||
if (defaultVal !== null) {
|
||||
const intVal = parseInt(defaultVal!, 10);
|
||||
if (!isNaN(intVal) && String(intVal) === defaultVal!.trim()) return intVal;
|
||||
return defaultVal!.replace(/^['"]|['"]$/g, '');
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
// Arithmetic: index + 1
|
||||
const arithMatch = expr.match(/^([\w][\w.]*)\s*([+\-*/])\s*(\d+)$/);
|
||||
if (arithMatch) {
|
||||
const [, varName, op, numStr] = arithMatch;
|
||||
const val = resolvePath(varName, { args, item, data, index });
|
||||
if (val !== null && val !== undefined) {
|
||||
const numVal = Number(val); const num = Number(numStr);
|
||||
if (!isNaN(numVal)) {
|
||||
switch (op) {
|
||||
case '+': return numVal + num; case '-': return numVal - num;
|
||||
case '*': return numVal * num; case '/': return num !== 0 ? numVal / num : 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// JS-like fallback expression: item.tweetCount || 'N/A'
|
||||
const orMatch = expr.match(/^(.+?)\s*\|\|\s*(.+)$/);
|
||||
if (orMatch) {
|
||||
const left = evalExpr(orMatch[1].trim(), ctx);
|
||||
if (left) return left;
|
||||
const right = orMatch[2].trim();
|
||||
return right.replace(/^['"]|['"]$/g, '');
|
||||
}
|
||||
|
||||
return resolvePath(expr, { args, item, data, index });
|
||||
}
|
||||
|
||||
export function resolvePath(pathStr: string, ctx: RenderContext): any {
|
||||
const args = ctx.args ?? {};
|
||||
const item = ctx.item ?? {};
|
||||
const data = ctx.data;
|
||||
const index = ctx.index ?? 0;
|
||||
const parts = pathStr.split('.');
|
||||
const rootName = parts[0];
|
||||
let obj: any; let rest: string[];
|
||||
if (rootName === 'args') { obj = args; rest = parts.slice(1); }
|
||||
else if (rootName === 'item') { obj = item; rest = parts.slice(1); }
|
||||
else if (rootName === 'data') { obj = data; rest = parts.slice(1); }
|
||||
else if (rootName === 'index') return index;
|
||||
else { obj = item; rest = parts; }
|
||||
for (const part of rest) {
|
||||
if (obj && typeof obj === 'object' && !Array.isArray(obj)) obj = obj[part];
|
||||
else if (Array.isArray(obj) && /^\d+$/.test(part)) obj = obj[parseInt(part, 10)];
|
||||
else return null;
|
||||
}
|
||||
return obj;
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize JavaScript source for browser evaluate() calls.
|
||||
*/
|
||||
export function normalizeEvaluateSource(source: string): string {
|
||||
const stripped = source.trim();
|
||||
if (!stripped) return '() => undefined';
|
||||
if (stripped.startsWith('(') && stripped.endsWith(')()')) return `() => (${stripped})`;
|
||||
if (/^(async\s+)?\([^)]*\)\s*=>/.test(stripped)) return stripped;
|
||||
if (/^(async\s+)?[A-Za-z_][A-Za-z0-9_]*\s*=>/.test(stripped)) return stripped;
|
||||
if (stripped.startsWith('function ') || stripped.startsWith('async function ')) return stripped;
|
||||
return `() => (${stripped})`;
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
/**
|
||||
* Tests for pipeline transform steps: select, map, filter, sort, limit.
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { stepSelect, stepMap, stepFilter, stepSort, stepLimit } from './steps/transform.js';
|
||||
|
||||
const SAMPLE_DATA = [
|
||||
{ title: 'Alpha', score: 10, author: 'Alice' },
|
||||
{ title: 'Beta', score: 30, author: 'Bob' },
|
||||
{ title: 'Gamma', score: 20, author: 'Charlie' },
|
||||
];
|
||||
|
||||
describe('stepSelect', () => {
|
||||
it('selects nested path', async () => {
|
||||
const data = { result: { items: [1, 2, 3] } };
|
||||
const result = await stepSelect(null, 'result.items', data, {});
|
||||
expect(result).toEqual([1, 2, 3]);
|
||||
});
|
||||
|
||||
it('selects array by index', async () => {
|
||||
const data = { list: ['a', 'b', 'c'] };
|
||||
const result = await stepSelect(null, 'list.1', data, {});
|
||||
expect(result).toBe('b');
|
||||
});
|
||||
|
||||
it('returns null for missing path', async () => {
|
||||
const result = await stepSelect(null, 'missing.path', { foo: 1 }, {});
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it('returns data as-is for non-object', async () => {
|
||||
const result = await stepSelect(null, 'foo', 'string-data', {});
|
||||
expect(result).toBe('string-data');
|
||||
});
|
||||
});
|
||||
|
||||
describe('stepMap', () => {
|
||||
it('maps array items', async () => {
|
||||
const result = await stepMap(null, {
|
||||
name: '${{ item.title }}',
|
||||
rank: '${{ index + 1 }}',
|
||||
}, SAMPLE_DATA, {});
|
||||
expect(result).toEqual([
|
||||
{ name: 'Alpha', rank: 1 },
|
||||
{ name: 'Beta', rank: 2 },
|
||||
{ name: 'Gamma', rank: 3 },
|
||||
]);
|
||||
});
|
||||
|
||||
it('handles single object', async () => {
|
||||
const result = await stepMap(null, {
|
||||
name: '${{ item.title }}',
|
||||
}, { title: 'Solo' }, {});
|
||||
expect(result).toEqual([{ name: 'Solo' }]);
|
||||
});
|
||||
|
||||
it('returns null/undefined as-is', async () => {
|
||||
expect(await stepMap(null, { x: '${{ item.x }}' }, null, {})).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('stepFilter', () => {
|
||||
it('filters by expression', async () => {
|
||||
const result = await stepFilter(null, 'item.score', SAMPLE_DATA, {});
|
||||
expect(result).toHaveLength(3); // all truthy
|
||||
});
|
||||
|
||||
it('returns non-array as-is', async () => {
|
||||
const result = await stepFilter(null, 'item.x', 'not-array', {});
|
||||
expect(result).toBe('not-array');
|
||||
});
|
||||
});
|
||||
|
||||
describe('stepSort', () => {
|
||||
it('sorts ascending by key', async () => {
|
||||
const result = await stepSort(null, 'score', SAMPLE_DATA, {});
|
||||
expect(result.map((r: any) => r.title)).toEqual(['Alpha', 'Gamma', 'Beta']);
|
||||
});
|
||||
|
||||
it('sorts descending', async () => {
|
||||
const result = await stepSort(null, { by: 'score', order: 'desc' }, SAMPLE_DATA, {});
|
||||
expect(result.map((r: any) => r.title)).toEqual(['Beta', 'Gamma', 'Alpha']);
|
||||
});
|
||||
|
||||
it('does not mutate original', async () => {
|
||||
const original = [...SAMPLE_DATA];
|
||||
await stepSort(null, 'score', SAMPLE_DATA, {});
|
||||
expect(SAMPLE_DATA).toEqual(original);
|
||||
});
|
||||
});
|
||||
|
||||
describe('stepLimit', () => {
|
||||
it('limits array to N items', async () => {
|
||||
const result = await stepLimit(null, '2', SAMPLE_DATA, {});
|
||||
expect(result).toHaveLength(2);
|
||||
});
|
||||
|
||||
it('limits using template expression', async () => {
|
||||
const result = await stepLimit(null, '${{ args.limit }}', SAMPLE_DATA, { limit: 1 });
|
||||
expect(result).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('returns non-array as-is', async () => {
|
||||
expect(await stepLimit(null, '5', 'string', {})).toBe('string');
|
||||
});
|
||||
});
|
||||
@@ -1,3 +0,0 @@
|
||||
/** Promote verified drafts. */
|
||||
import { registerCandidates } from './register.js';
|
||||
export function promoteCandidate(opts: any): any { return registerCandidates(opts); }
|
||||
@@ -1,2 +0,0 @@
|
||||
/** Register candidates, promote, scaffold, generate — stubs with exports. */
|
||||
export function registerCandidates(opts: any): any { return { ok: true, count: 0 }; }
|
||||
+4
-2
@@ -2,6 +2,8 @@
|
||||
* Core registry: Strategy enum, Arg/CliCommand interfaces, cli() registration.
|
||||
*/
|
||||
|
||||
import type { IPage } from './types.js';
|
||||
|
||||
export enum Strategy {
|
||||
PUBLIC = 'public',
|
||||
COOKIE = 'cookie',
|
||||
@@ -28,7 +30,7 @@ export interface CliCommand {
|
||||
browser?: boolean;
|
||||
args: Arg[];
|
||||
columns?: string[];
|
||||
func?: (page: any, kwargs: Record<string, any>, debug?: boolean) => Promise<any>;
|
||||
func?: (page: IPage | null, kwargs: Record<string, any>, debug?: boolean) => Promise<any>;
|
||||
pipeline?: any[];
|
||||
timeoutSeconds?: number;
|
||||
source?: string;
|
||||
@@ -43,7 +45,7 @@ export interface CliOptions {
|
||||
browser?: boolean;
|
||||
args?: Arg[];
|
||||
columns?: string[];
|
||||
func?: (page: any, kwargs: Record<string, any>, debug?: boolean) => Promise<any>;
|
||||
func?: (page: IPage | null, kwargs: Record<string, any>, debug?: boolean) => Promise<any>;
|
||||
pipeline?: any[];
|
||||
timeoutSeconds?: number;
|
||||
}
|
||||
|
||||
+3
-1
@@ -2,6 +2,8 @@
|
||||
* Runtime utilities: timeouts and browser session management.
|
||||
*/
|
||||
|
||||
import type { IPage } from './types.js';
|
||||
|
||||
export const DEFAULT_BROWSER_CONNECT_TIMEOUT = parseInt(process.env.OPENCLI_BROWSER_CONNECT_TIMEOUT ?? '30', 10);
|
||||
export const DEFAULT_BROWSER_COMMAND_TIMEOUT = parseInt(process.env.OPENCLI_BROWSER_COMMAND_TIMEOUT ?? '45', 10);
|
||||
export const DEFAULT_BROWSER_EXPLORE_TIMEOUT = parseInt(process.env.OPENCLI_BROWSER_EXPLORE_TIMEOUT ?? '120', 10);
|
||||
@@ -24,7 +26,7 @@ export async function runWithTimeout<T>(
|
||||
|
||||
export async function browserSession<T>(
|
||||
BrowserFactory: new () => any,
|
||||
fn: (page: any) => Promise<T>,
|
||||
fn: (page: IPage) => Promise<T>,
|
||||
): Promise<T> {
|
||||
const mcp = new BrowserFactory();
|
||||
try {
|
||||
|
||||
@@ -1,2 +0,0 @@
|
||||
/** Scaffold a draft CLI from endpoint evidence. */
|
||||
export function scaffoldCli(opts: any): any { return { ok: true }; }
|
||||
@@ -1,2 +0,0 @@
|
||||
/** Smoke testing, verification, register, promote, explore, synthesize, scaffold, generate — stubs. */
|
||||
export async function runSmoke(cmd: any, page: any, args?: any): Promise<any> { return { ok: true, result: null }; }
|
||||
@@ -0,0 +1,23 @@
|
||||
/**
|
||||
* Page interface: type-safe abstraction over Playwright MCP browser page.
|
||||
*
|
||||
* All pipeline steps and CLI adapters should use this interface
|
||||
* instead of `any` for browser interactions.
|
||||
*/
|
||||
|
||||
export interface IPage {
|
||||
goto(url: string): Promise<void>;
|
||||
evaluate(js: string): Promise<any>;
|
||||
snapshot(opts?: { interactive?: boolean; compact?: boolean; maxDepth?: number; raw?: boolean }): Promise<any>;
|
||||
click(ref: string): Promise<void>;
|
||||
typeText(ref: string, text: string): Promise<void>;
|
||||
pressKey(key: string): Promise<void>;
|
||||
wait(seconds: number): Promise<void>;
|
||||
tabs(): Promise<any>;
|
||||
closeTab(index?: number): Promise<void>;
|
||||
newTab(): Promise<void>;
|
||||
selectTab(index: number): Promise<void>;
|
||||
networkRequests(includeStatic?: boolean): Promise<any>;
|
||||
consoleMessages(level?: string): Promise<any>;
|
||||
scroll(direction?: string, amount?: number): Promise<void>;
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
import { defineConfig } from 'vitest/config';
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
include: ['src/**/*.test.ts'],
|
||||
},
|
||||
});
|
||||
Reference in new issue
Block a user