diff --git a/.release-please-manifest.json b/.release-please-manifest.json index 9a9ceb4..1c6c6d8 100644 --- a/.release-please-manifest.json +++ b/.release-please-manifest.json @@ -1,3 +1,4 @@ { - "dist/copilot": "0.1.1" + "dist/copilot": "0.1.1", + "dist/openclaw": "0.1.4" } diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0e0bb28..35e5695 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -35,8 +35,7 @@ Requires Node 22. Run `npm ci` once. ## Text that differs per host - **A word or phrase:** use a variable, for example `{{host}}`, and set it in `platforms//vars.yaml`. -- **A paragraph or a few lines:** use a slot, for example `{{> connect}}`, and write the host's text in `platforms//skills//connect.md`. Everything outside the slot stays shared, so later fixes still reach the host. -- **An optional paragraph:** `{{>? name}}` is a slot a host may leave out; without the file, its line goes. The README uses it for `readme/more.md`. +- **A paragraph or a few lines:** use a slot, for example `{{> connect}}`, and write the host's text in `platforms//skills//connect.md`. Everything outside the slot stays shared, so later fixes still reach the host. Every host needs the file for a `{{> slot}}`. Write `{{>? slot}}` for text only some hosts add, for example OpenClaw's frontmatter `metadata`: on a host without the file, the line disappears (and validate fails if no host has it, to catch a typo). - **The README:** every plugin's README is `shared/README.md`. A host writes only `readme/setup.md` (how to install and sign in) and, if it needs more, `readme/more.md`; its own `README.md` fails the render. Code hosts keep their own README. - **The whole file, as a last resort:** `npm run override -- ` copies the shared skill into `platforms//` and records which version it was copied from in `overrides.lock`. When the shared skill changes later, `npm run validate` fails until someone updates the copy, or confirms the change doesn't apply, and runs `npm run override:ack -- `. - **A skill only one host has:** put it in `platforms//skills//SKILL.md`, with no shared skill of that name. It ships only on that host and needs no `overrides.lock` entry. @@ -82,7 +81,7 @@ Each host is its own release-please package (`dist/`), with its own versio ### Mirror repos -Some installs name a repo of their own instead of a folder here: `copilot plugin marketplace add ZenRows/copilot-zenrows` reads that repo's root. Such a repo is a mirror: `mirror: ZenRows/copilot-zenrows` in the host's `vars.yaml` makes each release of the host replace the mirror's contents with `dist//`, plus the host's marketplace catalogs at the same paths, listing only that host with `source` set to `./` (the mirror's root), and a banner in its README pointing here. So the catalog is edited in one place, and `npm run validate` fails if a mirrored host has none. Nobody edits a mirror by hand; only its deploy key can push to it. +Some installs name a repo of their own instead of a folder here: `copilot plugin marketplace add ZenRows/copilot-zenrows` reads that repo's root, and so do the Cursor Marketplace and ClawHub. Such a repo is a mirror: `mirror: ZenRows/copilot-zenrows` in the host's `vars.yaml` makes each release of the host replace the mirror's contents with `dist//`, plus a banner in its README pointing here. If the host has marketplace catalogs, the mirror gets them at the same paths, listing only that host with `source` set to `./` (the mirror's root), so the catalog is edited in one place. A host that reads its plugin from the repo root, like Cursor or OpenClaw, needs no catalog. Nobody edits a mirror by hand; only its deploy key can push to it. - The release workflow runs `.github/workflows/mirror.yml` for each released host. It makes one normal commit (`release copilot-v0.1.2 from ZenRows/zenrows-integrations@`), never a force-push, and tags the mirror `v`. No commit is made when nothing changed, so a re-run is safe. - It pushes with the deploy key in the `MIRROR_KEY_` secret of the `release` environment. The key, the mirror's branch protection and its settings are in [iac-bootstrap](https://github.com/ZenRows/iac-bootstrap). diff --git a/README.md b/README.md index 93c61ef..6a48c8a 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,7 @@ Each host's guide covers installing the plugin and signing in to Zenrows. You si | Host | Setup guide | |---|---| | GitHub Copilot | [dist/copilot/README.md](dist/copilot/README.md) | +| OpenClaw | [dist/openclaw/README.md](dist/openclaw/README.md) | ## Skills diff --git a/dist/openclaw/.claude-plugin/plugin.json b/dist/openclaw/.claude-plugin/plugin.json new file mode 100644 index 0000000..62797b7 --- /dev/null +++ b/dist/openclaw/.claude-plugin/plugin.json @@ -0,0 +1,13 @@ +{ + "name": "zenrows", + "version": "0.1.4", + "description": "Live web data: fetch any public page as Markdown, HTML or text, pull structured fields from it as JSON, run batch jobs over many URLs, crawl a site from one start URL and drive a hosted browser for pages that need clicks, forms or pagination, with proxies, JavaScript rendering and anti-bot protection handled by Zenrows.", + "author": { + "name": "Zenrows", + "email": "integrations@zenrows.com" + }, + "homepage": "https://docs.zenrows.com/integrations/openclaw", + "repository": "https://github.com/ZenRows/zenrows-integrations", + "license": "MIT", + "mcpServers": "./.mcp.json" +} diff --git a/dist/openclaw/.clawhubignore b/dist/openclaw/.clawhubignore new file mode 100644 index 0000000..a1b82bc --- /dev/null +++ b/dist/openclaw/.clawhubignore @@ -0,0 +1,7 @@ +.git/ +.github/ +.claude/ +reports/ +node_modules/ +*.tgz +RELEASING.md diff --git a/dist/openclaw/.mcp.json b/dist/openclaw/.mcp.json new file mode 100644 index 0000000..43dd8e2 --- /dev/null +++ b/dist/openclaw/.mcp.json @@ -0,0 +1,10 @@ +{ + "mcpServers": { + "zenrows": { + "url": "https://mcp.zenrows.com/mcp", + "description": "Enable this plugin and connect your Zenrows account in OpenClaw. A Zenrows account is required; usage is billed by Zenrows.", + "auth": "oauth", + "type": "http" + } + } +} diff --git a/dist/openclaw/LICENSE b/dist/openclaw/LICENSE new file mode 100644 index 0000000..e7dd33a --- /dev/null +++ b/dist/openclaw/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Zenrows + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/dist/openclaw/README.md b/dist/openclaw/README.md new file mode 100644 index 0000000..f98936f --- /dev/null +++ b/dist/openclaw/README.md @@ -0,0 +1,94 @@ +# Zenrows for OpenClaw + +Live web data: fetch any public page as Markdown, HTML or text, pull structured fields from it as JSON, run batch jobs over many URLs, crawl a site from one start URL and drive a hosted browser for pages that need clicks, forms or pagination, with proxies, JavaScript rendering and anti-bot protection handled by Zenrows. This plugin connects the agent to the Zenrows MCP server and adds skills that tell the agent which Zenrows tool fits each task and how to keep requests efficient. + +## What you get + +- The Zenrows MCP server (`https://mcp.zenrows.com/mcp`). You sign in to your Zenrows account with OAuth; there is no API key to copy. +- Skills: + +| Skill | Use it for | +|---|---| +| `zenrows` | Overview: tools, defaults, errors and plan limits | +| `zenrows-getting-started` | First-run checks and troubleshooting | +| `zenrows-scrape` | Reading one page | +| `zenrows-extract` | Named fields or JSON from one page | +| `zenrows-batch` | A list of five or more URLs as one job | +| `zenrows-crawl` | Many pages under one site section | +| `zenrows-map` | Listing a site's URLs without fetching them | +| `zenrows-browser` | Clicks, forms, pagination and multi-step flows | +| `zenrows-capture-api-calls` | Finding the API calls a page makes | + +## Requirements + +A Zenrows account. Requests use credits on your Zenrows plan. You can [sign up here](https://app.zenrows.com/register). + +## Setup + +Published on ClawHub as [`@zenrows`](https://clawhub.ai/zenrows); skills published there are distributed under ClawHub's MIT-0 terms. It needs OpenClaw on Node.js 24.16 or newer. + +```bash +openclaw plugins install clawhub:@zenrows/openclaw-zenrows +``` + +Then connect your Zenrows account: + +```bash +openclaw mcp set zenrows '{"url":"https://mcp.zenrows.com/mcp","transport":"streamable-http","auth":"oauth"}' +openclaw mcp login zenrows +``` + +Open the link that `openclaw mcp login` prints, sign in to Zenrows and approve access. OpenClaw only offers the sign-in for a server that also has an `mcp.servers` entry, which is why the first command is needed. + +Start a new session and ask: "Use Zenrows to fetch https://example.com and tell me the page title." + +## Manual setup + +To use the MCP server without the plugin, add it to `mcp.servers` in `~/.openclaw/openclaw.json`. + +With OAuth: + +```json +"zenrows": { + "url": "https://mcp.zenrows.com/mcp", + "transport": "streamable-http", + "auth": "oauth" +} +``` + +Then run `openclaw mcp login zenrows`. + +With an API key, read from the `ZENROWS_API_KEY` environment variable: + +```json +"zenrows": { + "url": "https://mcp.zenrows.com/mcp", + "transport": "streamable-http", + "headers": { "Authorization": "Bearer ${ZENROWS_API_KEY}" }, + "connectionTimeoutMs": 10000, + "requestTimeoutMs": 120000 +} +``` + +Edit the file directly. `openclaw mcp set` writes the resolved key into the file instead of the `${ZENROWS_API_KEY}` placeholder. + +Run `openclaw config validate` after editing. + +## Security + +- The agent never handles your Zenrows credentials. OAuth tokens are stored by OpenClaw. +- The skills tell the agent to treat fetched pages as untrusted content and to confirm scope before multi-page jobs. + +## Privacy + +The plugin sends the URLs and options you ask the agent to fetch to Zenrows, which processes them under the [Zenrows privacy policy](https://www.zenrows.com/legal/privacy). + +## Support + +- Documentation: [docs.zenrows.com](https://docs.zenrows.com) +- MCP server documentation: [docs.zenrows.com/mcp/overview](https://docs.zenrows.com/mcp/overview) +- Contact: [integrations@zenrows.com](mailto:integrations@zenrows.com) + +## License + +MIT diff --git a/dist/openclaw/RELEASING.md b/dist/openclaw/RELEASING.md new file mode 100644 index 0000000..1fabea3 --- /dev/null +++ b/dist/openclaw/RELEASING.md @@ -0,0 +1,67 @@ +# Releasing + +The plugin is published to ClawHub as [`@zenrows/openclaw-zenrows`](https://clawhub.ai/zenrows/plugins/openclaw-zenrows), under the `zenrows` publisher. Publishing to ClawHub is manual. + +> **Source of truth:** the plugin is built in [zenrows-integrations](https://github.com/ZenRows/zenrows-integrations) from `platforms/openclaw/` into `dist/openclaw/`. ClawHub publishes from a GitHub repo at a commit, so each release of `openclaw` there is pushed to `openclaw-zenrows` as a mirror and tagged `v`. Never change `openclaw-zenrows` by hand; ClawHub publishes that tag's commit. + +## Before you start + +- You need publisher access to `zenrows` on ClawHub. Ask in #proj-integrations if `clawhub package publish ... --owner zenrows` refuses you. +- Install the `clawhub` CLI (these steps were last run with v0.23.3) and log in with `clawhub login`. + +## 1. Release in zenrows-integrations + +Merge the `openclaw` release PR in zenrows-integrations. release-please bumps the version in `package.json`, `openclaw.plugin.json` and `.claude-plugin/plugin.json` (ClawHub takes it from `package.json`), and the release workflow pushes `dist/openclaw/` to `openclaw-zenrows` and tags it `v`. Check that the tag is there before going on. + +## 2. Dry run against the release commit + +Publish the mirror's commit for the tag, not `main`, so what you check is what ships. + +```bash +clawhub whoami +VERSION= +SHA=$(gh api "repos/ZenRows/openclaw-zenrows/commits/v$VERSION" --jq .sha) + +clawhub package publish "ZenRows/openclaw-zenrows@$SHA" \ + --family bundle-plugin \ + --owner zenrows \ + --dry-run +``` + +Check that `Version` is the new one, `Name` is `@zenrows/openclaw-zenrows` and `Commit` is `$SHA`. + +## 3. Publish + +```bash +clawhub package publish "ZenRows/openclaw-zenrows@$SHA" \ + --family bundle-plugin \ + --owner zenrows \ + --changelog ' release notes>' \ + --wait +``` + +- Put the changelog in single quotes. In double quotes the shell runs anything between backticks, and the text loses those words. +- `--wait` returns only after ClawHub's security scan. Without it, a release the scan holds back looks published but isn't installable. +- If the CLI asks for `--manual-override-reason`, a trusted publisher is configured for this package. Stop and ask in #proj-integrations instead of overriding it. + +## 4. Prove it is live and is what was released + +```bash +curl -s https://clawhub.ai/api/v1/packages/@zenrows/openclaw-zenrows \ + | jq '.package | {latestVersion, scanStatus, sourceCommit: .verification.sourceCommit}' +``` + +`latestVersion` must be the new version, `scanStatus` `clean` and `sourceCommit` `$SHA`. + +Then compare the published files with the released files. This needs no local clone: + +```bash +VERSION= +cd "$(mktemp -d)" +curl -sSL -o pkg.zip "https://clawhub.ai/api/v1/packages/@zenrows/openclaw-zenrows/download?version=$VERSION" +mkdir pub repo && unzip -q pkg.zip -d pub +gh api "repos/ZenRows/openclaw-zenrows/tarball/$SHA" | tar -xz -C repo --strip-components=1 +diff -r $(sed -e 's:/$::' -e '/^$/d' -e 's/^/-x /' repo/.clawhubignore) repo pub/package && echo identical +``` + +The `-x` flags skip what `.clawhubignore` keeps out of the package (this file among them), so only shipped files are compared. Anything other than `identical` means the published package is not what was released. Then post the version in #proj-integrations so the Integrations brain picks it up. diff --git a/dist/openclaw/assets/icon.png b/dist/openclaw/assets/icon.png new file mode 100644 index 0000000..a32dc12 Binary files /dev/null and b/dist/openclaw/assets/icon.png differ diff --git a/dist/openclaw/assets/logo.png b/dist/openclaw/assets/logo.png new file mode 100644 index 0000000..a32dc12 Binary files /dev/null and b/dist/openclaw/assets/logo.png differ diff --git a/dist/openclaw/openclaw.plugin.json b/dist/openclaw/openclaw.plugin.json new file mode 100644 index 0000000..d3c8953 --- /dev/null +++ b/dist/openclaw/openclaw.plugin.json @@ -0,0 +1,20 @@ +{ + "id": "zenrows", + "name": "Zenrows", + "version": "0.1.4", + "description": "Live web data: fetch any public page as Markdown, HTML or text, pull structured fields from it as JSON, run batch jobs over many URLs, crawl a site from one start URL and drive a hosted browser for pages that need clicks, forms or pagination, with proxies, JavaScript rendering and anti-bot protection handled by Zenrows.", + "categories": ["web"], + "configSchema": { + "type": "object", + "additionalProperties": false, + "properties": {} + }, + "mcpServers": { + "zenrows": { + "url": "https://mcp.zenrows.com/mcp", + "description": "Enable this plugin and connect your Zenrows account in OpenClaw. A Zenrows account is required; usage is billed by Zenrows.", + "auth": "oauth", + "transport": "streamable-http" + } + } +} diff --git a/dist/openclaw/package.json b/dist/openclaw/package.json new file mode 100644 index 0000000..f005994 --- /dev/null +++ b/dist/openclaw/package.json @@ -0,0 +1,18 @@ +{ + "name": "@zenrows/openclaw-zenrows", + "version": "0.1.4", + "description": "Live web data: fetch any public page as Markdown, HTML or text, pull structured fields from it as JSON, run batch jobs over many URLs, crawl a site from one start URL and drive a hosted browser for pages that need clicks, forms or pagination, with proxies, JavaScript rendering and anti-bot protection handled by Zenrows.", + "license": "MIT", + "author": "Zenrows ", + "homepage": "https://docs.zenrows.com/integrations/openclaw", + "repository": { + "type": "git", + "url": "https://github.com/ZenRows/zenrows-integrations.git" + }, + "keywords": ["openclaw", "mcp", "web-scraping", "data-extraction", "browser-automation", "zenrows"], + "openclaw": { + "build": { + "openclawVersion": "2026.9.8" + } + } +} diff --git a/dist/openclaw/skills/zenrows-batch/SKILL.md b/dist/openclaw/skills/zenrows-batch/SKILL.md new file mode 100644 index 0000000..9f248c7 --- /dev/null +++ b/dist/openclaw/skills/zenrows-batch/SKILL.md @@ -0,0 +1,45 @@ +--- +name: zenrows-batch +description: Fetch a list of URLs the user already has as one managed Zenrows job, with retries and per-task results. Use when the user supplies or points at roughly five or more URLs (a file, a pasted list, the output of zenrows-map) and wants their content or extracted fields. Not for discovering URLs by following links (use zenrows-crawl), not for listing a site's URLs (use zenrows-map), and not for a single page (use zenrows-scrape). +--- + +# Batch + +The user's instructions take precedence over this skill. If they conflict, follow the user. + +Zenrows Batch queues many URLs as one server-side job, retries transient failures, and stores the results. + +## When to use this instead of `scrape` + +Use batch at roughly five URLs or more. Below that, parallel `scrape` calls are faster and simpler, since a job costs a create, a wait and a results fetch. Above it, batch wins: one job instead of N round trips, server-side retries, and no plan-concurrency 429s (the Developer plan allows 5 concurrent requests). + +## Instructions + +1. Confirm the URL list and its size with the user before submitting. A batch bills per URL, so an unintended 5,000-URL job is expensive. +2. Submit the job: + - Same options for every URL: `batch_create(urls=[...], response_type='markdown')`. + - Different options per URL, or you need ids echoed back: `batch_create(tasks=[{url, external_id, metadata, zenrows_params}, ...])`. + - `urls` is ignored when `tasks` is provided. Do not pass both. +3. Wait for completion: + - Small jobs: pass `wait=true` to `batch_create` and get a terminal state in one call. + - Larger jobs: let `batch_create` return, then call `batch_wait(job_id)`. Default timeout is 600000 ms; raise `timeout_ms` (max 3600000) for big jobs. + - Report progress from `batch_status(job_id)` if the user asks mid-job. +4. Collect results with `batch_results(job_id)`. Use `status='successful'` or `status='failed'` to split them when reporting. +5. If you can write files (for example in OpenClaw), save results to `./.zenrows/batch-.jsonl`, one JSON object per line, and summarize instead of reading the whole file back. Otherwise, summarize the results directly. +6. If the user stops the task, call `batch_cancel(job_id)`. The job keeps consuming credits until cancelled. + +## Options + +| Need | Where | Note | +|------|-------|------| +| Same settings for all URLs | job level: `js_render`, `premium_proxy`, `proxy_country`, `response_type` | Simplest form | +| Different settings per URL | `tasks[].zenrows_params` | Accepts `js_render`, `premium_proxy`, `extract`, `autoparse`, and other scrape params | +| Structured fields, not page content | `tasks[].zenrows_params` with `extract` or `autoparse` | Batch and Extract compose | +| Track which result is which | `tasks[].external_id` and `tasks[].metadata` | Echoed back on results | +| Geo-restricted pages | `proxy_country` | Works with the default settings | + +## Notes + +- `batch_create` has no required field, but a job with neither `tasks` nor `urls` does nothing. Always supply one. +- Always report the failed count alongside the successful one. Batch retries transient failures itself, so a URL in the failed set has genuinely not worked. +- `browser_batch` is unrelated: it batches actions inside one browser session. It is not this. diff --git a/dist/openclaw/skills/zenrows-browser/SKILL.md b/dist/openclaw/skills/zenrows-browser/SKILL.md new file mode 100644 index 0000000..4676f7e --- /dev/null +++ b/dist/openclaw/skills/zenrows-browser/SKILL.md @@ -0,0 +1,60 @@ +--- +name: zenrows-browser +description: 'Drive a page through steps before its content is available: clicking, filling forms, logging in, paginating, or multi-step navigation that a single fetch cannot do. Use when interaction is required to reach the content. Not for static page reads (use zenrows-scrape), and not for passively listing the network or API calls a page makes (use zenrows-capture-api-calls).' +--- + +# Browser automation + +The user's instructions take precedence over this skill. If they conflict, follow the user. + +## Instructions + +1. Open a session with `browser_navigate(url)`. Save the returned `session_id` and pass it to every later call. +2. Find the selectors you need with `browser_get_accessibility_tree` or `browser_query_selector_all`. Both are cheaper and more precise than a screenshot. Take `browser_screenshot` only when the page state is unclear or the user asked to see it. +3. Run the interaction as one `browser_batch` call whenever the steps are known in advance. Every round trip to the hosted browser costs seconds; a login sent as six separate calls is six times slower than it needs to be. +4. Fall back to individual `browser_*` calls only when a later step depends on reading an earlier result. +5. Always call `browser_close(session_id)` when done, to free a concurrency slot. + +## browser_batch + +`browser_batch(session_id, actions, stop_on_error)` runs 1 to 50 ordered actions in one call, reads included. A login and its result: + +```json +{ + "session_id": "", + "actions": [ + { "type": "wait_for_selector", "selector": "#email" }, + { "type": "fill", "selector": "#email", "value": "user@example.com" }, + { "type": "fill", "selector": "#password", "value": "secret" }, + { "type": "click", "selector": "button[type=submit]" }, + { "type": "wait_for_navigation" }, + { "type": "get_text" } + ] +} +``` + +Action names inside `actions` drop the `browser_` prefix: `navigate`, `go_back`, `go_forward`, `reload`, `click`, `hover`, `type`, `fill`, `select`, `check`, `uncheck`, `focus`, `press_key`, `scroll`, `drag`, `wait`, `wait_for_selector`, `wait_for_navigation`, `evaluate`, `screenshot`, `get_text`, `get_html`, `get_url`, `get_title`, `get_attribute`, `get_accessibility_tree`, `query_selector_all`. + +Three traps: + +- The action is `select`, not `select_option`. +- `fill` takes `value`. `type` takes `text`. +- `stop_on_error` defaults to `true`, so one bad selector halts the rest of the list. Set it to `false` only when the later actions do not depend on the earlier ones. + +`browser_batch` batches actions inside one browser session. It is unrelated to the Batch product and its `batch_*` tools. + +## Session management + +- Pass `session_id` to every subsequent `browser_*` call. +- Do not open multiple sessions unless the task requires it. +- Browser sessions hold concurrency slots on the user's plan. Close them promptly. +- Geo-targeting is set when the session is created. Pass `proxy_country` (ISO 3166-1 alpha-2, for example `de`) or `proxy_region` (`eu`, `na`, `ap`, `sa`, `af`, `me`) to `browser_navigate`. Later `browser_*` calls do not accept either, so set it on the first call or start a new session. + +## When to prefer browser over scrape + +| Scenario | Tool | +|----------|------| +| Read a static docs page | `scrape` | +| Log in, then navigate | `browser_*` | +| Click "Load more" pagination | `browser_*` | +| Fill a search form and read results | `browser_*` | diff --git a/dist/openclaw/skills/zenrows-capture-api-calls/SKILL.md b/dist/openclaw/skills/zenrows-capture-api-calls/SKILL.md new file mode 100644 index 0000000..e79db8e --- /dev/null +++ b/dist/openclaw/skills/zenrows-capture-api-calls/SKILL.md @@ -0,0 +1,35 @@ +--- +name: zenrows-capture-api-calls +description: Discover the background API or XHR endpoints a page calls, or reverse-engineer how it loads its data. Use when the user wants the underlying request a site makes, not the rendered page. Not for interacting with the page UI (use zenrows-browser), and not for reading rendered content (use zenrows-scrape). +--- + +# Capture API calls + +The user's instructions take precedence over this skill. If they conflict, follow the user. + +## Instructions + +1. Open a session with `browser_navigate(url)` and save the `session_id`. +2. Let the page finish loading: `browser_wait(session_id, ms=3000)`. Do not use `browser_wait_for_navigation` here; that must run before a navigation-triggering action, not after `browser_navigate`. +3. Collect network resource URLs with `browser_evaluate`: + + ```javascript + JSON.stringify( + performance.getEntriesByType('resource') + .filter(r => r.initiatorType === 'fetch' || r.initiatorType === 'xmlhttprequest' || r.name.includes('/api/')) + .map(r => ({ url: r.name, type: r.initiatorType })) + ) + ``` + +4. Identify the endpoint(s) most likely returning the target data (JSON paths, `/api/` URLs, GraphQL endpoints). +5. Present the endpoint URL, the method if inferable, and a sample response shape. +6. Optionally call the endpoint directly with `scrape(url=, response_type='html')` to get the raw response body. +7. Always call `browser_close(session_id)` when done. + +## Fallback + +If `performance.getEntriesByType` returns nothing (a single-page app with deferred loads), interact with the page (scroll, click load triggers) and re-run the evaluate script, or inspect inline `