Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 24 additions & 29 deletions src/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,17 @@ const DEFAULT_PREMIUM_PROXY = process.env.ZENROWS_PREMIUM_PROXY === "true";
const DEFAULT_RESPONSE_TYPE =
(process.env.ZENROWS_RESPONSE_TYPE as "markdown" | "plaintext" | "html" | undefined) ?? "markdown";

// Sent in the initialize response; clients put it in the model's context, so it says
// which tool to reach for first, not what every tool does.
// Sent in the initialize response; clients put it in the model's context, so it maps the
// groups of tools as facts, without telling the model which to reach for or in what order.
export const INSTRUCTIONS = [
"Zenrows returns the content of public web pages.",
"Start with scrape for one page's content, or extract for structured JSON fields.",
"For many URLs use batch_create, then batch_wait until it ends, batch_results to page through its rows and batch_content to read one page; to discover the URLs behind a listing or section page, use crawl_create.",
"Use the browser_* tools only when a page needs clicks, typing or several steps: browser_navigate opens a session, and browser_close frees it when done.",
"Call account_usage (free) before a large job or after a 402.",
].join(" ");
"Zenrows returns the content of public web pages. The tools come in five groups:",
"- scrape returns one page's content: markdown, plain text, HTML or a screenshot.",
'- extract returns structured JSON fields from one page; prepare_domain and prepared_domain_status set a domain up for its mode "auto".',
"- batch_* run a known list of URLs as one asynchronous job.",
"- crawl_* start from one URL and collect the URLs it links to on the same site, optionally with each page's content.",
"- browser_* drive a hosted browser session for pages that need clicks, typing or several steps; browser_navigate opens the session and browser_close ends it.",
"account_usage reports the plan's credits and is free.",
].join("\n");

export interface ScrapeParams {
url: string;
Expand Down Expand Up @@ -157,42 +159,35 @@ export function createServer(apiKey: KeyRef, clientName?: string, options: Serve
destructiveHint: false,
openWorldHint: true,
},
description: `Scrape any webpage and return its content using Zenrows (Fetch).
description: `Fetch one web page with Zenrows (Fetch) and return its content: markdown (default), plain text, HTML or a screenshot.

Returns full-page content (markdown/HTML/PDF/screenshot). For structured JSON
fields (products, articles, listings), the extract tool returns parsed fields
instead of a full page body.
For one page's full content.

By default the request uses automatic mode (mode=auto): Zenrows adds JS rendering
or premium proxies per page only when a plain request does not return the page,
and charges only for the configuration that succeeds. Pass just the URL.
Automatic mode (mode=auto) is the default: Zenrows adds JavaScript rendering or premium proxies to a page only when a plain request does not return it, and bills only the configuration that succeeded. A URL alone is enough.

Set js_render or premium_proxy only to force a fixed configuration. Doing so
switches off automatic mode, and every request is billed at that
configuration's cost (premium_proxy with js_render is 25x a basic request).
Setting js_render or premium_proxy turns automatic mode off and bills every request at that fixed configuration: 1 credit basic, 5 with js_render, 10 with premium_proxy, 25 with both.

Examples:
Default: { url: "https://example.com" }
Geo: { url: "https://example.com", proxy_country: "US" }
Forced: { url: "https://spa.com", js_render: true }`,
Default: { url: "https://example.com" }
Geo: { url: "https://example.com", proxy_country: "US" }
Forced rendering: { url: "https://spa.com", js_render: true }`,
inputSchema: {
url: z.string().url().describe("The webpage URL to scrape"),

js_render: z
.boolean()
.nullish()
.describe(
"Force JavaScript rendering on every request. Overrides automatic mode (mode=auto), " +
"which already renders JavaScript when a page needs it. Leave unset unless you need a fixed configuration."
"Optional. Renders JavaScript on every request (5 credits) and turns automatic mode off. " +
"Automatic mode already renders a page that needs it."
),

premium_proxy: z
.boolean()
.nullish()
.describe(
"Force premium residential proxies on every request (10x credit cost). Overrides " +
"automatic mode (mode=auto), which already uses premium proxies when a page needs them. " +
"Leave unset unless you need a fixed configuration."
"Optional. Uses premium residential proxies on every request (10 credits; 25 with js_render) " +
"and turns automatic mode off. Automatic mode already adds them to a page that needs them."
),

proxy_country: z
Expand All @@ -210,8 +205,8 @@ Examples:
.describe(
"Output format. 'markdown' (default) preserves structure and is ideal for LLMs. " +
"'plaintext' strips all formatting for pure text extraction. " +
"'pdf' returns a PDF of the page. " +
"'html' returns the raw HTML source (omits the response_type param; Zenrows default). " +
"'pdf' asks the API for a PDF, which this server decodes as text, so the result is not a usable PDF file. " +
"Ignored when autoparse, css_extractor, outputs, or screenshot params are set."
),

Expand All @@ -236,8 +231,8 @@ Examples:
.string()
.nullish()
.describe(
"CSS selector to wait for before capturing. Use when key content loads " +
"after the initial page render. Requires js_render=true; without it, including in automatic mode, it may be ignored."
"CSS selector to wait for before capturing, for content that loads after the first render. " +
"Applies with js_render=true; automatic mode may ignore it."
),

wait: z
Expand Down
18 changes: 9 additions & 9 deletions src/tools/batch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -374,7 +374,7 @@ const COSTS = `Cost: the same credits as the regular API, per successful task: 1

const LIMITS = `Limits: up to 100,000 tasks per call; 3 active jobs per account by default, shared with Crawl (BATCH_QUOTA_EXCEEDED past that); a job frees its slot when its run ends.`;

const RESULT_FACTS = `A result_url is presigned and expires after 2 hours: call batch_results again for a fresh one, or read the body with batch_content. Results are kept 14 days by default; after that a successful row reads as status "expired", with no result_url, and batch_content answers BATCH_RESULT_EXPIRED.`;
const RESULT_FACTS = `A result_url is presigned and expires after 2 hours: a new batch_results call returns a fresh one, and batch_content reads the body. Results are kept 14 days by default; after that a successful row reads as status "expired", with no result_url, and batch_content answers BATCH_RESULT_EXPIRED.`;

export function registerBatchTools(
server: McpServer,
Expand Down Expand Up @@ -430,15 +430,15 @@ export function registerBatchTools(
annotations: { title: "Create Batch Job", readOnlyHint: false, destructiveHint: false, openWorldHint: true },
description: `Submit a Batch job: many URLs scraped asynchronously in the cloud (Zenrows Batch API). Suited to large URL lists; scrape and extract handle single pages. Not the same as browser_batch.

Tasks use automatic mode (mode=auto) by default: Zenrows enables JS rendering and premium proxies only on the pages that need them. Pass just the URLs. Set js_render or premium_proxy (job-level, or per task) only to force a fixed configuration; that turns automatic mode off for those tasks.
Tasks use automatic mode (mode=auto) by default: Zenrows enables JS rendering and premium proxies only on the pages that need them. A list of URLs alone is enough. Setting js_render or premium_proxy (job-level, or per task) forces a fixed configuration and turns automatic mode off for those tasks.

${COSTS}
${LIMITS}

Creates a closed job with every task in this one call. Each task takes a url plus optional external_id, metadata, method/body (POST tasks) and scrape params. Every call sends an Idempotency-Key, so the bounded retries on network errors and 5xx never create a second job; an error reply carries the key to retry with.

Returns job_id, status and stats at once; the job runs asynchronously. Then: batch_wait until it ends, batch_results to list rows (paged), batch_content to read one task's page. A run that ends failed (out of credits, inactive subscription, key cap) answers ok:false with error.code and server_code; batch_rerun recovers it. Submits of 10,000+ tasks keep writing tasks for a while (ingesting:true).
Errors: BATCH_INVALID_REQUEST (400, server_code and invalid_tasks say what), BATCH_INSUFFICIENT_CREDITS (402), BATCH_KEY_CAP_REACHED (402, this key's cap), BATCH_QUOTA_EXCEEDED (429, job limit), AUTH_INVALID (401; server_code subscription_inactive means fix Billing), BATCH_ACCESS_DENIED (403, do not retry), BACKEND_UNAVAILABLE.
Returns job_id, status and stats at once; the job runs asynchronously. The job_id is what batch_status, batch_wait, batch_results and batch_content take: batch_wait reports when the run ends, batch_results lists its rows (paged) and batch_content returns one task's page. A run that ends failed (out of credits, inactive subscription, key cap) answers ok:false with error.code and server_code; batch_rerun recovers it. Submits of 10,000+ tasks keep writing tasks for a while (ingesting:true).
Errors: BATCH_INVALID_REQUEST (400, server_code and invalid_tasks say what), BATCH_INSUFFICIENT_CREDITS (402), BATCH_KEY_CAP_REACHED (402, this key's cap), BATCH_QUOTA_EXCEEDED (429, job limit), AUTH_INVALID (401; server_code subscription_inactive means the subscription is inactive), BATCH_ACCESS_DENIED (403; a retry gets the same answer), BACKEND_UNAVAILABLE.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: "subscription_inactive means the subscription is inactive" repeats itself, and the old "fix Billing" told the reader where the problem is. Maybe "means the subscription needs attention in Billing", which is a fact rather than an instruction.

Schedules, webhooks, exports, CSV inputs and open jobs are available through the Batch API and SDKs only.`,
inputSchema: {
tasks: z
Expand Down Expand Up @@ -612,7 +612,7 @@ A failed run answers ok:false with error.code BATCH_INSUFFICIENT_CREDITS, BATCH_
"batch_wait",
{
annotations: { title: "Wait for Batch Job", readOnlyHint: true, destructiveHint: false, openWorldHint: true },
description: `Poll a Batch job until its run ends (completed, stopped or failed) or pauses, and return its status and stats like batch_status. If the wait runs out first, it returns the current state with ok:true, timed_out:true and a note: call batch_wait again. A paused run returns at once with paused:true (it does not move until batch_resume). A failed run answers ok:false with error.code and server_code (insufficient_credits, subscription_inactive or api_key_cap_reached). Transient poll errors are retried. A call that carries a progressToken gets a progress notification after each poll.`,
description: `Poll a Batch job until its run ends (completed, stopped or failed) or pauses, and return its status and stats like batch_status. If the wait runs out first, it returns the current state with ok:true and timed_out:true; the job keeps running, and another batch_wait call goes on waiting. A paused run returns at once with paused:true (it does not move until batch_resume). A failed run answers ok:false with error.code and server_code (insufficient_credits, subscription_inactive or api_key_cap_reached). Transient poll errors are retried. A call that carries a progressToken gets a progress notification after each poll.`,
inputSchema: {
job_id: jobIdSchema,
timeout: waitSecondsSchema,
Expand All @@ -632,11 +632,11 @@ A failed run answers ok:false with error.code BATCH_INSUFFICIENT_CREDITS, BATCH_
"batch_results",
{
annotations: { title: "Batch Job Results", readOnlyHint: true, destructiveHint: false, openWorldHint: true },
description: `List a Batch run's result rows, a page at a time: up to limit rows (default ${DEFAULT_RESULTS_LIMIT}, max 1000) and next_cursor. Pass next_cursor back as cursor to continue; it is null once every row was read.
description: `List a Batch run's result rows, a page at a time: up to limit rows (default ${DEFAULT_RESULTS_LIMIT}, max 1000) and next_cursor. next_cursor, passed back as cursor, continues the list; it is null once every row was read.

Each row has task_id, external_id, url, metadata, status (successful, failed, pending, processing, expired), type, result_url when successful, and error when failed. ${RESULT_FACTS}

Filter with status: all (default), successful, failed, or processing (not finished yet). A sparse filter can return fewer rows than limit with next_cursor still set: keep paging. While the run is still running or writing its tasks, the list is partial.`,
Filter with status: all (default), successful, failed, or processing (not finished yet). A sparse filter can return fewer rows than limit with next_cursor still set; more rows follow. While the run is still running or writing its tasks, the list is partial.`,
inputSchema: {
job_id: jobIdSchema,
run_id: runIdSchema,
Expand Down Expand Up @@ -686,9 +686,9 @@ Filter with status: all (default), successful, failed, or processing (not finish
"batch_content",
{
annotations: { title: "Batch Task Content", readOnlyHint: true, destructiveHint: false, openWorldHint: true },
description: `Read one Batch task's stored result, by job_id and task_id (from batch_results). A text result (HTML, Markdown, plain text, JSON) comes back inline, cut to max_chars (default ${DEFAULT_CONTENT_MAX_CHARS}); a final note says when it was cut. A binary result (a PDF) is not inlined: the reply gives its result_url, presigned for 2 hours. Finding that URL reads the run's results; on a large run one call may run out of time first and answer with lookup_cursor: call batch_content again with it to go on.
description: `Read one Batch task's stored result, by job_id and task_id (from batch_results). A text result (HTML, Markdown, plain text, JSON) comes back inline, cut to max_chars (default ${DEFAULT_CONTENT_MAX_CHARS}); a final note says when it was cut. A binary result (a PDF) is not inlined: the reply gives its result_url, presigned for 2 hours. Finding that URL reads the run's results; on a large run one call may run out of time first and answer with lookup_cursor; passing it back continues the lookup.

BATCH_TASK_NOT_READY: the task has not finished; BATCH_TASK_FAILED: it failed (task_error holds its error); BATCH_RESULT_EXPIRED: past the job's retention (14 days by default); rerun to scrape it again.`,
BATCH_TASK_NOT_READY: the task has not finished; BATCH_TASK_FAILED: it failed (task_error holds its error); BATCH_RESULT_EXPIRED: past the job's retention (14 days by default); batch_rerun scrapes it again.`,
inputSchema: {
job_id: jobIdSchema,
task_id: z.string().min(1).describe("task_id from a batch_results row"),
Expand Down
17 changes: 9 additions & 8 deletions src/tools/browser.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,17 +44,18 @@ export function registerBrowserTools(
destructiveHint: false,
openWorldHint: true,
},
description: `Open a Zenrows Browser Sessions session and navigate to a URL.
description: `Open a Zenrows Browser session (a hosted browser) and navigate to a URL in one step.

This is the entry point for all browser automation. It creates a new session backed by
Zenrows' hosted browser infrastructure and navigates to the given URL in one step.
Every browser automation starts here. Returns a session_id that every other browser_* tool takes.

Returns a session_id that every other browser_* tool takes. The session holds one
concurrency slot until browser_close closes it.
Side effects: the session holds one concurrency slot and is billed for its open time
(5 credits per minute) and its traffic (25,000 credits per GB) until browser_close ends it;
a session closes after 3 minutes without a browser_* call, or 15 minutes after it opened,
and is billed until it closes.
Comment on lines +53 to +54

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do these two timeouts come from? The 5 credits per minute and 25,000 per GB match the docs, but I can't find "3 minutes without a browser_* call" or "15 minutes after it opened" in this repo. The docs only describe a session TTL of 3 minutes by default, configurable from 1 to 15: https://docs.zenrows.com/browser-sessions/features/session-ttl


When to use options:
Options:
- proxy_country: geo-restricted content (ISO 3166-1 alpha-2, e.g. "US", "DE")
- proxy_region: world region for geo-targeted proxy (eu, na, ap, sa, af, me)`,
- proxy_region: world region for a geo-targeted proxy (eu, na, ap, sa, af, me)`,
inputSchema: {
url: z.string().url().describe("The URL to navigate to"),
proxy_country: z
Expand Down Expand Up @@ -113,7 +114,7 @@ When to use options:
{
annotations: { title: "Close Browser Session", readOnlyHint: false, destructiveHint: false, openWorldHint: true },
description:
"Close a browser session and free all associated resources, including the concurrency slot it holds.",
"Close a browser session: frees its concurrency slot and ends its billed time.",
inputSchema: { session_id: sessionId },
},
async ({ session_id }) => {
Expand Down
Loading
Loading