diff --git a/.cursor-plugin/plugin.json b/.cursor-plugin/plugin.json index 6de6fcc..f7e9674 100644 --- a/.cursor-plugin/plugin.json +++ b/.cursor-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "cloro", "displayName": "cloro", - "version": "0.1.0", + "version": "0.2.0", "description": "Query ChatGPT, Perplexity, Gemini, Copilot, Grok, Google AI Mode, Google Search and Google News as agent tools, with country and state-level geo-targeting.", "author": { "name": "cloro", diff --git a/README.md b/README.md index 2f6c95b..a69b7eb 100644 --- a/README.md +++ b/README.md @@ -6,11 +6,13 @@ Runs hosted at `mcp.cloro.dev`, or locally over stdio. ## How do you connect an agent to the cloro MCP server? -1. Get a cloro API key from the [dashboard](https://dashboard.cloro.dev). -2. Point your client at the hosted server, passing the key. -3. Ask the agent a question that needs a live answer engine. +In Claude, you sign in with OAuth and do not need a key: -The hosted server speaks Streamable HTTP. Put the key in the URL path so it works in clients that cannot set custom headers: +1. Go to **Customize → Connectors** and select **Add custom connector**. +2. Enter `https://mcp.cloro.dev/mcp`. Under **Authentication**, choose **Sign in when needed**. +3. Ask Claude to use a cloro tool. Sign in to cloro and select the organization whose credits pay for the calls. + +Other clients use a cloro API key from the [dashboard](https://dashboard.cloro.dev). The hosted server speaks Streamable HTTP. Put the key in the URL path so it works in clients that cannot set custom headers: ```json { @@ -49,6 +51,8 @@ MCP_TRANSPORT=stdio CLORO_API_KEY=your_key node dist/index.js `MCP_TRANSPORT` is `http` (default) or `stdio`. In stdio mode the client spawns the process and the key comes from `CLORO_API_KEY`; in http mode keys arrive per request and no key is configured on the server. `PORT` defaults to `8095` and `CLORO_API_URL` to `https://api.cloro.dev`. +OAuth is off unless you set `CLERK_ISSUER`. The hosted server sets it to `https://clerk.cloro.dev`. `api.cloro.dev` accepts only tokens from that issuer, so a self-hosted server that forwards to it must use the same value. `MCP_PUBLIC_URL` (default `https://mcp.cloro.dev`) sets the resource URL in the protected resource metadata. + ## What tools does it expose? | Tool | Endpoint | @@ -64,13 +68,15 @@ MCP_TRANSPORT=stdio CLORO_API_KEY=your_key node dist/index.js | `list_countries` | [`GET /v1/countries`](https://cloro.dev/docs/api-reference/endpoint/countries) | | `list_states` | [`GET /v1/states`](https://cloro.dev/docs/api-reference/endpoint/states) | -The assistant tools take a `prompt` and a `country` (ISO 3166-1 alpha-2), plus an optional `state` for US state-level targeting. `scrape_google_ai_mode` targets sub-country with `location` or `uule` rather than `state`. `scrape_google` runs either from a `query` plus `country`, or from a complete `google.com/search` URL that carries the query and pagination itself, and accepts `include.aioverview` for Google's AI Overview. Every tool takes an optional `include` object for heavier payload fields; leave it unset for the leanest response. +The assistant tools take a `prompt` and a `country` (ISO 3166-1 alpha-2), plus an optional `state` for US state-level targeting. The Google tools take `gl` for the result geography and an optional `hl` for the interface language; `country` still works as a deprecated alias for `gl`. `scrape_google_ai_mode` targets sub-country with `location` or `uule` rather than `state`. `scrape_google` runs either from a `query` plus `gl`, or from a complete `google.com/search` URL that carries the query and pagination itself, and accepts `include.aioverview` for Google's AI Overview. Every tool takes an optional `include` object for heavier payload fields; leave it unset for the leanest response. `scrape_grok` is registered, but Grok availability varies. Check the [provider status page](https://cloro.dev/docs/guides/providers) before relying on it. ## How it works -The server holds no business logic and no secrets. Each request forwards the caller's own API key to `api.cloro.dev` as a Bearer token, so authentication, credit billing, rate limiting and concurrency are all enforced by the API rather than here. In http mode every POST builds a fresh server and transport bound to that caller's key, which is why one deployment serves everyone with no session state. +The server holds no business logic and no secrets. Each request forwards the caller's own credential, an API key or an OAuth access token, to `api.cloro.dev` as a Bearer token, so authentication, credit billing, rate limiting and concurrency are all enforced by the API rather than here. In http mode every POST builds a fresh server and transport bound to that caller's credential, which is why one deployment serves everyone with no session state. + +`initialize` and `tools/list` answer without a credential, so a client can list the tools before sign-in. The first `tools/call` without one returns `401` with a `WWW-Authenticate` header that points at `/.well-known/oauth-protected-resource/mcp`. An OAuth client reads the authorization server from there and runs the flow. The server checks a token's signature against the issuer's public JWKS (code in `src/oauth-tokens/`), so it needs no Clerk secret. Tool input schemas are the same zod schemas the API validates with, vendored into `src/schemas/` from cloro's backend. They are copied rather than imported because that package is not published. The [API reference](https://cloro.dev/docs/api-reference/introduction) is authoritative if the two ever disagree. @@ -86,7 +92,7 @@ The scrape tools call billable endpoints, so yes, at the same rate as the API. ` ### Why is my API key in the URL? -Because several MCP clients cannot set custom headers. If yours can, use the `Authorization` header form instead and keep the key out of the path. +Because several MCP clients cannot set custom headers. If yours can, use the `Authorization` header form instead and keep the key out of the path. If your client supports OAuth, use that and there is no key at all. The URL path accepts API keys only, never OAuth tokens. ### What is the request timeout? diff --git a/package.json b/package.json index 6ebab20..d38809d 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@cloro/mcp-server", - "version": "0.1.0", + "version": "0.2.0", "description": "MCP server for the cloro API. Query ChatGPT, Perplexity, Gemini, Copilot, Grok, Google AI Mode, Google Search and Google News as agent tools.", "license": "MIT", "type": "module", @@ -20,6 +20,7 @@ "@modelcontextprotocol/sdk": "^1.29.0", "cors": "^2.8.5", "express": "^4.20.0", + "jose": "^6.2.3", "zod": "^4.1.5" }, "devDependencies": { diff --git a/server.json b/server.json index 1fd7ca0..0f0ad87 100644 --- a/server.json +++ b/server.json @@ -3,8 +3,12 @@ "name": "dev.cloro/cloro", "title": "cloro", "description": "Scrape AI answer engines and Google Search/News with country and state-level geo-targeting.", - "version": "0.1.0", + "version": "0.2.0", "websiteUrl": "https://cloro.dev/docs/integrations/mcp", + "repository": { + "url": "https://github.com/cloro-dev/mcp-server", + "source": "github" + }, "icons": [ { "src": "https://cloro.dev/favicon-192.png", @@ -24,8 +28,8 @@ "headers": [ { "name": "Authorization", - "description": "Bearer . Create one at https://cloro.dev.", - "isRequired": true, + "description": "Optional. Bearer , for clients that do not run the OAuth flow. Create a key at https://cloro.dev. Clients that support OAuth should omit this and complete the authorization flow instead — the server advertises its authorization server at /.well-known/oauth-protected-resource.", + "isRequired": false, "isSecret": true } ] diff --git a/src/config.ts b/src/config.ts index 78392a4..c526a3a 100644 --- a/src/config.ts +++ b/src/config.ts @@ -9,3 +9,64 @@ export const config = { // Sync scrapes can take up to 5 minutes server-side; leave headroom on top. requestTimeoutMs: 320_000, }; + +/** + * OAuth settings, read on access rather than at import. + * + * Everything above is fixed for the life of the process. These are not: the + * tests flip `CLERK_ISSUER` to exercise both the configured and unconfigured + * server, and freezing them at import would make that impossible without + * module-registry games. + * + * Clerk is the authorization server. `CLERK_ISSUER` unset disables the OAuth + * branch entirely — the protected resource metadata advertises no + * authorization server and API keys stay the only credential — which is what + * makes this deployable ahead of the Clerk configuration. + */ +export const oauth = { + /** Public origin this server is reached at. */ + get publicUrl(): string { + return (process.env.MCP_PUBLIC_URL || "https://mcp.cloro.dev").replace( + /\/$/, + "", + ); + }, + + get issuer(): string | undefined { + return process.env.CLERK_ISSUER; + }, + + get jwksUrl(): string | undefined { + return process.env.CLERK_JWKS_URL; + }, + + /** + * Scopes named in the 401 challenge. `user:org:read` is the load-bearing + * one: it puts the organization picker on Clerk's consent screen and is what + * makes the token carry the `org_id` claim the API resolves an organization + * from. Without it a token verifies but can charge nobody. + */ + get scopes(): string[] { + return (process.env.MCP_OAUTH_SCOPES || "profile email user:org:read") + .split(" ") + .filter(Boolean); + }, + + /** + * The canonical resource URI (RFC 8707). The spec requires this to match the + * URL the user enters in their client exactly, so it is built from the + * public origin and not from the listen address. + */ + get resourceUri(): string { + return `${this.publicUrl}/mcp`; + }, + + /** Where the RFC 9728 document lives, for the 401 `resource_metadata` hint. */ + get resourceMetadataUrl(): string { + return `${this.publicUrl}/.well-known/oauth-protected-resource/mcp`; + }, + + get enabled(): boolean { + return Boolean(this.issuer); + }, +}; diff --git a/src/http.test.ts b/src/http.test.ts index 6edcdd7..5435c42 100644 --- a/src/http.test.ts +++ b/src/http.test.ts @@ -1,6 +1,9 @@ -import { describe, expect, it, afterAll } from "vitest"; +import { SignJWT, exportJWK, generateKeyPair } from "jose"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; -import { buildApp, extractApiKey } from "./http"; +import { config } from "./config"; +import { buildApp, extractCredential, isAnonymousRequest } from "./http"; +import { resetOAuthVerifier } from "./lib/oauth"; import type { Request } from "express"; import type { Server } from "node:http"; @@ -12,41 +15,115 @@ const fakeReq = (authorization?: string, pathKey?: string): Request => params: pathKey !== undefined ? { apiKey: pathKey } : {}, }) as unknown as Request; -describe("extractApiKey", () => { +describe("extractCredential", () => { it("parses a Bearer header", () => { - expect(extractApiKey(fakeReq("Bearer sk_live_abc"))).toBe("sk_live_abc"); + expect(extractCredential(fakeReq("Bearer sk_live_abc"))).toBe( + "sk_live_abc", + ); }); it("is case-insensitive on the scheme", () => { - expect(extractApiKey(fakeReq("bearer sk_live_abc"))).toBe("sk_live_abc"); + expect(extractCredential(fakeReq("bearer sk_live_abc"))).toBe( + "sk_live_abc", + ); }); it("parses an ApiKey header", () => { - expect(extractApiKey(fakeReq("ApiKey sk_live_abc"))).toBe("sk_live_abc"); + expect(extractCredential(fakeReq("ApiKey sk_live_abc"))).toBe( + "sk_live_abc", + ); }); it("accepts a bare key with surrounding whitespace", () => { - expect(extractApiKey(fakeReq(" sk_live_abc "))).toBe("sk_live_abc"); + expect(extractCredential(fakeReq(" sk_live_abc "))).toBe("sk_live_abc"); }); it("falls back to the path parameter", () => { - expect(extractApiKey(fakeReq(undefined, "sk_live_path"))).toBe( + expect(extractCredential(fakeReq(undefined, "sk_live_path"))).toBe( "sk_live_path", ); }); it("prefers the header over the path parameter", () => { - expect(extractApiKey(fakeReq("Bearer sk_header", "sk_path"))).toBe( + expect(extractCredential(fakeReq("Bearer sk_header", "sk_path"))).toBe( "sk_header", ); }); it("returns undefined when neither is present", () => { - expect(extractApiKey(fakeReq())).toBeUndefined(); + expect(extractCredential(fakeReq())).toBeUndefined(); + }); + + it("carries an OAuth token through unchanged", () => { + expect(extractCredential(fakeReq("Bearer eyJhbG.eyJzdWI.sig"))).toBe( + "eyJhbG.eyJzdWI.sig", + ); }); }); -describe("HTTP transport", () => { +describe("isAnonymousRequest", () => { + it("allows the handshake and the listings", () => { + expect(isAnonymousRequest({ method: "initialize" })).toBe(true); + expect(isAnonymousRequest({ method: "tools/list" })).toBe(true); + expect(isAnonymousRequest({ method: "ping" })).toBe(true); + }); + + it("refuses anything that reaches the cloro API", () => { + expect(isAnonymousRequest({ method: "tools/call" })).toBe(false); + expect(isAnonymousRequest({ method: "resources/read" })).toBe(false); + }); + + it("refuses methods this server does not implement", () => { + // The server registers tools only. A client has no reason to call these, + // because the handshake never advertises the capability -- so they are not + // on the anonymous list, and a caller that tries one needs a credential + // like any other unrecognised method. + expect(isAnonymousRequest({ method: "prompts/list" })).toBe(false); + expect(isAnonymousRequest({ method: "resources/list" })).toBe(false); + }); + + it("refuses a batch that smuggles a tool call past a handshake", () => { + expect( + isAnonymousRequest([{ method: "initialize" }, { method: "tools/call" }]), + ).toBe(false); + }); + + it("allows a batch only when every entry is a discovery method", () => { + expect( + isAnonymousRequest([{ method: "initialize" }, { method: "tools/list" }]), + ).toBe(true); + }); + + it("refuses a body with no method, an empty batch, or a non-object", () => { + expect(isAnonymousRequest({})).toBe(false); + expect(isAnonymousRequest([])).toBe(false); + expect(isAnonymousRequest(undefined)).toBe(false); + expect(isAnonymousRequest("initialize")).toBe(false); + }); +}); + +const initializeBody = JSON.stringify({ + jsonrpc: "2.0", + id: 1, + method: "initialize", + params: { + protocolVersion: "2025-03-26", + capabilities: {}, + clientInfo: { name: "test", version: "0" }, + }, +}); + +const toolsCallBody = JSON.stringify({ + jsonrpc: "2.0", + id: 2, + method: "tools/call", + params: { + name: "scrape_chatgpt", + arguments: { prompt: "hi", country: "US" }, + }, +}); + +describe("HTTP transport (OAuth not configured)", () => { const { app, beginShutdown } = buildApp(); const server: Server = app.listen(0); const port = (server.address() as AddressInfo).port; @@ -56,26 +133,51 @@ describe("HTTP transport", () => { server.close(); }); - const initializeBody = JSON.stringify({ - jsonrpc: "2.0", - id: 1, - method: "initialize", - params: { - protocolVersion: "2025-03-26", - capabilities: {}, - clientInfo: { name: "test", version: "0" }, - }, + it("completes the handshake with no credential at all", async () => { + const res = await fetch(`${base}/mcp`, { + method: "POST", + headers: { + "content-type": "application/json", + accept: "application/json, text/event-stream", + }, + body: initializeBody, + }); + expect(res.status).toBe(200); + expect(await res.text()).toContain('"name":"cloro"'); + }); + + it("lists tools with no credential, so a client can see what is on offer", async () => { + const res = await fetch(`${base}/mcp`, { + method: "POST", + headers: { + "content-type": "application/json", + accept: "application/json, text/event-stream", + }, + body: JSON.stringify({ jsonrpc: "2.0", id: 3, method: "tools/list" }), + }); + expect(res.status).toBe(200); + expect(await res.text()).toContain("scrape_chatgpt"); }); - it("rejects a POST without an API key with a 401 JSON-RPC error", async () => { + it("rejects a tool call without a credential, and sends no challenge", async () => { + // With no authorization server configured there is nowhere to send the + // client, so a challenge would name a document that advertises nothing. const res = await fetch(`${base}/mcp`, { method: "POST", headers: { "content-type": "application/json" }, - body: initializeBody, + body: toolsCallBody, }); expect(res.status).toBe(401); - const body = (await res.json()) as { error: { message: string } }; - expect(body.error.message).toContain("Missing API key"); + expect(res.headers.get("www-authenticate")).toBeNull(); + }); + + it("omits authorization_servers from the metadata", async () => { + const res = await fetch(`${base}/.well-known/oauth-protected-resource`); + expect(res.status).toBe(200); + const body = (await res.json()) as Record; + expect(body.resource).toBe("https://mcp.cloro.dev/mcp"); + expect(body).not.toHaveProperty("authorization_servers"); + expect(body).not.toHaveProperty("scopes_supported"); }); it("initializes with a path-embedded key", async () => { @@ -96,7 +198,6 @@ describe("HTTP transport", () => { expect(res.status).toBe(405); }); - it("serves /health ok, then 503 once shutdown begins", async () => { const healthy = await fetch(`${base}/health`); expect(healthy.status).toBe(200); @@ -107,3 +208,221 @@ describe("HTTP transport", () => { expect(draining.status).toBe(503); }); }); + +describe("HTTP transport (OAuth configured)", () => { + const ISSUER = "https://clerk.cloro.test"; + const JWKS_URL = `${ISSUER}/.well-known/jwks.json`; + + let server: Server; + let base: string; + type PrivateKey = Awaited>["privateKey"]; + let signingKey: PrivateKey; + let realFetch: typeof fetch; + let jwksDown = false; + /** Authorization header of the last request the tool layer made upstream. */ + let upstreamAuth: string | undefined; + + async function mint( + claims: { + org_id?: string | null; + expiresIn?: string; + /** `null` omits the claim. Defaults to this server's resource URI. */ + aud?: string | null; + } = {}, + ): Promise { + const { + org_id = "org_test", + expiresIn = "1h", + aud = "https://mcp.cloro.dev/mcp", + } = claims; + const jwt = new SignJWT({ + ...(org_id === null ? {} : { org_id }), + scope: "profile email user:org:read", + }) + .setProtectedHeader({ alg: "RS256", kid: "test-key" }) + .setIssuer(ISSUER) + .setSubject("user_test") + .setIssuedAt() + .setExpirationTime(expiresIn); + if (aud !== null) jwt.setAudience(aud); + return jwt.sign(signingKey); + } + + beforeAll(async () => { + vi.stubEnv("CLERK_ISSUER", ISSUER); + resetOAuthVerifier(); + + const pair = await generateKeyPair("RS256", { extractable: true }); + signingKey = pair.privateKey; + const jwk = await exportJWK(pair.publicKey); + const jwks = { + keys: [{ ...jwk, kid: "test-key", alg: "RS256", use: "sig" }], + }; + + // Nothing leaves the process: the JWKS and the cloro API are served from + // memory, and only the loopback server under test gets the real fetch. + realFetch = globalThis.fetch; + vi.stubGlobal( + "fetch", + async ( + input: Parameters[0], + init?: Parameters[1], + ) => { + const url = String(input); + if (url === JWKS_URL) { + if (jwksDown) throw new TypeError("fetch failed"); + return new Response(JSON.stringify(jwks), { + status: 200, + headers: { "content-type": "application/json" }, + }); + } + if (url.startsWith(config.apiUrl)) { + upstreamAuth = (init?.headers as Record | undefined) + ?.authorization; + return new Response( + JSON.stringify({ success: true, result: { text: "answer" } }), + { status: 200, headers: { "content-type": "application/json" } }, + ); + } + if (url.startsWith(base)) { + return realFetch(input, init); + } + throw new Error(`unexpected fetch: ${url}`); + }, + ); + + const { app } = buildApp(); + server = app.listen(0); + base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; + }); + + afterAll(() => { + server.close(); + vi.unstubAllEnvs(); + vi.unstubAllGlobals(); + resetOAuthVerifier(); + }); + + const callTool = (token?: string) => + realFetch(`${base}/mcp`, { + method: "POST", + headers: { + "content-type": "application/json", + accept: "application/json, text/event-stream", + ...(token ? { authorization: `Bearer ${token}` } : {}), + }, + body: toolsCallBody, + }); + + it("advertises the authorization server and scopes", async () => { + const res = await realFetch( + `${base}/.well-known/oauth-protected-resource/mcp`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as Record; + expect(body.resource).toBe("https://mcp.cloro.dev/mcp"); + expect(body.authorization_servers).toEqual([ISSUER]); + expect(body.scopes_supported).toEqual([ + "profile", + "email", + "user:org:read", + ]); + }); + + it("serves the same document at both well-known paths", async () => { + const [bare, suffixed] = await Promise.all([ + realFetch(`${base}/.well-known/oauth-protected-resource`), + realFetch(`${base}/.well-known/oauth-protected-resource/mcp`), + ]); + expect(await bare.json()).toEqual(await suffixed.json()); + }); + + it("challenges an unauthenticated tool call with WWW-Authenticate", async () => { + const res = await callTool(); + expect(res.status).toBe(401); + + const challenge = res.headers.get("www-authenticate") ?? ""; + expect(challenge).toContain("Bearer "); + expect(challenge).toContain('error="invalid_token"'); + expect(challenge).toContain( + 'resource_metadata="https://mcp.cloro.dev/.well-known/oauth-protected-resource/mcp"', + ); + expect(challenge).toContain('scope="profile email user:org:read"'); + }); + + it("still completes the handshake unchallenged", async () => { + const res = await realFetch(`${base}/mcp`, { + method: "POST", + headers: { + "content-type": "application/json", + accept: "application/json, text/event-stream", + }, + body: initializeBody, + }); + expect(res.status).toBe(200); + }); + + it("accepts a valid token and forwards it to the API verbatim", async () => { + const token = await mint(); + upstreamAuth = undefined; + const res = await callTool(token); + expect(res.status).toBe(200); + expect(await res.text()).not.toContain('"isError":true'); + expect(upstreamAuth).toBe(`Bearer ${token}`); + }); + + it("answers 503 with no challenge when the JWKS cannot be fetched", async () => { + // A fresh verifier has no cached keys, so the first token forces a fetch. + resetOAuthVerifier(); + jwksDown = true; + try { + const res = await callTool(await mint()); + expect(res.status).toBe(503); + expect(res.headers.get("www-authenticate")).toBeNull(); + } finally { + jwksDown = false; + resetOAuthVerifier(); + } + }); + + it("challenges an expired token", async () => { + const res = await callTool(await mint({ expiresIn: "-1h" })); + expect(res.status).toBe(401); + const challenge = res.headers.get("www-authenticate") ?? ""; + expect(challenge).toContain('error="invalid_token"'); + expect(challenge).toContain("(expired)"); + }); + + it("challenges a token minted for another resource", async () => { + // The MCP spec requires a resource server to accept only tokens issued + // for it. A token with no `aud` cannot prove that, so it fails the same + // way as one naming a different resource. + for (const aud of ["https://other.example/mcp", null]) { + const res = await callTool(await mint({ aud })); + expect(res.status, `aud=${aud}`).toBe(401); + expect(res.headers.get("www-authenticate")).toContain("(wrong_audience)"); + } + }); + + it("challenges a token with no organization using the RFC 6750 code", async () => { + // The reason is named in error_description, never in `error`: RFC 6750 + // registers three codes and a client may key re-authentication on them, + // so a custom `missing_organization` code could be treated as terminal. + const res = await callTool(await mint({ org_id: null })); + expect(res.status).toBe(401); + const challenge = res.headers.get("www-authenticate") ?? ""; + expect(challenge).toContain('error="invalid_token"'); + expect(challenge).not.toContain('error="missing_organization"'); + expect(challenge).toContain("(missing_organization)"); + }); + + it("forwards an API key without verifying it as a token", async () => { + const key = `sk_live_${"a1b2c3d4".repeat(4)}`; + upstreamAuth = undefined; + const res = await callTool(key); + // Not challenged: an sk_ key is not a JWT, so it goes upstream untouched + // and the API is the one that decides whether it is valid. + expect(res.status).toBe(200); + expect(upstreamAuth).toBe(`Bearer ${key}`); + }); +}); diff --git a/src/http.ts b/src/http.ts index 4e64cf9..3a997f1 100644 --- a/src/http.ts +++ b/src/http.ts @@ -1,18 +1,26 @@ import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js"; import { logger } from "./lib/logger"; +import { looksLikeOAuthToken } from "./oauth-tokens"; import cors from "cors"; import express, { Express, Request, Response } from "express"; -import { config } from "./config"; +import { config, oauth } from "./config"; +import { buildChallenge, getOAuthVerifier } from "./lib/oauth"; +import { buildProtectedResourceMetadata } from "./lib/protected-resource-metadata"; import { buildServer } from "./server"; /** - * Extract the caller's cloro API key. Two forms, mirroring the public API + * Extract the caller's credential. Three forms, mirroring the public API * (and SerpApi's hosted MCP) so clients that can't set headers still work: * 1. `Authorization: Bearer sk_...` (or `ApiKey sk_...`, or the bare key) - * 2. URL path: POST //mcp + * 2. `Authorization: Bearer ` + * 3. URL path: POST //mcp + * + * The path form stays API-key-only. It exists for clients that cannot set a + * header, and an OAuth client is not one of those — the spec forbids putting + * an access token in a URL, because URLs end up in logs and history. */ -export function extractApiKey(req: Request): string | undefined { +export function extractCredential(req: Request): string | undefined { const header = req.headers.authorization; if (header) { const match = /^(?:Bearer|ApiKey)\s+(.+)$/i.exec(header.trim()); @@ -29,25 +37,125 @@ function jsonRpcError(res: Response, status: number, message: string): void { }); } +/** + * Refuse at the HTTP layer with the challenge the MCP spec defines. + * + * This has to happen before the JSON-RPC message reaches the SDK: once a tool + * handler is running its return value is already destined for a 200, and a 200 + * produces no Connect card. + */ +function unauthorized( + res: Response, + message: string, + description?: string, +): void { + if (oauth.enabled) { + res.set("WWW-Authenticate", buildChallenge(description)); + } + jsonRpcError(res, 401, message); +} + +/** + * JSON-RPC methods answerable with no credential. + * + * Handshake and listing describe the server; they read nothing and charge + * nothing. Requiring a credential for them made `initialize` return 401, which + * reads to any client -- and to any readiness audit -- as a server that is not + * running: there is no way to tell a credential problem from a dead + * deployment when the handshake itself is what fails. Everything that reaches + * the cloro API still needs one, and `tools/call` is not on this list. + * + * This is also exactly the lazy-authentication shape Claude expects: a client + * connects, lists the tools, and is challenged only when it first calls one. + * + * The list covers what this server implements and nothing else. It registers + * tools only, so `prompts/list` and `resources/list` are not on it: a client + * has no reason to call a method whose capability the handshake never + * advertised, and adding one here would be guessing at a client that does. + */ +const ANONYMOUS_METHODS = new Set([ + "initialize", + "notifications/initialized", + "ping", + "tools/list", +]); + +/** + * True when every request in the body is a discovery method. A batch is + * all-or-nothing on purpose: letting a batch through because its first entry + * is `initialize` would carry an unauthenticated `tools/call` with it. + */ +export function isAnonymousRequest(body: unknown): boolean { + const entries = Array.isArray(body) ? body : [body]; + if (entries.length === 0) return false; + return entries.every( + (entry) => + typeof entry === "object" && + entry !== null && + typeof (entry as { method?: unknown }).method === "string" && + ANONYMOUS_METHODS.has((entry as { method: string }).method), + ); +} + +const MISSING_CREDENTIAL_MESSAGE = + "Authentication required. Connect with OAuth, or send a cloro API key as an Authorization: Bearer header."; + /** * Stateless Streamable HTTP: every POST gets a fresh server + transport bound - * to the caller's API key, so one deployment serves all customers with no + * to the caller's credential, so one deployment serves all customers with no * session state. */ async function handleMcpRequest(req: Request, res: Response): Promise { - const apiKey = extractApiKey(req); - if (!apiKey) { - // Never log header or path contents here — both can carry API keys. - logger.warn("MCP request rejected: missing API key", "MCP"); - jsonRpcError( - res, - 401, - "Missing API key. Send it as an Authorization: Bearer header, or use the //mcp path.", - ); - return; + const credential = extractCredential(req); + const anonymous = isAnonymousRequest(req.body); + + if (!credential) { + if (!anonymous) { + // Never log header or path contents here — both can carry credentials. + logger.warn("MCP request rejected: no credential", "MCP"); + unauthorized(res, MISSING_CREDENTIAL_MESSAGE); + return; + } + } else if (looksLikeOAuthToken(credential)) { + // Verify here rather than letting the API do it alone. A token that fails + // upstream comes back inside a tool result, which cannot carry the 401 + // that drives re-authentication. + const verify = getOAuthVerifier(); + if (!verify) { + logger.warn("OAuth token presented but OAuth is not configured", "MCP"); + unauthorized(res, MISSING_CREDENTIAL_MESSAGE); + return; + } + + const result = await verify(credential); + if (!result.ok && result.reason === "unavailable") { + // Nothing is known about the token. No challenge: a 401 would make the + // client discard it and re-run the OAuth flow, which fails the same way. + logger.error("OAuth token verification unavailable", "MCP"); + jsonRpcError( + res, + 503, + "OAuth token verification is temporarily unavailable. Retry shortly.", + ); + return; + } + if (!result.ok) { + logger.warn("MCP request rejected: OAuth token invalid", "MCP", { + reason: result.reason, + }); + const message = + result.reason === "missing_organization" + ? "This OAuth token is not bound to an organization. Reconnect and grant organization access." + : "Invalid or expired OAuth access token."; + unauthorized(res, message, `${message} (${result.reason})`); + return; + } } - const server = buildServer(apiKey); + // An anonymous handshake gets a server whose tools are wired to an empty + // credential. Listing them never calls the API, and `tools/call` cannot + // reach here without one, so the empty string is never sent anywhere. + const server = buildServer(credential ?? ""); const transport = new StreamableHTTPServerTransport({ sessionIdGenerator: undefined, }); @@ -78,7 +186,9 @@ function methodNotAllowed(_req: Request, res: Response): void { /** The Express app plus the switch that flips /health to 503 during drain. */ export function buildApp(): { app: Express; beginShutdown: () => void } { const app = express(); - app.use(cors()); + // The challenge header is the whole authorization signal, and a browser + // client cannot read it unless it is explicitly exposed. + app.use(cors({ exposedHeaders: ["WWW-Authenticate"] })); app.use(express.json({ limit: "1mb" })); let isShuttingDown = false; @@ -90,6 +200,25 @@ export function buildApp(): { app: Express; beginShutdown: () => void } { res.status(200).json({ status: "ok" }); }); + // RFC 9728 protected resource metadata. Unauthenticated: its job is to tell + // a client with no credentials how to get one. + // + // Both paths serve the same document. RFC 9728 section 3.1 has a client try + // the path-suffixed form first when the resource URL has a path component, + // and ours is `/mcp` — so the suffixed path is the one Claude actually + // probes, and the bare path is the fallback for clients that don't. + for (const path of [ + "/.well-known/oauth-protected-resource", + "/.well-known/oauth-protected-resource/mcp", + ]) { + app.get(path, (_req, res) => { + res + .type("application/json") + .set("Cache-Control", "public, max-age=3600") + .json(buildProtectedResourceMetadata()); + }); + } + for (const path of ["/mcp", "/:apiKey/mcp"]) { app.post(path, (req, res) => { void handleMcpRequest(req, res); @@ -111,6 +240,12 @@ export function runHttpServer(): void { const server = app.listen(config.port, () => { logger.info(`cloro MCP server listening on port ${config.port}`, "MCP"); + logger.info( + oauth.enabled + ? `OAuth enabled — authorization server ${oauth.issuer}` + : "OAuth disabled (CLERK_ISSUER unset) — API keys only", + "MCP", + ); }); // Sync scrapes can run up to ~5 min; the client AbortSignal (requestTimeoutMs) diff --git a/src/lib/oauth.ts b/src/lib/oauth.ts new file mode 100644 index 0000000..2cee9e0 --- /dev/null +++ b/src/lib/oauth.ts @@ -0,0 +1,86 @@ +import { logger } from "./logger"; +import { createOAuthTokenVerifier } from "../oauth-tokens"; + +import { oauth } from "../config"; + +import type { OAuthTokenVerifier } from "../oauth-tokens"; + +/** + * OAuth support for the hosted MCP server. + * + * This server is a resource server, not an authorization server — Clerk issues + * the tokens. Its two jobs are to refuse an unauthenticated call in the exact + * shape the MCP spec defines, and to reject a bad token here rather than + * forwarding it to the API and having to translate an upstream failure back + * into a transport-level 401 from inside a tool handler. + * + * Verification is signature-only against Clerk's public JWKS, so this app + * still holds no secret. + */ + +let cached: OAuthTokenVerifier | null | undefined; + +export function getOAuthVerifier(): OAuthTokenVerifier | null { + if (cached !== undefined) return cached; + + const issuer = oauth.issuer; + if (!issuer) { + cached = null; + return cached; + } + + try { + cached = createOAuthTokenVerifier({ + issuer, + jwksUrl: oauth.jwksUrl, + audience: oauth.resourceUri, + }); + } catch (error) { + // A malformed issuer or JWKS URL disables OAuth rather than crashing the + // first request that carries a token. + logger.error( + "OAuth disabled: CLERK_ISSUER / CLERK_JWKS_URL is not a valid URL", + "MCP", + { issuer, jwksUrl: oauth.jwksUrl }, + error, + ); + cached = null; + } + return cached; +} + +/** Test seam. */ +export function resetOAuthVerifier(): void { + cached = undefined; +} + +/** + * The `WWW-Authenticate` challenge that turns a refusal into a Connect card. + * + * A 200 wrapping `isError: true` is an application-level tool failure: the + * client hands the text to the model and moves on, with no authorization + * prompt. Only a transport-level 401 carrying this header makes a client stop, + * run the OAuth flow, and retry the call. + * + * `error` is always `invalid_token`. RFC 6750 section 3.1 registers exactly + * three codes (`invalid_request`, `invalid_token`, `insufficient_scope`) and a + * client is entitled to key its re-authentication on them; a custom code such + * as `wrong_audience` is a spec violation that a strict client may treat as + * terminal. Every failure this server can name — expired, wrong audience, no + * organization — is cured the same way, by running the flow again, so they + * share the code and differ only in `error_description`. + */ +export function buildChallenge( + description = "Authentication required for this tool", +): string { + const params = [ + `error="invalid_token"`, + `error_description="${description.replace(/"/g, "'")}"`, + `resource_metadata="${oauth.resourceMetadataUrl}"`, + ]; + const scopes = oauth.scopes; + if (scopes.length > 0) { + params.push(`scope="${scopes.join(" ")}"`); + } + return `Bearer ${params.join(", ")}`; +} diff --git a/src/lib/protected-resource-metadata.ts b/src/lib/protected-resource-metadata.ts new file mode 100644 index 0000000..148242a --- /dev/null +++ b/src/lib/protected-resource-metadata.ts @@ -0,0 +1,40 @@ +import { oauth } from "../config"; + +/** + * RFC 9728 protected resource metadata for the hosted MCP server. + * + * The MCP authorization spec builds on RFC 9728, and a client may probe this + * well-known path directly rather than waiting to be pointed at it. That is + * the reason this document exists and its counterpart on the public API does + * not: an OpenAPI `securitySchemes.bearerAuth` entry already tells an HTTP + * client what api.cloro.dev wants, and nothing probes a `.well-known` path + * for a plain REST API unprompted. MCP has no OpenAPI to carry that. + * + * `authorization_servers` names the cloro Clerk instance. A client reads it + * after a 401, fetches Clerk's RFC 8414 metadata to find `/authorize` and + * `/token`, and runs the flow from there — nothing about the authorization + * server is hard-coded in the client. Only the first entry is used, so this + * stays a single-element list. + * + * The document degrades on purpose when `CLERK_ISSUER` is unset: with no + * authorization server configured there is nothing to point a client at, and + * advertising one that does not answer is worse than advertising none. API + * keys keep working either way. + */ +export function buildProtectedResourceMetadata(): Record { + const metadata: Record = { + resource: oauth.resourceUri, + resource_name: "cloro MCP server", + resource_documentation: "https://cloro.dev/docs/integrations/mcp", + bearer_methods_supported: ["header"], + resource_policy_uri: "https://cloro.dev/privacy/", + resource_tos_uri: "https://cloro.dev/terms/", + }; + + if (oauth.enabled) { + metadata.authorization_servers = [oauth.issuer]; + metadata.scopes_supported = oauth.scopes; + } + + return metadata; +} diff --git a/src/oauth-tokens/classify.ts b/src/oauth-tokens/classify.ts new file mode 100644 index 0000000..6c8305d --- /dev/null +++ b/src/oauth-tokens/classify.ts @@ -0,0 +1,57 @@ +/** + * Both verifiers in this package do the same JWKS signature check, so they + * fail in the same ways and say so with the same words. + */ + +/** + * Skew allowed on `exp` / `nbf`. Clerk session tokens live 60 seconds and + * carry `nbf`, so a pod whose clock lags Clerk's by a second would otherwise + * refuse every fresh token. Matches the default in Clerk's own verifier. + */ +export const CLOCK_TOLERANCE = "5s"; + +export type VerifyFailure = + | "malformed" + | "invalid_token" + | "expired" + | "wrong_issuer" + /** + * The issuer's key set could not be fetched, so nothing is known about the + * token. Callers must answer this with a 5xx and no re-authentication + * challenge: a fresh token would fail the same way. + */ + | "unavailable"; + +export function classifyVerifyError(error: unknown): VerifyFailure { + const code = + typeof error === "object" && error !== null && "code" in error + ? (error as { code: unknown }).code + : undefined; + + switch (code) { + case undefined: + case "ERR_JOSE_GENERIC": + case "ERR_JWKS_TIMEOUT": + // A code-less error is the network failure fetch itself threw; the + // generic code is jose reporting a non-200 or unparsable JWKS response. + return "unavailable"; + case "ERR_JWT_EXPIRED": + return "expired"; + case "ERR_JWT_CLAIM_VALIDATION_FAILED": + return isIssuerClaim(error) ? "wrong_issuer" : "invalid_token"; + case "ERR_JWS_INVALID": + case "ERR_JWT_INVALID": + return "malformed"; + default: + return "invalid_token"; + } +} + +function isIssuerClaim(error: unknown): boolean { + return ( + typeof error === "object" && + error !== null && + "claim" in error && + (error as { claim: unknown }).claim === "iss" + ); +} diff --git a/src/oauth-tokens/credential.test.ts b/src/oauth-tokens/credential.test.ts new file mode 100644 index 0000000..2f9eea1 --- /dev/null +++ b/src/oauth-tokens/credential.test.ts @@ -0,0 +1,43 @@ +import { describe, expect, it } from "vitest"; + +import { looksLikeApiKey, looksLikeOAuthToken } from "./credential"; + +const API_KEY = `sk_live_${"a1b2c3d4".repeat(4)}`; +const JWT = "eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJ1c2VyXzEifQ.c2lnbmF0dXJl"; + +describe("looksLikeApiKey", () => { + it("accepts live and test keys", () => { + expect(looksLikeApiKey(API_KEY)).toBe(true); + expect(looksLikeApiKey(API_KEY.replace("live", "test"))).toBe(true); + }); + + it("rejects the wrong length, uppercase hex, and a bare prefix", () => { + expect(looksLikeApiKey("sk_live_deadbeef")).toBe(false); + expect(looksLikeApiKey(`sk_live_${"A1B2C3D4".repeat(4)}`)).toBe(false); + expect(looksLikeApiKey("sk_live_")).toBe(false); + }); +}); + +describe("looksLikeOAuthToken", () => { + it("accepts a three-segment JWT", () => { + expect(looksLikeOAuthToken(JWT)).toBe(true); + }); + + it("rejects an API key", () => { + expect(looksLikeOAuthToken(API_KEY)).toBe(false); + }); + + it("rejects anything that is not three segments", () => { + expect(looksLikeOAuthToken("not.a")).toBe(false); + expect(looksLikeOAuthToken("a.b.c.d")).toBe(false); + expect(looksLikeOAuthToken("")).toBe(false); + }); + + it("never classifies the same credential as both", () => { + for (const credential of [API_KEY, JWT, "garbage"]) { + expect( + looksLikeApiKey(credential) && looksLikeOAuthToken(credential), + ).toBe(false); + } + }); +}); diff --git a/src/oauth-tokens/credential.ts b/src/oauth-tokens/credential.ts new file mode 100644 index 0000000..d81bec2 --- /dev/null +++ b/src/oauth-tokens/credential.ts @@ -0,0 +1,27 @@ +/** + * Tells a cloro API key apart from an OAuth access token. + * + * Both arrive in the same `Authorization: Bearer ...` position, so every + * entry point that accepts either has to decide which one it is holding + * before it can validate it. The two formats do not overlap: a key is + * `sk_live_` / `sk_test_` plus 32 hex characters, and a JWT is three + * base64url segments separated by dots. Nothing is a valid instance of both. + * + * The check is deliberately shape-only. It says which validator to run, not + * whether the credential is good — `looksLikeOAuthToken` returning true means + * "verify this as a JWT", and that verification is still free to reject it. + */ + +/** `sk_live_` / `sk_test_` plus the 32 hex chars `generateApiKey` produces. */ +const API_KEY_PATTERN = /^sk_(live|test)_[a-f0-9]{32}$/; + +/** Three non-empty base64url segments. */ +const JWT_PATTERN = /^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/; + +export function looksLikeApiKey(credential: string): boolean { + return API_KEY_PATTERN.test(credential); +} + +export function looksLikeOAuthToken(credential: string): boolean { + return !looksLikeApiKey(credential) && JWT_PATTERN.test(credential); +} diff --git a/src/oauth-tokens/index.ts b/src/oauth-tokens/index.ts new file mode 100644 index 0000000..9b16ad0 --- /dev/null +++ b/src/oauth-tokens/index.ts @@ -0,0 +1,4 @@ +export * from "./credential"; +export * from "./resource"; +export * from "./session"; +export * from "./verify"; diff --git a/src/oauth-tokens/resource.ts b/src/oauth-tokens/resource.ts new file mode 100644 index 0000000..31e62fd --- /dev/null +++ b/src/oauth-tokens/resource.ts @@ -0,0 +1,7 @@ +/** + * The canonical resource URI (RFC 8707) of the hosted MCP server, and so the + * `aud` every access token must carry. Fixed rather than configured: the API + * accepts tokens for exactly one resource server, and having two services + * read it from two env vars is how they drift apart. + */ +export const MCP_RESOURCE_URI = "https://mcp.cloro.dev/mcp"; diff --git a/src/oauth-tokens/session.test.ts b/src/oauth-tokens/session.test.ts new file mode 100644 index 0000000..b9c6f72 --- /dev/null +++ b/src/oauth-tokens/session.test.ts @@ -0,0 +1,162 @@ +import { SignJWT, exportJWK, generateKeyPair } from "jose"; +import { afterEach, beforeAll, describe, expect, it, vi } from "vitest"; + +import { createSessionTokenVerifier, looksLikeSessionToken } from "./session"; + +const ISSUER = "https://clerk.cloro.dev"; +const JWKS_URL = `${ISSUER}/.well-known/jwks.json`; + +let signingKey: CryptoKey; +/** A second, unrelated key pair — its tokens must never verify. */ +let foreignKey: CryptoKey; + +interface Claims { + sub?: string; + /** `null` omits the claim. */ + sid?: string | null; + /** The version 2 organization claim. `null` omits it. */ + o?: { id: string } | null; + /** The version 1 organization claim. */ + org_id?: string; + iss?: string; + expiresIn?: string; + /** `nbf` claim, as an offset from now. */ + notBefore?: string; + key?: CryptoKey; +} + +async function mint(claims: Claims = {}): Promise { + const { + sub = "user_123", + sid = "sess_789", + o = { id: "org_456" }, + org_id, + iss = ISSUER, + expiresIn = "1h", + notBefore, + key = signingKey, + } = claims; + + const jwt = new SignJWT({ + ...(sid === null ? {} : { sid }), + ...(o === null ? {} : { o, v: 2 }), + ...(org_id === undefined ? {} : { org_id }), + }) + .setProtectedHeader({ alg: "RS256", kid: "test-key" }) + .setIssuer(iss) + .setSubject(sub) + .setIssuedAt() + .setExpirationTime(expiresIn); + if (notBefore !== undefined) jwt.setNotBefore(notBefore); + return jwt.sign(key); +} + +beforeAll(async () => { + const pair = await generateKeyPair("RS256", { extractable: true }); + const foreign = await generateKeyPair("RS256", { extractable: true }); + signingKey = pair.privateKey; + foreignKey = foreign.privateKey; + + const jwk = await exportJWK(pair.publicKey); + const jwks = { + keys: [{ ...jwk, kid: "test-key", alg: "RS256", use: "sig" }], + }; + + vi.stubGlobal("fetch", async (input: RequestInfo | URL) => { + if (String(input) !== JWKS_URL) { + throw new Error(`unexpected fetch: ${String(input)}`); + } + return new Response(JSON.stringify(jwks), { + status: 200, + headers: { "content-type": "application/json" }, + }); + }); +}); + +afterEach(() => { + vi.clearAllMocks(); +}); + +describe("looksLikeSessionToken", () => { + it("is true for a token with a session id", async () => { + expect(looksLikeSessionToken(await mint())).toBe(true); + }); + + it("is false for a token without one, such as an OAuth access token", async () => { + expect(looksLikeSessionToken(await mint({ sid: null }))).toBe(false); + }); + + it("is false for a value that is not a JWT", () => { + expect(looksLikeSessionToken("sk_live_0123456789abcdef")).toBe(false); + }); +}); + +describe("createSessionTokenVerifier", () => { + it("accepts a version 2 token and returns the principal", async () => { + const verify = createSessionTokenVerifier({ issuer: ISSUER }); + + expect(await verify(await mint())).toEqual({ + ok: true, + principal: { + userId: "user_123", + organizationId: "org_456", + sessionId: "sess_789", + }, + }); + }); + + it("reads the organization from org_id on a version 1 token", async () => { + const verify = createSessionTokenVerifier({ issuer: ISSUER }); + const result = await verify(await mint({ o: null, org_id: "org_v1" })); + + expect(result).toEqual({ + ok: true, + principal: { + userId: "user_123", + organizationId: "org_v1", + sessionId: "sess_789", + }, + }); + }); + + it("derives the JWKS URL from the issuer and tolerates a trailing slash", async () => { + const verify = createSessionTokenVerifier({ issuer: `${ISSUER}/` }); + expect((await verify(await mint())).ok).toBe(true); + }); + + it("rejects an expired token", async () => { + const verify = createSessionTokenVerifier({ issuer: ISSUER }); + const result = await verify(await mint({ expiresIn: "-1h" })); + expect(result).toEqual({ ok: false, reason: "expired" }); + }); + + it("tolerates a few seconds of clock skew on nbf, but not more", async () => { + const verify = createSessionTokenVerifier({ issuer: ISSUER }); + expect((await verify(await mint({ notBefore: "3s" }))).ok).toBe(true); + const tooFar = await verify(await mint({ notBefore: "30s" })); + expect(tooFar).toEqual({ ok: false, reason: "invalid_token" }); + }); + + it("rejects a token from another issuer", async () => { + const verify = createSessionTokenVerifier({ issuer: ISSUER }); + const result = await verify(await mint({ iss: "https://evil.example" })); + expect(result).toEqual({ ok: false, reason: "wrong_issuer" }); + }); + + it("rejects a token signed by an unknown key", async () => { + const verify = createSessionTokenVerifier({ issuer: ISSUER }); + expect((await verify(await mint({ key: foreignKey }))).ok).toBe(false); + }); + + it("rejects a token with no organization — there is no account to charge", async () => { + const verify = createSessionTokenVerifier({ issuer: ISSUER }); + const result = await verify(await mint({ o: null })); + expect(result).toEqual({ ok: false, reason: "missing_organization" }); + }); + + it("rejects a token with no session id", async () => { + const verify = createSessionTokenVerifier({ issuer: ISSUER }); + const result = await verify(await mint({ sid: null })); + expect(result).toEqual({ ok: false, reason: "malformed" }); + }); +}); diff --git a/src/oauth-tokens/session.ts b/src/oauth-tokens/session.ts new file mode 100644 index 0000000..8005bb9 --- /dev/null +++ b/src/oauth-tokens/session.ts @@ -0,0 +1,128 @@ +import { + createRemoteJWKSet, + decodeJwt, + jwtVerify, + type JWTPayload, +} from "jose"; + +import { + CLOCK_TOLERANCE, + classifyVerifyError, + type VerifyFailure, +} from "./classify"; + +/** + * Verifies Clerk *session* tokens — the credential the dashboard already + * holds for the signed-in user. + * + * The playground sends monitor requests as the user rather than with an API + * key (ENG-989): keys are stored only as hashes, so the dashboard cannot send + * one on the user's behalf. Verification is the same JWKS signature check the + * OAuth verifier does, so this needs no Clerk secret either. + * + * A session token is not an OAuth access token and the two cannot share a + * verifier: a session token carries no `aud`, and the organization is in the + * `o.id` claim on version 2 tokens rather than `org_id`. + */ + +/** Session token v2 nests the organization; version 1 used `org_id`. */ +interface OrganizationClaimV2 { + id?: unknown; +} + +export interface SessionPrincipal { + /** Clerk user id — `User.id` in the cloro schema. */ + userId: string; + /** Clerk organization id — `Organization.id` in the cloro schema. */ + organizationId: string; + /** Clerk session id. */ + sessionId: string; +} + +export type SessionTokenFailure = VerifyFailure | "missing_organization"; + +export type SessionVerifyResult = + | { ok: true; principal: SessionPrincipal } + | { ok: false; reason: SessionTokenFailure }; + +export interface SessionVerifierConfig { + /** Clerk Frontend API issuer, e.g. https://clerk.cloro.dev */ + issuer: string; + /** JWKS endpoint. Defaults to `/.well-known/jwks.json`. */ + jwksUrl?: string; +} + +export type SessionTokenVerifier = ( + token: string, +) => Promise; + +/** + * Tells a session token from an OAuth access token, so the caller knows which + * verifier to run. Only a session token carries `sid`. + * + * This reads the payload without checking the signature, which is safe + * because it only routes: whichever verifier runs still rejects a token it + * does not trust. + */ +export function looksLikeSessionToken(token: string): boolean { + try { + return typeof decodeJwt(token).sid === "string"; + } catch { + return false; + } +} + +function organizationId(payload: JWTPayload): string | null { + const nested = (payload.o as OrganizationClaimV2 | undefined)?.id; + if (typeof nested === "string" && nested) return nested; + if (typeof payload.org_id === "string" && payload.org_id) { + return payload.org_id; + } + return null; +} + +export function createSessionTokenVerifier( + config: SessionVerifierConfig, +): SessionTokenVerifier { + const issuer = config.issuer.replace(/\/$/, ""); + const jwksUrl = config.jwksUrl ?? `${issuer}/.well-known/jwks.json`; + + const jwks = createRemoteJWKSet(new URL(jwksUrl)); + + return async function verifySessionToken( + token: string, + ): Promise { + let payload: JWTPayload; + try { + ({ payload } = await jwtVerify(token, jwks, { + issuer, + clockTolerance: CLOCK_TOLERANCE, + })); + } catch (error) { + return { ok: false, reason: classifyVerifyError(error) }; + } + + const userId = payload.sub; + const sessionId = payload.sid; + if ( + typeof userId !== "string" || + !userId || + typeof sessionId !== "string" || + !sessionId + ) { + return { ok: false, reason: "malformed" }; + } + + // No organization means the user has none active in the dashboard, so + // there is no account to charge. + const organization = organizationId(payload); + if (!organization) { + return { ok: false, reason: "missing_organization" }; + } + + return { + ok: true, + principal: { userId, organizationId: organization, sessionId }, + }; + }; +} diff --git a/src/oauth-tokens/verify.test.ts b/src/oauth-tokens/verify.test.ts new file mode 100644 index 0000000..e617e76 --- /dev/null +++ b/src/oauth-tokens/verify.test.ts @@ -0,0 +1,201 @@ +import { SignJWT, exportJWK, generateKeyPair } from "jose"; +import { afterEach, beforeAll, describe, expect, it, vi } from "vitest"; + +import { createOAuthTokenVerifier } from "./verify"; + +const ISSUER = "https://clerk.cloro.dev"; +const JWKS_URL = `${ISSUER}/.well-known/jwks.json`; +const RESOURCE = "https://mcp.cloro.dev/mcp"; + +let signingKey: CryptoKey; +/** A second, unrelated key pair — its tokens must never verify. */ +let foreignKey: CryptoKey; +/** What the JWKS endpoint answers; tests swap it to simulate an outage. */ +let jwksResponder: () => Response | Promise; +let serveJwks: () => Response; + +interface Claims { + sub?: string; + /** `null` omits the claim — `undefined` would just re-apply the default. */ + org_id?: string | null; + scope?: string; + aud?: string | string[]; + iss?: string; + expiresIn?: string; + key?: CryptoKey; +} + +async function mint(claims: Claims = {}): Promise { + const { + sub = "user_123", + org_id = "org_456", + scope = "profile email user:org:read", + aud, + iss = ISSUER, + expiresIn = "1h", + key = signingKey, + } = claims; + + const jwt = new SignJWT({ + ...(org_id === null ? {} : { org_id }), + scope, + client_id: "client_789", + }) + .setProtectedHeader({ alg: "RS256", kid: "test-key" }) + .setIssuer(iss) + .setIssuedAt() + .setExpirationTime(expiresIn); + + if (sub !== undefined) jwt.setSubject(sub); + if (aud !== undefined) jwt.setAudience(aud); + + return jwt.sign(key); +} + +beforeAll(async () => { + const pair = await generateKeyPair("RS256", { extractable: true }); + const foreign = await generateKeyPair("RS256", { extractable: true }); + signingKey = pair.privateKey; + foreignKey = foreign.privateKey; + + const jwk = await exportJWK(pair.publicKey); + const jwks = { + keys: [{ ...jwk, kid: "test-key", alg: "RS256", use: "sig" }], + }; + + serveJwks = () => + new Response(JSON.stringify(jwks), { + status: 200, + headers: { "content-type": "application/json" }, + }); + jwksResponder = serveJwks; + + vi.stubGlobal("fetch", async (input: RequestInfo | URL) => { + if (String(input) !== JWKS_URL) { + throw new Error(`unexpected fetch: ${String(input)}`); + } + return jwksResponder(); + }); +}); + +afterEach(() => { + jwksResponder = serveJwks; + vi.clearAllMocks(); +}); + +describe("createOAuthTokenVerifier", () => { + it("accepts a well-formed token and returns the principal", async () => { + const verify = createOAuthTokenVerifier({ issuer: ISSUER }); + const result = await verify(await mint()); + + expect(result).toEqual({ + ok: true, + principal: { + userId: "user_123", + organizationId: "org_456", + scopes: ["profile", "email", "user:org:read"], + clientId: "client_789", + }, + }); + }); + + it("derives the JWKS URL from the issuer and tolerates a trailing slash", async () => { + const verify = createOAuthTokenVerifier({ issuer: `${ISSUER}/` }); + const result = await verify(await mint()); + expect(result.ok).toBe(true); + }); + + it("rejects an expired token", async () => { + const verify = createOAuthTokenVerifier({ issuer: ISSUER }); + const result = await verify(await mint({ expiresIn: "-1h" })); + expect(result).toEqual({ ok: false, reason: "expired" }); + }); + + it("rejects a token from another issuer", async () => { + const verify = createOAuthTokenVerifier({ issuer: ISSUER }); + const result = await verify(await mint({ iss: "https://evil.example" })); + expect(result).toEqual({ ok: false, reason: "wrong_issuer" }); + }); + + it("rejects a token signed by an unknown key", async () => { + const verify = createOAuthTokenVerifier({ issuer: ISSUER }); + const result = await verify(await mint({ key: foreignKey })); + expect(result.ok).toBe(false); + }); + + it("rejects a token with no org_id — there is no account to charge", async () => { + const verify = createOAuthTokenVerifier({ issuer: ISSUER }); + const result = await verify(await mint({ org_id: null })); + expect(result).toEqual({ ok: false, reason: "missing_organization" }); + }); + + it("rejects a mismatched audience when one is configured", async () => { + const verify = createOAuthTokenVerifier({ + issuer: ISSUER, + audience: RESOURCE, + }); + const result = await verify(await mint({ aud: "https://elsewhere.test" })); + expect(result).toEqual({ ok: false, reason: "wrong_audience" }); + }); + + it("accepts a matching audience, including the array form", async () => { + const verify = createOAuthTokenVerifier({ + issuer: ISSUER, + audience: RESOURCE, + }); + + expect((await verify(await mint({ aud: RESOURCE }))).ok).toBe(true); + expect( + (await verify(await mint({ aud: ["https://other.test", RESOURCE] }))).ok, + ).toBe(true); + }); + + it("rejects a token with no aud claim when an audience is configured", async () => { + // Clerk writes the `resource` parameter into `aud` once aud_claim_enabled + // is on. A missing claim therefore means the instance is misconfigured, + // and the resource server must fail closed rather than accept a token it + // cannot prove was minted for it. + const verify = createOAuthTokenVerifier({ + issuer: ISSUER, + audience: RESOURCE, + }); + expect(await verify(await mint())).toEqual({ + ok: false, + reason: "wrong_audience", + }); + }); + + it("ignores aud entirely when no audience is configured", async () => { + const verify = createOAuthTokenVerifier({ issuer: ISSUER }); + expect((await verify(await mint({ aud: "https://anything" }))).ok).toBe( + true, + ); + }); + + it("reports the key set as unavailable when the JWKS fetch fails", async () => { + const outages: Array<[string, () => Response | Promise]> = [ + ["network error", () => Promise.reject(new TypeError("fetch failed"))], + [ + "timeout", + () => + Promise.reject( + Object.assign(new Error("aborted"), { name: "TimeoutError" }), + ), + ], + ["non-200", () => new Response("upstream down", { status: 503 })], + ["not JSON", () => new Response("", { status: 200 })], + ]; + for (const [label, responder] of outages) { + jwksResponder = responder; + const verify = createOAuthTokenVerifier({ issuer: ISSUER }); + const result = await verify(await mint()); + expect(result, label).toEqual({ ok: false, reason: "unavailable" }); + } + }); + + it("rejects a syntactically broken token", async () => { + const verify = createOAuthTokenVerifier({ issuer: ISSUER }); + const result = await verify("not-a-jwt"); + expect(result.ok).toBe(false); + }); +}); diff --git a/src/oauth-tokens/verify.ts b/src/oauth-tokens/verify.ts new file mode 100644 index 0000000..b97824e --- /dev/null +++ b/src/oauth-tokens/verify.ts @@ -0,0 +1,137 @@ +import { createRemoteJWKSet, jwtVerify, type JWTPayload } from "jose"; + +import { + CLOCK_TOLERANCE, + classifyVerifyError, + type VerifyFailure, +} from "./classify"; + +/** + * Verifies OAuth 2.1 access tokens minted by the cloro Clerk instance. + * + * Two services need this and they must agree: apps/mcp verifies so it can + * answer an unauthenticated `tools/call` with a spec-shaped 401 challenge + * instead of forwarding a bad token, and apps/api verifies because it is the + * internet-facing resource server that actually spends credits. A single + * implementation keeps the two from drifting into disagreeing about what a + * valid token is. + * + * Verification is signature-based against the instance JWKS, which is public. + * That is the reason this package needs no Clerk secret key, and the reason + * apps/mcp can stay secretless. + */ + +/** The `org_id` claim that carries the organization the user consented for. */ +const ORGANIZATION_CLAIM = "org_id"; + +export interface OAuthPrincipal { + /** Clerk user id — `User.id` in the cloro schema. */ + userId: string; + /** Clerk organization id — `Organization.id` in the cloro schema. */ + organizationId: string; + scopes: string[]; + clientId?: string; +} + +export type OAuthTokenFailure = + | VerifyFailure + | "wrong_audience" + | "missing_organization"; + +export type OAuthVerifyResult = + | { ok: true; principal: OAuthPrincipal } + | { ok: false; reason: OAuthTokenFailure }; + +export interface OAuthVerifierConfig { + /** Clerk Frontend API issuer, e.g. https://clerk.cloro.dev */ + issuer: string; + /** JWKS endpoint. Defaults to `/.well-known/jwks.json`. */ + jwksUrl?: string; + /** + * Canonical resource URI this token must have been minted for + * (RFC 8707), e.g. https://mcp.cloro.dev/mcp. + * + * When set, the token must carry a matching `aud` claim — a token with no + * `aud` at all is rejected. Clerk writes the `resource` parameter into + * `aud` verbatim once `aud_claim_enabled` is on for the instance (verified + * against a real token from the dev instance), so an absent claim means + * the instance is misconfigured, not that the token is fine. Failing + * closed there is the point: the MCP spec requires a resource server to + * accept only tokens issued for it. + */ + audience?: string; +} + +function toScopes(payload: JWTPayload): string[] { + const scope = payload.scope; + if (typeof scope === "string") { + return scope.split(" ").filter(Boolean); + } + // Some authorization servers emit the plural array form instead. + if (Array.isArray(payload.scopes)) { + return payload.scopes.filter((s): s is string => typeof s === "string"); + } + return []; +} + +function audienceMatches(aud: JWTPayload["aud"], expected: string): boolean { + if (typeof aud === "string") return aud === expected; + if (Array.isArray(aud)) return aud.includes(expected); + return false; +} + +export type OAuthTokenVerifier = (token: string) => Promise; + +export function createOAuthTokenVerifier( + config: OAuthVerifierConfig, +): OAuthTokenVerifier { + const issuer = config.issuer.replace(/\/$/, ""); + const jwksUrl = config.jwksUrl ?? `${issuer}/.well-known/jwks.json`; + + // createRemoteJWKSet caches keys and refetches on an unknown `kid`, so key + // rotation needs no redeploy and the happy path makes no network call. + const jwks = createRemoteJWKSet(new URL(jwksUrl)); + + return async function verifyOAuthToken( + token: string, + ): Promise { + let payload: JWTPayload; + try { + ({ payload } = await jwtVerify(token, jwks, { + issuer, + clockTolerance: CLOCK_TOLERANCE, + })); + } catch (error) { + return { ok: false, reason: classifyVerifyError(error) }; + } + + if (config.audience && !audienceMatches(payload.aud, config.audience)) { + return { ok: false, reason: "wrong_audience" }; + } + + const userId = payload.sub; + if (typeof userId !== "string" || !userId) { + return { ok: false, reason: "malformed" }; + } + + // No org_id means the user never granted `user:org:read`, or granted it + // without picking an organization. Either way there is no account to + // charge, so the token cannot authorize a scrape. + const organizationId = payload[ORGANIZATION_CLAIM]; + if (typeof organizationId !== "string" || !organizationId) { + return { ok: false, reason: "missing_organization" }; + } + + const clientId = payload.client_id; + + return { + ok: true, + principal: { + userId, + organizationId, + scopes: toScopes(payload), + clientId: typeof clientId === "string" ? clientId : undefined, + }, + }; + }; +} diff --git a/src/schemas/aimode.schema.ts b/src/schemas/aimode.schema.ts index 9d2037d..984817a 100644 --- a/src/schemas/aimode.schema.ts +++ b/src/schemas/aimode.schema.ts @@ -1,5 +1,5 @@ import { z } from "zod"; -import { createCountrySchema } from "./country.schema"; +import { geoFields, refineGeo, applyGeo } from "./geo.schema"; // Full list available in https://support.google.com/websearch/answer/16011537#zippy=%2Csupported-countries-territories @@ -259,7 +259,12 @@ export const aimodeSchema = z .min(1, "Prompt cannot be empty") .max(10_000, "Prompt is too long (max 10,000 characters)") .trim(), - country: createCountrySchema(AIMODE_AVAILABLE_COUNTRIES, []), + // `hl` is the interface-language override (ENG-828): a country alone + // cannot express "German interface from a US exit", and multilingual + // countries (CH, BE, CA) resolve to one dominant language. `gl` is the + // result geography, validated against AI Mode's own supported-country + // list. `country` is the deprecated alias normalized into `gl`. + ...geoFields(AIMODE_AVAILABLE_COUNTRIES), location: z.string().trim().optional(), uule: z.string().trim().optional(), device: z.enum(["desktop", "mobile"]).default("desktop"), @@ -272,7 +277,9 @@ export const aimodeSchema = z .partial() .optional(), }) + .superRefine((data, ctx) => refineGeo(data, ctx)) .refine((data) => !(data.location && data.uule), { message: "Cannot set both location and uule; provide only one", path: ["uule"], - }); + }) + .transform(applyGeo); diff --git a/src/schemas/chatgpt.schema.ts b/src/schemas/chatgpt.schema.ts index caecbca..433f06c 100644 --- a/src/schemas/chatgpt.schema.ts +++ b/src/schemas/chatgpt.schema.ts @@ -7,7 +7,6 @@ import { getAvailableCountries } from "./utils"; const CHATGPT_BLOCKED_COUNTRIES: string[] = [ "CN", // China - "CZ", // Czechia "HK", // Hong Kong "IR", // Iran "MO", // Macao @@ -38,4 +37,9 @@ export const chatgptSchema = withState({ }) .partial() .optional(), + // Serve ChatGPT's legacy (desktop) UI instead of the default mobile-web UI. + // The desktop UI is where ads/shopping render, so callers that need them pass + // this. Defaults to the mobile-web UI when unset. + legacy: z.boolean().optional(), + disableWebSearch: z.boolean().optional(), }); diff --git a/src/schemas/copilot.schema.ts b/src/schemas/copilot.schema.ts index 03045e9..0ac425c 100644 --- a/src/schemas/copilot.schema.ts +++ b/src/schemas/copilot.schema.ts @@ -7,6 +7,7 @@ import { getAvailableCountries } from "./utils"; const COPILOT_BLOCKED_COUNTRIES: string[] = [ "BY", // Belarus "CN", // China + "IR", "RU", // Russia "SY", // Syria "VE", // Venezuela diff --git a/src/schemas/geo.schema.ts b/src/schemas/geo.schema.ts new file mode 100644 index 0000000..0977d50 --- /dev/null +++ b/src/schemas/geo.schema.ts @@ -0,0 +1,146 @@ +import { z } from "zod"; +import { createCountrySchema } from "./country.schema"; + +/** + * Google's own geo vocabulary — `hl` (interface language) and `gl` (result + * geography) — as first-class request fields. + * + * `country` used to do three jobs at once: seed `gl`, seed `hl` through the + * worker's country-to-language table, and act as a soft preference when + * leasing a farmed session. Callers could not separate them, so a German + * interface was only reachable by asking for DE and multilingual countries + * collapsed to one dominant language. Integrators also arrive already + * speaking `hl`/`gl`, because Google's docs and every competing SERP API use + * those names. + * + * `country` stays accepted and behaves exactly as before; it is normalized + * into `gl` here and is deprecated in favour of it. + * + * Two constraints shape this module: + * + * 1. **Normalization must be idempotent.** `validateBodyMiddleware` replaces + * `req.body` with the parsed output, and the async scheduler re-validates + * persisted payloads against this same schema (ENG-662). Parsing an + * already-normalized payload must produce an identical result and must not + * raise a conflict — hence the case-insensitive comparison below. + * 2. **`gl` is emitted lowercase, validated uppercase.** Google's wire format + * is lowercase; the supported-country constants are uppercase. + */ + +/** `hl` — free-form. Google owns the language-code list, so a closed enum here + * would reject codes it accepts (`pt-br`, `zh-tw`, ...). */ +export const hlField = z.string().trim().toLowerCase().optional(); + +/** + * `gl` — validated against the endpoint's own supported-country list, since + * Cloro owns which exits it stocks and which countries each model supports. + * Compared uppercase; the list constants are uppercase ISO codes. + */ +export const createGlSchema = (availableCountries: string[]) => + z + .string() + .trim() + .superRefine((val, ctx) => { + if (!availableCountries.includes(val.toUpperCase())) { + ctx.addIssue({ + code: "custom", + message: + "We currently do not support this country code. If you'd like us to add support, please reach out to our support team and we will implement it.", + }); + } + }) + .optional(); + +/** + * The geo field trio for an endpoint's object shape. `country` keeps its + * existing validation and message; it is optional here because `gl` can stand + * in for it. Presence is enforced by `refineGeo`. + */ +export const geoFields = (availableCountries: string[]) => ({ + country: createCountrySchema(availableCountries, []).optional(), + gl: createGlSchema(availableCountries), + hl: hlField, +}); + +export type GeoInput = { + country?: string; + gl?: string; + hl?: string; +}; + +export type GeoOutput = { + country: string; + gl: string; + hl?: string; +}; + +/** + * Cross-field geo rules. `country` and `gl` are two channels for one fact, so + * a disagreement is rejected rather than resolved by a silent precedence rule + * — the same reasoning the `location` / `uule` refinement already applies. + * An agreeing pair is the alias round-tripping and is accepted, which is what + * keeps normalization idempotent. + * + * `requirePresence: false` is for callers that resolve geo from another + * channel (Google's `url` mode reads it from the URL's `gl`) and emit their + * own message when nothing resolves. + */ +export function refineGeo( + data: GeoInput, + ctx: z.RefinementCtx, + { requirePresence = true }: { requirePresence?: boolean } = {}, +): void { + if (data.country && data.gl) { + if (data.country.toUpperCase() !== data.gl.toUpperCase()) { + ctx.addIssue({ + code: "custom", + path: ["gl"], + message: `Cannot set both country and gl to different values (country: "${data.country}", gl: "${data.gl}"); country is deprecated — send gl alone`, + }); + } + return; + } + + if (requirePresence && !data.country && !data.gl) { + ctx.addIssue({ + code: "custom", + path: ["gl"], + message: "Provide gl (or country, which is deprecated)", + }); + } +} + +/** + * Collapse whichever geo channels were supplied into the canonical trio. + * + * `country` is still emitted because the browsers worker reads it off the + * queue message today; dropping it would silently serve every request from + * the wrong geography. It is derived here rather than caller-authored, and + * comes out once browsers consumes `gl`. + * + * Returns `null` when no geo resolved, so callers that allow that (Google's + * `url` mode before URL derivation) can decide what to do. + */ +export function normalizeGeo(data: GeoInput): GeoOutput | null { + const source = data.gl ?? data.country; + if (!source) return null; + + return { + gl: source.toLowerCase(), + country: source.toUpperCase(), + hl: data.hl, + }; +} + +/** + * Transform helper: collapse the geo channels and merge them back over the + * parsed body. Endpoints whose transform does nothing else can pass this + * directly to `.transform()`. + * + * Non-null assertion is safe wherever `refineGeo` ran with + * `requirePresence: true` — a body carrying neither channel never reaches here. + */ +export function applyGeo(data: T): T & GeoOutput { + const geo = normalizeGeo(data)!; + return { ...data, country: geo.country, gl: geo.gl, hl: geo.hl }; +} diff --git a/src/schemas/google-news.schema.ts b/src/schemas/google-news.schema.ts index def9202..ba85146 100644 --- a/src/schemas/google-news.schema.ts +++ b/src/schemas/google-news.schema.ts @@ -1,23 +1,28 @@ import { z } from "zod"; -import { createCountrySchema } from "./country.schema"; +import { geoFields, refineGeo, applyGeo } from "./geo.schema"; import { googleDeviceSchema } from "./device.schema"; import { ALL_COUNTRY_CODES } from "./all-countries"; export const GOOGLE_NEWS_AVAILABLE_COUNTRIES = ALL_COUNTRY_CODES; -export const googleNewsSchema = z.object({ - query: z - .string("Query cannot be empty") - .min(1, "Query cannot be empty") - .max(10_000, "Query is too long (max 10,000 characters)") - .trim(), - country: createCountrySchema(GOOGLE_NEWS_AVAILABLE_COUNTRIES, []), - device: googleDeviceSchema, - pages: z.number().positive().max(10).default(1), - include: z - .object({ - html: z.boolean(), - }) - .partial() - .optional(), -}); +export const googleNewsSchema = z + .object({ + query: z + .string("Query cannot be empty") + .min(1, "Query cannot be empty") + .max(10_000, "Query is too long (max 10,000 characters)") + .trim(), + // Same geo contract as Google search and AI Mode: `hl` is the interface + // language, `gl` the result geography, `country` the deprecated alias. + ...geoFields(GOOGLE_NEWS_AVAILABLE_COUNTRIES), + device: googleDeviceSchema, + pages: z.number().positive().max(10).default(1), + include: z + .object({ + html: z.boolean(), + }) + .partial() + .optional(), + }) + .superRefine((data, ctx) => refineGeo(data, ctx)) + .transform(applyGeo); diff --git a/src/schemas/google-url.ts b/src/schemas/google-url.ts index 739e940..fee8f25 100644 --- a/src/schemas/google-url.ts +++ b/src/schemas/google-url.ts @@ -95,6 +95,29 @@ const pagesForResultCount = (num: number): number => { const readTrimmed = (params: URLSearchParams, key: string): string | null => params.get(key)?.trim() || null; +/** + * Read a param without form decoding. `URLSearchParams` turns a literal `+` + * into a space, which corrupts a UULE token (`w+CAIQ...` becomes `w CAIQ...`) + * so Google ignores it and falls back to the session's location history. + */ +const readRaw = (search: string, key: string): string | null => { + const decode = (s: string): string => { + try { + return decodeURIComponent(s); + } catch { + return s; + } + }; + for (const pair of search.replace(/^\?/, "").split("&")) { + const eq = pair.indexOf("="); + const name = eq === -1 ? pair : pair.slice(0, eq); + if (decode(name) !== key) continue; + const value = eq === -1 ? "" : decode(pair.slice(eq + 1)); + return value.trim() || null; + } + return null; +}; + export const parseGoogleSearchUrl = (raw: string): GoogleUrlParseResult => { let url: URL; try { @@ -131,7 +154,7 @@ export const parseGoogleSearchUrl = (raw: string): GoogleUrlParseResult => { query, country: readTrimmed(url.searchParams, "gl")?.toUpperCase() ?? null, language: readTrimmed(url.searchParams, "hl")?.toLowerCase() ?? null, - uule: readTrimmed(url.searchParams, "uule"), + uule: readRaw(url.search, "uule"), startOffset: readNonNegativeInt(url.searchParams, "start", 0), pages: pagesForResultCount( readNonNegativeInt(url.searchParams, "num", RESULTS_PER_PAGE), diff --git a/src/schemas/google.schema.ts b/src/schemas/google.schema.ts index 01f7cd2..175e47e 100644 --- a/src/schemas/google.schema.ts +++ b/src/schemas/google.schema.ts @@ -1,5 +1,5 @@ import { z } from "zod"; -import { createCountrySchema } from "./country.schema"; +import { geoFields, refineGeo, normalizeGeo, applyGeo } from "./geo.schema"; import { googleDeviceSchema } from "./device.schema"; import { ALL_COUNTRY_CODES } from "./all-countries"; import { parseGoogleSearchUrl } from "./google-url"; @@ -39,9 +39,11 @@ export const googleSchema = z // A complete google.com/search URL. Cloro forwards it as the fetch target // instead of assembling one from the structured fields. url: z.string().trim().optional(), - // Optional only because `url` mode can derive it from the URL's `gl`. - // Structured mode still requires it — enforced below. - country: createCountrySchema(GOOGLE_AVAILABLE_COUNTRIES, []).optional(), + // `hl` / `gl` are the caller-facing geo contract; `country` is the + // deprecated alias normalized into `gl`. All three are optional at the + // field level because `url` mode can derive geo from the URL's `gl` — + // presence is enforced per-mode below. + ...geoFields(GOOGLE_AVAILABLE_COUNTRIES), location: z.string().trim().optional(), uule: z.string().trim().optional(), device: googleDeviceSchema, @@ -52,8 +54,6 @@ export const googleSchema = z include: z .object({ html: z.boolean(), - // undocumented; inlines raw HTML into `html` when combined with include.html - rawHtml: z.boolean(), aioverview: z.union([ z.boolean(), z @@ -84,6 +84,10 @@ export const googleSchema = z path: ["uule"], }) .superRefine((data, ctx) => { + // Conflict between the two geo channels is always an error, in either + // mode. Presence is mode-specific, so it is enforced below instead. + refineGeo(data, ctx, { requirePresence: false }); + if (!data.query && !data.url) { ctx.addIssue({ code: "custom", @@ -95,11 +99,11 @@ export const googleSchema = z if (!data.url) { // Structured mode has no URL to derive geo from. - if (!data.country) { + if (!data.country && !data.gl) { ctx.addIssue({ code: "custom", - message: "Invalid input: expected string, received undefined", - path: ["country"], + message: "Provide gl (or country, which is deprecated)", + path: ["gl"], }); } return; @@ -144,17 +148,35 @@ export const googleSchema = z return; } - // `country` selects the proxy and session, so it must resolve to something - // we support. The URL's `gl` stands in when the caller omits it — they have - // already stated the geo once and should not have to repeat it. - if (data.country) return; + // `hl` / `gl` (or the deprecated `country`) next to a URL that carries its + // own copy is two channels for one fact, like the URL-owned fields above. + // An agreeing pair is the transform output round-tripping and is accepted. + if (data.hl && parsed.language && data.hl !== parsed.language) { + ctx.addIssue({ + code: "custom", + message: "Cannot set both url and hl; the URL owns this value", + path: ["hl"], + }); + } + + const explicitGl = (data.gl ?? data.country)?.toUpperCase(); + if (explicitGl) { + if (parsed.country && parsed.country !== explicitGl) { + const field = data.gl ? "gl" : "country"; + ctx.addIssue({ + code: "custom", + message: `Cannot set both url and ${field}; the URL owns this value`, + path: [field], + }); + } + return; + } if (!parsed.country) { ctx.addIssue({ code: "custom", - message: - "Provide country, or include gl= in the url so it can be derived", - path: ["country"], + message: "Provide gl, or include gl= in the url so it can be derived", + path: ["gl"], }); return; } @@ -162,8 +184,8 @@ export const googleSchema = z if (!GOOGLE_AVAILABLE_COUNTRIES.includes(parsed.country)) { ctx.addIssue({ code: "custom", - message: `Derived country "${parsed.country}" from the url's gl is not supported; pass a supported country explicitly`, - path: ["country"], + message: `Derived country "${parsed.country}" from the url's gl is not supported; pass a supported gl explicitly`, + path: ["gl"], }); } }) @@ -175,10 +197,8 @@ export const googleSchema = z // set so the inferred output type stays a single object, not a union. if (!parsed?.ok) { return { - ...data, - // Guaranteed present in structured mode by the refinement above. - country: data.country as string, - language: undefined as string | undefined, + // Geo presence is guaranteed in structured mode by the refinement above. + ...applyGeo(data), pages: data.pages ?? DEFAULT_PAGES, startOffset: undefined as number | undefined, tbs: undefined as string | undefined, @@ -188,16 +208,23 @@ export const googleSchema = z // Decompose the URL into the fields the worker already consumes, so `url` // mode needs no special handling downstream: `q` becomes the prompt, `gl` - // the country, `hl` the language override, `uule` the location, and `tbs` / - // `safe` ride along as their own keys. + // and `hl` the geo pair, `uule` the location, and `tbs` / `safe` ride along + // as their own keys. Explicit `hl` / `gl` either agree with the URL's + // copies (enforced above) or fill in when the URL has none. // // `pages` / `startOffset` are the pagination authority — `num` is capped at // 10 by Google, so depth is fulfilled by paginating `start` instead. + const geo = normalizeGeo({ + gl: data.gl ?? data.country ?? (parsed.country as string), + hl: data.hl ?? parsed.language ?? undefined, + })!; + return { ...data, query: parsed.query, - country: data.country ?? (parsed.country as string), - language: parsed.language ?? undefined, + country: geo.country, + gl: geo.gl, + hl: geo.hl, uule: parsed.uule ?? undefined, pages: parsed.pages, startOffset: parsed.startOffset as number | undefined, diff --git a/src/schemas/index.ts b/src/schemas/index.ts index d3327bf..eafdf30 100644 --- a/src/schemas/index.ts +++ b/src/schemas/index.ts @@ -19,6 +19,7 @@ export * from "./perplexity.schema"; export * from "./country.schema"; export * from "./device.schema"; export * from "./state.schema"; +export * from "./geo.schema"; export * from "./country-models"; export * from "./headers"; export * from "./all-countries"; diff --git a/src/server.test.ts b/src/server.test.ts index b1fc737..b5adac0 100644 --- a/src/server.test.ts +++ b/src/server.test.ts @@ -99,7 +99,6 @@ describe("cloro MCP server", () => { "scrape_copilot", "scrape_perplexity", "scrape_grok", - "scrape_google_ai_mode", ]) { const tool = tools.find((t) => t.name === name)!; expect(tool.inputSchema.required).toEqual( @@ -108,16 +107,32 @@ describe("cloro MCP server", () => { } }); + it("offers hl and gl on the Google-family tools", () => { + for (const name of [ + "scrape_google", + "scrape_google_news", + "scrape_google_ai_mode", + ]) { + const tool = tools.find((t) => t.name === name)!; + expect(Object.keys(tool.inputSchema.properties ?? {})).toEqual( + expect.arrayContaining(["hl", "gl", "country"]), + ); + // `country` is the deprecated alias for `gl`, so neither is required on + // its own — the schema's cross-field rule enforces that one is present. + expect(tool.inputSchema.required ?? []).not.toContain("country"); + expect(tool.inputSchema.required ?? []).not.toContain("gl"); + } + }); + it("keeps scrape_google's fields optional for query/url dual mode", () => { const tool = tools.find((t) => t.name === "scrape_google")!; expect(tool.inputSchema.required ?? []).toEqual([]); }); - it("hides include.rawHtml from scrape_google but keeps public flags", () => { + it("advertises scrape_google's public include flags", () => { const tool = tools.find((t) => t.name === "scrape_google")!; const includeProps = tool.inputSchema.properties?.include?.properties; expect(includeProps).toBeDefined(); - expect(Object.keys(includeProps!)).not.toContain("rawHtml"); expect(Object.keys(includeProps!)).toEqual( expect.arrayContaining(["html", "aioverview", "paaAioverview"]), ); @@ -219,5 +234,4 @@ describe("cloro MCP server", () => { "INSUFFICIENT_CREDITS", ); }); - }); diff --git a/src/server.ts b/src/server.ts index dfe7b31..9a71912 100644 --- a/src/server.ts +++ b/src/server.ts @@ -12,7 +12,7 @@ import { registerReferenceTools } from "./tools/reference-tools"; export function buildServer(apiKey: string): McpServer { const server = new McpServer({ name: "cloro", - version: "0.1.0", + version: "0.2.0", }); const client = new CloroClient(apiKey); diff --git a/src/tools/monitor-tools.ts b/src/tools/monitor-tools.ts index 4ce4148..c098a9f 100644 --- a/src/tools/monitor-tools.ts +++ b/src/tools/monitor-tools.ts @@ -23,7 +23,11 @@ const COUNTRY_DOC = const STATE_DOC = 'Optional state code for state-level targeting (e.g. "CA" when country is "US"). Only some countries support this — call list_states for the supported countries and their codes.'; const INCLUDE_DOC = - "Optional flags to include heavier payload fields in the response. Leave unset for the leanest response."; + "Optional flags for heavier payload fields, each off by default: markdown (the answer rendered as markdown), html (the answer page HTML), rawResponse (the engine's unprocessed response payload). Leave unset for the leanest response."; +const CHATGPT_INCLUDE_DOC = + "Optional flags for heavier payload fields, each off by default: markdown (the answer rendered as markdown), html (the answer page HTML), rawResponse (the engine's unprocessed response payload), searchQueries (the web searches ChatGPT issued while answering), ads and shopping (sponsored and product results; these render only on the desktop UI, so pair them with legacy: true). Leave unset for the leanest response."; +const CHATGPT_LEGACY_DOC = + "Serve ChatGPT's desktop UI instead of the default mobile-web UI. Needed for include.ads and include.shopping, which only render on desktop. Defaults to false."; const promptDoc = (engine: string) => `The prompt to submit to ${engine}.`; const QUERY_DOC = "The search query."; @@ -31,6 +35,15 @@ const LOCATION_DOC = 'Optional location name to target search results (e.g. "Austin, Texas, United States"). Mutually exclusive with uule.'; const UULE_DOC = "Optional Google UULE location parameter. Mutually exclusive with location."; +const HL_DOC = + 'Optional Google interface-language code, sent as hl (e.g. "de", "pt-br"). Defaults to the language derived from gl — set it when the geography\'s dominant language is not the one you want.'; +const GL_DOC = + 'ISO 3166-1 alpha-2 code for the result geography, sent to Google as gl (e.g. "us"). Use list_countries to see supported codes per model.'; +// Only the Google-family tools deprecate `country`; the prompt engines still +// take it as their only geo field, so they keep COUNTRY_DOC unchanged. +const GOOGLE_COUNTRY_DOC = + COUNTRY_DOC + + " Deprecated — use gl instead; country is kept for compatibility and must not disagree with gl."; const DEVICE_DOC = "Device type to emulate. Defaults to desktop."; const GOOGLE_DEVICE_DOC = "Device to emulate: desktop, mobile, ios (Safari on iPhone), or android (Chrome on Android). mobile is an alias for android. Defaults to desktop."; @@ -63,6 +76,7 @@ const promptEngineTool = (tool: { engine: string; shape: z.ZodRawShape; description?: string; + extraDocs?: Record; }): MonitorTool => ({ name: tool.name, path: tool.path, @@ -75,6 +89,7 @@ const promptEngineTool = (tool: { country: COUNTRY_DOC, state: STATE_DOC, include: INCLUDE_DOC, + ...tool.extraDocs, }), }); @@ -87,6 +102,7 @@ const MONITOR_TOOLS: MonitorTool[] = [ description: "Submit a prompt to ChatGPT from a chosen country (and optionally US state) and return the full answer: text, cited sources, and optionally markdown, search queries, shopping results, and ads. Use this to see how ChatGPT answers a prompt and which brands/sources it mentions.", shape: chatgptSchema.shape, + extraDocs: { include: CHATGPT_INCLUDE_DOC, legacy: CHATGPT_LEGACY_DOC }, }), promptEngineTool({ name: "scrape_gemini", @@ -122,9 +138,13 @@ const MONITOR_TOOLS: MonitorTool[] = [ title: "Scrape Google AI Mode", description: "Submit a prompt to Google AI Mode from a chosen country and return the AI answer with cited sources. Supports location or UULE targeting and desktop/mobile emulation.", - inputSchema: describeFields(aimodeSchema.shape, { + // aimodeSchema is a ZodPipe (object -> transform) since geo + // normalization; the tool advertises the pipe's input object. + inputSchema: describeFields(aimodeSchema.in.shape, { prompt: promptDoc("Google AI Mode"), - country: COUNTRY_DOC, + country: GOOGLE_COUNTRY_DOC, + gl: GL_DOC, + hl: HL_DOC, location: LOCATION_DOC, uule: UULE_DOC, device: DEVICE_DOC, @@ -139,30 +159,23 @@ const MONITOR_TOOLS: MonitorTool[] = [ "Run a Google search from a chosen country and return organic results, with optional AI Overview extraction (include.aioverview) and People-Also-Ask AI answers (include.paaAioverview). Two modes: structured (query + country, with optional location/uule/pages) or url (a complete google.com/search URL that owns query, location, and pagination). Supports desktop, mobile, iOS, and Android emulation and multi-page results.", // googleSchema is a ZodPipe (object -> transform) since url-mode // decomposition (ENG-536); the tool advertises the pipe's input object. - // include.rawHtml is an undocumented internal flag — the API still - // accepts it, but agents shouldn't be offered it. - inputSchema: describeFields( - { - ...googleSchema.in.shape, - include: googleSchema.in.shape.include - .unwrap() - .omit({ rawHtml: true }) - .optional(), - }, - { - query: "The search query. Required unless url is provided.", - url: "A complete google.com/search URL to fetch instead of building one from structured fields. When set, query/location/uule/pages must be omitted (the URL owns them); country can be derived from the URL's gl parameter.", - country: - COUNTRY_DOC + - " Required in query mode; in url mode it can be derived from the URL's gl parameter.", - location: LOCATION_DOC, - uule: UULE_DOC, - device: GOOGLE_DEVICE_DOC, - pages: PAGES_DOC, - include: - "Optional flags. Set aioverview: true to extract Google's AI Overview (or { markdown: true } for markdown), paaAioverview: true to hydrate AI answers in People Also Ask. Leave unset for organic results only.", - }, - ), + inputSchema: describeFields(googleSchema.in.shape, { + query: "The search query. Required unless url is provided.", + url: "A complete google.com/search URL to fetch instead of building one from structured fields. When set, query/location/uule/pages must be omitted (the URL owns them); gl and hl are read from the URL's gl/hl parameters; an explicit gl or hl must agree with the URL's copy.", + country: GOOGLE_COUNTRY_DOC, + gl: + GL_DOC + + " Required in query mode; in url mode it is read from the URL's gl parameter unless supplied here, and an explicit value wins.", + hl: + HL_DOC + + " In url mode it is read from the URL's hl parameter unless supplied here, and an explicit value wins.", + location: LOCATION_DOC, + uule: UULE_DOC, + device: GOOGLE_DEVICE_DOC, + pages: PAGES_DOC, + include: + "Optional flags. Set aioverview: true to extract Google's AI Overview (or { markdown: true } for markdown), paaAioverview: true to hydrate AI answers in People Also Ask. Leave unset for organic results only.", + }), }, { name: "scrape_google_news", @@ -170,9 +183,13 @@ const MONITOR_TOOLS: MonitorTool[] = [ title: "Scrape Google News", description: "Run a Google News search from a chosen country and return news results. Supports desktop, mobile, iOS, and Android emulation and multi-page results.", - inputSchema: describeFields(googleNewsSchema.shape, { + // googleNewsSchema is a ZodPipe (object -> transform) since geo + // normalization; the tool advertises the pipe's input object. + inputSchema: describeFields(googleNewsSchema.in.shape, { query: QUERY_DOC, - country: COUNTRY_DOC, + country: GOOGLE_COUNTRY_DOC, + gl: GL_DOC, + hl: HL_DOC, device: GOOGLE_DEVICE_DOC, pages: PAGES_DOC, include: INCLUDE_DOC, diff --git a/tsconfig.json b/tsconfig.json index dfc0ba0..9a08e1b 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -1,7 +1,7 @@ { "compilerOptions": { "target": "ES2022", - "lib": ["ES2023"], + "lib": ["ES2023", "DOM"], "module": "ESNext", "moduleResolution": "bundler", "resolveJsonModule": true,