feat: add optional you.com search helper

README.md

@@ -7,6 +7,7 @@ This is a Model Context Protocol (MCP) server that integrates [AgentQL](https://
 ### Tools
 
 - `extract-web-data` - extract structured data from a given 'url', using 'prompt' as a description of actual data and its fields to extract.
+- `search-web` - optional You.com web search helper for finding candidate pages before extraction. Requires `YDC_API_KEY`.
 
 ## Installation
 
@@ -33,7 +34,8 @@ npm install -g agentql-mcp
       "command": "npx",
       "args": ["-y", "agentql-mcp"],
       "env": {
-        "AGENTQL_API_KEY": "YOUR_API_KEY"
+        "AGENTQL_API_KEY": "YOUR_API_KEY",
+        "YDC_API_KEY": "YOUR_YOUCOM_API_KEY"
       }
     }
   }
@@ -46,7 +48,7 @@ Read more about MCP configuration in Claude [here](https://modelcontextprotocol.
 
 For one-click installation, click one of the install buttons below:
 
-[![Install with NPX in VS Code](https://img.shields.io/badge/VS_Code-NPM-0098FF?style=flat-square&logo=visualstudiocode&logoColor=white)](https://insiders.vscode.dev/redirect/mcp/install?name=agentql&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22agentql-mcp%22%5D%2C%22env%22%3A%7B%22AGENTQL_API_KEY%22%3A%22%24%7Binput%3AapiKey%7D%22%7D%7D&inputs=%5B%7B%22type%22%3A%22promptString%22%2C%22id%22%3A%22apiKey%22%2C%22description%22%3A%22AgentQL+API+Key%22%2C%22password%22%3Atrue%7D%5D) [![Install with NPX in VS Code Insiders](https://img.shields.io/badge/VS_Code_Insiders-NPM-24bfa5?style=flat-square&logo=visualstudiocode&logoColor=white)](https://insiders.vscode.dev/redirect/mcp/install?name=agentql&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22agentql-mcp%22%5D%2C%22env%22%3A%7B%22AGENTQL_API_KEY%22%3A%22%24%7Binput%3AapiKey%7D%22%7D%7D&inputs=%5B%7B%22type%22%3A%22promptString%22%2C%22id%22%3A%22apiKey%22%2C%22description%22%3A%22AgentQL+API+Key%22%2C%22password%22%3Atrue%7D%5D&quality=insiders)
+[![Install with NPX in VS Code](https://img.shields.io/badge/VS_Code-NPM-0098FF?style=flat-square&logo=visualstudiocode&logoColor=white)](https://insiders.vscode.dev/redirect/mcp/install?name=agentql&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22agentql-mcp%22%5D%2C%22env%22%3A%7B%22AGENTQL_API_KEY%22%3A%22%24%7Binput%3AapiKey%7D%22%2C%22YDC_API_KEY%22%3A%22%24%7Binput%3AydcApiKey%7D%22%7D%7D&inputs=%5B%7B%22type%22%3A%22promptString%22%2C%22id%22%3A%22apiKey%22%2C%22description%22%3A%22AgentQL+API+Key%22%2C%22password%22%3Atrue%7D%2C%7B%22type%22%3A%22promptString%22%2C%22id%22%3A%22ydcApiKey%22%2C%22description%22%3A%22You.com+API+Key%22%2C%22password%22%3Atrue%7D%5D) [![Install with NPX in VS Code Insiders](https://img.shields.io/badge/VS_Code_Insiders-NPM-24bfa5?style=flat-square&logo=visualstudiocode&logoColor=white)](https://insiders.vscode.dev/redirect/mcp/install?name=agentql&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22agentql-mcp%22%5D%2C%22env%22%3A%7B%22AGENTQL_API_KEY%22%3A%22%24%7Binput%3AapiKey%7D%22%2C%22YDC_API_KEY%22%3A%22%24%7Binput%3AydcApiKey%7D%22%7D%7D&inputs=%5B%7B%22type%22%3A%22promptString%22%2C%22id%22%3A%22apiKey%22%2C%22description%22%3A%22AgentQL+API+Key%22%2C%22password%22%3Atrue%7D%2C%7B%22type%22%3A%22promptString%22%2C%22id%22%3A%22ydcApiKey%22%2C%22description%22%3A%22You.com+API+Key%22%2C%22password%22%3Atrue%7D%5D&quality=insiders)
 
 #### Manual Installation
 
@@ -63,14 +65,21 @@ Add the following JSON block to your User Settings (JSON) file in VS Code. You c
         "id": "apiKey",
         "description": "AgentQL API Key",
         "password": true
+      },
+      {
+        "type": "promptString",
+        "id": "ydcApiKey",
+        "description": "You.com API Key",
+        "password": true
       }
     ],
     "servers": {
       "agentql": {
         "command": "npx",
         "args": ["-y", "agentql-mcp"],
         "env": {
-          "AGENTQL_API_KEY": "${input:apiKey}"
+          "AGENTQL_API_KEY": "${input:apiKey}",
+          "YDC_API_KEY": "${input:ydcApiKey}"
         }
       }
     }
@@ -88,14 +97,21 @@ Optionally, you can add it to a file called `.vscode/mcp.json` in your workspace
       "id": "apiKey",
       "description": "AgentQL API Key",
       "password": true
+    },
+    {
+      "type": "promptString",
+      "id": "ydcApiKey",
+      "description": "You.com API Key",
+      "password": true
     }
   ],
   "servers": {
     "agentql": {
       "command": "npx",
       "args": ["-y", "agentql-mcp"],
       "env": {
-        "AGENTQL_API_KEY": "${input:apiKey}"
+        "AGENTQL_API_KEY": "${input:apiKey}",
+        "YDC_API_KEY": "${input:ydcApiKey}"
       }
     }
   }
@@ -110,7 +126,7 @@ Optionally, you can add it to a file called `.vscode/mcp.json` in your workspace
 - Enter the following:
   - Name: "agentql" (or your preferred name)
   - Type: "command"
-  - Command: `env AGENTQL_API_KEY=YOUR_API_KEY npx -y agentql-mcp`
+  - Command: `env AGENTQL_API_KEY=YOUR_API_KEY YDC_API_KEY=YOUR_YOUCOM_API_KEY npx -y agentql-mcp`
 
 Read more about MCP configuration in Cursor [here](https://docs.cursor.com/context/model-context-protocol).
 
@@ -128,7 +144,8 @@ Read more about MCP configuration in Cursor [here](https://docs.cursor.com/conte
       "command": "npx",
       "args": ["-y", "agentql-mcp"],
       "env": {
-        "AGENTQL_API_KEY": "YOUR_API_KEY"
+        "AGENTQL_API_KEY": "YOUR_API_KEY",
+        "YDC_API_KEY": "YOUR_YOUCOM_API_KEY"
       }
     }
   }
@@ -145,6 +162,12 @@ Give your agent a task that will require extracting data from the web. For examp
 Extract the list of videos from the page https://www.youtube.com/results?search_query=agentql, every video should have a title, an author name, a number of views and a url to the video. Make sure to exclude ads items. Format this as a markdown table.
 ```
 
+Or use the optional You.com helper first:
+
+```text
+search-web query="playwright locator best practices" count=5
+```
+
 > [!TIP]
 > In case your agent complains that it can't open urls or load content from the web instead of using AgentQL, try adding "use tools" or "use agentql tool" hint.
 
@@ -176,7 +199,8 @@ If you want to try out development version, you can use the following config ins
     "agentql": {
       "command": "/path/to/agentql-mcp/dist/index.js",
       "env": {
-        "AGENTQL_API_KEY": "YOUR_API_KEY"
+        "AGENTQL_API_KEY": "YOUR_API_KEY",
+        "YDC_API_KEY": "YOUR_YOUCOM_API_KEY"
       }
     }
   }
@@ -185,13 +209,3 @@ If you want to try out development version, you can use the following config ins
 
 > [!NOTE]
 > Don't forget to remove the default AgentQL MCP server config to not confuse Claude with two similar servers.
-
-## Debugging
-
-Since MCP servers communicate over stdio, debugging can be challenging. We recommend using the [MCP Inspector](https://github.com/modelcontextprotocol/inspector), which is available as a package script:
-
-```bash
-npm run inspector
-```
-
-The Inspector will provide a URL to access debugging tools in your browser.

dist/index.js

@@ -0,0 +1,161 @@
+#!/usr/bin/env node
+import { Server } from '@modelcontextprotocol/sdk/server/index.js';
+import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
+import { CallToolRequestSchema, ListToolsRequestSchema } from '@modelcontextprotocol/sdk/types.js';
+import fetch from 'node-fetch';
+const server = new Server({
+    name: 'agentql-mcp',
+    version: '1.0.1',
+}, {
+    capabilities: {
+        tools: {},
+    },
+});
+const EXTRACT_TOOL_NAME = 'extract-web-data';
+const SEARCH_TOOL_NAME = 'search-web';
+const AGENTQL_API_KEY = process.env.AGENTQL_API_KEY;
+const YDC_API_KEY = process.env.YDC_API_KEY;
+if (!AGENTQL_API_KEY) {
+    console.error('Error: AGENTQL_API_KEY environment variable is required');
+    process.exit(1);
+}
+function buildToolList() {
+    const tools = [
+        {
+            name: EXTRACT_TOOL_NAME,
+            description: 'Extracts structured data as JSON from a web page given a URL using a Natural Language description of the data.',
+            inputSchema: {
+                type: 'object',
+                properties: {
+                    url: {
+                        type: 'string',
+                        description: 'The URL of the public webpage to extract data from',
+                    },
+                    prompt: {
+                        type: 'string',
+                        description: 'Natural Language description of the data to extract from the page',
+                    },
+                },
+                required: ['url', 'prompt'],
+            },
+        },
+    ];
+    if (YDC_API_KEY) {
+        tools.push({
+            name: SEARCH_TOOL_NAME,
+            description: 'Searches the web with You.com and returns the top web and news results as JSON.',
+            inputSchema: {
+                type: 'object',
+                properties: {
+                    query: {
+                        type: 'string',
+                        description: 'Search query to send to You.com',
+                    },
+                    count: {
+                        type: 'number',
+                        description: 'Number of results per section to request',
+                        default: 10,
+                    },
+                },
+                required: ['query'],
+            },
+        });
+    }
+    return tools;
+}
+async function searchWeb(query, count = 10) {
+    const endpoint = new URL('https://ydc-index.io/v1/search');
+    endpoint.searchParams.set('query', query);
+    endpoint.searchParams.set('count', String(count));
+    const response = await fetch(endpoint.toString(), {
+        headers: {
+            'X-API-Key': `${YDC_API_KEY}`,
+        },
+        signal: AbortSignal.timeout(30_000),
+    });
+    if (!response.ok) {
+        throw new Error(`You.com API error: ${response.statusText}\n${await response.text()}`);
+    }
+    return (await response.json());
+}
+server.setRequestHandler(ListToolsRequestSchema, async () => {
+    return { tools: buildToolList() };
+});
+server.setRequestHandler(CallToolRequestSchema, async (request) => {
+    switch (request.params.name) {
+        case EXTRACT_TOOL_NAME: {
+            const rawUrl = request.params.arguments?.url;
+            const rawPrompt = request.params.arguments?.prompt;
+            if (rawUrl === undefined || rawUrl === null || rawUrl === '' || rawPrompt === undefined || rawPrompt === null || rawPrompt === '') {
+                throw new Error("Both 'url' and 'prompt' are required");
+            }
+            const url = String(rawUrl);
+            const prompt = String(rawPrompt);
+            const endpoint = 'https://api.agentql.com/v1/query-data';
+            const response = await fetch(endpoint, {
+                method: 'POST',
+                headers: {
+                    'X-API-Key': `${AGENTQL_API_KEY}`,
+                    'X-TF-Request-Origin': 'mcp-server',
+                    'Content-Type': 'application/json',
+                },
+                body: JSON.stringify({
+                    url: url,
+                    prompt: prompt,
+                    params: {
+                        wait_for: 0,
+                        is_scroll_to_bottom_enabled: false,
+                        mode: 'fast',
+                        is_screenshot_enabled: false,
+                    },
+                }),
+            });
+            if (!response.ok) {
+                throw new Error(`AgentQL API error: ${response.statusText}\n${await response.text()}`);
+            }
+            const json = (await response.json());
+            return {
+                content: [
+                    {
+                        type: 'text',
+                        text: JSON.stringify(json.data, null, 2),
+                    },
+                ],
+            };
+        }
+        case SEARCH_TOOL_NAME: {
+            if (!YDC_API_KEY) {
+                throw new Error('YDC_API_KEY environment variable is required for search-web');
+            }
+            const rawQuery = request.params.arguments?.query;
+            if (rawQuery === undefined || rawQuery === null || rawQuery === '') {
+                throw new Error("'query' is required");
+            }
+            const query = String(rawQuery);
+            const count = Number(request.params.arguments?.count ?? 10);
+            const json = await searchWeb(query, Number.isFinite(count) && count > 0 ? count : 10);
+            const results = {
+                web: json.results?.web ?? [],
+                news: json.results?.news ?? [],
+            };
+            return {
+                content: [
+                    {
+                        type: 'text',
+                        text: JSON.stringify(results, null, 2),
+                    },
+                ],
+            };
+        }
+        default:
+            throw new Error(`Unknown tool: '${request.params.name}'`);
+    }
+});
+async function main() {
+    const transport = new StdioServerTransport();
+    await server.connect(transport);
+}
+main().catch((error) => {
+    console.error('Server error:', error);
+    process.exit(1);
+});