Give LangChain agents live web data — Google SERPs, page fetching, and crawling — using the official langchain-scrapeless integration, powered by Scrapeless.
Two examples, in the order you should run them: tools first, agent second.
An LLM cannot answer a question about a page it has never seen. These tools are the retrieval layer for:
- Research agents — search, open the results, and answer from what the pages actually say.
- RAG ingestion — crawl a documentation site into Markdown, then chunk and embed it.
- Competitive monitoring — fetch a rival's pricing page on a schedule and let the model summarize the delta.
- Grounded answers — replace "as of my training data" with a live fetch.
- Python 3.9+
- A Scrapeless API key — create a free account
- An LLM key (OpenAI by default) — only for example 2's full run
git clone https://github.com/<owner>/scrapeless-langchain-examples.git
cd scrapeless-langchain-examples
pip install -r requirements.txt
cp .env.example .env
export SCRAPELESS_API_KEY=your_key_hereLangChain tools are plain callables. Run them with no model, no agent, and no LLM key. Do this first: it proves the data layer works before you spend tokens on the reasoning layer.
python3 01_tools_direct.py
python3 01_tools_direct.py --only search,unlockerReal output (2026-08-11):
=== search ===
ok 16,640 chars
=== trends ===
KNOWN BAD ValueError: ... /api/v1/scraper/result/... failed with status 400
known issue: result polling returns HTTP 400 (failed 3/3 attempts on 2026-08-11)
=== unlocker ===
ok 167 chars
=== scrape ===
ok 535 chars
=== crawl ===
ok 26,871 chars
4/5 tools returned data, 1 known-broken skipped
| Tool | Class | Status |
|---|---|---|
| Google Search | ScrapelessDeepSerpGoogleSearchTool |
✅ working |
| Universal Scraping | ScrapelessUniversalScrapingTool |
✅ working |
| Crawler Scrape | ScrapelessCrawlerScrapeTool |
|
| Crawler Crawl | ScrapelessCrawlerCrawlTool |
|
| Google Trends | ScrapelessDeepSerpGoogleTrendsTool |
❌ broken upstream |
Works with OpenAI or OpenRouter — whichever key is set. OpenRouter is picked up automatically, which keeps a test run to a fraction of a cent:
python3 02_react_agent.py --dry-run # no LLM key needed
export OPENROUTER_API_KEY=your_key # or OPENAI_API_KEY
python3 02_react_agent.py "What does scrapeless.com claim it does?"
python3 02_react_agent.py --model mistralai/mistral-nemo "..."A verified run on the cheap default (mistralai/mistral-nemo, ~$0.019/1M input tokens):
provider: OpenRouter · model: mistralai/mistral-nemo
================================ Human Message =================================
Use scrapeless_universal_scraping to fetch https://example.com and tell me the exact h1 text.
================================== Ai Message ==================================
Tool Calls:
scrapeless_universal_scraping
Args:
url: https://example.com
================================= Tool Message =================================
{"headings":["Example Domain"]}
================================== Ai Message ==================================
The h1 text on the page is: Example Domain
1 tool call(s) made.
The script counts tool calls and exits 1 if the model answered without calling one — an ungrounded answer is a failure, not a success.
Measured, same model: a one-tool instruction ("fetch this URL and report the h1") worked perfectly. A two-step instruction ("search, then fetch the top result") produced only one tool call — it ran the search and then answered from the result snippets instead of fetching the page, while sounding confident about what "the page claims."
If you need a multi-step chain, either:
- Split the steps yourself — call
ScrapelessDeepSerpGoogleSearchTool, pick the URL in code, then callScrapelessUniversalScrapingTool. Deterministic and cheaper than paying a model to decide. - Or use a stronger model — pass
--modelwith something above the bargain tier.
Small models vary enormously here, and the failure is quiet: you get a fluent answer built from snippets. That is exactly the case the tool-call counter catches.
A single SERP response is ~15 KB of JSON, and only ~10 KB of it is organic_results. The rest — related_searches, video blocks, pagination — goes straight into the model's context if you hand it the raw response, crowding out your instruction and inflating cost. Filter before passing to the LLM:
response = ScrapelessDeepSerpGoogleSearchTool().invoke({"q": "web scraping api"})
slim = [{"title": r["title"], "link": r["link"]} for r in response.get("organic_results", [])[:5]]Bug to know about: every link inside related_searches comes back with a malformed scheme — htts:// instead of https:// (8 occurrences per response, consistently). organic_results links are unaffected. If you follow related-search links programmatically, repair the scheme first.
--dry-run builds the real agent graph and prints the tools the model would be offered, without calling a model. Use it to check wiring before spending tokens:
graph built without calling a model.
tools offered to the model (2):
- scrapeless_deepserp_google_search
- scrapeless_universal_scraping
nodes in graph: ['__end__', '__start__', 'model', 'tools']
The agent gets two tools, not five, on purpose: a ReAct loop holding five overlapping scrapers picks the wrong one far more often than one holding an obviously-distinct search tool and fetch tool.
langgraph.prebuilt.create_react_agent is deprecated since LangGraph v1.0 in favor of langchain.agents.create_agent. Both accept (model, tools), so this repo prefers the new import and falls back:
try:
from langchain.agents import create_agent as build_agent
except ImportError: # langchain < 1.0
from langgraph.prebuilt import create_react_agent as build_agentCopying the deprecated import from older tutorials still works today but prints a warning and breaks at LangGraph v2.
Every item below was reproduced against the live API on 2026-08-11 with langchain-scrapeless 0.1.3.
1. Google Trends is broken. ScrapelessDeepSerpGoogleTrendsTool failed 3 of 3 attempts. The job is created, then the result fetch fails:
GET https://api.scrapeless.com/api/v1/scraper/result/<uuid> failed with status 400
Example 1 marks it KNOWN BAD — it still runs so you can tell when it's fixed, but it does not fail the script. If you need trends data, call the scraper.google.trends actor directly and handle the polling yourself.
2. ScrapelessCrawlerScrapeTool silently drops a URL. Pass N>1 URLs and you get N−1 results back:
| URLs sent | total returned |
Result |
|---|---|---|
| 3 | 2 | dropped one silently |
| 2 | 1 | dropped one silently |
| 1 | 1 | correct |
Which URL is dropped varies between runs, consistent with a race on job completion rather than a positional bug. Nothing in the response signals the loss — success is still true. Always pass one URL per call and loop; scrape_many() in example 1 shows the workaround.
3. Both crawler tools return empty successes. The same single-URL scrape that had just worked came back as:
{ "success": true, "status": "completed", "completed": 0, "total": 0, "data": [] }success: true with nothing in it. Checking the flag gives you silent blanks in your pipeline; check data instead. Example 1's require_data() rejects these and retry_until_data() retries up to 3 times — which is what made the run above pass.
| Symptom | Cause and fix |
|---|---|
SCRAPELESS_API_KEY is not set |
Export the key or fill in .env. |
OPENAI_API_KEY is not set |
Only example 2's full run needs it. Use --dry-run to verify wiring without a model. |
| Trends tool raises a 400 | Known upstream break. See issue 1. |
| Fewer pages than URLs you passed | Known N−1 bug. See issue 2 — pass one URL per call. |
Empty data: [] with success: true |
Known flake. See issue 3 — retry. |
LangGraphDeprecatedSinceV10 warning |
You are on the old import. See the v1 note above. |
| Agent ignores the tools | Be explicit in the prompt: name the tool you want used. |
- Example 1: fully verified live. 4 of 5 tools returned real data; the 5th fails upstream and is labeled as such. Output committed to
results/tools-output.json. - Example 2: fully verified live via OpenRouter on
mistralai/mistral-nemo— the agent calledscrapeless_universal_scraping, received real page data, and answered from it.--dry-runis separately verified and needs no LLM key. The multi-step limitation documented above is a measured result from the same run, not a caveat added for safety.
scrapeless-langchain-examples/
├── 01_tools_direct.py # all five tools, no LLM, with retry + empty-payload guards
├── 02_react_agent.py # LangGraph/LangChain agent, --dry-run needs no LLM key
├── results/tools-output.json
├── requirements.txt
└── LICENSE
- Product: Deep SERP API · Universal Scraping API
- Guides: LangChain + Scrapeless · Web scraping with LangChain · LangChain Scrapeless MCP
- Package:
langchain-scrapelesson PyPI · scrapeless-ai/langchain-scrapeless
MIT — see LICENSE.