feat: Advanced web search — deep page reading instead of snippet-only search

[rexlunae]

Inspiration: Unsloth

Unsloth's web search doesn't just scan summaries — it actually visits pages to collect relevant information. This provides much more accurate/in-depth context. Uses DuckDuckGo for private search, then fetches and reads each result page.

The result: a local Qwen3.5-4B model searched 20+ websites and cited sources, with web search happening inside its thinking trace.

What RustyClaw could do

Upgrade RustyClaw's web_search tool (or add a new deep_search tool) that:

1. Search: Query search engine (Brave/Google via API, or DuckDuckGo for privacy)
2. Fetch top N results: Actually visit each result URL with web_fetch
3. Extract relevant passages: Find the parts of each page most relevant to the query
4. Synthesize: Return a structured response with findings + source citations
5. Crawl depth: Option to follow links one level deep for thorough research

Why this matters

Current web_search returns snippets. Agents frequently need to web_fetch those results manually. A combined search+read pipeline would be a compound tool that's more efficient (one tool call vs N+1) and higher quality (structured extraction vs raw HTML).

Scope

• rustyclaw-core: deep_search tool (or enhance web_search with a deep=true parameter)
• Configurable: max pages to fetch, max depth, timeout per page
• Respects robots.txt and rate limits
• Caches fetched pages within a session to avoid re-fetching

Prior art

• Unsloth: https://unsloth.ai/docs/new/studio/chat.md#advanced-web-search
• Perplexity's search+read pipeline
• Jina Reader API for content extraction
• Crawlee (already on RustyClaw's roadmap via issue feat: Crawlee integration — web scraping and browser automation tool #196)