AgentWeb Next

AgentWeb Next is a safer, smaller rebuild of the original AgentWeb idea: web access for AI agents with structured search, fetch, and research outputs.

Why this rewrite exists

This repo fixes the main issues found in thienduy2211/agentweb:

• Modular architecture instead of giant core.py / deep_research.py files.
• Tests are first-class and must pass in CI.
• Docs and install paths point to thienduy2211/agentweb-next.
• Localhost/private IP fetches are blocked by default to reduce SSRF risk.
• No heavy reranker/model download in the default path.
• No undocumented auth/stealth promises. Experimental features must be exposed before being documented.

Install

uv tool install 'agentweb-next @ git+https://github.com/thienduy2211/agentweb-next.git'

Or for development:

git clone https://github.com/thienduy2211/agentweb-next.git
cd agentweb-next
uv sync --extra dev
uv run pytest

Commands

agentweb-next search "Hermes Agent" --format json
agentweb-next fetch https://example.com --format json
agentweb-next research "Python agents web access" --format markdown

Python SDK

from agentweb_next import AgentWeb

aw = AgentWeb()
print(aw.search("python agents"))
print(aw.fetch("https://example.com"))
print(aw.research("python agents"))

Security posture

fetch blocks private, loopback, link-local, unspecified, multicast, and cloud metadata IPs by default. For local development tests only, pass an explicit policy in Python:

from agentweb_next.safety import URLPolicy

CLI intentionally does not expose an allow-localhost flag yet.

Scope

This is an initial clean MVP, not a drop-in clone of the old repo. It prioritizes correctness, safety, packaging, tests, and maintainability.