research-agent-harness

research-agent-harness is a minimal, readable research agent with pluggable retrieval toolkits. It includes example data-analysis queries that you can try.

The agent is built on pydantic-ai. It follows the same contract with every toolkit:

• Gather data with the retrieval tools.
• Do all computation in a sandboxed run_python tool (pandas and numpy, no network access).
• Save the working datasets as CSV files.
• Finish with a markdown report and structured key_findings.

When you swap the retrieval toolkit, everything else stays fixed. Differences in the output therefore come from the toolkit, not from the scaffolding.

Toolkits

Toolkit	Retrieval tools	Key
tako	tako_search + tako_contents, via the Tako Python SDK	TAKO_API_KEY
openai_web	OpenAI's built-in web search (Responses API)	—
exa	exa_search + exa_contents	EXA_API_KEY
parallel	parallel_search + parallel_extract	PARALLEL_API_KEY

The agent uses OpenAI as its model provider for every toolkit, so OPENAI_API_KEY is always required.

Every retrieval tool is a function tool that this harness authors over the provider's API. There are no MCP servers. Tool names, descriptions, result shaping, and payload capture therefore use the same mechanism in every toolkit, and this repo's git rev pins them. Two asymmetries remain, and we disclose them:

• Tool descriptions follow a fair-play rule. The Exa and Parallel descriptions carry the usage guidance that their vendors ship. The Tako descriptions carry ours, because we maintain this repo and each vendor ships its own best guidance.
• openai_web is a fused server-side tool. Its search results exist only in model context, so the harness cannot capture payloads from it.

Install

The harness requires Python 3.12+ and uv.

uv sync
cp .env.example .env   # fill in keys

Quickstart

# Run one query with one toolkit
uv run python -m harness.run --query energy_spain_vietnam --toolkit tako

# Run every query against every toolkit, 3 repeats each
uv run python -m harness.run --query all --toolkit all --n 3

Then read the artifacts under runs/ (see below). To point the agent at your own question, add an entry to queries/queries.yaml.

How a run works

Each run writes a self-contained artifact directory:

runs/<query>/<toolkit>/run_NN/
  trace.json         # full message history: tool calls, args, timings
  report.md          # the agent's final markdown analysis
  findings.json      # the structured key_findings it committed to
  meta.json          # model, tokens, cost, wall-clock, tool-call counts,
                     #   plus reproducibility snapshots (system prompt,
                     #   git rev, resolved model, API host)
  workspace/         # the agent's working dir: scripts + CSVs it wrote
    retrieved/       # harness-captured copy of every retrieval payload

Two details do most of the work:

• Payload capture. The harness saves every retrieval-tool result to workspace/retrieved/ at the tool boundary. It then appends a marker that tells the model where the file is, so the model can pd.read_csv() its data instead of retyping numbers out of the conversation. The harness controls the record of what data each toolkit HAD; the working CSV files show what data the agent chose to USE.
• Structured findings. Every run ends with the same typed output: a report plus key_findings (name, value, unit, source URL). The findings make the agent commit to specific numbers. You can then compare its claims with the data it retrieved.

meta.json records operational stats only: tool-call counts, per-call latency, wall-clock time, token usage, and model cost. There is no automated scoring. To read a run, read its report against the data in workspace/.

The shared system prompt is deliberately minimal and toolkit-neutral. Any retrieval-strategy hint would come from the harness, not from the toolkit. The prompt carries only the analyst framing and the artifact contract.

Example queries

queries/queries.yaml ships with three queries, one per vertical: macro, housing, and energy. Each is a real data-analysis task. A good answer needs a complete time series or a cross-entity panel — superlatives, counts, streaks, and crossover years that no single page states. The three queries are a starting point. Add your own with a new id and prompt.

Adding your own toolkit

You need one module and one registry line. Create harness/toolkits/mytool.py:

from harness.capture import PayloadRecorder, wrap_tool
from harness.toolkits.base import Toolkit, ToolkitParts
from harness.toolkits.errors import tool_errors_to_text


@tool_errors_to_text
def mytool_search(query: str) -> str:
    """Your model-facing tool description."""
    ...  # call your API, return readable text


def build(recorder: PayloadRecorder) -> ToolkitParts:
    return ToolkitParts(tools=[wrap_tool(mytool_search, recorder)])


TOOLKIT = Toolkit(name="mytool", build=build)

Then add it to TOOLKITS in harness/toolkits/registry.py. Three rules apply:

• Tools return plain, readable text. They never raise — use tool_errors_to_text.
• Wrap retrieval tools with wrap_tool so that payload capture stays uniform.
• Put your best usage guidance in the docstring. The docstring is the description the model sees.

Notes and limitations

• Run variance is real. Use --n 3 and read the spread. A single run proves little.
• The sandbox is not a security boundary. run_python executes in a subprocess with a cooperative socket guard. The guard blocks network access so that the model cannot route around its retrieval tools. But the guard is at the same trust tier as a timeout — it is not isolation. If that matters to you, run untrusted queries in a container.
• The default model is set by AGENT_MODEL in .env. Use OpenAI Responses API models. The harness records the resolved model snapshot for each run.

License

The code is licensed under Apache 2.0. The example queries (queries/) are licensed under CC BY 4.0.