Skip to content

Add Siftdog vs Tavily search benchmark - #16

Merged
khsarvar merged 2 commits into
mainfrom
bench/tavily-benchmark
Sep 25, 2026
Merged

khsarvar merged 2 commits into
mainfrom
bench/tavily-benchmark

Conversation

@khsarvar

@khsarvar khsarvar commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Adds a reproducible benchmark comparing Siftdog's /search with Tavily's, so any published numbers can be checked by running one command. Results themselves aren't in this PR; a dated results page will follow separately.

What's in bench/

  • queries.jsonl: 45 queries. 25 factual questions with a known answer, 10 docs questions with an official documentation site, 5 news, 5 open-ended.
  • run.py: runs every query at basic and advanced depth against both services through the official tavily-python client; only api_base_url differs. Service order alternates per query, requests are paced (--delay), and failures are counted rather than stopping the run. --services siftdog runs without a Tavily key; --limit N runs a subset.
  • report.py: writes REPORT.md with latency (median, 95th percentile), errors, factual answer found (anywhere and top 3), official docs site found, URL and domain overlap (Jaccard) and Tavily cost per 1,000 searches, plus per-query scores. No returned content goes into the report.
  • README.md: method, how to run, and caveats: pacing (SearXNG engines block bursts), small samples, latency including network path, results as a dated snapshot.
  • requirements.txt: tavily-python, kept out of Siftdog's own dependencies.
  • .gitignore: bench/results/, because raw responses contain third-party web content.

Not shipped: the PyPI source package includes only src/siftdog, tests, README, LICENSE and pyproject, and the wheel only the package. CI lints bench/ but doesn't run it.

Verification

  • Full run against Tavily and Siftdog 0.2.1: 180 requests, 0 errors. Report generated and spot-checked against the raw URLs (a surprising docs result was confirmed from the raw data, not a scoring bug).
  • Report checked on a partial run (--limit), Siftdog-only, and synthetic two-service data (overlap and cost columns).
  • ruff clean; 49 tests pass.

@khsarvar
khsarvar merged commit 7f21e2b into main Sep 25, 2026
4 checks passed
@khsarvar
khsarvar deleted the bench/tavily-benchmark branch September 25, 2026 18:23
@khsarvar khsarvar changed the title Add Sifthound vs Tavily search benchmark Add Siftdog vs Tavily search benchmark Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant