Fetch Details agent: LLM evaluation framework (DeepEval, dataset, CI workflow)

[rafacm]

Follow-up to #114. Introduce an LLM evaluation framework for the Fetch Details step that lets us compare agent performance across iterations of prompt, model, and tools.

The structured-heavy report schema landed in #114 was designed with this in mind — outcome, source_kind, audio_url, extraction_confidence, cross_linked, discovered_* flags are all field-level evaluable without LLM-as-judge. This issue is about the framework, dataset, and CI gating that turn those fields into a regression signal.

Scope

Dataset

Hand-curated YAML golden set, ~10-15 cases, in episodes/tests/fixtures/fetch_details/cases.yaml. Cover:

• All five outcomes (ok, partial, not_a_podcast_episode, unreachable, extraction_failed).
• Canonical-URL cases (rtve.es, ardsounds.de, similar publisher sites).
• Aggregator-URL cases (Apple Podcasts).
• Cross-linking cases (canonical → iTunes lookup; aggregator → canonical link).
• A 404 / non-podcast-page case for not_a_podcast_episode.
• A Spotify-style JS-only page for extraction_failed (or partial if static fetch returns enough).
• 2-3 absolute-fact cases (URL X should yield title='...') so v0 flaws don't get baked in as the de facto baseline.

Framework

Sharp split:

• Field-level metrics (pytest, no LLM judge): outcome match, source_kind match, aggregator_provider match, required-field population, audio_url resolves via HTTP HEAD when expected, cross_linked flag matches, extraction_confidence calibration across the suite.
• Behavioral metrics (pytest, from tool trace): "for canonical-with-no-audio cases, did the agent call search_apple_podcasts?", median tool calls per case, ratio of correct cross-linking decisions.
• Prose quality (DeepEval, gated marker): narrative faithfulness vs trace, hints_for_next_step usefulness. Run on demand; not in the regular CI loop.

Test files:

• episodes/tests/test_fetch_details_eval.py — pytest, parametrized over the YAML cases.
• episodes/tests/test_fetch_details_prose_eval.py — DeepEval, behind @pytest.mark.deepeval or similar.

CI workflow

New separate workflow .github/workflows/fetch-details-eval.yml:

on:
  push:
    branches: [main]
    paths:
      - episodes/agents/fetch_details*.py
      - episodes/agents/_model.py
      - episodes/fetch_details_step.py
      - episodes/podcast_aggregators/**
      - episodes/tests/test_fetch_details*.py
      - episodes/tests/fixtures/fetch_details/**
      - .github/workflows/fetch-details-eval.yml
  pull_request:
    branches: [main]
    paths: [...same as above...]
  workflow_dispatch:

Mirror the Postgres service from ci.yml. Pin RAGTIME_FETCH_DETAILS_API_KEY as a GitHub Actions secret.

Stability

• VCR cassettes (vcrpy) for the HTTP layer (iTunes, fyyd, target podcast pages) so a third-party outage doesn't break the eval. LLM calls stay live.
• Cassettes regenerated on demand with a make target / pytest flag; reviewed in PRs that touch them.

Open decisions

• CI model: production vs cheaper. Production model gives faithful signal but costs more per CI run; a cheaper model (e.g., gpt-4.1-mini if not already production) tests infra but not production quality. Decide based on actual cost data once we run a few real evals.

Versioning add-on

Add agent_config_hash column to FetchDetailsRun:

agent_config_hash = models.CharField(max_length=16, blank=True, default="", db_index=True)
# = sha256(system_prompt + model_string + sorted_tool_names)[:16]

Computed once at get_agent() init, stamped on each run. Eval groups runs by hash to compare across versions. Caveat: tool names not implementations — coarse signal, but eval observes implementation behavior anyway.

Acceptance criteria

• pytest -m fetch_details_eval runs the field-level + behavioral suite locally and in CI on path-filtered triggers.
• pytest -m deepeval runs the prose suite on demand.
• VCR cassettes covering the dataset cases.
• agent_config_hash migration + write-on-run wiring.
• Documentation: doc/plans/, doc/sessions/, doc/features/, CHANGELOG.

Related

• Implementation PR: Fetch Details agent: cross-linking, outcomes, and per-run history #114
• Plan in Fetch Details agent: cross-linking, outcomes, and per-run history #114: doc/plans/2026-04-29-fetch-details-cross-linking.md ("Deferred to follow-up issue" section)

[rafacm]

Regression case to add to the dataset when this lands:

Unreachable submitted URL must not fall through to aggregator search.

A URL on a non-resolving domain (e.g. https://foo.foo/wiki/Django_Reinhardt) was producing a fabricated ok outcome by extracting Django_Reinhardt from the slug, calling search_apple_podcasts, and adopting an unrelated Apple result as the "right" episode. Tightened the system prompt in #116 to require:

1. fetch_url is always the first tool call, on the submitted URL
2. If it returns FETCH_FAILED, the run is unreachable and all EpisodeDetails are null/empty — no aggregator fall-through
3. Aggregator search queries must come from titles/show names extracted out of a successfully fetched page, never from URL slugs / query params / domain names

Suggested eval cases:

• unreachable_dns_failure: https://this-domain-does-not-exist-xyz.example/foo → outcome=unreachable, tool_trace == ["fetch_url"], all details null.
• unreachable_404: a known-404 path on a real host → same expectation.
• slug_does_not_seed_search: any unreachable URL whose slug looks like a real podcast title (e.g. .../django-reinhardt-episode-1) → assert search_apple_podcasts is NOT in the tool trace.

Verified manually post-fix: outcome=unreachable, only fetch_url called, no fabricated metadata. Behavioral assertion ("for unreachable cases, tool trace contains only fetch_url") fits cleanly under the "Behavioral metrics" bucket already scoped in this issue.

[rafacm]

A second regression case for the dataset — sibling of the unreachable one above.

Loaded-but-not-a-podcast page must not fall through to aggregator search.

https://en.wikipedia.org/wiki/Django_Reinhardt (any non-podcast page that loads cleanly: wiki, blog post, search result, homepage) was producing a fabricated ok outcome by treating the article's subject as an episode title, calling search_apple_podcasts for "Django Reinhardt", and adopting an unrelated Apple result as the match. Tightened the prompt in #116 (commit e2a36a9) so:

• A page that loads but isn't a podcast episode page yields outcome=not_a_podcast_episode with all EpisodeDetails null/empty.
• Cross-linking is gated on the submitted page being itself a podcast episode page (source_kind in {canonical, aggregator}); searching aggregators using the page's subject as a query is explicitly forbidden.

Suggested eval cases:

• not_a_podcast_episode_wiki: a Wikipedia article URL → outcome=not_a_podcast_episode, tool_trace == ["fetch_url"], all details null.
• not_a_podcast_episode_homepage: a publisher's homepage (not an episode page) → same expectation.
• not_a_podcast_episode_blog_post: a non-podcast blog post that mentions a podcast topic → assert search_apple_podcasts is NOT in the trace, all details null.

Verified manually post-fix: outcome=not_a_podcast_episode, only fetch_url called, no Apple fallback. Same "tool trace contains only fetch_url" behavioral assertion as the unreachable cases.

[rafacm]

A third regression case for the dataset — an extraction-validation failure rather than a fall-through one.

Extracted image_url must actually serve an image.

For https://www.ardsounds.de/episode/urn:ard:section:18e7d9e4486970f4/ the agent extracts an Apple is1-ssl.mzstatic.com/image/thumb/... URL whose path looks legitimate (absolute https, plausible JPG filename) but whose endpoint requires an imgShow query parameter. Hitting it returns HTTP 400 with body:

com.apple.amp.img.GraalJSBadRequestException: 400:Missing imgShow query parameter

So the agent emits outcome=ok with extraction_confidence=high and a broken image_url. The schema-level validators (URL_RE, absolute-https check) can't catch this — the URL is well-formed, just non-resolvable as an image.

Suggested eval cases / metric:

• Field-level metric: for any case where image_url is non-null, HEAD it and assert 200 with Content-Type: image/*. Same shape as the planned audio_url HEAD check in this issue's "Field-level metrics" bullet — extending it to image_url is cheap.
• image_url_resolves_ardsounds: ardsounds.de episode URL → image HEAD must return 200 + image/*. Catches the mzstatic-thumb-without-imgShow regression specifically.
• image_url_resolves_canonical_preferred: when both publisher-canonical and Apple mzstatic images are present on the page, the resolved one should be the canonical (publisher) image. Apple's mzstatic thumb URLs have a history of breaking when the API contract changes — so if both are available, prefer the canonical to reduce dataset-rot. (Soft assertion: a quality-of-extraction signal, not a hard pass/fail.)

Verified manually: curl -I of the extracted URL returns HTTP/2 400 with the GraalJSBadRequestException above.