Skip to content

feat: eval --query, and verify relevance splits across stub and store - #14

Merged
hardy30894 merged 1 commit into
mainfrom
feat-eval-query
Oct 23, 2026
Merged

hardy30894 merged 1 commit into
mainfrom
feat-eval-query

Conversation

@hardy30894

Copy link
Copy Markdown
Member

Stacked on #12.

What

secondwind eval --query replays each tool result with the user's latest message as the query, the way the proxy calls the optimizer on live traffic.

What it found

  1. The JSON relevance split is value-complete but my check missed it. It keeps the rows that match the query inline in the stub and stores only the rest, so resolve alone returns a subset. The faithfulness check now recognizes a split and requires the inline rows plus the stored rows to be exactly the original's rows, in any order. On the author's 1,491 sessions with --query: 0 of 138,448 blocks unfaithful. A test proves the check also catches one row missing or one row too many.
  2. A text version of the split does not pay, so it is not in this PR. I built one (keep the most relevant quarter of the lines inline with their line numbers, store the whole block) and replayed the corpus:
replay net context saved reopens per session
no query 24.6% 23.7
query, with a text split 8.5% 21.4

The larger stubs cost far more than the few reopens they prevent, so the query description stays as #12 left it: JSON record arrays only.

Tests

a_relevance_split_is_faithful_across_stub_and_store; the existing eval and gate tests run through the new options struct. Clippy clean.

@hardy30894
hardy30894 added this pull request to stack #15 October 23, 2026 02:36
Base automatically changed from feat-reopen-gate to main October 23, 2026 02:39
The proxy passes the user's message to the optimizer; --query lets eval measure that path. Its first run flagged five JSON blocks as unfaithful: a relevance split keeps the matching rows inline and stores only the rest, so the store alone never equals the original. The faithfulness check now recognizes a split and requires the inline and stored rows together to be exactly the original's rows. With it, 0 of 138,448 blocks are unfaithful with --query.
@hardy30894
hardy30894 merged commit 98f4b2e into main Oct 23, 2026
3 checks passed
@hardy30894
hardy30894 deleted the feat-eval-query branch October 23, 2026 02:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant