Repository navigation
feat: eval --query, and verify relevance splits across stub and store - #14
Merged
Merged
Conversation
hardy30894
force-pushed
the
feat-eval-query
branch
from
October 23, 2026 02:36
3dec260 to
b9b92a3
Compare
The proxy passes the user's message to the optimizer; --query lets eval measure that path. Its first run flagged five JSON blocks as unfaithful: a relevance split keeps the matching rows inline and stores only the rest, so the store alone never equals the original. The faithfulness check now recognizes a split and requires the inline and stored rows together to be exactly the original's rows. With it, 0 of 138,448 blocks are unfaithful with --query.
hardy30894
force-pushed
the
feat-eval-query
branch
from
October 23, 2026 02:39
b9b92a3 to
7a3cb1f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #12.
What
secondwind eval --queryreplays each tool result with the user's latest message as the query, the way the proxy calls the optimizer on live traffic.What it found
resolvealone returns a subset. The faithfulness check now recognizes a split and requires the inline rows plus the stored rows to be exactly the original's rows, in any order. On the author's 1,491 sessions with--query: 0 of 138,448 blocks unfaithful. A test proves the check also catches one row missing or one row too many.The larger stubs cost far more than the few reopens they prevent, so the
querydescription stays as #12 left it: JSON record arrays only.Tests
a_relevance_split_is_faithful_across_stub_and_store; the existing eval and gate tests run through the new options struct. Clippy clean.