Deepen architecture: dead-code removal, eval judge rewire, vector-store seam fix - #17
Merged
Merged
Conversation
ResponseGenerator, the utils.py helpers, and the traced_stage decorator all had zero production callers (verified by repo-wide grep, not just the review that flagged them). traced_stage's helpers (_coerce_attr, _OTEL_SCALARS) and its dedicated tests go with it; the one call site that referenced it in a comment (backend.py) is updated to describe the inline-span pattern actually in use. Resolves part of spec #16 sequencing step 1.
judge_faithfulness/judge_answer_relevancy/judge_context_precision in eval.metrics.generation were pure reshape wrappers around src.evaluation's judges — runner._score_question was their only caller. Deleting them concentrates the reshape at that one call site instead of leaving it as an indirection with nothing behind it. Also hoists the markdown-fence-stripping JSON parser (previously duplicated between evaluation.py and generation.py's factual-match judge) into evaluation.parse_json_response, the shared implementation. Adds direct coverage of _score_question's judge-metric wiring (previously only exercised indirectly via the runner's end-to-end test), guarding this rewire and any future one. Resolves part of spec #16 sequencing step 1.
…leak RAGBackend.get_document_chunks reached into self.vector_store._collection directly and parsed ChromaDB's raw batch response — the one place backend crossed the vector store's seam instead of going through its public interface. get_by_doc_id returns plain dicts (chunk_id, content, metadata) rather than SearchResult: this is a metadata filter, not a similarity search, so there's no meaningful score to attach, and forcing one onto SearchResult would misrepresent what the result is. Adds direct backend-level coverage of get_document_chunks, which had none before this change. Resolves the remainder of spec #16 sequencing step 1.
- Architecture.md's generation.py row still described the deleted wrapper functions; correct it to describe the direct-call path. - _score_question duplicated the (reasoning, details_json) -> dict reshape for faithfulness and context_precision; extract _judge_details as the single reshape used by both (answer_relevancy's 2-tuple return doesn't fit the same shape, so it stays inline). - Tighten the judge-metrics test: the previous assertion vacuously passed for answer_correctness instead of checking its value.
This was referenced Jul 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements sequencing step 1 (of 6) of #16.
Test plan
279 passed