fix(graders): record refused verdicts from the LLM graders - #46
Open
Krishnachaitanyakc wants to merge 1 commit into
Open
fix(graders): record refused verdicts from the LLM graders#46Krishnachaitanyakc wants to merge 1 commit into
Krishnachaitanyakc wants to merge 1 commit into
Conversation
`_parse_grade_response` collapsed every non-`correct` verdict to INCORRECT, so the `refused` verdict that both OPEN_ENDED_GRADING_PROMPT and OPEN_ENDED_RANGE_GRADING_PROMPT instruct the grading model to emit could never be returned. `GradeResult.refusal` was therefore always False on `_grade_llm_verifier` and `_grade_range_llm_verifier`, making an abstention indistinguishable from a wrong answer in every open-ended result. `_grade_str_verifier` already handled all three cases. This is visible in the committed baselines: all 205 rows of each open-ended grader CSV carry `sure=False`, including answers such as "The specific p-value cannot be determined without the actual data." Map the three documented verdicts explicitly. Anything else (a missing or malformed <grade> tag, or an unexpected verdict string) keeps its current INCORRECT result, since an unreadable verdict is a formatting failure by the grading model rather than an abstention by the answerer. No accuracy figure changes: GradeType.REFUSED.numeric_grade is already 0 and is_correct is already False, so the `grade` and `correct` columns are identical for every input. Only `refusal` (persisted as the `sure` column by grade_outputs.py) gains True values, which moves precision, coverage and n_sure, the numbers those metrics were always meant to report.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both grading prompts ask for
correct,incorrectorrefused, but_parse_grade_responsemaps everything exceptcorrecttoINCORRECT, sorefusedis never returned andGradeResult.refusalstaysFalseon both LLM paths.Visible in the committed baselines: all 205 rows in both
*-grader-openended.csvfiles havesure=False, including answers like "The specific p-value cannot be determined without the actual data."gradeandcorrectare unchanged for every input sinceREFUSED.numeric_gradeis already0. OnlyrefusalgainsTruevalues, so regenerating open ended results would moveprecision,coverageandn_sure.