Skip to content

Make parseGraderSelection permissive; keep validateEvalYAML strict for unknown-field detection#134

Draft
Copilot wants to merge 2 commits intolarryo/validate_yamlfrom
copilot/sub-pr-133
Draft

Make parseGraderSelection permissive; keep validateEvalYAML strict for unknown-field detection#134
Copilot wants to merge 2 commits intolarryo/validate_yamlfrom
copilot/sub-pr-133

Conversation

Copy link
Contributor

Copilot AI commented Mar 17, 2026

parseGraderSelection used KnownFields(true) when decoding the pass-1 LLM response, causing it to silently drop valid grader selections whenever the LLM included extra keys (e.g. reason:, explanation:) alongside graders:. Unknown-field validation belongs in validateEvalYAML, not here.

Changes

  • parseGraderSelection — removed KnownFields(true) from both structured and bare-list decoders. Extra keys in LLM output no longer prevent grader-type extraction.
  • validateEvalYAML — unchanged; retains KnownFields(true) as the authoritative place to catch unknown/invalid fields in the final generated eval YAML.
  • New testTestParseGraderSelectionExtraFields covers the previously-broken case:
# LLM response with extra field — was silently dropping "code" and "text"
graders:
  - code
  - text
reason: these graders cover the task

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

Co-authored-by: LarryOsterman <8220172+LarryOsterman@users.noreply.github.com>
Copilot AI changed the title [WIP] [WIP] Address feedback on yaml validation and configuration issues detection Make parseGraderSelection permissive; keep validateEvalYAML strict for unknown-field detection Mar 17, 2026
Copilot AI requested a review from LarryOsterman March 17, 2026 00:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants