Skip to content

smt_solver: report an evaluator crash as invalid - #45

Draft
findalexli wants to merge 1 commit into
ByteDance-Seed:mainfrom
findalexli:fix/smt-evaluator-status
Draft

findalexli wants to merge 1 commit into
ByteDance-Seed:mainfrom
findalexli:fix/smt-evaluator-status

Conversation

@findalexli

Copy link
Copy Markdown
Contributor

Task finding: #42. Merge after #43, which removes the crash this finding reports; until then the solver in #42 would go from a valid 0 to invalid.

What and why

When evaluate.py crashes, parse_result.py finds no score file and reports valid: true, score 0, so a judge failure looks like a solver that scored nothing. run.sh now passes on the evaluator's exit status, and no score file with a status other than 0 or 2 gives valid: false. evaluate.py's "solver is not executable" exit changes from 1 to 2, so a missing solve stays a valid 0, as today. This looks only at a missing score file, so Known issue 1 stays.

Old vs new judge

Old key cd3a5bf1f538, new 3ef6c9298492; sforge eval, one worker, order swapped, n=2 per judge. The new image is the published old tag plus this branch's judge files, because two full builds don't fit a worker's disk.

Submission Old New
The reproducer in #42, evaluator crashes 0, valid, correct=0/0 Invalid, "evaluator exited with status 1 before writing a score file"
Fails to build, or no solve, or rejected by preflight 0, valid, correct=0/0 Same
Always unknown, or wrong models 0, valid, correct=0/774 Same
Replay fixture, correct models on 42 files 6.28, correct=42/774 Same

A valid 0 from a crashed evaluation has no score to map; regrade it. No other score moves.

Checklist for task changes

  • version in task.toml is bumped (any change that can move scores). If smt_solver: validate deep models on a large stack #43 merges first, this becomes 1.2.0.
  • NOTES.md: the change and its score effect are under Changes. No Known issue covered this.
  • Old vs new judge on the same submissions, on the same host, results above.
  • New blobs: none.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant