Repository navigation
Add proofcheck: evidence checks for pull requests - #39
Merged
Merged
Conversation
proofcheck runs each PR's changed tests on the base and head commits, replays reproducers from linked issues, maps which changed hunks the evidence depends on, and posts a proof card. Two projects: the Rust workspace (cargo) and the Python SDK in adapters/python (pytest). The agent workflow is opt-in and needs the PROOFCHECK_LLM_* secret and variables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017bQP5nwJVPMXqMhewzhQv5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds proofcheck to provio. On every pull request it runs the PR's changed tests on the base and head commits, replays reproducers from linked issues, maps which changed hunks the evidence depends on, and posts one comment (a "proof card") with a verdict label.
Generated with
proofcheck init --agent. It found two projects:rustat.: cargo. Changed integration tests (crates/*/tests/*.rs) and changed#[cfg(test)]modules are checked.pythonatadapters/python: pytest. Installed the same way asci.ymldoes (--no-deps -e, plus the test dependencies).It skipped
adapters/typescript(its tests usenode --test, which proofcheck doesn't support) andcrates/provio-cli/pyproject.toml(packaging for the Rust binary).The workflows, which pin
bhaskargurram-ai/proofcheck@v1.0while 1.0 is being tested:proofcheck.ymlruns onpull_requestwith a read-only token and no secrets. This is the only place PR code runs.proofcheck-comment.ymlruns onworkflow_runand posts the card and label. It never checks out or runs PR code.proofcheck-agent.ymlis opt-in. When a "fixes #N" claim has no evidence, it asks an LLM to write a reproduction test, then runs that test on base and head in a job without secrets. It needs thePROOFCHECK_LLM_API_KEYsecret and thePROOFCHECK_LLM_PROVIDER,PROOFCHECK_LLM_MODELandPROOFCHECK_LLM_REASONING_EFFORTvariables. It only runs for owners, members and collaborators, or for PRs labelledproofcheck:agent.The comment and agent workflows only run once this is on
main.🤖 Generated with Claude Code
https://claude.ai/code/session_017bQP5nwJVPMXqMhewzhQv5
Generated by Claude Code