The corpus runs when someone remembers, which means it runs after a change rather than against one. Every bug it has found, and it has found twenty, was found by a human deciding to run it.
What it would take
rust-corpus is private and large, so this is not a matter of adding a step to this repo's CI. Options worth weighing:
- a scheduled run in rust-corpus itself, reporting against the released plugin
- a run triggered by a release tag here, so every release has a pass rate
- a reduced set that runs per-commit, with the full graph on a schedule
The full graph is 1,051 crates and a cold build takes minutes, so per-commit on the whole thing is unlikely to be the answer.
What to publish
The pass rate, and the list of failures with a reason each. A number with no failure list is not evidence, since the interesting part is which crates and why.
Depends on
Worth doing after or alongside #30, so the number published is one of four rather than one alone.
How you would know
A pass rate that updates without anyone running anything, and a regression in it that nobody triggered by hand.
The corpus runs when someone remembers, which means it runs after a change rather than against one. Every bug it has found, and it has found twenty, was found by a human deciding to run it.
What it would take
rust-corpus is private and large, so this is not a matter of adding a step to this repo's CI. Options worth weighing:
The full graph is 1,051 crates and a cold build takes minutes, so per-commit on the whole thing is unlikely to be the answer.
What to publish
The pass rate, and the list of failures with a reason each. A number with no failure list is not evidence, since the interesting part is which crates and why.
Depends on
Worth doing after or alongside #30, so the number published is one of four rather than one alone.
How you would know
A pass rate that updates without anyone running anything, and a regression in it that nobody triggered by hand.