This applies to scanpy as well, but it's easier to try these things here since the test-suite runs faster.
There should be some kind of check on PRs for coverage. It should be really obvious if new code is tested, or if coverage goes down. There have a been a few cases in scanpy (mostly with external) where code just wasn't being tested on CI due to configuration issues. I'd like to avoid this.
Codecov
Codecov seems like a popular solution. I think we had it enabled at one point, but don't remember why we stopped. I remember finding it annoying, and that it gave a lot of notifications. I assume there are ways to make it less annoying, since I haven't been bothered by it recently when I contribute to repositories with it on.
It would be pretty easy to turn on, just add this to test coverage jobs.
It seems very configurable, but without a huge amount of documentation.
Options for integrations include:
- a bot comment on a PR with coverage information that gets updated
- Using github checks
Features I'd like
- I'd like to be able to see which test covered which lines. I don't remember seeing this happen with codecov, but maybe they allow this now. We can generate this output with contexts.
- A diff of coverage for a PR
- A badge for the repo
- Not necessarily failing a PR if some coverage isn't met, but definitely making it visible
This applies to scanpy as well, but it's easier to try these things here since the test-suite runs faster.
There should be some kind of check on PRs for coverage. It should be really obvious if new code is tested, or if coverage goes down. There have a been a few cases in scanpy (mostly with external) where code just wasn't being tested on CI due to configuration issues. I'd like to avoid this.
Codecov
Codecov seems like a popular solution. I think we had it enabled at one point, but don't remember why we stopped. I remember finding it annoying, and that it gave a lot of notifications. I assume there are ways to make it less annoying, since I haven't been bothered by it recently when I contribute to repositories with it on.
It would be pretty easy to turn on, just add this to test coverage jobs.
It seems very configurable, but without a huge amount of documentation.
Options for integrations include:
Features I'd like