Repository navigation
feat: report instrumentation coverage from the calls a bundle records - #40
Merged
utsab345 merged 1 commit intoNov 4, 2026
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #30.
stepfork coverage <bundle>reports the dependency calls a bundle recorded and how much evidence it carries for each. Every observed call is in exactly one of three states:captured— the outcome is present and carries a digest replay can verify.unfingerprinted— the outcome is present but carries no digest, so replay substitutes it while nothing lets you check it.missing— the call was recorded and no outcome was captured for it.The design decisions, so they can be argued with
The denominator is the calls the bundle holds, never the calls the agent made. #30 asks what StepFork can reliably observe, and the one thing it cannot observe is a call made without instrumentation: no event is written, so nothing in the bundle records that it happened. The report leaves that count
null, with the reason printed beside it, instead of0— a printed zero reads as "the agent made no uninstrumented calls", which is the strongest possible claim from the weakest possible evidence. The word observed is in the key name and in the prose for the same reason. If you would rather the key were absent from the JSON entirely, that is a one-line change.unfingerprintedis not coverage. A bundle can carry an outcome with no digest. Replay still substitutes it, so it looks covered, but nothing can say whether it is the value the tool returned. Counting it as captured would inflate the number on exactly the bundles that deserve the lower one.An empty denominator reports
NOT MEASURED, not0.0. A trace with no dependency calls has no rate. Formatting that as zero announces a measurement where none was taken.It is a report, not a gate. A bundle with missing calls still exits
0; the reading is in the counts. Exit2is reserved for a bundle that cannot be read.Measured on this branch
Head
94dca23, parente6875608.pytest: 714 passed, 0 failed. The 8 tests intests/unit/test_coverage.pyare new; the same tree passes 705 without them (713 vs 714 elsewhere istests/unit/test_storage.py:418, which skips outside a git checkout).ruff checkclean;ruff format --check200 files clean;mypystrict clean across 135 files;mkdocs build --strictexit 0.examples/coverage/demo.pyruns offline and prints both cases: 2/2 captured, then 1/2 with onemissing.stepfork coverageagainst a real recorded bundle reports1/1 observed calls (100.0%),captured 1,missing 0.The tests are meant to fail on the defect they name. With the denominator changed to count only captured calls — the cheapest way to make a coverage number look good —
test_uncaptured_call_is_counted_not_dropped,test_unfingerprinted_outcome_is_not_counted_as_capturedand both CLI tests fail (4 failed, 4 passed). Restored, 8 pass.What this does not claim
It does not enumerate uninstrumented calls; the module states that in a constant that both the human and JSON output print. And
capturedmeans the bundle holds a verifiable outcome, not that it is what the tool really returned — for that the digest has to travel by a route the bundle's author cannot edit, which is whatvalidate --expect-digestcovers.docs/cli.mdsays both, under "What this report does not cover".#30 asks for a design comment before a PR. I put the design here rather than in a comment so it can be read against the code it decides. Happy to move it to the issue first, or to drop whichever part you would shape differently — the definition of coverage is the part most worth your view.