Skip to content

feat: report instrumentation coverage from the calls a bundle records - #40

Merged
utsab345 merged 1 commit into
utsab345:mainfrom
how2how2how2-arch:feat/instrumentation-coverage
Nov 4, 2026
Merged

utsab345 merged 1 commit into
utsab345:mainfrom
how2how2how2-arch:feat/instrumentation-coverage

Conversation

@how2how2how2-arch

Copy link
Copy Markdown
Contributor

Refs #30.

stepfork coverage <bundle> reports the dependency calls a bundle recorded and how much evidence it carries for each. Every observed call is in exactly one of three states:

  • captured — the outcome is present and carries a digest replay can verify.
  • unfingerprinted — the outcome is present but carries no digest, so replay substitutes it while nothing lets you check it.
  • missing — the call was recorded and no outcome was captured for it.

The design decisions, so they can be argued with

The denominator is the calls the bundle holds, never the calls the agent made. #30 asks what StepFork can reliably observe, and the one thing it cannot observe is a call made without instrumentation: no event is written, so nothing in the bundle records that it happened. The report leaves that count null, with the reason printed beside it, instead of 0 — a printed zero reads as "the agent made no uninstrumented calls", which is the strongest possible claim from the weakest possible evidence. The word observed is in the key name and in the prose for the same reason. If you would rather the key were absent from the JSON entirely, that is a one-line change.

unfingerprinted is not coverage. A bundle can carry an outcome with no digest. Replay still substitutes it, so it looks covered, but nothing can say whether it is the value the tool returned. Counting it as captured would inflate the number on exactly the bundles that deserve the lower one.

An empty denominator reports NOT MEASURED, not 0.0. A trace with no dependency calls has no rate. Formatting that as zero announces a measurement where none was taken.

It is a report, not a gate. A bundle with missing calls still exits 0; the reading is in the counts. Exit 2 is reserved for a bundle that cannot be read.

Measured on this branch

Head 94dca23, parent e6875608.

  • pytest: 714 passed, 0 failed. The 8 tests in tests/unit/test_coverage.py are new; the same tree passes 705 without them (713 vs 714 elsewhere is tests/unit/test_storage.py:418, which skips outside a git checkout).
  • ruff check clean; ruff format --check 200 files clean; mypy strict clean across 135 files; mkdocs build --strict exit 0.
  • examples/coverage/demo.py runs offline and prints both cases: 2/2 captured, then 1/2 with one missing.
  • stepfork coverage against a real recorded bundle reports 1/1 observed calls (100.0%), captured 1, missing 0.

The tests are meant to fail on the defect they name. With the denominator changed to count only captured calls — the cheapest way to make a coverage number look good — test_uncaptured_call_is_counted_not_dropped, test_unfingerprinted_outcome_is_not_counted_as_captured and both CLI tests fail (4 failed, 4 passed). Restored, 8 pass.

What this does not claim

It does not enumerate uninstrumented calls; the module states that in a constant that both the human and JSON output print. And captured means the bundle holds a verifiable outcome, not that it is what the tool really returned — for that the digest has to travel by a route the bundle's author cannot edit, which is what validate --expect-digest covers. docs/cli.md says both, under "What this report does not cover".

#30 asks for a design comment before a PR. I put the design here rather than in a comment so it can be read against the code it decides. Happy to move it to the issue first, or to drop whichever part you would shape differently — the definition of coverage is the part most worth your view.

@utsab345
utsab345 merged commit 0ed307a into utsab345:main Nov 4, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants