Skip to content

feat(fork): report recorded outputs an override may have invalidated - #42

Open
how2how2how2-arch wants to merge 1 commit into
utsab345:mainfrom
how2how2how2-arch:feat/classify-stale-downstream-outputs
Open

how2how2how2-arch wants to merge 1 commit into
utsab345:mainfrom
how2how2how2-arch:feat/classify-stale-downstream-outputs

Conversation

@how2how2how2-arch

Copy link
Copy Markdown
Contributor

Refs #33.

What this changes

A fork copies every event at or before the fork point, and an override replaces one of those outputs. A recorded output captured after the overridden step was produced under the upstream state the override changed, so replay substituting it is not evidence for the modified execution path. Today those outputs are indistinguishable from an ordinary substitution: MatchedCall.action is only substituted or executed, and nothing on the fork says which recorded outputs an override upstream invalidated.

fork_trace() now reports both facts on its result:

  • overridden_steps — which steps were replaced (a set, not the existing overrides_applied count)
  • stale_event_ids — the retained recorded outputs that follow the earliest of them

The CLI fork report shows them, and docs/fork.md gained a section on what the classification does and does not prove.

The limitation is stated, not implied

This is an upper bound, not dependency analysis: every recorded output after the earliest overridden step is reported, whether or not its inputs actually changed. The event the override replaced is not reported as stale, because its value is the override rather than a reused recording.

Replay does not consult the classification yet, so a call reported as substituted is still not by itself proof that it belongs to the modified path. Neither is the classification persisted into a bundle, so it is available only from the fork_trace() call that produced it. Both are written into the docs rather than left to be discovered.

Tests

tests/unit/test_fork_staleness.py, offline and deterministic:

  • outputs after an override are reported stale
  • a fork without overrides reports nothing stale
  • the override target is not stale itself
  • an override late in the prefix does not mark earlier outputs stale
  • two overrides report the steps, not a count
  • tool calls are never classified as outputs

The plant: replacing overridden_steps[0] with 0 in the classifier — classifying from "an override happened" instead of "which step it was" — fails test_only_outputs_after_the_earliest_override_are_stale with exactly one extra id, and passes again once restored.

Compatibility

No trace or receipt format change: the new fields are additive on ForkResult with defaults, and existing traces load, validate and replay unchanged. ruff check, ruff format --check and mypy src are clean, and the full suite is 722 passed.

Design note

This follows the approach I proposed in the issue thread on 2026-10-28: the classification needs the set of overridden steps, and no arithmetic on overrides_applied can recover it.

A fork keeps every event at or before the fork point, and an override
replaces one of those outputs. Recorded outputs captured after the
overridden step were produced under the upstream state the override
changed, so replay substituting them is not evidence for the modified
execution path, yet they were indistinguishable from an ordinary
substitution: the only per-call classification was substituted vs
executed, and nothing on the fork or the manifest said which recorded
outputs an override upstream had invalidated.

fork_trace() now reports the overridden steps and the retained recorded
outputs that follow the earliest of them, and the CLI fork report shows
both, with a note that replaying them is not evidence for the modified
path. The classification is an upper bound rather than a dependency
analysis: it marks every recorded output after the earliest overridden
step whether or not that output's inputs changed, and it does not mark
the event the override replaced, because that value is the override
rather than a reused recording.

What it does not do, and docs/fork.md now says so: replay does not
consult the classification yet, so a call reported as substituted is not
by itself proof that it belongs to the modified path, and the
classification is not persisted into a bundle, so it is available only
from the fork_trace() call that produced it.

The approach follows the design proposed in the issue thread on
2026-10-28: the classification needs the set of overridden steps, which
no arithmetic on overrides_applied can recover.

Refs utsab345#33

Tests: tests/unit/test_fork_staleness.py covers the stale case, the
still-matching control, the override target itself, two overrides in one
fork, and that tool calls are never classified as outputs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant