[gauntlet:plan_approval] #903 re-measure: all 27 estates with stage 12 active - #939
Merged
Merged
Conversation
The runner rewrote every row. `go run ./tools/gauntlet run -set all -parallel 4` at 70e2722, against the pinned emulator ghcr.io/lex00/floci@sha256:a39185cc, 44m11s wall for the 27 estates. Board: core 0/26 clear, all 0/27 clear (every row carried plan_approval=not_run behind #936's flip) -> core 26/26 clear, all 27/27 clear. Every estate reports `GAUNTLET stage=plan_approval verdict=pass`; no estate lost a stage that was pass before. One stage moved backwards, and it is not plan_approval and not this change's doing: reference-ec2-vpc's day2_crash (stage 10, StatusPlanned, so the bars are unaffected) went pass -> fail. It reproduces deterministically - twice, on fresh instance ids, once in the parallel run and once in a solo `gauntlet run reference-ec2-vpc` - and the shape points at #920 (issue #901) landing the suppress half of a tombstone without the write half: H1 crashes a create_before_destroy replace; the old object is deposed and alive, and #920 correctly writes NO tombstone for it. H2's recovery apply destroys that deposed object (terminated, confirmed through the AWS CLI) and clears the deposed record entry - but supersedeIdentity is called from exactly one site (internal/live/projection/writeback.go:821, the replace path), so nothing records the object this apply just destroyed. H3's plan then sees the terminated instance's marker still readable with no tombstone covering it and refuses: "Two live aws_instance resources carry estate ec2-reference and address aws_instance.main at once". The same plan WARNS rather than errors about an earlier replace's old object, which does have a tombstone - the two objects differ only in whether anything recorded them. Left as the runner wrote it, unacknowledged in live/gauntlet/regressions.json: that ledger is for a human acknowledging a regression their own change earns, and this one is a defect to fix, not one to sign off. Reported in the PR body and on #903 instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018QSftBd7j3Dp4mnB6feRKL
This was referenced Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The catch-up re-measure #903 asks for, and the one #936's flip needed behind it.
Artifact and rendered docs only: no script, no Go file, no
stages.go.The board
Read from
bash scripts/pickup.sh, not from the runner's own summary:Before, at #936 (70e2722) - every one of the 27 rows still carried
plan_approval: not_runfrom before the flip:After, this commit:
Runner's own summary line, the parallel run:
Wall clock: 44m11s for all 27 at
-parallel 4(20:16:39 -> 21:00:50),plus 4m08s for the one solo re-run below. Emulator
ghcr.io/lex00/floci@sha256:a39185cc, oracle terraform 1.15.8 / tofu 1.12.5.The 27 verdicts
Every estate in the manifest,
verdict=pass. Each line'sdetail=runs 1-3 KBand is in
live/gauntlet.jsonandlive/gauntlet/logs/<estate>.logverbatim;elided here for length.
Diffing every stage of every row against the artifact at 70e2722, rather
than trusting the summary: 27 rows before and after,
plan_approvalnot-passon none of them,
clear: falseon none of them, and exactly one stage movedbackwards.
The one estate that moved backwards
reference-ec2-vpc,day2_crash(stage 10,StatusPlanned, so the two barsare unaffected and the row still reads
clear: true): pass -> fail,detail=the post-recovery plan exited 1. Itsplan_approvalpassed on thesame run.
Not a flake, and not this change's doing. It reproduced twice on fresh
instance ids - once in the parallel run, once in a solo
go run ./tools/gauntlet run reference-ec2-vpc- and the row this PR carriesis the solo re-run's. The row it replaced was measured at eec6fb4 on
2026-09-06T01:27Z, so the cause is in main between there and 70e2722.
The log, from the recovery apply through the plan that refuses:
Two objects, one plan, opposite treatment, and the difference between them is
whether anything ever recorded them as destroyed:
i-49b4438...(an earlierreplace's old object) has a tombstone and draws the warning;
i-65a7966...(the deposed object H2 destroyed 30 seconds earlier) has none and draws the
refusal. Both are terminated instances whose marker tags EC2 still serves.
Where the tombstone went: #920 (issue #901) taught the write-back not to
tombstone a deposed object whose destroy leg had not completed - correctly,
since at H1 that object is superseded at the address and alive, and
writeback.go's own log line says as much ("That object is NOT recorded asdestroyed, because it is alive and the next apply's job is to destroy it").
But the next apply then destroys it and records nothing:
supersedeIdentityhas exactly one call site,internal/live/projection/writeback.go:821, on the replace path. Nothingwrites a tombstone when a deposed object's outstanding destroy finally runs.
Before #920 the (at-that-moment wrong) H1 tombstone happened to cover H3;
removing it exposed the missing write half. That is the read-half-without-
its-write-half shape HANDOFF names, pointed the other way, and it lands in
the table's first row: choudoufu refuses where nothing is wrong with the
world.
Left exactly as the runner wrote it, and deliberately not entered in
live/gauntlet/regressions.json: that ledger is a human acknowledging aregression their own change earns, and this is a defect for its own unit -
signing it off here would hide it.
gauntlet runwill keep exiting nonzeroon this estate until it is fixed; the artifact is written and rendered either
way, which is the ratchet behaving as
live/GAUNTLET.mddescribes.Tests
Scoped to what this change touches, per the brief:
Both re-run after the rebase onto
origin/main(which was clean -live/gauntlet.jsonwas untouched on main since 70e2722, so nomerge-artifactwas needed).Reproduce
The one regression alone:
Closes the measurement half of #903.
🤖 Generated with Claude Code
https://claude.ai/code/session_018QSftBd7j3Dp4mnB6feRKL