[gauntlet:reference-ec2-vpc/day2_crash] destroying a deposed object recorded no tombstone, so its lingering tag was a second claimant

[lex00]

The unit

reference-ec2-vpc/day2_crash, tracked as #938. The stage is planned, so
it is not on either headline bar and gauntlet next does not name it; the
estate reads clear either way.

The five-row class

Row 1: choudoufu refuses where stock proceeds. Stock's state file names
the surviving object outright and replans the estate without a word. This
fork refused it, totally, until AWS forgot the terminated instance:

Error: Two live resources claiming one address

2 live aws_instance resources carry estate "ec2-reference" and address
"aws_instance.main" at once: i-557a11917316ed797, i-c3fd6bf4eb2de323a. A
human has to resolve the collision before this estate can be planned

The bisect

Three PRs merged into the window between eec6fb4282 (last pass) and
22d9a3f099 (fail). Done by reading, not by assuming the lead:

PR	what it touches	verdict
#932 (12ab0e5493)	identity.Class handler tables in projection/command/mv/liveimport/lint	not it. A dispatch refactor; writeback.go is untouched and no tombstone or deposed path moves.
#935 (c6aebde9f5)	discovery's double-sighting dedupe	not it. It merges ONE live object seen by two enumeration legs. The two claimants here are two distinct instance IDs, which no dedupe can or should merge.
#920 (001fd5dd8a, issue #901)	writeback.go's tombstone gate	it, and it is correct about the apply it fixed.

git diff 001fd5dd8a^1 001fd5dd8a -- internal/live/projection/writeback.go
is the whole change: deposedMayStillHold was added to the switch that
decides whether a replace records a destroyed identity.

What was wrong

#920 fixed H1 and uncovered H2. The stage's three applies:

• H1 — the create_before_destroy replace is interrupted between the
  create committing and the destroy dispatching. The record correctly ends
  up current=i-new, deposed=i-old, and tombstones: no tombstone for a deposed object whose destroy leg failed (#901) #920 correctly writes no tombstone:
  the deposed object is alive.
• H2 — the recovery apply destroys the deposed object. Its plan schedules
  a deposed destroy, not a replace, so ReplacedAddrs is empty; the
  address's recorded identity has named the new object since H1, so
  identitySuperseded is false. Neither of supersedeIdentity's two
  facts holds, so the identity this estate's own apply terminated was
  recorded nowhere at all.
• H3 — the terminated instance's tags are still readable through the
  tagging API (discovery: a replace should tombstone the object it destroys, so a superseded claimant is provably dead rather than merely unrecorded #670's lingering-tag case, and AWS's own documented behaviour
  for a terminated instance), so the sweep hands the address two claimants
  and nothing says one of them is dead. Refusal.

Before #920 this passed for the wrong reason: H1 wrote a tombstone for a
deposed-and-alive object, which #901 is right to call a lie about a live
resource. The fix is not to put that lie back.

The generic rule

An apply that destroys a deposed object records that identity as destroyed
by this estate, on two facts and no type name:

1. the plan's own Delete of that deposed key — WriteBackRequest.DestroyedDeposed,
   derived by backend/local's new destroyedDeposedInstances(plan) exactly
   as tombstones: derive the replace signal from the plan, not from the record's identity changing (import and mv wording) #854's replace set is derived by replacedInstances(plan). The two
   partition the plan's destroy evidence: IsReplace() already excludes a
   deposed-key change, and this takes only deposed-key Deletes.
2. the final state no longer carrying the key as deposed — because a plan
   verb says a destroy was scheduled, which is precisely the gap tombstones: a create_before_destroy replace whose destroy fails records a live deposed object as destroyed #901
   exists for. A destroy leg that fails again leaves the key in ri.Deposed
   and records nothing.

The identity written is the one the record already holds under that key,
rendered on the crashed apply's own write-back by LocatedRecordFrom. So the
rule reaches every recordable type by copying a payload an existing writer
produced rather than deriving a second one, and it is called from the one
place both write-back loops already share, beside diffDeposedForWrite.

How many types it reaches. No aws_* name appears anywhere in the change,
and no entry is needed in live/derivation_guard_test.go's registry. Measured
against a real hashicorp/aws 6.59.0 GetProviderSchema response
(pluginschema.Acquire, throwaway probe, not committed): of 1699 managed
resource types, 1462 have a recordable identity plan (90 composite, 1372
scalar) — that is the population whose deposed entry carries an identity, and
therefore the population this rule can write an entry for. The remaining 237
render no identity, and addTombstoneEntry's own empty check leaves them
silent, which is the same "unproven, so stay quiet" direction
deposedMayStillHold takes.

Red, then green

Both directions, so neither guard is one that cannot fail.

With tombstoneDestroyedDeposed returning 0 (main's behaviour):

--- FAIL: TestWriteBackDestroyingADeposedObjectRecordsItAsDestroyed
    after the apply that destroyed the deposed object the record names [] as
    destroyed by this estate, want exactly ["eipassoc-00112233445566778"].
--- FAIL: TestWriteBackDeposedDestroyRecordsOnlyTheKeyThePlanNamed
    the record names [] as destroyed, want exactly ["eipassoc-00112233445566778"]
--- FAIL: TestDiscover_crashWindowClosedLeavesOneClaimant
    after the recovery apply the record names [] as destroyed by this estate,
    want [vpc-old] - GitHub issue #938

With the still-deposed guard removed (i.e. #901's suppression broken):

--- FAIL: TestWriteBackDeposedDestroyThatFailedAgainRecordsNothing
    a deposed destroy that failed again recorded [eipassoc-00112233445566778]
    as destroyed, want none
--- FAIL: TestDiscover_crashWindowStillOpenKeepsBothClaimants
    a deposed destroy that did not happen recorded map[vpc-old:...] as
    destroyed: vpc-old is deposed and still running

With destroyedDeposedInstances' deposed filter inverted:

--- FAIL: TestWriteBackSeesTheDeposedDestroySetAfterApply
    WriteBack was handed the deposed-destroy set [], want
    [test_instance.foo deadbeef].

internal/live/discovery/crashwindowclose_test.go is deliberately the only
test in that package that builds its record by running
projection.WriteBack instead of seeding one. Every seeded superseded-claimant
test passed throughout this regression, because the shape the write half
actually produced after a crashed replace was one no seeded test ever asked
for: no deposed entry, and no tombstone either.

The estate: reproduce, then fix

Not just unit tests. With tombstoneDestroyedDeposed neutered to return 0 and
everything else identical, on this same worktree and emulator:

GAUNTLET stage=day2_crash verdict=fail duration_s=37 detail=the post-recovery plan exited 1

and with the fix in:

GAUNTLET stage=day2_crash verdict=pass duration_s=35

Before (main's row, written by #939)

"day2_crash": "fail"   detail: "the post-recovery plan exited 1"

After (this branch, all thirteen verdict lines)

GAUNTLET stage=cold_deploy verdict=pass
GAUNTLET stage=greenfield verdict=pass
GAUNTLET stage=migrate verdict=pass
GAUNTLET stage=test_plan verdict=pass
GAUNTLET stage=test_apply verdict=pass
GAUNTLET stage=drift_reconverge verdict=pass
GAUNTLET stage=plan_approval verdict=pass
GAUNTLET stage=day2_replace verdict=pass
GAUNTLET stage=day2_rename verdict=pass
GAUNTLET stage=day2_remove verdict=pass
GAUNTLET stage=day2_count verdict=pass
GAUNTLET stage=day2_crash verdict=pass
GAUNTLET stage=strict verdict=pass

Nothing moved backwards. The board's day2_crash tally goes 0 pass / 1 fail /
26 not_run to 1 pass / 0 fail / 26 not_run; the two headline bars stay at core
26 of 26 and all 27 of 27, since this stage is planned.

The rebase onto #939 conflicted on live/gauntlet.json and the three rendered
files. Resolved by taking main's copies wholesale and re-running the estate,
never by hand-merging.

#920's open question: is the read side's deposed-over-tombstone ordering still needed?

Yes, and it is now a compatibility leg rather than a live one.

pruneSupersededEntry keeps a claimant that matches a deposed record even
when a tombstone names the same object. After this change the write side can
no longer produce that record from one estate's own applies: the entry is
written only for a key that has LEFT ri.Deposed, and diffDeposedForWrite
deletes that key inside the same mutate closure, so deposed-and-tombstoned is
not a state a current build reaches.

What still reaches it is what #920's own comment named: a record written by an
estate running an older build (every pre-#920 crashed apply wrote exactly that
shape), a hand-edited record, or a future writer. The ordering costs one
refusal when it is wrong and prevents a live object being pruned out of a
collision set when it is right, so it stays.
TestDiscover_crashWindowStillOpenKeepsBothClaimants now pins the live
direction of it end to end.

Reproduce

git checkout gauntlet/reference-ec2-vpc-day2_crash
go build -o .bin/choudoufu ./cmd/choudoufu
TOFU_BIN="$PWD/.bin/choudoufu" go run ./tools/gauntlet run reference-ec2-vpc
grep 'GAUNTLET stage=' live/gauntlet/logs/reference-ec2-vpc.log

Unit level:

go test -count=1 ./internal/live/projection/ ./internal/live/discovery/ ./internal/backend/local/

Testing

gofmt -l clean, go build ./..., go vet ./..., and
go test -count=1 ./internal/live/projection/ ./internal/live/discovery/ ./internal/live/check/ ./live/ ./tools/gauntlet/ ./internal/backend/local/ ./internal/live/marksafe/ all green. gauntlet check reports rendered files
current.

Refs #938, #901, #920, #670, #854, #908, #939.

🤖 Generated with Claude Code

https://claude.ai/code/session_018QSftBd7j3Dp4mnB6feRKL