tombstones: no tombstone for a deposed object whose destroy leg failed (#901)

[lex00]

Closes #901.

What was wrong

supersedeIdentity rests on two facts, and issue #854 added the second:
identitySuperseded says the record's identity moved, and
WriteBackRequest.ReplacedAddrs says this run's plan scheduled a replace for
that address. A create_before_destroy replace whose destroy leg FAILS
satisfies both and destroyed nothing: the create commits, the old object is
deposed, the destroy fails, and the planned action was CreateThenDelete
throughout. The apply's final state finishes with Current naming the new
object and ri.Deposed still holding the old one - alive, running, billed -
and a tombstone was written for it.

A tombstone's own meaning is "this estate's apply destroyed this identity".
Nothing renders it today, because pruneSupersededEntry's deposed leg runs
before its tombstone leg, so the protection was one leg's ordering rather than
the write being correct.

The fix

writeBackRecordEnvelopes now asks the address's own final state whether the
identity being superseded may still be one of its deposed objects, and writes no
entry when it may (deposedMayStillHold, internal/live/projection/writeback.go).
It reads ri.Deposed - the same map diffDeposedForWrite reads one closure
later - through the same LocatedRecordFrom every identity writer in that file
already uses, and compares on tombstoneKey, the key the entry would have been
written under and the key discovery's claimantMatchesAnyTombstone later matches
a live claimant against. One if became a four-arm switch, plus the helper and
its doc.

A deposed object whose identity cannot be rendered at all answers "may be", not
"is someone else": it may BE the superseded identity, and supersedeIdentity's
own rule for that doubt is that writing no entry is the strictly louder
direction. What suppressing costs is a refusal the next plan makes anyway on
that deposed object's own account.

Genericity. No provider type name appears in the change: the rule is a
property of the address's final state (ri.Deposed, which every managed resource
instance has) and of the identity the type's own schema renders. It reaches every
type writeBackRecordEnvelopes's identity half reaches - every non-record-backed
managed type the provider schema admits - because the only thing it asks about a
type is what LocatedRecordFrom already answers for it. Nothing was added to
live/derivation_guard_test.go's registry because nothing was hand-wired.

Five-row table. Row 2 (the plans or the resulting cloud differ), one step
back: the record an apply writes is what the next plan reads, and this apply
wrote a false fact about a live object into it.

Red, then green

The test (internal/live/projection/tombstonedeposed_test.go) builds the exact
shape with fakes: a first apply records eipassoc-0011..., then a replace whose
final state carries the new object as Current and the old one under
Deposed["deadbeef"], with the address in ReplacedAddrs.

Against the unfixed write side:

=== RUN   TestWriteBackReplaceWithAFailedDestroyLegTombstonesNothing
    tombstonedeposed_test.go:156: a create_before_destroy replace whose destroy leg failed recorded [eipassoc-00112233445566778] as destroyed, want none. "eipassoc-00112233445566778" is deposed and still running: an entry naming it says this estate's apply destroyed a live object, and the only thing keeping that out of an operator's report is which leg of pruneSupersededEntry runs first.
--- FAIL: TestWriteBackReplaceWithAFailedDestroyLegTombstonesNothing (0.00s)
=== RUN   TestWriteBackReplaceStillTombstonesWhenTheDeposedObjectIsSomeoneElse
--- PASS: TestWriteBackReplaceStillTombstonesWhenTheDeposedObjectIsSomeoneElse (0.00s)
=== RUN   TestWriteBackReplaceSuppressesTheTombstoneForAnUnreadableDeposedObject
    tombstonedeposed_test.go:246: a replace at an address carrying a deposed object whose identity could not be rendered recorded [eipassoc-00112233445566778] as destroyed, want none: that deposed object may BE "eipassoc-00112233445566778", and nothing here can tell
--- FAIL: TestWriteBackReplaceSuppressesTheTombstoneForAnUnreadableDeposedObject (0.00s)
FAIL
FAIL	github.com/intentius/choudoufu/internal/live/projection	2.990s

The middle one is the control that keeps the suppression about THE IDENTITY
BEING SUPERSEDED rather than about the address having any deposed object: a
replace at an address carrying an unrelated, left-over deposed object must still
write the entry, and it passes on both sides of the fix.

After the fix:

=== RUN   TestWriteBackReplaceTombstonesTheDestroyedIdentity
--- PASS: TestWriteBackReplaceTombstonesTheDestroyedIdentity (0.00s)
=== RUN   TestWriteBackReplaceTombstonesAreBounded
--- PASS: TestWriteBackReplaceTombstonesAreBounded (0.00s)
=== RUN   TestWriteBackReplaceWithNoPlanSignalTombstonesNothing
--- PASS: TestWriteBackReplaceWithNoPlanSignalTombstonesNothing (0.00s)
=== RUN   TestWriteBackReplaceWithAFailedDestroyLegTombstonesNothing
--- PASS: TestWriteBackReplaceWithAFailedDestroyLegTombstonesNothing (0.00s)
=== RUN   TestWriteBackReplaceStillTombstonesWhenTheDeposedObjectIsSomeoneElse
--- PASS: TestWriteBackReplaceStillTombstonesWhenTheDeposedObjectIsSomeoneElse (0.00s)
=== RUN   TestWriteBackReplaceSuppressesTheTombstoneForAnUnreadableDeposedObject
--- PASS: TestWriteBackReplaceSuppressesTheTombstoneForAnUnreadableDeposedObject (0.00s)
PASS
ok  	github.com/intentius/choudoufu/internal/live/projection	0.656s

The headline test also asserts the second half of #901's "Do": the deposed object
is still recorded under deposed after the fixed write-back, by key and by id.
That is what keeps it a live claimant for pruneSupersededEntry and findable for
matchDeposedClaimant. A "fix" that suppressed the tombstone by dropping the
deposed entry would have satisfied the tombstone assertion and lost the object.

Both touched packages, uncached:

$ env -u PWD go test -count=1 ./internal/live/projection/ ./internal/live/discovery/
ok  	github.com/intentius/choudoufu/internal/live/projection	5.646s
ok  	github.com/intentius/choudoufu/internal/live/discovery	11.347s

gofmt -l is clean on all four touched files.

Is pruneSupersededEntry's deposed-over-tombstone ordering still needed?

Yes, and it is now pinned - it was not before.

TestSupersededClaimant_theGuardsCanFail's case "a recorded deposed claimant is
kept, not pruned" says in its own comment that deposed wins over a tombstone
naming the same object, but it sets no tombstone, so the claimant it keeps is
kept by the tombstone leg's own "nothing records it as destroyed" exit. No case
in that table set both for one object. This PR adds the one that does, and proves
it load-bearing by mutation rather than by assertion: disabling
claimantMatchesAnyDeposed in pruneSupersededEntry fails the new case and
nothing else in the table.

--- FAIL: TestSupersededClaimant_theGuardsCanFail/a_deposed_claimant_is_kept_even_when_a_tombstone_names_it
    supersededclaimant_test.go:793: kept claimants [vpc-new], want [vpc-old vpc-new]
    supersededclaimant_test.go:796: 1 displaced reports, want 0

(every other subtest still PASS; the mutation was reverted before committing).

The leg stays as the second line of defence. The write is correct from this
commit forward, but the record outlives the build that wrote it: an estate whose
record was written by an older build carries exactly this shape, and so does a
hand-edited one. What the ordering costs when the tombstone is the honest one is
a refusal; what dropping it would cost is a live, running object pruned out of
the collision it belongs in.

Smoke claim

Filed as #919: a step on claim 18 (live/smoke/scenarios/a-shadow-is-not-a-claimant.sh)
that reads the record off disk after a create_before_destroy replace whose
destroy leg failed and asserts tombstone does not name the deposed instance,
with a BREAK=1 arm that patches the pre-#901 entry back in and requires the run
to catch it. Claim 18's existing step 4 is its control, so the new assertion
cannot pass vacuously.

Reproduce

env -u PWD go test -count=1 ./internal/live/projection/ -run 'TestWriteBackReplace' -v
env -u PWD go test -count=1 ./internal/live/discovery/ -run 'TestSupersededClaimant_theGuardsCanFail' -v

Per the maintainer's brief for this unit, the full tier was not run here.

🤖 Generated with Claude Code

https://claude.ai/code/session_018QSftBd7j3Dp4mnB6feRKL

[lex00]

Answering this PR's open question - whether the read side's deposed-over-tombstone ordering in pruneSupersededEntry is still needed - from #943, which closes the write-side gap next to it (#938).

Yes, and it is now a compatibility leg rather than a live one.

#943 makes the apply that DESTROYS a deposed object record it as destroyed, from the plan's own Delete of that deposed key plus the final state no longer carrying the key. The entry is written only for a key that has left ri.Deposed, and diffDeposedForWrite deletes that key inside the same mutate closure - so "deposed and tombstoned at once" is no longer a state one estate's own applies can produce.

What still reaches it is exactly what this PR's own comment named: a record written by an estate on a pre-#920 build (every crashed create_before_destroy apply wrote that shape), a hand-edited record, or a future writer. The ordering costs one refusal when it is wrong and keeps a live object out of a prune when it is right, so it stays. TestDiscover_crashWindowStillOpenKeepsBothClaimants in #943 now pins the live direction of it end to end, built by running projection.WriteBack rather than by seeding a record.

Also worth recording here: this PR's suppression was right, and the day2_crash stage passing before it was passing for the wrong reason - H1 was writing a tombstone for a deposed-and-alive object. #943 does not put that back.