#919 smoke: claim 18 step 7, a failed destroy leg writes no tombstone

[lex00]

Closes #919.

What this adds

A step on a-shadow-is-not-a-claimant (claim 18) that reads the record back off the emulator after a create_before_destroy replace whose destroy leg failed, the write-side change #901 (PR #920) landed with fake-backed tests only. Step 4 is the control: it already shows an ordinary replace DOES write a tombstone, so an empty list here cannot pass vacuously.

The mechanism, confirmed with the plain CLI first

The issue named two candidates. Only one works on the pinned image:

candidate	what floci does
disable_api_termination	accepts the ModifyInstanceAttribute, reads the attribute back as false, terminates anyway. Dead on this image. (The provider at v6.59.0 would not have cleared it either: resourceInstanceDelete only does that under force_destroy.)
IAM deny of ec2:TerminateInstances	with enforcement on, a role granted Allow * + Deny ec2:TerminateInstances can RunInstances and CreateTags, and TerminateInstances answers AccessDenied ... User is not authorized to perform: ec2:TerminateInstances with the instance still running.

So the scenario exports FLOCI_IAM_ENFORCEMENT=true before stack_up (the harness's test key bypasses the filter, so steps 1 to 6 run exactly as before) and step 7 creates a no-terminate role, proves its fence on a throwaway instance with the CLI, then applies a third replace under the role.

What step 7 asserts

• the apply exits non-zero, the replacement's Creation complete line is present, and the error is the platform's denial of the destroy leg
• the CLI lists exactly two running instances marked for the address: the deposed one and its replacement, and the deposed one reads running
• the record file: identity.import_id is the replacement, deposed[] is the old instance, tombstone[] still carries step 4's two entries and does not carry the old instance
• the next plan proposes aws_instance.web (deposed object ...) will be destroyed rather than pruning it; a plan that exits 0 without mentioning the deposed instance fails the step

BREAK=1 now runs two arms on one estate

The step-5 duplicate arm used to tear down and exit 0. It now terminates the duplicate and deletes its markers (confirmed with the CLI that an untagged terminated instance leaves describe-instances, describe-tags and the tagging API), so step 7's arm can run. Step 7's arm patches the record to list the running, deposed instance under tombstone, the entry the write side produced before #901, and the read must catch it. The patch is reverted before teardown.

Placement

Step 7 sits before teardown rather than directly after step 4. Converging the estate for step 5 would need a recovery apply that destroys the deposed object, and on today's main that apply records no tombstone (#938), so the shadow-arm plan would refuse on the lingering tag. With the deposed-alive state last, only the teardown has to deal with it, and it does.

Measured

Both arms, FLOCI_PORT=4750, pinned image sha256:a39185cc:

$ bash live/smoke/smoke.sh a-shadow-is-not-a-claimant
=== 7. a failed destroy leg writes no tombstone ===
    aws: [ERROR]: An error occurred (AccessDenied) when calling the TerminateInstances operation: User is not authorized to perform: ec2:TerminateInstances
    aws_instance.web: Creation complete after 10s [id=i-8bdd7bc5f639c6f07]
    Error: terminating EC2 Instance (i-ab39d21e298740497): operation error EC2: TerminateInstances, https response error StatusCode: 403, ...
    i-ab39d21e298740497 is running (the deposed object, the destroy the platform refused); i-8bdd7bc5f639c6f07 is its replacement
    identity.import_id = i-8bdd7bc5f639c6f07   (the replacement, live)
    deposed[]          = i-ab39d21e298740497   (the old object, live, the destroy that was refused)
    tombstone[]        = i-2272f959adbca418b i-67c9f4ac99301f1d8
      # aws_instance.web (deposed object d6d3e90b) will be destroyed
PASS: smoke scenario 'a-shadow-is-not-a-claimant' - every claim held (smoke v1.0.0)

$ BREAK=1 bash live/smoke/smoke.sh a-shadow-is-not-a-claimant
-> caught - the plan refused with "Two live resources claiming one address", naming both i-46adb6f07004367d2 and i-7a5e2cf4e369816a4. ...
-> caught - the record named the running, deposed i-46adb6f07004367d2 as destroyed, and the read refused it. ...
PASS: smoke scenario 'a-shadow-is-not-a-claimant' - every claim held (smoke v1.0.0)

Docs updated: claim 18's step list and BREAK paragraph in site/content/docs/claims.md, the table's minute count, and the entry in live/smoke/README.md.

Refs #901, #920, #900, #854, #670, #938.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KXQqcqYiLU513jwXziuQnh