lore-aws: Avoid rewriting S3 on fragment deduplication and fix data corruption from torn metadata/blob representation

[mjansson]

Deduplicate S3 payloads across partitions, and fix two fragment write corruption cases

Summary

The AWS immutable store uploads a fragment payload whenever the exact
(partition, context, hash) row is absent, even when the payload for that hash is
already stored. The S3 object key is the bare content hash, so that upload replaces
content other partitions reference. The object and the metadata describing it are
written as two independent, unordered operations, which permits two corruption cases.

This change makes a put that finds the content already stored record a reference to it
and perform no S3 write, and makes the object and its metadata a matched pair under
concurrency and partial failure.

No transactions, no new DynamoDB permissions, and no change to the fragment metadata
table.

Root causes

Current write path (AwsImmutableStore::put → write_payload):

1. do_query(MatchFull) — resolves to MatchFull or MatchNone only.
2. MatchFull → return.
3. MatchNone + payload → PutObject (unconditional) → PutItem metadata
   (unconditional) → PutItem association.

1. Parallel interleaved puts — low likelihood, permanent, no failure required

Two writers store the same content in different representations (for example different
compression) on different partitions concurrently. S3 and DynamoDB are written
independently, so the writer that wins S3 need not be the writer that wins DynamoDB.
The stored blob ends up described by the other writer's fragment. All writes succeed;
the result is permanently inconsistent.

2. Overwrite where only the S3 put succeeds — higher likelihood

A second partition storing content that is already stored has no exact row, so it
uploads, replacing the object other partitions reference. If the subsequent metadata
write is lost — DynamoDB throttling, or process termination between the two writes, a
case the existing code comments already anticipate — the blob and the published
metadata no longer describe each other.

The affected partition cannot repair this: its own re-put matches on the association and
returns without writing. Only a put from a third partition rewrites both sides.

Improvement

A put for content already stored writes an association only. No S3 request is issued on
the deduplicating path.

The fix

Probe

exists_exact (fragments table) and metadata_lookup (metadata table) are issued
concurrently: two strongly consistent single-item reads against different tables, one
round trip. The metadata table is keyed by hash alone, so probe cost does not scale with
the number of partitions referencing the content. S3 is not consulted.

Resolution

Probed state	Action
Association present, metadata published and healthy	No writes
Metadata published and healthy, no association	Deduplicate: PutItem association only
Metadata absent	Upload
Marked for obliteration	Back off (SlowDown); the mark is transient
PayloadObliterated tombstone	Upload, conditional on the tombstone
No payload supplied, not already associated	Reject (Payload buffer required)

Deduplication adopts the stored fragment, never the incoming one. A payload is still
required, so presenting bytes remains proof of possession; ingress verifies the payload
hashes to the address.

Upload

1. PutObject with If-None-Match: *. The object becomes write-once: exactly one writer
   can create the bytes for a hash. S3 provides the mutual exclusion; no external lock is
   used.
2. Metadata published conditionally, only after this writer's own upload created the
   object. The published fragment therefore always describes the stored bytes. A rejected
   publish re-reads the row and re-conditions rather than re-uploading.
3. PutItem association.

Key already present (HTTP 412)

1. Re-read metadata. Published → deduplicate. Obliteration flags → discard the remnant.
2. Otherwise HeadObject. Younger than the abandonment threshold → return SlowDown;
   the caller retries. Older → the writer that stored it is gone.
3. Reclaim with If-Match on the observed ETag, making reclaim single-winner, then
   publish.

Only a writer that displaced bytes (reclaim or force_write) may overwrite another
writer's metadata after a rejected publish. A writer whose upload was conditional stands
down, since its bytes may since have been reclaimed.

Obliteration

Reordered so that association writes need no transaction:

1. Load metadata. Tombstone → return. Marked by another obliteration → remove this
   partition's association and return, leaving the payload and metadata to the mark
   holder.
2. Mark the metadata row.
3. Remove this partition's association.
4. Wait for puts that read the row before the mark to finish writing.
5. Count remaining associations. Any → clear the mark and return. None → obliterate
   sub-fragments, delete the payload, write the tombstone.

A put that reads the mark backs off, so no reference can appear while the payload is
being removed, and the mark is set before the association is removed so the partition
being obliterated cannot write its own reference back. Association reads are strongly
consistent, so the wait is not for consistency — it is for in-flight writes to land, and
is sized above the DynamoDB request timeout.

Removing the reference is what obliteration must do; the payload and metadata are cleanup
that applies only once nothing references the content. Sub-fragments are therefore visited
only once the parent is known to be going away.

Only a tombstone refuses a read. A row that is merely marked still has its payload, so
refusing reads there would hide live content from every partition holding it.

How this resolves the root causes

• Case 1: the conditional upload permits exactly one writer to create the bytes, and
  metadata is published only by that writer. The S3 winner and the metadata winner are
  the same writer by construction.
• Case 2: the second partition performs no S3 write and no metadata write, so there is
  no upload to replace the blob and no metadata write to lose.

Related changes

• force_write respects obliteration flags, replaces the object in place, and fails
  visibly if it cannot publish.
• A put that stored bytes and then loses the publish to an obliteration withdraws them.
• New counter store.immutable.missing_payload, incremented with an error! log when a
  read finds published metadata with no object behind it. Published metadata is
  authoritative and deliberately not verified on the read or deduplication paths; this
  condition is an operational failure and is now reported rather than surfacing as an
  ordinary not-found.

Flow and call comparison against main

Put

Case	main	This change
Re-put, same partition and context	2 × GetItem, sequential	2 × GetItem, concurrent — one round trip
Content already stored elsewhere	1 × GetItem + PutObject + 2 × PutItem	2 × GetItem + 1 × PutItem, no S3
New content	1 × GetItem + PutObject + 2 × PutItem	2 × GetItem + PutObject (If-None-Match) + 1 × conditional PutItem + 1 × PutItem
Key taken, writer in flight	n/a	+ 1 × GetItem + HeadObject, then SlowDown
Key taken, writer abandoned	n/a	+ 1 × GetItem + HeadObject + PutObject (If-Match)

DynamoDB write units, items under 1 KB:

Case	main	This change
Re-put	0	0
Deduplicate	2	1
New content	2	2

Deduplication drops an PutObject and one write unit. New content is unchanged on
writes and costs one additional single-item read. Re-puts are unchanged in cost and take
one round trip instead of two.

Obliterate

Step	main	This change
Load metadata	GetItem	GetItem
Mark	conditional PutItem	conditional PutItem
Sub-fragments	recursed before the reference count	recursed only once nothing references the parent
Remove association	DeleteItem	DeleteItem
Drain	—	wait, sized above the DynamoDB request timeout
Count references	Query (strongly consistent)	Query (strongly consistent)
Still referenced	revert metadata	clear the mark
Not referenced	ListObjectVersions + DeleteObject + conditional PutItem	unchanged

Same call set plus the drain. Obliteration is rare and not latency sensitive.

Requirements on S3 and DynamoDB

S3

Requirement	Used for
PutObject with If-None-Match: *	Write-once objects; mutual exclusion between competing writers
PutObject with If-Match: <etag>	Single-winner reclaim of an abandoned object
HeadObject returning ETag and Last-Modified	Deciding whether an unpublished object is abandoned, and identifying it for reclaim

Both conditional write headers are supported by S3. An endpoint that ignores either
header, or omits Last-Modified, degrades silently:

• ignoring If-None-Match removes the mutual exclusion and restores case 1;
• ignoring If-Match removes single-winner reclaim;
• omitting Last-Modified prevents an abandoned object from ever being reclaimed, so
  puts for that hash back off indefinitely (logged at error!).

S3-compatible endpoints must be confirmed to honour all three before use. No integration
test covers this; unit tests use mocks.

No bucket configuration change is required. Object keys are unchanged.

DynamoDB

Requirement	Used for
GetItem, PutItem, DeleteItem, Query, BatchGetItem, DescribeTable	Existing usage
Strongly consistent reads	Probe correctness and the obliteration reference count
Conditional PutItem	Publishing metadata; the obliteration mark

No new IAM permissions. Association writes are plain puts, so
dynamodb:ConditionCheckItem is not required and TransactWriteItems is not used by
this path.

Schema and compatibility

No table change and no migration. The fragment metadata table keeps its exact attribute
set (hash, flags, size_payload, size_content); no attributes are added, and rows
written by this change are byte-identical to existing rows. Asserted by
published_rows_keep_their_original_shape.

Tests

Reproduction on main

Branch repro/fragment-dedup-corruption, one commit off main:

cargo test -p lore-aws --lib corruption

Both tests fail. All 71 pre-existing tests are unaffected.

• concurrent_cross_partition_writers_tear_blob_and_metadata — six writers, distinct
  partitions and contexts, same content hash, six distinct representations, 512
  iterations on a multi-threaded runtime against an in-memory S3 and DynamoDB in which
  each operation is individually atomic. No injected failures. Fails, for example:
  published size_payload 69 does not match the stored blob length 66.
  The interleaving comes from the scheduler, so this samples a real race; reproduced on
  6/6 full-suite runs at 512 iterations. A single green run does not disprove it.

• a_failed_cross_partition_write_corrupts_the_first_partition — deterministic, no
  concurrency. Partition 1 stores content and reads it back. Partition 2 stores the same
  content in a different representation; its upload lands and its metadata write is
  throttled. Partition 1, which never wrote again, can no longer read what it stored:
  Failed to load from immutable store, size mismatch (load 65, expected 64).
  This is a real get through the read path. That the read breaks is itself evidence the
  blob was replaced.

Verification on this branch

96 tests pass. Counterparts run the same scenarios with the same fault injected:

• concurrent_writers_with_different_representations_stay_cohesive — the case 1 race;
  asserts the stored blob and published metadata agree and that exactly one upload occurs.
• a_failed_cross_partition_write_cannot_corrupt_the_first_partition — the case 2
  sequence with the metadata-write fault armed. Asserts the fault is still armed
  afterwards (no metadata write occurred), that exactly one upload occurred, and that
  partition 1's read succeeds.

Deduplication:
test_put_immutable_deduplicates_across_partitions,
test_put_immutable_full_match_writes_nothing,
cross_partition_put_of_another_representation_leaves_the_blob_alone,
deduplicating_writers_adopt_the_stored_representation,
test_put_immutable_payload_required_to_deduplicate.

Reclaim and abandonment:
test_put_immutable_reclaims_abandoned_object,
test_put_immutable_backs_off_from_an_unpublished_upload,
test_put_immutable_losing_the_reclaim_publishes_nothing,
test_put_immutable_conditional_writer_does_not_publish_over_a_reclaimer,
test_put_immutable_publish_reconditions_without_reuploading,
test_put_immutable_force_write_publish_exhaustion_is_loud,
abandonment_is_decided_conservatively,
stored_time_keeps_sub_second_precision.

Obliteration:
test_obliterate_marks_before_removing_the_association,
test_obliterate_already_obliterating,
test_put_immutable_force_write_respects_obliteration,
test_put_immutable_discards_object_left_by_obliteration,
test_put_immutable_discards_reclaimed_bytes_when_obliteration_wins,
test_get_immutable_marked_for_obliteration_is_still_readable,
test_get_immutable_reports_published_metadata_with_no_object.

Negative controls, asserting the checks are not vacuous:
legacy_write_order_tears_blob_and_metadata,
legacy_cross_partition_overwrite_tears_blob_and_metadata,
without_metadata_conditions_writers_upload_over_each_other.

Each behavioural guard was verified by reverting it and confirming a specific test fails.

Known limitations

• An obliteration interrupted between marking the row and finishing leaves the mark set,
  and nothing clears it. Such a hash remains readable but cannot be written until an
  operator intervenes. Intended to be resolved separately by a durable obliteration work
  table that servers pick up at startup and at intervals; that resume path will need to
  take over the mark explicitly rather than deferring to it.
• Published metadata is treated as proof the payload exists. If an object is removed
  while its row survives, reads of that hash fail for every partition referencing it and
  writing the content again does not repair it. Reported via
  store.immutable.missing_payload.

[peter-lockhart-pub]

LGTM - I agree with the jist of it. Lets chat about this to make sure.

Some nits commented below (and after a few I stopped commenting). I think all traces should use fields - especially if we ever have to enable debug then filtering them out will be much easier as we debug something at scale.

[peter-lockhart-pub]

We should instrument and alert on this. The hash doesn't need to be part of the label but draw our attention to the logs

[mjansson]

It will never fire on real S3, not worth the effort imo

[peter-lockhart-pub]

Probably a good idea to cache the labels like what the DDB impl does

[mjansson]

I think that's a change to the S3Impl signature and a separate change - no S3 code currently does that.

[epic-lore-bot]

Imported as Lore CR-280.

[peter-lockhart-pub]

We chatted on a call and LGTM. Pre-approving re the tracing fields