lore-aws: Carry fragment metadata on the S3 object

[mjansson]

Problem

The AWS store keys S3 objects by the content hash while the object holds one representation of that content. Two writers may hold different valid representations of the same content (LZ4 and Zstd of the same bytes) and both address the same key. The fragment describing which representation is stored lives in a separate DynamoDB record, so two independently written records are required to agree. They can fail to:

1. Concurrent cross-partition writers — two writers upload different representations and publish different fragments. The interleaving leaving writer A's fragment beside writer B's payload is permanent and requires no failure.
2. A lost metadata write — a writer replaces the object, then fails to publish. The stored fragment describes bytes that are gone, and the affected partition cannot repair it: its re-put sees a full match and does nothing.

Both yield a payload that cannot be decompressed, reported as an internal size mismatch, undetected until a read fails.

Separately, lookup short-circuits to MatchNone when MatchFull is requested, so put's
MatchPartition and MatchHash arms are unreachable — content already durable in one partition is uploaded again for another.

Change

The fragment travels on the object as x-amz-meta-lore-fragment: <flags hex>:<size_payload>:<size_content>. Object metadata is part of the object version, so a GetObject returns headers and body from the same version, and a full-object PUT is atomic. Last-writer-wins becomes safe; the disagreement is removed rather than policed.

The DynamoDB fragment metadata table becomes a fragment state table: row presence means the hash exists, plus obliteration state. The existence probe stays a single GetItem with no S3 request.

No S3 or DynamoDB feature beyond the plainest ones — no conditional S3 write, no ETag
compare-and-set, no object-age heuristic, no transaction. The one conditional DynamoDB write is an attribute_not_exists create guarding an obliteration mark.

Flows

Put — probe association ∥ state, in parallel:

Probe	Action
state = Obliterating	SLOWDOWN
state present, associated	OK
state present, not associated, payload supplied	associate only, no S3
state present, not associated, no payload	Payload buffer required
no state, or Obliterated	upload → conditional create → associate

Ordering is load-bearing: object, then state row, then association. The reverse leaves a hash claiming to exist with no bytes, unrepairable by retry because the next put takes the
already-stored branch. Every step is idempotent, so a retry converges from any interruption.

Get — association GetItem ∥ GetObject. The fragment arrives on the response carrying the bytes; the size check is self-consistency on one object.

Query — association ∥ state, no S3. It is called once per fragment stored on the ingress path (query_match_full), which ADR-00008 identifies as the busiest path in the server.

ImmutableStore::get_metadata — new, defaulting to a MatchFull query; the AWS store overrides it with a HeadObject. The only path spending an S3 request purely on metadata.

Obliterate — mark → delete association (compliance discharged) → drain → count → release mark, or recurse sub-fragments, delete payload, tombstone.

Costs

Operation	main	This branch
Put, new content	1–3 DDB reads, 1 S3 PUT, 2 DDB writes	2 DDB reads (parallel), 1 S3 PUT, 2 DDB writes
Put, dedup across partitions	not supported — re-uploads	2 DDB reads, 1 DDB write, no S3
Get	2 DDB reads, 1 S3 GET	1 DDB read, 1 S3 GET (parallel)
Query	2 DDB reads	2 DDB reads (parallel), no S3
Copy	2 DDB reads, 1 DDB write	1 DDB read, 1 DDB write
get_metadata	2 DDB reads	2 DDB reads + 1 HeadObject

Production code in lore-aws grows ~460 lines; dynamodb.rs is untouched. This is not a
simplification measured in lines.

Rollout

Full stop, then full start — no mixed fleet. That removes the dual-write phase: nothing writes the old shape, and the guarantee holds from the first write. There is no clean rollback once writes are served — content written after cut-over is described only on its object.

Configuration is non-breaking: dynamodb_metadata_table is accepted as an alias for
dynamodb_fragment_state_table. The optional dynamodb_fragment_metadata_table gates the fallback read for pre-cut-over objects; leaving it unset declares no such object exists, so an object with no metadata is reported as damaged rather than described from a row that cannot be about it.

Testing

94 tests in lore-aws. The fake supports fault injection across nine operations, which makes error paths reachable. Every behavioural guard was verified by removing it and confirming the test fails.

Notable: concurrent_writers_cannot_tear_the_fragment_from_its_payload (4 writers × 4
representations × 64 randomized rounds), sub-fragment obliteration and its failure aggregation, the drain window, the post-upload obliteration race, and lost-payload detection and repair.

Known limitations

A lost payload (object gone, reference remains) is detected on both reads and repaired by clearing the state row; reconciling the whole population is left to the planned obliteration work table. A crashed obliteration leaves a mark that nothing clears. The obliteration drain narrows but does not close its race.

Design and migration: docs/proposals/2026-08-03-fragment-metadata-on-the-s3-object.md. Decision and alternatives: ADR-00018, which supersedes ADR-00006.

[epic-lore-bot]

Imported as Lore CR-288.