Replication part 2: verified stores, log shipping, promotion, geo-replication #166

Activity

1. Hafeok

   added a commit that references this issue on Oct 28, 2026

   chore(release): v0.1.0-preview.16, Replication within a store

   ...

   b898f22
2. Hafeok commented on Nov 11, 2026

   

   Hafeok

   on Nov 11, 2026 – with Claude

   OwnerAuthor

   More actions

   Starting part 2. This is a cloud session on branch claude/zen-hopper-5e32m5, against main at 2cad18d (v0.1.0-preview.20). The session was developed with AI assistance under human review.

   The first deliverable is a delta plan on this issue: what in the approved plan (#97, sections 5–8) no longer matches main, and what I would change. I won't write any code until the maintainer has answered it.

   ---

   Generated by Claude Code
3. Hafeok commented on Nov 11, 2026

   

   Hafeok

   on Nov 11, 2026 – with Claude

   OwnerAuthor

   More actions

   Delta plan: Replication part 2 against main at 2cad18d (v0.1.0-preview.20)

   This was developed with AI assistance under human review. It covers the plan approved on #97 (sections 5–8), checked against main today. Each claim about the code carries a path. Nothing below re-opens a ruling. Where main forces a change, I say what it is and what I would do. Where I need you to decide, it is marked D1–D9, and they are collected at the end. No code until you answer.

   Found and filed on the way: #283 (two passages of the storage format spec that disagree with ADRs 0217 and 0161) and #284 (ADR 0241 is filed by both #259 and #264).

   ---

   1. What in sections 5–8 no longer matches main

   §5, tier 3

   Plan said	main today	Change
   "a set's nine blobs, as they are"	Still true. ExtentSet.BlobsOf (src/Varve.Store/Extents.cs:87-117) names order0–order5, terms, graphs and commits, but commits is absent in pre-0161 sets and graphs may be absent. Format 5's seek columns live inside each orderK (storage format §7.5). There is no separate seek-directory object.	The shipper copies exactly the blobs BlobsOf names for the set, not a fixed nine.
   (the memory work)	Nothing has landed: #268 (#222) and #264 (#227) are open, and #256 is an open issue. #268 changes only how a format-5 seek directory is read, not what a set is. #264 makes a checkpoint flush the memtable, so there are more and smaller sets, which means more objects shipped but no new shape. #256 concerns the projection's runs under derived/index/runs/, which are local cache and never in the store (ObjectDerivedStore.cs:111, Shared: extents and catalog only).	No design change. I check this again at the head of pull request 2. If #268 or #256 changes a set's blobs by then, the shipper follows BlobsOf and nothing else.
   reader record readers/<id>-shipper	ADR 0160's record (ReaderRecordFormat.cs, storage format §8.1) has catalogOldest, catalogNewest, state and maxDerived. A follower uses <ReaderId>-follower (ADR 0164, amendment 2026-10-20).	One record per target: <ReaderId>-ship-<target-id>, where target-id is a short digest of varve:store and varve:prefix. Two targets of one dataset must not share an interval. The record still protects what it copies: the shipper announces the catalog version it copies before its first read, and moves the interval forward only after the target's catalog is replaced.
   (not in the plan) ADR 0213 point 7	A shared set's terms and commits are written as version 4 while any reader record says maxDerived < 6. The source writer rewrites old sets oldest first, and an older build treats a version-6 blob as a miss.	Byte-identical copying stays. The shipper never rewrites a set on the target: point 7's rewrite is the source writer's, and its result reaches the target as a new set under a new name. The shipper verifies each blob's checks (ADR 0213) before its create. A damaged source blob is not shipped. That catalog version waits until the source writer has rewritten the set under a fresh name (0213 point 5), and the shipper reports shipping: waiting-on-damaged-set. A target reader of an older build is answered by replay, never by a wrong result.
   (not in the plan)	Dataset.ShipAsync (src/Varve.Store/Replicas.cs:29-94) re-creates segments through Append/Seal and writes a fresh catalog, so its output is not byte-identical to the source.	A ShipAsync copy is not a shipping target: resuming against one is shipping-diverged at slot 0, or shipping-target-not-empty. ShipAsync stays as the one-shot bootstrap of ADR 0083, and the guide says which to use when.
   Shipper failures (not in the plan)	Part 1 abdicates on Dataset.AppendFailed and Dataset.SpillBound (ADR 0179 point 6).	A target failure is never a source failure. It never abdicates, never releases and never touches the source's writer record. See §2 below.
   varve apply refuses unverified providers; runtime store-unverified	#166 took your Q2 answer, the stricter one: a verification record at that exact endpoint, with the suite version at or above the build's minimum, and the provider verified on the compiled page.	Both checks run at run time. varve apply can check only the page, so it does. D1: where the verification record lives (below).

   D1. I propose that the verification record is the object varve-verification at the root of the store URL given to varve storage verify, the container or the bucket. The shipper reads it at the target's varve:store. It falls under ADR 0217: a format this build doesn't read is store-unverified, never "verified". The alternative is per prefix, which is stricter but makes every new target prefix need its own run.

   §6, promotion and lineage

   • promotion is reserved in no code. ClaimTrigger (src/Varve.Store.ObjectStorage/Model/ClaimTrigger.cs:8-26) has Restart, Operator, Abdication, Probe and Unknown. ADR 0179 point 1 rules only that promotion is an operator's act (Replication part 2: verified stores, log shipping, promotion, geo-replication #166). ADR 0185 adds the trigger, and ADR 0136 gets the amendment. Older builds read it as Unknown and never act on it (ADR 0217 point 2), so the record stays format: 1.
   • The lineage record is named in ADR 0217 point 4 (TheLineageRecordFollowsTheSameRule) and in ADR 0216's inventory as "not on main". ADR 0185 cites both, and the format inventory gains its fixture.
   • "Who claims after promote" has changed under part 1. The plan had varve promote claim on the target's record directly. In a fleet, part 1's start rule (MemberDecision.cs:64-77) would then see a holder that is not in the writer set and claim again at once. That is two epochs for one promotion, and the warmth is lost. D2 below proposes promotion as a request to an instance, like the takeover.
   • The epoch rule (highest adopted chunk epoch, plus one) stands. ObjectStorage.AdoptedEpoch (ObjectStorage.cs:243) already exposes it on a follower, and a target with no writer record gets one created at that epoch.

   §7, tier 2

   • Nothing exists yet: no secondary endpoint, no LastSyncTime, no comp=stats anywhere in src/ or tests/. The Bicep templates offer Standard_GZRS but not RA-GRS or RA-GZRS (deploy/azure/container-apps/main.bicep:52-59), and there is no geo template.

   • A custom role has a precedent: container-apps-handover/main.bicep:653 (jobStarter). The stats role follows its shape.

   • Azurite and the secondary. I don't know whether Azurite 3.37.0 serves -secondary reads and Get Blob Service Stats, so I won't assume it. The first slice of pull request 3 is a spike that answers it. If it does, the geo double runs over Azurite as well as memory. If it doesn't, memory only, and the adapter's secondary path is proved only on the real-account leg, which says so.

   • The real-account leg. azure-objects.yml exists (dispatch only, environment azure-objects, AZURE_TENANT_ID, AZURE_CLIENT_ID and VARVE_AZURE_CONTAINER), but it gates on one variable (ServiceLegs.Require, AzureAccountTests.cs:64-76), not on ADR 0223's rule. The tier-2 leg follows ADR 0223 decision 5, as the provider legs do (tests/Varve.Server.Tests/ProviderTests.cs:271-328):

     • none of VARVE_GEO_ACCOUNT, VARVE_GEO_CONTAINER or VARVE_GEO_TENANT_ID (plus AZURE_CLIENT_ID) set: it skips, naming each;
     • some set: it fails.

     It is a separate workflow, azure-geo.yml, in an environment azure-geo. Its jobs: follow the secondary, report the last-sync time, and promote after a failover. The failover is dispatch only, because a customer-initiated failover takes hours and converts the account. A Bicep file and docs/azure-geo-setup.md, written step by step in the shape of docs/auth-providers.md, are for you to set it up. Until you do, the close-out says tier 2's real-account gate has not run.

   §8, RustFS

   • Done in part 1. Varve:ObjectStorage:StsEndpoint (ADR 0137, amendment 2026-10-27), RustFS behind mock-oauth2 in .github/storage/compose.yaml, and RustFsFleetTests (tests/Varve.Server.Tests/FleetTests.cs:1844). ADR 0182 is unused (docs/adr/README.md:382). Tier 3's S3 leg uses web identity, with no key.
   • ADR 0182 stays free. I don't take it.

   ---

   2. The promotion claim, the hold-off and the writer set (change 1)

   Facts:

   • What the hold-off is. ClaimsHoldOffAfterAStoreFailure (ADR 0163, amendment 2026-11-10, Fleet: a writer whose first open fails never opens again until restart (Started set whatever the outcome) #251) and ClaimAfterAReleaseByItsReason restated (ADR 0180, amendment 2026-11-10, Replication part 1 follow-ups: the releaser stands out of its own release's race, and a forward answers by what happened #216). It is per member, per dataset, in memory, and elapsed-time with jitter (src/Varve.Server/ClaimHoldOff.cs:22-79). Its round is Varve:Fleet:FollowInterval, with a cap of 30 s.
   • What it holds off and what it doesn't. It holds off claims at start, the placed writer's claim after a placement release, and claims in place of a release. A takeover request "is answered whatever the hold-off" (0163:427), and TakeoverAsync (WriterSet.cs:313-349) never reads Held.

   D2. Promotion is a request to an instance, as the takeover is (recommended).

   • The request. It is PUT /manage/datasets/{d}/state {"promote":{"source":"<address>","position":p}} on the target fleet, with the operator role. The receiving instance answers it and never forwards it.
   • Which instance can answer. The receiving instance must be in the dataset's writer set in the target fleet and following the replica (a tier-1 follower of the target, which is active-active across copies anyway). It then:
     1. replaces the lineage with promoted;
     2. claims in place, at max(record epoch, highest adopted chunk epoch) + 1, with trigger promotion (ObjectStorage.ClaimInPlaceAsync, extended to create the record when it is absent);
     3. fences, and becomes the writer. Its projection and cache stay warm.
   • The client. varve promote is the client of that route (ManagementClient.RequestPromotionAsync), as varve manage takeover is of the takeover.
   • No fleet. For a directory target, or a target served by no fleet, varve promote --embedded <target> acts on the storage directly and then releases with the new reason promotion. The release lets the target's placed member start by the start rule ("the record is released") rather than by claiming over a live holder.
   • Refusals reuse 409 takeover-refused's reasons where they apply (not-in-writer-set, cold), plus not-a-replica and source-mismatch.
   • Rejected alternative: the CLI claims directly, as the plan had it. The target fleet's start rule then takes a second epoch, and the warm projection is thrown away.

   How promotion sits with the hold-off:

   • A promotion is an operator's act, answered whatever the hold-off, exactly as the takeover is (ADR 0163's amendment gets a dated line saying so).
   • A promotion claim that fails after winning the record releases its epoch, by part 1's fix 7453a21. The release reason is promotion, not storage-error. A storage-error release would start a standby race (0180's restatement), and a standby would then finish a promotion on its own, which would break "nothing in Varve promotes a replica on its own".
   • Hence a new open rule, beside dataset-is-replica. An open for write of a dataset whose lineage is promoted, but whose record has no promotion claim that fenced, is refused with 409 dataset-is-replica, reason promotion-incomplete. Only re-running the promotion completes it, and that is idempotent on the same lineage.
   • dataset-is-replica is not a failure for the hold-off. A member placed on a replica opens it as a follower (it is not a failed open), so it never enters the hold-off, and it logs nothing at warning. Without this, every placed member of a target fleet would back off to 30 s and log a failure for as long as the replica exists.

   Does a shipper's failed write to the target start a hold-off of its own?

   • Not the claim hold-off. The shipper never claims anything on the source or the target, and a target failure is not a source failure.
   • It does back off its own retries. It uses the same shape (one ship round, doubling to a cap, with jitter), as a separate instance of ClaimHoldOff keyed by target, not as a ruling about claims.
   • What the back-off decides. It decides only when the next round runs, never whether shipping is correct. That is the session's "no timer decides anything".
   • How it shows. Readiness reports shipping: failing with the target and the last error, and never turns unready, as retention doesn't (ADR 0162).

   The writer set on the target:

   • A target fleet declares the dataset with its own placedOn and standbyOn.
   • While the lineage says replica, every member opens it as a follower: no member claims, and the start rule never runs.
   • After promotion, part 1's rules apply unchanged.
   • Writes to a replica. A write forwarded to a replica's writer finds none, which is 503 today. On a replica it becomes 409 dataset-is-replica (ADR 0168 amendment), because retrying won't help.

   ---

   3. What governs a read of a shipped copy (change 2)

   The target fleet's own declarations. Nothing that governs access is shipped.

   • What is shipped: log/, extent sets and the catalog. The log carries:
     • the dataset's own graphs, including a classifier binding's annotated shapes graph and a TBox binding's graphs, since a boundGraph is a graph of the same dataset (ADR 0234 point 1, docs/operator/inference.md:44);
     • Settings commits ("every copy of the dataset agrees on them", spec §2);
     • validator attachments;
     • the management position as a bare integer ("p"^^varve:managementPosition, src/Varve.Protocol/Endpoints/Writes.cs:278,309-312).
   • What is not shipped: grants, varve:ClassifierBinding, varve:TBoxBinding, shapes bindings and moved event declarations. All of these live in the source fleet's _fleet/ (ADR 0166). Also not shipped: the classification and validation projections, which are local cache only (ADR 0233 point 6, ADR 0174 point 7). A target follower rebuilds them from its own fleet's bindings.
   • Therefore, the operator guide says: a replica is only as protected as the declarations made where it is served. A replica in a fleet with no classifier binding shows every row to anyone who can read the graph. Active-active across copies needs the same grants and bindings on both sides. I'll open a follow-up issue for a drift check that compares the two fleets' declarations for a shipped dataset. It is not in part 2's gate.

   A defect the plan didn't see: the recorded management position belongs to another fleet. A dataset on another store is necessarily served by another fleet, since a fleet's system dataset is _fleet/ on its own store. The integer recorded in each commit is a position in the source fleet's system dataset, and nothing records which fleet. On the target today:

   • Writes after a promotion. _order.Observe (src/Varve.Server/OpenDatasets.cs:692-694) and the write-path advance (ServerHost.cs:987-996, Fleet.Recorded) would seed the target's watermark from a source position. Writes are refused 503 management-behind until the target's system head passes that number, or accepted against a meaningless one.
   • Event rendering. EventCatalogues.AtCommitAsync (src/Varve.Protocol/Endpoints/EventCatalogues.cs:184-212) would resolve moved declarations (ADR 0167) at a source position in the target's system dataset. That gives the wrong catalogue, silently, or management-behind for ever.
   • Audit. The as-of grants read (ADR 0165 point 6) would answer with the wrong fleet's grants.

   D3. The fix (in ADR 0185, amending ADR 0165).

   • Which positions are foreign. A recorded management position belongs to the fleet that wrote the commit. The lineage record names the source fleet: the _fleet/log/MANIFEST dataset id, read by the shipper. Every commit at or below the lineage's promoted position, and every commit of a replica, carries a foreign position.
   • Which checks ignore them. The watermark seed, the write-path advance and Recorded read only positions above the promoted one.
   • Events of foreign commits (recommended). Moved declarations are resolved against a shipped copy of the source fleet's system dataset, which is just another varve:ShippingTarget, on _fleet, into fleets/<source-fleet-id>/ on the target store. It is followed read-only. The same copy is what an operator needs after losing the source store, because without it the source's grants and placement are gone with it. Without the copy, an event of a foreign commit whose declarations had moved answers a new 503 events-source-fleet-unavailable, and the guide says why.
   • The alternative. Part 2 doesn't ship _fleet at all, and the problem above is the only answer. That is simpler, but a region's loss then loses its declarations.
   • Audit of a foreign commit answers from the same copy, or says it can't.

   ---

   4. The erasure precondition (change 3), as ADR 0184 would state it

   Erasure mode across copies (an open question owned by #203). A shipped log and a shipped checkpoint carry only ciphertext of private terms (spec §9, I9: no file in log/ and no checkpoint holds plaintext; the extent sets the shipper copies are the checkpoint). Local derived/ caches, which may hold tagged plaintext, are never shipped. The key store is never inside a dataset directory, so it is never shipped. Erasure must take effect on every copy (I10). Before erasure mode may be enabled on a dataset that has a varve:ShippingTarget, or a varve:ShippingTarget declared on a dataset in erasure mode, milestone 9 part 2 must meet one of:
   (a) the copies share the key store's destruction: every copy resolves its subject keys from a key store whose destruction of a key is destruction for every copy, its backups included, before the Erasure commit is acknowledged; or
   (b) erasure mode refuses a varve:ShippingTarget, at apply time and at run time, with a problem naming this ADR.
   A per-copy key store destroyed when the copy adopts the shipped Erasure commit (spec §9 T4's "replicas … destroy their copy of the key") meets neither. A tier-3 target that nothing follows never adopts it, so I10 would hold for that copy only once someone reads it. Until #203 chooses, varve:erasureMode stays refused (ADR 0166), so part 2 cannot meet the case.

   One tension to hand to #203, not to resolve here. Spec §9 T4 has each replica destroy its own copy of a key. The roadmap's milestone 9 part 2 text has one key store outside every dataset, "so destroying a key erases on every replica at once". (a) is the roadmap's. ADR 0184 names the tension, and I'll comment it on #203.

   ---

   5. The split, and each pull request's gate

   The three-pull-request split holds. I found no better seam. Two notes:

   • Pull request 2 grows by D3. I'd keep it there, because the foreign position is a promotion defect.
   • Pull request 3 depends on two open pull requests. The descriptor needs A release descriptor's date follows its digests: release.cs --sbom, and land.cs's three days (#248) #261 (eng/release.cs --sbom), and the problem pages need Problems that say what to do (#194, ADR 0242 filed unaccepted) #267 (ADR 0242's "What to do"). I'll write every new page with ## What to do from the start, so it passes either way.

   Pull request 1, #117 (ADR 0183). Completes: #117, Closes #117 on its last commit.

   • The suite moves to src/Varve.Store.ObjectStorage/Verification/ (layer 5), with:
     • its seven cases;
     • Racers and Rounds as parameters;
     • LosesRaces, a test double of a store that passes ADR 0134's probe (ObjectStorage.cs:838-901) and loses the race.
   • tests/…/ObjectStoreContractCases.cs becomes a call into it, so there is one suite.
   • varve storage verify <url> [--rounds n] [--record <file>] writes the result record and the endpoint's verification record.
   • testdata/object-stores/results/ gets the Azurite and RustFS records.
   • eng/object-stores.cs generates docs/operator/object-stores.md, and eng/status.cs gains check 6.
   • store-unverified gets its problem page.
   • Gate:
     • verify passes against Azurite and RustFS (web identity) in storage (services);
     • it fails with the counterexample against IgnoresConditions and LosesRaces (memory, every OS);
     • the page check is red on a hand-edited page;
     • the format inventory carries fixtures for the result record and the verification record.
   • Supply chain: no new package. Varve.Server already references Varve.Store.ObjectStorage, .Azure and .S3 (src/Varve.Server/Varve.Server.csproj), so the CLI's closure gains code, not dependencies. The tool SBOM's digest changes, and the register doesn't. I run dotnet run eng/lockfiles.cs and expect no diff, and I'll show it.

   Pull request 2, tier 3 and promotion (ADRs 0184, 0185). Completes: none.

   • Code:
     • the ShippingTarget vocabulary and shapes;
     • the shipper;
     • the lineage record;
     • the promotion trigger and release reason;
     • the promote route and varve promote;
     • dataset-is-replica, replica-promoted, shipping-diverged and shipping-target-not-empty;
     • D3.
   • Gate:
     • tier 3 under failure injection (memory↔memory with generated schedules: shipper crashes at every step, target faults, source consolidation and maintenance racing it). A target follower equals the reference model at its reported position;
     • byte identity up to the shipped position;
     • the tier-3 promotion rehearsal (promote, write, the old primary's shipper refused by replica-promoted, and by shipping-diverged when it raced);
     • the same between Azurite and RustFS, and from each to a directory, in the new job shipping (services).

   Pull request 3, tier 2, active-active across copies and the close-out (ADR 0186). Completes: #166, #97 (#117 is closed by pull request 1). The descriptor commit carries Closes #166 and Closes #97.

   • Gate:
     • the geo double (out-of-order replication with a last-sync watermark), with the follower and the failover-promotion schedules;
     • the tier-2 promotion rehearsal;
     • a dataset shipped to a second store and followed there by a second fleet, with writes through either fleet reaching the source's writer, in shipping (services);
     • the real-account leg, or the plain statement that it hasn't run.

   ---

   6. CI jobs, durations, nightly depth

   The durations below are estimates from the existing jobs. I'll replace them with measured per-OS figures (eng/test-durations.cs --label) on each pull request.

   Job	New?	Pull request	OS	Pull request depth, est.	Nightly
   storage (services)	+verify against Azurite and RustFS	1	Linux	+1–2 min (16 racers × 8 rounds × 2 stores)	—
   test-store (in eng/ci.cs)	+verify against the broken stores; +shipping, promotion and geo properties	1–3	Linux, Windows, macOS	+60–90 s each OS	full depth, below
   shipping (services)	new, its own named job	2–3	Linux	6–10 min (Azurite↔RustFS, each↔directory, the cross-fleet case)	—
   geo (account) in azure-geo.yml	new, main only, ADR 0223 rule	3	Linux	skips until set up	dispatch: the failover rehearsal

   A failure in shipping (services) that I can't reproduce goes on its own issue with the job id. I won't retry it until it's green.

   Nightly (.github/workflows/nightly.yml, job store), each with its seed printed (docs/testing.md §8):

   Variable	Pull request	Nightly
   VARVE_VERIFY_ROUNDS	8	64
   VARVE_SHIPPING_SCHEDULES	40	1,000
   VARVE_PROMOTION_SCHEDULES	40	1,000
   VARVE_GEO_SCHEDULES	40	1,000

   Each comes with its budget check at 80%.

   ---

   7. ADRs and dated amendments

   New ADRs, filed unaccepted. I checked every open pull request today, and none uses 0183–0186. I'll check again before filing.

   ADR	Title
   0183	Object store verification and the compatibility page
   0184	Log shipping (with §4's erasure precondition)
   0185	Promotion across copies and the lineage record (with D2 and D3)
   0186	Tier 2: following a geo secondary

   I propose no ADR beyond these. No new package needs one, and the Bicep and the leg sit under ADR 0223 and ADR 0142's amendments.

   Dated amendments (ADR 0068):

   • 0134: the suite is shipped code; LosesRaces.

   • 0135: the target's consolidation is done by its shipper.

   • 0136:

     • the trigger promotion and the release reason promotion;
     • an open for write refused on a replica lineage, and on promoted without a fenced promotion claim;
     • creating the record at max adopted epoch + 1.

   • 0137:

     • the secondary endpoint;
     • the custom blobServices/read role.

   • 0142: the geo layout beside the default and the handover layouts.

   • 0160:

     • the shipper as a reader of the source, one record per target;
     • as the target's maintenance writer, which deletes retired sets against the target's records;
     • a tier-2 follower reads unannounced.

   • 0163:

     • promotion is answered whatever the hold-off;
     • dataset-is-replica is not a failure for the hold-off;
     • a replica's members open as followers.

   • 0164:

     • a follower of a replica;
     • a tier-2 follower over a read-only secondary.

     (Memory bounds for writing, checkpointing and reopening: a dataset written under a limit reopens under it (#227) #264 amends 0164 as well, and whichever lands second merges.)

   • 0165: a recorded management position belongs to the fleet that wrote it (D3).

   • 0166: varve:shipsTo and varve:ShippingTarget, and their shapes.

   • 0167: events of foreign commits (D3).

   • 0168: a write to a replica is 409 dataset-is-replica.

   • 0213: the shipper copies sets as they are, verifies checks before copying, and never rewrites.

   • 0216: the lineage record, the verification record and the result record.

   • 0217: the verification record follows its rule (point 4 already covers lineage).

   • 0113 and 0119: readiness gains shipping, and the problem catalogue gains its entries.

   Problem types, each with a docs/problems/ page with "What to do":

   • store-unverified;
   • shipping-diverged;
   • shipping-target-not-empty;
   • dataset-is-replica (with reasons replica and promotion-incomplete);
   • replica-promoted;
   • events-source-fleet-unavailable, only under D3's fallback.

   ---

   Decisions for you

   • D1. Should the verification record sit at the root of the store URL (proposed), or per prefix?
   • D2. Should promotion be a request answered by a warm member of the target fleet's writer set (proposed), with varve promote --embedded plus a promotion release where no fleet serves the target? Or should the CLI claim directly, as the plan had it?
   • D3. A recorded management position belongs to its fleet, and the lineage bounds it. For events and audit of foreign commits, should the target ship the source's _fleet (proposed), or answer 503 events-source-fleet-unavailable?
   • D4. The erasure precondition's wording in §4, and the spec-versus-roadmap tension, which goes to Milestone 9, part 2: erasure mode (preview) #203.
   • D5. Does the three-pull-request split hold as in §5, with D3 in pull request 2?
   • D6. Should the tier-2 leg be a new workflow azure-geo.yml in a new environment azure-geo (proposed), or a job in azure-objects.yml's environment?
   • D7. The shipper's retry back-off is a separate instance of the hold-off's shape and never a claim hold-off. Agreed?
   • D8. Writes forwarded to a replica answer 409 dataset-is-replica rather than today's 503. Agreed?
   • D9. Should I open the follow-up issue for a cross-fleet declaration drift check now, outside part 2's gate?

   On your answer I file ADR 0183 and start pull request 1.

   ---

   Generated by Claude Code