[RFC] Remote Store Fence: object-store-backed primary fencing to enable zero-replica auto-recovery

[Bukhtawar]

Intent

Make the remote store — not a replica — the authority on "who may acknowledge writes" for a shard.

Today, segrep + remote store indices rely on replicas for primary fencing: every write fans out a no-op PRIMARY_TERM_VALIDATION request (ReplicationModeAwareProxy), and a stale primary fails because a replica that has learned a newer term rejects it. This makes at least one replica a safety requirement, which blocks the end goal:

Zero-replica remote-backed indices that recover automatically on node loss — the shard is reassigned immediately, hydrates from the object store, and (with partial-locality serving) answers queries within seconds like a warm shard while data re-heats in the background. In that world the new primary starts acknowledging writes seconds after node-left, while the old primary is very likely still alive and unaware — so a fencing mechanism that works with zero replicas is the critical prerequisite, not an optimization.

Writes acked by a stale primary today are silently lost: remote store files are primary-term-scoped (no physical corruption), but the new primary's lineage never includes them.

Scope of this RFC: Phase 0 (the fence) is the critical change, specified in full below. Phase 1 (auto-restore trigger) and Phase 2 (warm search during hydration) are outlined as delivery phases; each will get its own detailed design.

The critical change: a per-shard fence object

One small mutable object per shard in the translog repository (the repository that owns the acknowledgement path):

fence = { term, allocationId, seq }     — updated only via compare-and-swap (ETag If-Match)

• The fencing token is the CAS chain (opaque version token / ETag), not the term. Every acknowledged sync advances it; any second writer — stale primary, zombie relocation source, same term or not — holds a stale token and fails its next CAS.
• The fence is control-plane only: never read by snapshot restore, pinned-timestamp resolution, or GC. All snapshot-referenced files (segment/translog data and metadata) remain immutable with unchanged names — the shallow-snapshot (pinned timestamp) contract is untouched.

Write path (request durability, term T)

1. PUT translog data files                       (unchanged)
2. PUT immutable translog metadata file          (unchanged)      ─┐ concurrent
3. CAS fence {term:T, alloc, seq+1}                                ─┘
4. ack ⇐ (2) AND (3) succeeded
   CAS rejected → re-read fence → newer term / different owner → failShard("primary fenced")

PUT overhead — called out explicitly

The fence adds exactly one small PUT (the CAS) per translog sync on the ack path, and nothing else:

Path	Extra requests	Today	Delta
Translog sync (ack)	+1 PUT per sync	~2 PUTs (tlog data + metadata)	~+50% PUT count on this path
Segment/refresh upload	0	—	freshness check rides the translog CAS; on-demand CAS only if stale
Idle shard	0	—	no uploads ⇒ nothing to fence; no heartbeat
Reads / GETs	0	—	ETag threaded from each PUT response; fence GETs only at startup, seal, and after a rejection
Storage	~0	—	one tiny fence object per shard + one seal marker per failover

Magnitude: RemoteFsTranslog coalesces concurrent bulks, so sync rate is bounded by upload latency (~30–50 ms ⇒ ≤20–25 syncs/s even on the hottest shard). Worst case ≈ 2M extra PUTs/day ≈ $10/day for the hottest possible shard (S3 PUT $0.005/1k); a typical shard at 1–5 syncs/s is $0.40–$2/day, and cost scales with write throughput (idle ≈ $0). Conditional PUTs are billed the same as normal PUTs. Latency impact ≈ 0: the CAS runs concurrently with the metadata PUT and the translog data PUT dominates.

Context: replica term validation is a free intra-cluster RPC, so at ≥1 replica this is net-new request cost — but for the motivating zero-replica case the comparison is against running a replica at all (compute + storage + inter-AZ traffic), which the fence makes safe to drop. Knob for cost-sensitive users: a lease-amortized mode (CAS every N seconds, acks valid within the lease) caps overhead at ~1 PUT/N-sec regardless of throughput, trading a bounded N-second fencing window (fencing.mode: per_sync | lease; lease is the natural fit for async durability, whose loss window stays bounded by the async interval as today).

Failover (term bump to T+1)

1. New primary: CAS fence → {term:T+1, alloc:new}     — the SEAL (strictly-greater term enforced)
2. AFTER the seal: read latest metadata = restore point
3. Recover, open for writes; first CAS continues the chain

Invariant: every acked write required a pre-seal CAS, so the post-seal restore point contains all acked writes. Zero acked-write loss, no replica required, no timeout/lease wait — the seal is one CAS (~10–50 ms) and recovery can begin the instant node-left is observed.

Relocation (same term, ownership handoff)

Target CASes the fence to itself (same term, new allocationId) inside the existing permit-drained handoff window. A zombie source's next CAS fails → failShard. This upgrades the same-(term, generation) two-writer case from detected during restore/GC (RemoteStoreUtils.verifyNoMultipleWriters throwing after data is already damaged) to prevented.

Orphan handling

Metadata PUT and fence CAS are not atomic (object stores condition a PUT only on its own key), so a fenced writer can land a never-acked orphan metadata file. Mitigation: an immutable per-failover seal marker seal__inv(T+1)__<last-valid-md-file>; pinned-timestamp resolution, restore, and verifyNoMultipleWriters skip old-term files beyond it, and GC deletes them. (A future evolution — folding metadata into a single versioned CASed key with object-store versioning providing point-in-time history — removes orphans by construction and is cost-neutral; deferred.)

API surface

• BlobContainer: opaque-version-token conditional writes — getWithVersion(key), compareAndSwap(key, expectedToken, bytes), create-if-absent. Implementations: S3 (If-Match/If-None-Match, GA since 2024), GCS (generation preconditions), Azure (ETags), FS (tests). Fencing is gated on the repository advertising this capability.
• Fence CAS in TranslogTransferManager (ack path); freshness check in RemoteStoreRefreshListener; seal in the remote-store restore path; ownership CAS in relocation handoff; seal-marker awareness in RemoteStoreUtils.
• Settings: remote_store.fencing.enabled, freshness interval.

Compatibility

• Rolling upgrade: run fence and replica term validation until cluster min version supports fencing, then drop the fanout (ReplicationModeAwareProxy is the seam). Fence bootstrap via create-if-absent; no fence object ⇒ legacy behavior.
• Behavioral delta at ≥1 replica: replicas leave the write path entirely; replica health falls to follower checks + segrep lag detection. No new availability dependency — with remote store, writes already fail when the object store is down.

Delivery phases

Phase 0 — Fence (this RFC) — implemented in #22774

The safety foundation, specified above. See #22774 for the design as built, which refines this RFC in three ways. Detailed rationale, invariants, sequence diagrams and formal models live there rather than being duplicated here.

Refinements over the specification above:

1. The acknowledgement path is keyed per primary term (fence__<invertLong(term)>), not a single mutable object. A takeover creates its own key with create-if-absent and then deletes every lower-term key — deletion, not the CAS, is what fences the incumbent. This was necessary because a single shared object gives safety but not deterministic liveness: both writers contend with the same primitive, so the rightful new primary's read→CAS window is a race it can lose, and a hot incumbent can drive it into a repeated recovery-failure loop. With term-scoped keys a writer's only destructive act is deleting keys below its own term, so it can never touch a higher-term writer's path.
2. Seal-before-restore is explicit and asserted, on both the recovery and the promotion paths. The promotion path was a real gap: a superseded promoted replica kept serving as a started primary, because postActivatePrimaryMode logs and swallows the fenced upload's IOException.
3. Relocation ownership transfer is performed by the source, as its last act before the primary context leaves the node and after its uploads have drained, retaining the resulting token so an aborted handoff can be reverted. Letting the target claim the shared object spuriously fenced a healthy source, since IndexShard#relocated releases the upload drains on abort and the cluster keeps the source as primary.

Gating shipped as a dynamic cluster setting cluster.remote_store.fencing.enabled baked into the final index setting index.remote_store.fencing.enabled at creation time, so operators can flip it for new indices while no live index ever switches its write witness mid-flight.

Exit criteria status:

Criterion	Status
Fence CAS on every ack path	done
Seal on failover	done — recovery and promotion
Ownership CAS on relocation	done — source-performed transfer with revert on abort
Seal-marker-aware resolution/GC	open — orphan metadata files from a fenced write are term-scoped and harmless to acked data, but resolution and GC are not yet marker-aware
Observe-only mode first	open
Replica term validation still active	deviation — #22774 switches fenced request-durability indices to NO_REPLICATION immediately rather than after Phase 1 bakes. Flagged there for reviewer input; this staging may need revisiting

Formal verification (in-repo, model-checked by TLC as part of ./gradlew check): seal ordering and acked-write loss, cross-term takeover determinism, and the equal-term relocation handoff, each over a complete state space. Two alternative designs were refuted by the checker along the way — a single shared acknowledgement-path object, and a claim-object escalation scheme that moved the livelock rather than removing it.

Phase 1 — Auto-restore trigger

On node-left, the cluster manager reassigns a lost zero-replica remote-backed primary immediately and the new node hydrates from the remote store — no manual _remotestore/_restore, shard goes RED → recovering → STARTED without operator action.

• Allocation: allow unassigned primaries of remote-backed indices to be assigned to any eligible node with a remote-store recovery source (auto version of RemoteStoreRestoreService), gated on index.remote_store.auto_restore.enabled + fencing enforced + repository CAS capability.
• Recovery sequence: seal (fence CAS, strictly-greater term) → post-seal restore-point read → full hydration → translog replay → open for writes.
• RTO in this phase: node-left detection + full segment download (proportional to shard size). Safety is complete here; speed comes in Phase 2.

Phase 2 — Warm search during hydration

Decouple RTO from shard size: the shard opens immediately in partial locality and serves like a warm shard while re-heating.

• Open the engine over CompositeDirectory + FileCache with block-level on-demand fetch from the remote segment store; background promotion to full locality (PARTIAL → FULL becomes a shard lifecycle state rather than an index tier property).
• Reads can open before translog replay completes (search visibility is post-refresh anyway); writes open after replay.
• RTO: seconds, independent of shard size; query latency degraded until the cache warms.
• Dependencies: promotion of the writable-warm machinery (WRITABLE_WARM_INDEX_EXPERIMENTAL_FLAG), file-cache-headroom-aware allocation, merge throttling during partial locality, and locality/hydration observability.
• Note: this phase makes the fence more critical, not less — the new primary acks within seconds of node-left while the old primary is almost certainly still alive, so the multi-writer overlap the fence closes occurs on essentially every failover.

Phases 1 and 2 are unsafe without Phase 0; neither requires further changes to the fence.

Phase 1's recovery sequence — seal, then post-seal restore-point read, then hydration and replay — is already implemented as part of #22774, so what remains for Phase 1 is the allocation-level trigger plus its index.remote_store.auto_restore.enabled gating.