[scotwells]

Summary

Search results often showed a resource as it was when it was created, so a load balancer that received its hostname seconds later could never be found by that hostname, and resources deleted days ago kept appearing. This happened because the indexer's replicas each batch events for a few seconds and replace the whole document on flush, so whichever replica flushed last decided what the index held, and nothing checked whether that state was the newest. With this change a document only ever moves forward: results reflect the current state of a resource, deleted resources disappear and stay gone, and hostnames or addresses assigned after creation become findable. Dry-run requests, which never create anything, no longer leave documents behind for objects that never existed.

What changed

Ordering guard (compare-and-set, with a shared arbiter). The indexer now creates and shares a JetStream KV bucket (SEARCH_INDEXER_DOC_STATE, flag --nats-doc-state-bucket) that records, per index and document, the newest state any replica has accepted: its resourceVersion, whether it is indexed, unindexed or deleted, a copy of the document, and the latest Meilisearch task known to carry a state for it. Every flush runs three steps in internal/indexer/ordering.go:

1. Claim each operation with a compare-and-set. An operation older than the recorded state is replaced by that state, so a late create re-applies the newest update rather than the create response.
2. Enqueue the claimed states and learn the task uid that carries them.
3. Record that task. Meilisearch applies tasks in uid order, so if the bucket already names a newer state, it was enqueued before this task and is about to be overwritten; the flush enqueues it again behind its own task. Only the flush holding the latest recorded task repairs, and a flush that fails between steps is nacked so the redelivery runs the protocol from the newest recorded state.

Within one buffer, an operation older than the one already buffered for its document is dropped instead of replacing it.

Why not read the current document from Meilisearch. Its writes are queued tasks; in production a task takes 47-79 s to apply at a healthy queue depth, longer than the 10 s flush window, so a read of the current document misses exactly the newer write it would need to see. The bucket is linearizable per key and has no lag. Keeping a copy of the document in the record is what lets the flush that would clobber a newer state re-enqueue that state itself, which closes the race between two replicas that both claim before either enqueues (covered by TestBatcher_SameObject_ConcurrentFlushes_RepairsOverwrite).

Why not partition by object. Audit events are published on one literal subject (audit.k8s.activity) with no per-object token, so JetStream cannot partition them by uid without changing the Vector sink, the stream, the consumers and the Deployment, and replacing the HPA with a fixed partition count. Partitioning alone also does not survive redelivery: a nacked older batch redelivered after a newer one still regresses the document, so a version guard is needed either way.

Delete wins over late writes. A delete verb, or a write carrying a deletionTimestamp, is recorded as a tombstone that supersedes any later-flushed write regardless of resourceVersion, since a uid is never reused. Removal because an object stopped matching a policy stays an ordinary versioned state (DeleteOp.ObjectDeleted false), so a newer matching write brings the document back and a policy change cannot strand an object.

Dry-run events are skipped. Events whose requestURI carries dryRun are acked without indexing, for deletes as well as creates, and so are writes whose response object has no resourceVersion. The one e2e fixture that publishes hand-built audit events now carries a resourceVersion, as every real API server response does.

Operations. The bucket is created by the indexer at startup (CreateOrUpdateKeyValue), file-backed, history 1, with a TTL of 7 days (--doc-state-ttl, matching the audit stream's retention so no redelivered event can outlive the record that orders it) and --nats-doc-state-replicas (default 1; production can raise it to match its streams). Its size is roughly one entry per live indexed document. Two new counters, search_indexer_stale_writes_superseded_total and search_indexer_order_repairs_total, show how often the guard acts. The 400 HTTPProxy documents already frozen in production are not touched by this change; a reindex of HTTPProxy after deploy brings them up to date or removes the ones whose objects are gone.

Test plan

• Unit tests pass: go test -timeout 5m ./...
• Lint passes: go vet ./...
• Regression tests cover an older update flushed after a newer one (same window, later batch, split across two batchers, and two flushes claiming before either enqueues), a delete followed by a late create or update, a policy removal followed by a newer matching write, and a dry-run create or delete
• Staging: after deploy, an HTTPProxy created fresh is findable by its canonical hostname within a minute, and a deleted one disappears from results

Fixes #123