Skip to content

fix: Check watch caches against etcd once an hour - #846

Merged
vvoytovych merged 1 commit into
mainfrom
feat/watch-cache-consistency-check-period
Oct 15, 2026
Merged

vvoytovych merged 1 commit into
mainfrom
feat/watch-cache-consistency-check-period

Conversation

@vvoytovych

Copy link
Copy Markdown
Contributor

Milo keeps one watch cache per project per resource type, and since Kubernetes 1.34 each compares itself with etcd every 5 to 7.5 minutes. Each check is a full LIST, so with tens of thousands of caches per server the checks are nearly all of its etcd LISTs. In production none has found a difference.

This adds a flag for the period, one hour by default, applied before any storage is built.

Change Effect
a flag for the consistency check period, default one hour about twelve times fewer checks: in production about 55 a second, not about 670
  • The check stays on, and so does the protection it gives snapshot LISTs.
  • The period is one setting for the whole process. It covers every server in Milo's binary: the extensions server, the control plane and the aggregator. All their caches are Milo's, and nothing else sets it.
  • A period that is not positive stops the server at start.

Test plan

  • Unit tests: the default is one hour, the flag sets the period, a non-positive period is refused; removing each fails its test
  • On kind, with the flag at one minute, the checks run at that pace
  • On kind, no check runs in 14 minutes at the default; set late, checks start within 7.5 minutes
  • CI passes on this branch

Related to datum-cloud/infra#7071

Milo keeps one watch cache per project per resource type, and since
Kubernetes 1.34 each cache compares itself with etcd every 5 to 7.5
minutes through a full LIST. With tens of thousands of caches per
server, those checks are nearly all of its etcd LISTs.

Add --watch-cache-consistency-check-period, default one hour, and set
the cacher's period from it before any storage is built, since each
cache reads the period when its checker starts. The check stays on.

Related to datum-cloud/infra#7071
@vvoytovych
vvoytovych merged commit 6cab515 into main Oct 15, 2026
7 checks passed
@vvoytovych
vvoytovych deleted the feat/watch-cache-consistency-check-period branch October 15, 2026 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants