Repository navigation
List before watching, so an empty cluster counts as synced - #6
Merged
Merged
Conversation
The CR index went straight into a watch and marked itself synced on the first event. On a cluster with no LLMSLORequirement there is none: the API server sends no ADDED event and its first BOOKMARK can take minutes. slo-api then waited its 30s for a sync that never came, the liveness probe killed it before it served, and the pod crash-looped; had it lived, /readyz would have stayed 503. List first, rebuild the index from the result and mark it synced -- an empty list is a complete one -- then watch from the list's resourceVersion. The test fake always sent a BOOKMARK, which is why this never showed; the new tests use a watch that stays silent, and fail on the old code.
The release with the CR index that lists before it watches.
_replace cleared the index and then re-added every CR outside the lock. It runs on every watch reconnect, so a request in between saw no CRs and answered a known route as unknown. Build the new index aside and swap it in under the lock: a reader sees the old index or the new one. The test hands _replace a generator that records what a reader sees at each step; with the clear-then-add version it sees an empty index.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom. On a fresh cluster with no
LLMSLORequirement, slo-api crash-loops: it logscr-index not synced after 30s; serving anyway, but the liveness probe (2s delay, 10s period, 3 failures) has already killed it; a pod that survives answers/readyzwith 503. Seen on a new arm64 cluster right after install.Cause. The CR index went straight into a watch and set
syncedon its first event. With no CRs the API server sends no ADDED event, and its first BOOKMARK can take minutes. Clusters that already have SLOs never showed it: their ADDED events arrive at once. The test fake always sent a BOOKMARK, so the tests could not show it either.Change.
_watch_oncelists first, rebuilds the index from the result (which also drops CRs deleted while no watch ran), marks it synced -- an empty list is a complete one -- and then watches from the list'sresourceVersion.Tests.
tests/test_cr_index_sync.pyuses a watch that stays silent, like a real API server on an empty cluster: an empty cluster must report synced, and listed CRs must be indexed before any watch event. Both fail on the old code (wait_syncedstays False); with this change all 55 tests pass.Release. Also sets
pyproject.tomlto 0.2.1, so thev0.2.1tag can be cut once this merges.