Repository navigation
Coalesce circuit-breaker health writes - #304
Conversation
BaaRai
left a comment
There was a problem hiding this comment.
Thanks for moving the health writes onto one goroutine, it is a clear step forward from per-call writes. Four problems remain.
1. Writes can still reach the queue out of order. The registry calls forget and persist after releasing r.mu (llmpool/health.go:127-130 in Pass, 156-159 in Trip, 168-171 in Reset). Two concurrent calls on the same profile can therefore enqueue in the reverse order of their state changes, for example a Trip snapshot landing after a Pass clear. Coalescing keeps the last enqueued op, so the database ends up with a tripped row while memory is clean, and the next start restores that stale cooling window.
Fix: enqueue while holding r.mu, or stamp each op with a per-profile sequence number taken under the lock and drop older ops in enqueue.
2. The writer has no lifecycle. newHealthSync takes no context and has no flush or Close (server/llmpool.go:67-76). It is created in newLLMHealthRegistry (server/llmpool.go:27), which is called from server.New, where ctx is in scope. On shutdown nothing drains pending, so the last trip can be lost or written against closed stores, while keeping the cooling window across a restart is the reason this state is persisted (llmpool/health.go:48-49). Each server.New in tests also leaks the goroutine.
Fix: pass ctx to newHealthSync, flush pending when it is done, and let the server wait for that flush before the stores close.
3. The test does not run loop(). TestHealthSyncCoalescesToLatest re-implements the loop body inline (server/llmpool_health_test.go:37-47). The real dispatch (clear vs save, the OpenUntil mapping, the wake handling) is never executed. Please drive newHealthSync with fake save/clear functions and wait on a channel for the expected rows.
4. A failed save is not retried. The op is removed from pending before the write, so when save fails, the state is only retried at the next transition for that profile, which may never come (server/llmpool.go:130-132).
Fix: put the op back in pending when no newer op exists for that profile; otherwise let the newer op win.
BaaRai
left a comment
There was a problem hiding this comment.
Thanks, the ordering, lifecycle, retry and test points are addressed, and the tests now drive the real loop.
One problem is introduced by the ordering fix: persist and forget are now called while r.mu is held (llmpool/health.go), and apply holds h.mu across the SaveLLMHealth / ClearLLMHealth call (server/llmpool.go). So enqueue blocks on an in-flight database write while the registry lock is held, and Trip/Pass — the hot path — stalls for the duration of a slow or hung write, which is exactly when the breaker matters. Keep the ordering, but release h.mu around the I/O: do the seq check and take the op under the lock, write outside it, then re-take the lock to record applied when it is still the newest.
A newer state for the same profile replaces one that has not been written yet, and a failed write is logged instead of discarded on a detached goroutine.
…database closes. A later update can no longer be queued behind an older one, a failed save is retried, and shutdown waits for the last cooling window.
8eeaaac to
c9d81a4
Compare
BaaRai
left a comment
There was a problem hiding this comment.
Thanks, the write now runs outside mu with writeMu ordering the calls, and the superseded check keeps the newest state, so enqueue no longer waits on database I/O. I rebased the branch on current main (the shutdown order moved in #300) and kept FlushLLMHealth after the drain. Approved.
Summary
Test plan
TestHealthSyncCoalescesToLatest.