Skip to content

Coalesce circuit-breaker health writes - #304

Merged
BaaRai merged 4 commits into
BaaRai:mainfrom
Protonmatter:protonmatter/llmpool-health-queue
Nov 17, 2026
Merged

BaaRai merged 4 commits into
BaaRai:mainfrom
Protonmatter:protonmatter/llmpool-health-queue

Conversation

@Protonmatter

Copy link
Copy Markdown
Contributor

Summary

Test plan

  • TestHealthSyncCoalescesToLatest.

@BaaRai BaaRai left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for moving the health writes onto one goroutine, it is a clear step forward from per-call writes. Four problems remain.

1. Writes can still reach the queue out of order. The registry calls forget and persist after releasing r.mu (llmpool/health.go:127-130 in Pass, 156-159 in Trip, 168-171 in Reset). Two concurrent calls on the same profile can therefore enqueue in the reverse order of their state changes, for example a Trip snapshot landing after a Pass clear. Coalescing keeps the last enqueued op, so the database ends up with a tripped row while memory is clean, and the next start restores that stale cooling window.
Fix: enqueue while holding r.mu, or stamp each op with a per-profile sequence number taken under the lock and drop older ops in enqueue.

2. The writer has no lifecycle. newHealthSync takes no context and has no flush or Close (server/llmpool.go:67-76). It is created in newLLMHealthRegistry (server/llmpool.go:27), which is called from server.New, where ctx is in scope. On shutdown nothing drains pending, so the last trip can be lost or written against closed stores, while keeping the cooling window across a restart is the reason this state is persisted (llmpool/health.go:48-49). Each server.New in tests also leaks the goroutine.
Fix: pass ctx to newHealthSync, flush pending when it is done, and let the server wait for that flush before the stores close.

3. The test does not run loop(). TestHealthSyncCoalescesToLatest re-implements the loop body inline (server/llmpool_health_test.go:37-47). The real dispatch (clear vs save, the OpenUntil mapping, the wake handling) is never executed. Please drive newHealthSync with fake save/clear functions and wait on a channel for the expected rows.

4. A failed save is not retried. The op is removed from pending before the write, so when save fails, the state is only retried at the next transition for that profile, which may never come (server/llmpool.go:130-132).
Fix: put the op back in pending when no newer op exists for that profile; otherwise let the newer op win.

@BaaRai BaaRai left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the ordering, lifecycle, retry and test points are addressed, and the tests now drive the real loop.

One problem is introduced by the ordering fix: persist and forget are now called while r.mu is held (llmpool/health.go), and apply holds h.mu across the SaveLLMHealth / ClearLLMHealth call (server/llmpool.go). So enqueue blocks on an in-flight database write while the registry lock is held, and Trip/Pass — the hot path — stalls for the duration of a slow or hung write, which is exactly when the breaker matters. Keep the ordering, but release h.mu around the I/O: do the seq check and take the op under the lock, write outside it, then re-take the lock to record applied when it is still the newest.

A newer state for the same profile replaces one that has not been written yet, and a failed write is logged instead of discarded on a detached goroutine.
…database closes.

A later update can no longer be queued behind an older one, a failed save is retried, and shutdown waits for the last cooling window.
@BaaRai
BaaRai force-pushed the protonmatter/llmpool-health-queue branch from 8eeaaac to c9d81a4 Compare November 17, 2026 21:40

@BaaRai BaaRai left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the write now runs outside mu with writeMu ordering the calls, and the superseded check keeps the newest state, so enqueue no longer waits on database I/O. I rebased the branch on current main (the shutdown order moved in #300) and kept FlushLLMHealth after the drain. Approved.

@BaaRai
BaaRai merged commit eddf03c into BaaRai:main Nov 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants