feat(health): report a recall_log source that went quiet, not just a dead table - #27
Merged
Merged
Conversation
…dead table The existing stall check asks "has ANYTHING written recently", which is only true once every writer is dead at the same time. In practice they die one at a time: a refactor drops a CLI branch, that caller's recalls stop, and the other callers keep the table busy — so the table reads healthy forever and the dead caller is invisible. Two real instances on one deployment, both found by hand weeks late, and both sitting in recall_log the whole time as a source whose last_ts stopped on a date and never moved again: tool-recall-hook 2484 rows 2026-05-05 .. 2026-07-15 silent 20d mcp 668 rows ... .. 2026-07-10 silent 24d detectStalledSources() reports any source with >= sourceStallMinHistory rows (200) and >= sourceStallDays (3) of silence. The volume floor is what keeps it usable: one-off CLI probes and debug labels live below it and would otherwise flood every run. Verified against the real 21k-row table — the two above are reported, the three live sources are not, and `cli` (23 rows, silent 21d) is correctly treated as an occasional caller rather than a stall. A stall is a signal, not a verdict: a renamed source looks exactly like a dead one from here. The report says so and leaves the call to a human. Which is the second half of this commit. Chasing the `mcp` entry showed it was a rename in effect — 3044bb2 dropped `_source: 'mcp'` from recall_memory's buildMemoryContext call, so those recalls kept happening under the default 'context-builder' label. No traffic lost, but the by-source breakdown quietly stopped meaning what it says, and 668 rows' worth of MCP calls have been hiding inside another caller's 12k for 24 days. Label restored. 289 passed / 0 failed. Without the change the two new assertions go red while the whole-table ones stay green — that gap is the bug. Co-authored-by: 千夏 <qianxia@clawgamers.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
The existing stall check asks "has anything written recently" — only true once every writer is dead at the same time.
In practice they die one at a time. A refactor drops a CLI branch, that caller's recalls stop, the other callers keep the table busy, and the table reads healthy forever while the dead caller is invisible.
Two real instances, both found by hand weeks late
Both were sitting in
recall_logthe whole time as a source whoselast_tsstopped on a date and never moved:What this adds
detectStalledSources()— any source with ≥sourceStallMinHistory(200) rows and ≥sourceStallDays(3) of silence.The volume floor is what makes it usable: one-off CLI probes and debug labels live below it and would otherwise flood every run.
Verified against the real 21k-row table:
tool-recall-hookmcpcontext-builderprompt-recall-hookcliA stall is a signal, not a verdict. A renamed source looks exactly like a dead one from here, so the report says so and leaves the call to a human.
Which is the second half of this PR
Chasing the
mcpentry showed it was a rename in effect.3044bb2dropped_source: 'mcp'fromrecall_memory'sbuildMemoryContextcall:Those recalls kept happening — under the default
'context-builder'label. No traffic lost, but the by-source breakdown quietly stopped meaning what it says, and 668 rows' worth of MCP calls have been hiding inside another caller's 12k for 24 days. Label restored.Verification
289 passed / 0 failed. Without the change the two new assertions go red while every whole-table assertion stays green — that gap is the bug:
🤖 Generated with Claude Code