Skip to content

feat(health): report a recall_log source that went quiet, not just a dead table - #27

Merged
DottytheHomeless merged 1 commit into
mainfrom
feat/per-source-stall-detection
Aug 4, 2026
Merged

DottytheHomeless merged 1 commit into
mainfrom
feat/per-source-stall-detection

Conversation

@MXAntian

@MXAntian MXAntian commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

The gap

The existing stall check asks "has anything written recently" — only true once every writer is dead at the same time.

In practice they die one at a time. A refactor drops a CLI branch, that caller's recalls stop, the other callers keep the table busy, and the table reads healthy forever while the dead caller is invisible.

Two real instances, both found by hand weeks late

Both were sitting in recall_log the whole time as a source whose last_ts stopped on a date and never moved:

tool-recall-hook   2484 rows   2026-05-05 .. 2026-07-15   silent 20d
mcp                 668 rows   ...        .. 2026-07-10   silent 24d

What this adds

detectStalledSources() — any source with ≥ sourceStallMinHistory (200) rows and ≥ sourceStallDays (3) of silence.

The volume floor is what makes it usable: one-off CLI probes and debug labels live below it and would otherwise flood every run.

Verified against the real 21k-row table:

source rows silent reported
tool-recall-hook 2484 20d
mcp 668 24d
context-builder 12083 0d
prompt-recall-hook 3051 0d
cli 23 21d — (occasional caller, not a stall)

A stall is a signal, not a verdict. A renamed source looks exactly like a dead one from here, so the report says so and leaves the call to a human.

Which is the second half of this PR

Chasing the mcp entry showed it was a rename in effect. 3044bb2 dropped _source: 'mcp' from recall_memory's buildMemoryContext call:

-        _source: 'mcp',
-        _sessionId: null,

Those recalls kept happening — under the default 'context-builder' label. No traffic lost, but the by-source breakdown quietly stopped meaning what it says, and 668 rows' worth of MCP calls have been hiding inside another caller's 12k for 24 days. Label restored.

Verification

289 passed / 0 failed. Without the change the two new assertions go red while every whole-table assertion stays green — that gap is the bug:

✗ (d) a high-volume source that stopped is reported as stalled — []
✗ (d) stalled source carries volume and silence so it can be triaged

🤖 Generated with Claude Code

…dead table

The existing stall check asks "has ANYTHING written recently", which is only
true once every writer is dead at the same time. In practice they die one at a
time: a refactor drops a CLI branch, that caller's recalls stop, and the other
callers keep the table busy — so the table reads healthy forever and the dead
caller is invisible.

Two real instances on one deployment, both found by hand weeks late, and both
sitting in recall_log the whole time as a source whose last_ts stopped on a
date and never moved again:

  tool-recall-hook   2484 rows   2026-05-05 .. 2026-07-15   silent 20d
  mcp                 668 rows   ...        .. 2026-07-10   silent 24d

detectStalledSources() reports any source with >= sourceStallMinHistory rows
(200) and >= sourceStallDays (3) of silence. The volume floor is what keeps it
usable: one-off CLI probes and debug labels live below it and would otherwise
flood every run. Verified against the real 21k-row table — the two above are
reported, the three live sources are not, and `cli` (23 rows, silent 21d) is
correctly treated as an occasional caller rather than a stall.

A stall is a signal, not a verdict: a renamed source looks exactly like a dead
one from here. The report says so and leaves the call to a human.

Which is the second half of this commit. Chasing the `mcp` entry showed it was
a rename in effect — 3044bb2 dropped `_source: 'mcp'` from recall_memory's
buildMemoryContext call, so those recalls kept happening under the default
'context-builder' label. No traffic lost, but the by-source breakdown quietly
stopped meaning what it says, and 668 rows' worth of MCP calls have been hiding
inside another caller's 12k for 24 days. Label restored.

289 passed / 0 failed. Without the change the two new assertions go red while
the whole-table ones stay green — that gap is the bug.

Co-authored-by: 千夏 <qianxia@clawgamers.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@MXAntian
MXAntian marked this pull request as ready for review August 4, 2026 04:12
@DottytheHomeless
DottytheHomeless merged commit dcd6d1c into main Aug 4, 2026
2 checks passed
@DottytheHomeless
DottytheHomeless deleted the feat/per-source-stall-detection branch August 4, 2026 04:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants