Repository navigation
feat(health): measure Cue speed, timeouts and lost tool calls daily (#57) - #61
Conversation
) The daily health check now reports, per harness and moment kind, Cue's unavailable, timeout and lost counts and its round trip, the server's own time, over_budget cuts and unsettled served replies. It compares Agent Observer's tool calls with the serving log's pre-tool rows, read-only, and counts an id-bearing over_budget row as a server cut, never as lost. A rate files only when it got worse than the previous window. Adds the systemd templates for Linux machines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
What: Independent Sol review round 1 requests changes on Required fixesR1 [P2] Freshness is not checked for every judged sessionLocation:
Reproduced through the real health subprocess with fixture databases: moments at This can put unqualified events into both the denominator and gap, create a false R2 [P2] An exact five-percentage-point rise can fail to fileLocation:
Compare the count ratios without losing the inclusive boundary, and test exactly five points as well as a rise just below it. The existing test exercises approximately six points, so it passes with this defect. R3 [P2] Budget cuts lose their moment-kind dimensionLocation: The design section 3 explicitly requires Count and render by R4 [P2] The unavailable Cue rate has no reported trendLocation: The design requires the previous-window comparison for the filed rates, including Render that previous rate independently of the available-moment denominators. Add an outage trend case, including an all-unavailable previous window: its unavailable denominator is valid even when timeout/lost denominators are zero. Explain unavailable trend reasons as specified by design section 5. Verified and limits
Elon recordRequirements and who asked: The user requests an independent exact-head review against the final #57 design, with contract proof, a findings comment and |
|
Independent review of Checked
Not blocking
Elon record
|
…ind, unavailable trend - Every judged session must have Observer sources imported at or after its served activity in the window, fully read; a session with no source, or one whose moments only bracket the window, is no longer judged unchecked. - The regression rule compares exact fractions, so a rise of exactly five points files. - over_budget cuts are counted per harness and kind, with an unknown-kind bucket for cuts before parsing. - The unavailable Cue rate shows its previous-window rate, with a reason when there is none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
What: All four findings from Sol's round 1 review are fixed in head
Proof at
|
|
What: Independent Sol review round 2 passes exact head Round 1 findings
Independent proof and scope
Elon recordRequirements and who asked: The user requests an independent round 2 review at the exact SHA, closure of R1-R4, failing-before evidence, regression checks, a findings comment and |
|
What: Round 2 review passes FindingsNo required fixes. R1 closed
Checked at the health seam, and with probes that are not part of the PR: an import exactly at the last moment is judged; one second earlier is not; a bracketing session imported exactly at the shifted window end is judged; a bracketing session with no source is not judged. R2 closed
R3 closed
R4 closedEach Cue line prints Proof
Not blocking
Elon record
|
What: The daily health check now reports Cue speed, timeouts, and lost and unavailable Cue moments per harness and moment kind. It also reports the server's own time, budget cuts, unsettled replies, and tool calls Agent Observer saw that have no pre-tool row. It files a rate only when the rate got worse.
Why: Until now the check counted only moment statuses and outcomes, so Cue regressions and calls that never reached the server were found by hand (toolboxmd/chromeria#248).
So what: Sol and Grok review this head now. It merges only after toolboxmd/chromeria#251, including #252's ids on
over_budgetrows, merges. Deploying to rocky, cavallo and bestia is the planner's later step, with a backup, a runbook entry and a rollback.Closes #57.
Design and reviews
The design went through three roast rounds before this build. The files are local on rocky and not published, because they hold machine paths and usernames:
~/scratch/kw-resolver-test/design/DESIGN-cue-measurement.md. Earlier versions are beside it:-v1.md,-v2.md,-v3.md.ROAST-cue-measurement-sol.md,ROAST-cue-measurement-grok.mdROAST2-cue-measurement-sol.md,ROAST2-cue-measurement-grok.mdROAST3-cue-measurement-sol.md(build),ROAST3-cue-measurement-grok.md. Its blocker is closed by matchingover_budgetrows by id.Every finding's disposition is in the design's sections 8 and 10.
Change
contextmd/serve/measures.py(new), read-only:okrows.server_ms, or fromserver_secondsif contextmd serving log: moments.server_ms holds seconds, not milliseconds chromeria#250 renames it.over_budgetcounts, plus served (200) deliveries the hook never settled.exec-results) with pre-tool rows. Only calls inside the session's first-to-last moment count, and only in sessions whose every Observer source is imported past the session's last moment with no unread tail. A call whose pre-tool row isover_budgetcounts as a server cut, never as lost. Excluded calls are counted and printed. An unreadable ledger fails the run, so the window is not stored.health.py: owners forcue/timeoutandcue/lost(chromeria) and forcue/unavailableandmoment/unlogged(contextmd). Issue rows can now say their unit and the previous window's rate.__main__.py: adds--observer-db, with the same default asreport.deploy/health/: systemd user service and timer templates for Linux, and Linux install, proof and rollback steps. Rollback stops the timer and any running service, and never touchescontextmd-cue.service.report.py: corrects the stale docstring about Observer's CLI. README documents the new measures. Version 0.5.7 -> 0.6.0 (minor: new capability).The hook is unchanged. A hook failure log, design Appendix A, follows only if tool calls with no pre-tool row, excluding server cuts, stay above 2% for a week after #251 deploys, as #57 now says.
Proof (head 5ae62ff; review round 1 fixes below)
healthseam. The 11 new cases intests/test_health.py(CueHealthTest) were written before the code. They and the 16 existing health cases failed first, because the--observer-dboption the fixture now passes did not exist yet. Their fixtures include:okrows, timeouts,lostand unavailable rows;over_budgetpre-tool row, plus an id-less one;exec-ids;over_budgetas lost, skipping the freshness check, and filing every over-the-bar rate. The last one first passed undetected, so I raised that test's counts above the floor.HealthTestcases are unchanged. Only the shared fixture changed: it adds the Cue columns, adeliveriestable, and an empty Observer ledger passed as--observer-db, so the tests never read this machine's ledger.python3 -m unittest discover -s tests -t .gave 130 tests: OK, 18 skipped.tests/test_contract.py, which needsCONTEXTMD_CHROMERIA. Not run. Chromeria's main checkout has nonode_modules, so its dev server exits, and I did not install dependencies into a shared checkout. This PR changes neither the hook nor the server path that suite covers.versionctl release-check: release candidate OK, v0.6.0 at 5ae62ff.healthseam; 5 new cases. At 5ae62ff,CONTEXTMD_CHROMERIA=<disposable clone of chromeria at 2cad774, #251 with #252's ids, locked deps> python3 -m unittest discover -s tests -t .gave 135 tests: OK, 0 skipped.Read-only dry run, rocky's real log and ledger
Command:
bin/contextmd-serve health --base ~/.t3 --dry-run, 2026-10-13 18:45Z, 1.4 s, exit 0. The stored window did not move. #251 is not deployed yet, soserver cutis 0 everywhere.The new classes file nothing here: there is no previous window with Cue statuses, and Observer's rates rose under 5 points.
candidate/hiddenis an existing class.Limits (design section 6)
catchCause204, missing hook config, or a transfer stall all look the same. A stale heartbeat separates only "server down".Website impact: none. This is an internal health check with no public site or documentation page.
Elon record
over_budgetrows matched by id as server cuts (chromeria#252 folded into #251).over_budgetbranch at 65552bf.🤖 Generated with Claude Code