Skip to content

feat(health): measure Cue speed, timeouts and lost tool calls daily (#57) - #61

Merged
lukemaj merged 2 commits into
mainfrom
feat/health-cue-latency
Oct 13, 2026
Merged

lukemaj merged 2 commits into
mainfrom
feat/health-cue-latency

Conversation

@lukemaj

@lukemaj lukemaj commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

What: The daily health check now reports Cue speed, timeouts, and lost and unavailable Cue moments per harness and moment kind. It also reports the server's own time, budget cuts, unsettled replies, and tool calls Agent Observer saw that have no pre-tool row. It files a rate only when the rate got worse.
Why: Until now the check counted only moment statuses and outcomes, so Cue regressions and calls that never reached the server were found by hand (toolboxmd/chromeria#248).
So what: Sol and Grok review this head now. It merges only after toolboxmd/chromeria#251, including #252's ids on over_budget rows, merges. Deploying to rocky, cavallo and bestia is the planner's later step, with a backup, a runbook entry and a rollback.

Closes #57.

Design and reviews

The design went through three roast rounds before this build. The files are local on rocky and not published, because they hold machine paths and usernames:

  • Design v3.1 (built): ~/scratch/kw-resolver-test/design/DESIGN-cue-measurement.md. Earlier versions are beside it: -v1.md, -v2.md, -v3.md.
  • Round 1: ROAST-cue-measurement-sol.md, ROAST-cue-measurement-grok.md
  • Round 2: ROAST2-cue-measurement-sol.md, ROAST2-cue-measurement-grok.md
  • Round 3: ROAST3-cue-measurement-sol.md (build), ROAST3-cue-measurement-grok.md. Its blocker is closed by matching over_budget rows by id.

Every finding's disposition is in the design's sections 8 and 10.

Change

  • contextmd/serve/measures.py (new), read-only:
    • Cue buckets per harness and kind. Unavailable is rated over all scored moments. Timeout, lost, ok and other are rated over available moments. Round trip is the nearest-rank p50/p90 over ok rows.
    • Server time. p50/p90/max, read as seconds from server_ms, or from server_seconds if contextmd serving log: moments.server_ms holds seconds, not milliseconds chromeria#250 renames it.
    • Budget cuts and replies. over_budget counts, plus served (200) deliveries the hook never settled.
    • Observer comparison. Over the health window shifted 2 h back, it compares tool calls (Codex: completed exec- results) with pre-tool rows. Only calls inside the session's first-to-last moment count, and only in sessions whose every Observer source is imported past the session's last moment with no unread tail. A call whose pre-tool row is over_budget counts as a server cut, never as lost. Excluded calls are counted and printed. An unreadable ledger fails the run, so the window is not stored.
    • Filing. Each harness and kind is judged separately, never pooled. A rate files only on a regression: at least 10 events, over 2%, and either the previous window was at or under 2% or the rate rose at least 5 points.
  • health.py: owners for cue/timeout and cue/lost (chromeria) and for cue/unavailable and moment/unlogged (contextmd). Issue rows can now say their unit and the previous window's rate.
  • __main__.py: adds --observer-db, with the same default as report.
  • deploy/health/: systemd user service and timer templates for Linux, and Linux install, proof and rollback steps. Rollback stops the timer and any running service, and never touches contextmd-cue.service.
  • report.py: corrects the stale docstring about Observer's CLI. README documents the new measures. Version 0.5.7 -> 0.6.0 (minor: new capability).

The hook is unchanged. A hook failure log, design Appendix A, follows only if tool calls with no pre-tool row, excluding server cuts, stay above 2% for a week after #251 deploys, as #57 now says.

Proof (head 5ae62ff; review round 1 fixes below)

  • Test first at the health seam. The 11 new cases in tests/test_health.py (CueHealthTest) were written before the code. They and the 16 existing health cases failed first, because the --observer-db option the fixture now passes did not exist yet. Their fixtures include:
    • slow ok rows, timeouts, lost and unavailable rows;
    • Observer calls with no pre-tool row;
    • an id-bearing over_budget pre-tool row, plus an id-less one;
    • no-moment, outside-span, stale-source and unread-tail sessions;
    • Codex exec- ids;
    • crossings, standing rates and 5-point rises, and one harness regressing while the pooled rate stays under 2%;
    • two consecutive runs.
  • Mutation checks. Each of these breaks one test: treating over_budget as lost, skipping the freshness check, and filing every over-the-bar rate. The last one first passed undetected, so I raised that test's counts above the floor.
  • Fixture changes. The existing HealthTest cases are unchanged. Only the shared fixture changed: it adds the Cue columns, a deliveries table, and an empty Observer ledger passed as --observer-db, so the tests never read this machine's ledger.
  • Full suite. python3 -m unittest discover -s tests -t . gave 130 tests: OK, 18 skipped.
    • The 18 skips are tests/test_contract.py, which needs CONTEXTMD_CHROMERIA. Not run. Chromeria's main checkout has no node_modules, so its dev server exits, and I did not install dependencies into a shared checkout. This PR changes neither the hook nor the server path that suite covers.
  • versionctl release-check: release candidate OK, v0.6.0 at 5ae62ff.
  • Review round 1 (Sol, 4 findings): each was fixed after a failing test was added at the health seam; 5 new cases. At 5ae62ff, CONTEXTMD_CHROMERIA=<disposable clone of chromeria at 2cad774, #251 with #252's ids, locked deps> python3 -m unittest discover -s tests -t . gave 135 tests: OK, 0 skipped.

Read-only dry run, rocky's real log and ledger

Command: bin/contextmd-serve health --base ~/.t3 --dry-run, 2026-10-13 18:45Z, 1.4 s, exit 0. The stored window did not move. #251 is not deployed yet, so server cut is 0 everywhere.

Serving health on rocky: 2026-10-13 13:50:49 UTC to 2026-10-13 18:45:46 UTC
Heartbeat: 1 s old
claude: 3396 moments. unjoined 15 (0.4%). outcomes below_threshold 7022, cut_budget 3, deduped 273, delivery_unverified 8, failed_by_output 3, holdout 76, injected 117, lifted 28, passed 223, suppressed 137, unsupported_key 399
codex: 2990 moments. unjoined 22 (0.7%). outcomes below_threshold 4170, deduped 51, failed_by_output 10, holdout 1, injected 20, lifted 9, passed 31, suppressed 9, unsupported_key 74
grok: 2776 moments. unjoined 47 (1.7%). outcomes below_threshold 6090, carried 10, deduped 71, deferred 2, delivery_unverified 1, failed_by_output 6, hidden 10, holdout 30, injected 10, truncated 3, unsupported_key 267
opencode: no calls, coverage unknown
Cue claude tool failure: scored 11, unavailable 2 (18.2%), available 9, timeout 4 (44.4%), lost 0 (0.0%), ok 5; round trip p50/p90 450/597 ms; previous unavailable
Cue claude tool result: scored 1166, unavailable 101 (8.7%) *, available 1065, timeout 157 (14.7%) *, lost 130 (12.2%) *, ok 778; round trip p50/p90 648/956 ms; previous unavailable
Cue claude user message: scored 90, unavailable 23 (25.6%) *, available 67, timeout 24 (35.8%) *, lost 0 (0.0%), ok 43; round trip p50/p90 414/623 ms; previous unavailable
Cue codex tool failure: scored 2, unavailable 1 (50.0%), available 1, timeout 1 (100.0%), lost 0 (0.0%), ok 0; round trip -; previous unavailable
Cue codex tool result: scored 702, unavailable 105 (15.0%) *, available 597, timeout 62 (10.4%) *, lost 79 (13.2%) *, ok 456; round trip p50/p90 507/963 ms; previous unavailable
Cue codex user message: scored 17, unavailable 3 (17.6%), available 14, timeout 1 (7.1%), lost 0 (0.0%), ok 13; round trip p50/p90 302/340 ms; previous unavailable
Cue grok tool failure: scored 33, unavailable 6 (18.2%), available 27, timeout 13 (48.1%) *, lost 0 (0.0%), ok 14; round trip p50/p90 548/636 ms; previous unavailable
Cue grok tool result: scored 946, unavailable 210 (22.2%) *, available 736, timeout 23 (3.1%) *, lost 51 (6.9%) *, ok 662; round trip p50/p90 533/1000 ms; previous unavailable
Cue grok user message: scored 20, unavailable 3 (15.0%), available 17, timeout 4 (23.5%), lost 0 (0.0%), ok 13; round trip p50/p90 301/334 ms; previous unavailable
Round trips are over ok rows only, so they understate the slow tail.
Server claude pre-tool: p50/p90/max 383/659/1397 ms
Server claude tool result: p50/p90/max 451/670/1321 ms
Server claude user message: p50/p90/max 480/874/1184 ms
Server codex pre-tool: p50/p90/max 362/607/1791 ms
Server codex tool result: p50/p90/max 390/637/1536 ms
Server codex user message: p50/p90/max 187/702/937 ms
Server grok pre-tool: p50/p90/max 399/692/2445 ms
Server grok tool result: p50/p90/max 369/596/2025 ms
Server grok user message: p50/p90/max 481/1053/1458 ms
Served replies not confirmed by the hook's settle: claude 5 of 91, codex 0 of 15, grok 1 of 16
Observer claude: judged 1061 tool calls, server cut 0, no pre-tool row 92 (8.7%) *; previous unlogged 5.9%
Observer codex: judged 1705 completed shell execs, server cut 0, no pre-tool row 178 (10.4%) *; previous unlogged 7.8%
Observer grok: judged 1743 tool calls, server cut 0, no pre-tool row 167 (9.6%) *; previous unlogged 4.7%
Not judged: 3 calls in sessions with no moments (coverage unknown), 7 outside their session's first-to-last moment, 1077 in 9 sessions not synced, 4 served sessions with no Observer source. over_budget rows without ids: 0
* at least 10 events and over 2%; filed only when it got worse than the previous window.
would file toolboxmd/chromeria candidate/hidden

The new classes file nothing here: there is no previous window with Cue statuses, and Observer's rates rose under 5 points. candidate/hidden is an existing class.

Limits (design section 6)

  • Cause unknown. It cannot tell why a call left no row: connect failure, an event-loop stall, the route's catchCause 204, missing hook config, or a transfer stall all look the same. A stale heartbeat separates only "server down".
  • Non-tool moments. User messages and other non-tool moments are not compared with Observer.
  • Silent sessions. Sessions where every hook failed show only as "coverage unknown".
  • Codex. Execs that never completed are absent.
  • Observer sync. Lost-call counts need an Observer sync. No machine runs one on a schedule yet (No machine runs a scheduled sync, so the ledger lags hours behind native records agent-observer#78), so sessions that were not synced are printed, not judged.
  • Trend. The previous-window comparison covers the filed rates, not server-time percentiles.

Website impact: none. This is an internal health check with no public site or documentation page.

Elon record

  • Requirements and who asked:
  • Deleted:
    • a hook failure log, deferred on evidence;
    • Observer sync inside health;
    • global freshness watermarks;
    • pooled rates;
    • a second cursor;
    • stored metrics history;
    • latency limits;
    • a dashboard.
  • Bottleneck: the cause of a call with no row. With #251 and #252, server cuts are logged with ids, so the remaining no-row count is measured daily, and it decides whether hook-side evidence is needed.
  • Checked myself:
    • test-first red, then green;
    • three mutations;
    • the full suite;
    • the dry run above against rocky's real data;
    • #251's over_budget branch at 65552bf.

🤖 Generated with Claude Code

)

The daily health check now reports, per harness and moment kind, Cue's
unavailable, timeout and lost counts and its round trip, the server's own
time, over_budget cuts and unsettled served replies. It compares Agent
Observer's tool calls with the serving log's pre-tool rows, read-only, and
counts an id-bearing over_budget row as a server cut, never as lost. A rate
files only when it got worse than the previous window. Adds the systemd
templates for Linux machines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lukemaj

lukemaj commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

What: Independent Sol review round 1 requests changes on 5405c36a2506c7961c1e08d7d382c48766ac9af2 (v0.6.0), with four findings.
Why: Freshness can be bypassed, an exact five-point regression can be missed, and two promised reporting dimensions are absent.
So what: Fix these findings and add their regression cases, then review the new head; merge remains blocked until toolboxmd/chromeria#251, including #252, merges.

Required fixes

R1 [P2] Freshness is not checked for every judged session

Location: contextmd/serve/measures.py:122-149,162-166.

spans includes the entire serving history, but last includes only sessions with moments inside the shifted window. Therefore a session whose logged moments bracket that window can have Observer calls judged even when its sources are stale or have unread tails: that session never enters the freshness loop. A missing source also bypasses the check because imported is not None is required before adding a session to stale.

Reproduced through the real health subprocess with fixture databases: moments at now - 27h and now - 1h, a tool call at now - 5h, and a source imported at now - 48h with read_offset=1,size_bytes=10. The shifted daily window is [now - 26h, now - 2h). Output was judged 1 tool calls ... no pre-tool row 1 (100.0%), with 0 in 0 sessions not synced. Separately, a session with in-window moments and an Observer event but no source was both judged and printed as a served session with no source.

This can put unqualified events into both the denominator and gap, create a false moment/unlogged regression, and advance the cursor past them. Establish freshness for every candidate judged session, and exclude sessions without enough source/moment evidence. Add cases with no in-window moments and with missing sources. The existing freshness test covers only sessions that have in-window moments and source rows.

R2 [P2] An exact five-percentage-point rise can fail to file

Location: contextmd/serve/measures.py:197-198.

0.15 - 0.10 is slightly below 0.05 in binary floating point. Thus regressed((15,100,[]), (10,100,[])) returns False, despite meeting all three design requirements. Reproduced at the health seam with 10 lost / 100 available in the previous window and 15 / 100 now: the current row prints lost 15 (15.0%) *; ... previous ... lost 10.0%, but the report says Nothing to file and the fake forge receives zero writes.

Compare the count ratios without losing the inclusive boundary, and test exactly five points as well as a rise just below it. The existing test exercises approximately six points, so it passes with this defect.

R3 [P2] Budget cuts lose their moment-kind dimension

Location: contextmd/serve/measures.py:111-112,253-254.

The design section 3 explicitly requires over_budget counts per harness and kind, but the query selects only harness and the Counter pools kinds. One Claude pre-tool cut and one Claude tool-result cut print only Over budget claude: 2. This prevents the daily report from distinguishing which moment path is being cut, even after #252 provides kind.

Count and render by (harness, kind), retaining an explicit unknown-kind bucket for cuts before parsing, and add a multi-kind test. Matching an id-bearing pre-tool budget cut as a server cut is correct and should remain so.

R4 [P2] The unavailable Cue rate has no reported trend

Location: contextmd/serve/measures.py:245-248.

The design requires the previous-window comparison for the filed rates, including cue/unavailable, but render requests only cue/timeout and cue/lost. Reproduced with unavailable rising from 1 / 100 to 15 / 100: filing correctly produces cue/unavailable, while the daily row reports only previous timeout 0.0%, lost 0.0% and omits the previous unavailable rate. A standing outage also has no unavailable comparison in the daily output.

Render that previous rate independently of the available-moment denominators. Add an outage trend case, including an all-unavailable previous window: its unavailable denominator is valid even when timeout/lost denominators are zero. Explain unavailable trend reasons as specified by design section 5.

Verified and limits

  • Read the final local design, Health check: measure Cue latency, timeouts and lost moments every day #57, the full PR body, the complete diff, surrounding health/Observer code, and #251/#252's live contract. No private design contents, session text, usernames or source paths are reproduced here. The intentionally cited local design paths are not a finding. No secrets or transcript data found in the diff.
  • Cue buckets and available/scored denominators, nearest-rank ok round trips, seconds-to-ms server durations under either column name, unsettled 200s, Codex completed exec- result matching, and matched pre-tool over_budget exclusion from the no-row gap are consistent with the design in the exercised cases.
  • Observer open/schema/query failures return before filing or saving the cursor. New classes have the specified owners. The 10-event floor, strict over-2% star and regression-only filing work apart from R2.
  • Independently reran the three claimed mutations in the disposable ContextMD clone. Counting matched budget cuts as gap, bypassing freshness, and filing all standing rates each failed the relevant existing test. Those tests do not cover R1-R4's reproduced cases.
  • The systemd service/timer templates are appropriately scoped to the health job, use the supplied absolute executable paths, and schedule a persistent daily run. Rollback stops both timer and active service before restoring state. No deployment or live systemd activation was performed. The three-machine reports required by Health check: measure Cue latency, timeouts and lost moments every day #57 remain the explicitly deferred deployment proof.
  • ContextMD exact head: 5405c36a2506c7961c1e08d7d382c48766ac9af2; comparison base: 27d21ed2dad97ce62f93ee0a4ec9362926f394f6.
  • Chromeria contract candidate: 2cad7749d59e2d917a40547087f03c60ebafcbc4, #251 still OPEN at review time. Its known identity is passed to logUnserved on budget expiry, including kind, session and call id after selection. Installed its locked dependencies only in the disposable /tmp clone. CONTEXTMD_CHROMERIA=<temporary clone> python3 -m unittest tests.test_contract: 18 passed.
  • CONTEXTMD_CHROMERIA=<temporary clone> python3 -m unittest discover -s tests -t .: 130 passed, zero skips, on the exact ContextMD head. The initial run without the dependency had 18 skips.
  • No main checkout was modified, no live serving log was opened, and Chromeria was not restarted. Contract tests launched and stopped only their own isolated dev server and temporary databases.

Elon record

Requirements and who asked: The user requests an independent exact-head review against the final #57 design, with contract proof, a findings comment and review/independent-sol status.
Deleted: No implementation changes, deployment, restart, hook-side instrumentation or new measurement store.
Bottleneck: Whether the reported and filed rates use qualified populations and preserve the specified thresholds.
Checked myself: Exact-head code and dependency contract, fixture subprocess reproductions of all four findings, full Python suite, all 18 real hook/server contract tests, and three mutation checks.

@lukemaj

lukemaj commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Independent review of 5405c36a2506c7961c1e08d7d382c48766ac9af2 (v0.6.0). No blocking findings. Do not merge until toolboxmd/chromeria#251 is merged with #252's ids on over_budget rows. That gate is already in the PR body, and the code matches the contract.

Checked

  • Design v3.1, Health check: measure Cue latency, timeouts and lost moments every day #57, and the diff. Cue buckets are exclusive per harness and kind. Unavailable is not_called or connect_failed over scored moments. Timeout, lost, ok, and other are over available moments. Round trip is nearest-rank p50/p90 of cue_ms on ok rows only. server_ms and server_seconds are both read as seconds. The live log's server_ms is seconds (about 0.007 to 2.4).
  • Codex: judged calls are tool_result rows whose id starts with exec-. call_ ids are ignored. Live pre_tool ids for Codex are exec- (3,569) plus 2 call_ rows. An id-bearing over_budget pre-tool row is a server cut and is not in the gap or in moment/unlogged. An id-less over_budget row is only printed beside the gap. #251 is not deployed, so the live cut count is 0.
  • Served span is the session's first-to-last moment, endpoints included. Freshness requires every Observer source for that session to be imported at or after the session's last moment in the shifted window, with no unread tail. Excluded calls are printed and are not counted as lost. An Observer open or schema error exits 1 before save_window.
  • Filing is per harness and kind, never pooled. A rate files only at 10 or more events, over 2%, and a previous rate at or under 2% or a rise of at least 5 points. Day one files nothing. cue/timeout and cue/lost go to chromeria. cue/unavailable and moment/unlogged stay here. over_budget status routing stays with health: file over_budget moments to Chromeria #60, as the design says.
  • Tests: python3 -m unittest tests.test_health at this SHA, 27 tests, OK. Breaking each rule in a /tmp copy failed one test: counting over_budget as lost, skipping freshness, and filing every over-the-bar rate. A read-only 6 hour measure on this machine's log and ledger returned sane rates and filed nothing, which matches the rule (the unlogged rises were under 5 points, and there is no previous Cue window).
  • systemd user service and timer match the launchd schedule (06:00, oneshot). Rollback stops the timer and the service and never names contextmd-cue.service. Install fills absolute paths, then enables only the timer. No secrets or session text in the diff. The local design paths in the PR body are intentional.

Not blocking

  • I did not run tests/test_contract.py. This PR does not change the hook or the server path that suite covers. Chromeria's main checkout still has no node_modules.
  • The README says a source must be imported "after" the session's last moment. The code and the design treat an equal timestamp as fresh.
  • The printed Cue line shows the previous timeout and lost rates, not the previous unavailable rate. Filing still compares unavailable, and a filed issue includes that previous rate.

Elon record

  • Requirements and who asked: the user, this review round: code matches the design, counts and filing rules hold, tests fail on the broken rules, systemd install and rollback are sound, no private data.
  • Deleted: nothing. Re-running the 18 contract tests was not needed to judge this diff.
  • Bottleneck: whether an over_budget row is counted as lost. It is not, once it has an id.
  • Checked myself: the diff, the health tests, the three mutations, and a read-only measure of the live log.

…ind, unavailable trend

- Every judged session must have Observer sources imported at or after its
  served activity in the window, fully read; a session with no source, or
  one whose moments only bracket the window, is no longer judged unchecked.
- The regression rule compares exact fractions, so a rise of exactly five
  points files.
- over_budget cuts are counted per harness and kind, with an unknown-kind
  bucket for cuts before parsing.
- The unavailable Cue rate shows its previous-window rate, with a reason when
  there is none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lukemaj

lukemaj commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

What: All four findings from Sol's round 1 review are fixed in head 5ae62ffc63c4d0a28a908b8139efbbc684b16548. Each fix has a test at the health seam that failed on 5405c36 and passes now.
Why: The review found a freshness bypass, a missed exact five-point rise, budget cuts pooled across kinds, and no trend for the unavailable rate.
So what: Sol and Grok review the new head. Merge still waits for toolboxmd/chromeria#251.

Finding Fix Test (failed before)
R1 freshness bypass Every judged session is now checked, not only sessions with moments inside Observer's window. All of its sources must be imported at or after min(last moment, window end) with no unread tail. A session with no Observer source is not judged. test_a_session_bracketing_the_window_still_needs_fresh_sources, test_a_session_with_no_observer_source_is_not_judged. Before the fix, both judged the call (judged 1 ... no pre-tool row 1).
R2 exact five points BAR and RISE are exact Fractions, and rates are compared as fractions. test_exactly_five_points_up_files_and_just_under_does_not: 10 → 15 of 100 files, 10 → 14 does not. Before the fix, 10 → 15 filed nothing.
R3 cuts per kind over_budget is counted per harness and kind, with an unknown kind bucket for cuts made before parsing. Id-bearing pre-tool cuts still match as server cuts. test_budget_cuts_are_counted_per_kind: Over budget claude: pre-tool 1, tool result 1, unknown kind 1.
R4 unavailable trend Each Cue row prints previous: unavailable x%, timeout y%, lost z%. A rate with no moments in the previous window prints -, and a row with none at all prints previous unavailable: <reason>. test_the_unavailable_rate_has_a_trend: an all-unavailable previous window prints unavailable 100.0%, timeout -, lost -; 1% → 15% prints the trend and files cue/unavailable.

Proof at 5ae62ff:

  • CONTEXTMD_CHROMERIA=<disposable clone at chromeria 2cad774, with pnpm's locked dependencies> python3 -m unittest discover -s tests -t . gave 135 tests: OK, 0 skipped. The 18 contract tests ran.
  • The read-only dry run on rocky's real log exits 0. Under the stricter freshness rule it judges fewer Claude calls: 794, against 1,061 before. The new Not judged line reads: 1344 in 10 sessions not synced, 10 served sessions with no Observer source. rocky's ledger was last synced hours ago, and agent-observer#78 tracks that.
  • versionctl release-check: v0.6.0 OK at 5ae62ff.

@lukemaj

lukemaj commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

What: Independent Sol review round 2 passes exact head 5ae62ffc63c4d0a28a908b8139efbbc684b16548; R1-R4 are closed and no blocking regression was found.
Why: The five new regression tests expose the original defects on 5405c36a2506c7961c1e08d7d382c48766ac9af2, and the corrected candidate passes the complete suite.
So what: Merge only after toolboxmd/chromeria#251 merges with the required budget-cut identity contract; recheck the integration if that contract changes. Deployment and the three-machine reports remain the planner's follow-up.

Round 1 findings

  • R1 closed: Every candidate interior Observer call now goes through freshness qualification. All sources must be fully read and their minimum import timestamp must reach min(session last moment, shifted window end). Missing sources exclude the session. Both new cases, test_a_session_bracketing_the_window_still_needs_fresh_sources and test_a_session_with_no_observer_source_is_not_judged, fail on the old code because it judges the call and reports a 100% gap. Both pass now. Independently exercised the positive bracketing case: an import exactly at the shifted window end qualifies; one second earlier is excluded. The stronger check can conservatively exclude ongoing sessions, as documented.
  • R2 closed: The bar and rise comparisons use exact fractions. test_exactly_five_points_up_files_and_just_under_does_not fails on the old code because the exact 10% to 15% rise files no issue, producing the expected unpacking error from an empty result. It passes now, including the non-filing 10% to 14% case. Additional checks confirmed a 14.99% current rate against 10% does not qualify, exactly 2% does not qualify, and a zero previous denominator does not qualify.
  • R3 closed: Budget cuts are counted by harness and kind, with an explicit unknown-kind bucket. test_budget_cuts_are_counted_per_kind fails on the old pooled Over budget claude: 3 output and passes now. The existing test still verifies that an id-bearing pre-tool cut is a server cut, excluded from the no-row gap, while an id-less cut is reported separately.
  • R4 closed: Previous unavailable, timeout and lost rates are rendered independently. test_the_unavailable_rate_has_a_trend fails on the old output and passes now, including the all-unavailable previous window and the 1% to 15% unavailable regression. Additional fixture subprocess checks verified both missing-trend reasons: no Cue statuses, and no corresponding harness/kind population in an otherwise populated previous window.

Independent proof and scope

  • Read the design in full, Health check: measure Cue latency, timeouts and lost moments every day #57, the complete PR body, all three existing comments including the worker's reply, and review/inline-comment records (none). Read the complete gh pr diff and exact git show changes, surrounding health/Observer code, and Chromeria's budget-cut identity implementation.
  • In an isolated /tmp baseline worktree, copied only the candidate's test file onto 5405c36 and ran the five new cases: 4 assertion failures and 1 error, each caused by its reported defect rather than setup. Candidate python3 -m unittest -v tests.test_health: 32 passed.
  • Installed locked Chromeria dependencies only in a disposable /tmp clone at 2cad7749d59e2d917a40547087f03c60ebafcbc4. Ran CONTEXTMD_CHROMERIA=<that clone> python3 -m unittest discover -s tests -t . at the exact ContextMD head: 135 passed in 42.626 s, zero skipped, including all 18 contract tests. git diff --check passed.
  • The inspected behavior preserves exclusive Cue buckets, scored/available denominators, nearest-rank ok-only percentiles, seconds-to-ms conversion, unsettled-200 reporting, Observer exclusions and Codex completed-exec matching, regression-only filing and class ownership. Observer failures still return before filing or saving the cursor. No new blocking finding in the Linux templates or rollback instructions.
  • Live PR head remained 5ae62ff; Chromeria #251 was still OPEN at 2cad774 when checked. This review does not approve a future changed identity contract. Before merge, verify that the merged dependency still supplies kind, session and tool_use_id for parsed pre-tool budget cuts, and rerun affected integration proof if it changes.
  • No main checkout was written, no live serving log was opened, no live dry run was repeated, and no live Chromeria process was restarted. Contract tests started and stopped only their own isolated servers and temporary databases. Deployment proof is not claimed.

Elon record

Requirements and who asked: The user requests an independent round 2 review at the exact SHA, closure of R1-R4, failing-before evidence, regression checks, a findings comment and review/independent-sol status.
Deleted: Implementation changes, live-system mutations, deployment, restart and additional instrumentation are outside this review.
Bottleneck: Whether the four fixes qualify the measured populations, preserve the inclusive threshold and expose the promised reporting dimensions.
Checked myself: Exact-head and baseline code, all five failing-before cases, positive freshness and threshold boundaries, trend reasons, the complete 135-test suite and the current dependency identity contract.

@lukemaj

lukemaj commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

What: Round 2 review passes 5ae62ffc63c4d0a28a908b8139efbbc684b16548. Sol's four findings are closed, and the new tests fail on 5405c36.
Why: Freshness now covers every judged session, the five-point rise is an exact fraction, budget cuts keep their kind, and the unavailable rate shows its previous window.
So what: Nothing here blocks the review. Merge still waits for toolboxmd/chromeria#251, including #252's ids on over_budget rows.

Findings

No required fixes.

R1 closed

synced runs for every call inside the session span. A session is judged only when every Observer source is imported at or after min(last moment, shifted window end), with no unread tail. min(imported_at) and max(unread tail) still combine every source. A missing source is not judged.

Checked at the health seam, and with probes that are not part of the PR: an import exactly at the last moment is judged; one second earlier is not; a bracketing session imported exactly at the shifted window end is judged; a bracketing session with no source is not judged.

R2 closed

BAR, RISE, and both rates are Fraction. 15/100 against 10/100 files. 14/100 does not. 13/100 against 8/100 files. 2/100 is not over the bar. A previous rate of exactly 2/100 still counts as at or under the bar.

R3 closed

over_budget is counted per harness and kind. A null kind prints unknown kind, after the named kinds. An id-bearing pre-tool cut is still a server cut and stays out of the gap. The existing server-cut test still passes.

R4 closed

Each Cue line prints previous: unavailable x%, timeout y%, lost z%. A zero denominator prints -. When none of the three has a denominator, the line is previous unavailable: plus the reason (no Cue statuses in the previous window, or no Cue moments of this kind in the previous window).

Proof

  • Five new tests at 5ae62ff: pass. With only contextmd/serve/measures.py restored from 5405c36: all five fail. The bracket session and the sourceless session are judged, 15/100 files nothing, the cut line is Over budget claude: 3, and the all-unavailable row still says previous unavailable with no unavailable rate.
  • python3 -m unittest tests.test_health: 32 tests, OK.
  • Disposable Chromeria clone at 2cad7749d59e2d917a40547087f03c60ebafcbc4, dependencies already installed: python3 -m unittest discover -s tests -t . gave 135 tests, OK, none skipped. Those tests started only their own server. Chromeria was not restarted, and that clone stayed clean.
  • Read-only dry run of this tree rendered the new lines, including previous unavailable: no Cue statuses in the previous window and previous: unlogged .... The new classes filed nothing. The only finding was the pre-existing candidate/hidden. The health state file and the serving log kept the same mtime. The process then exited 1 in unchanged health.file because gh issue list returned empty stdout for that existing class. A direct gh issue list just after returned JSON. That is not a defect in this commit.

Not blocking

  • synced's docstring says "imported after". The code, the module docstring, and the README use at or after (>=). An equal timestamp is fresh, and a probe confirms it.
  • A session whose moments only bracket the window and that has no Observer source is not judged, but "served sessions with no Observer source" still counts only sessions with a moment inside the window. The call is in "not synced", not in the gap, so it cannot file moment/unlogged.

Elon record

  • Requirements and who asked: the user, this round: confirm R1 to R4 are closed at exact head 5ae62ff, that the new tests fail without the fixes, and that nothing regressed.
  • Deleted: no code changes, no live restart, no new store.
  • Bottleneck: whether freshness, the exact five-point boundary, per-kind cuts, and the unavailable trend hold on the path the tests run.
  • Checked myself: the round-2 diff, design sections 3 to 5, the five tests on both SHAs, equal-timestamp probes, the 135-test suite, and a read-only dry run.

@lukemaj
lukemaj merged commit 806cd73 into main Oct 13, 2026
2 checks passed
@lukemaj
lukemaj deleted the feat/health-cue-latency branch October 13, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Health check: measure Cue latency, timeouts and lost moments every day

1 participant