feat(health): measure Cue speed, timeouts and lost tool calls daily (#57)

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## [0.6.0] - 2026-10-13
+
+### Added
+
+- The daily health check reports Cue speed, timeouts, lost and unavailable Cue moments per harness and kind, server time, budget cuts, unsettled replies and tool calls with no pre-tool row from Agent Observer, and files a rate only when it got worse (#57).
+
 ## [0.5.7] - 2026-10-13
 
 ### Changed

README.md

@@ -205,6 +205,31 @@ repository that owns it. It reads only stored values, read-only:
 - **Heartbeats.** An expected harness with no moments reads
   `no calls, coverage unknown` while the heartbeat is fresh, and
   `server stale` when it is not.
+- **Cue, per harness and moment kind (#57).** For user messages, tool
+  results and tool failures with a Cue status: *unavailable*
+  (`not_called` or `connect_failed`) over all scored moments, then
+  *timeout* and *lost* over the available ones, and the p50/p90 round trip
+  of `ok` rows. Round trips leave out timeouts, so they understate the slow
+  tail.
+- **Server time and budget cuts.** p50/p90/max of the server's own time per
+  kind, `over_budget` moments per harness, and served (200) replies the hook
+  never settled.
+- **Tool calls with no pre-tool row.** Agent Observer's ledger
+  (`--observer-db`, default as for `report`), read-only, over the window two
+  hours back. It judges tool calls (Codex: completed shell execs) between
+  their session's first and last logged moment, in sessions whose every
+  Observer source was imported at or after that session's served activity in
+  the window (its last moment, or the window's end while it goes on) with no
+  unread tail. A session with no Observer source is not judged. A call whose pre-tool row is `over_budget` is a server cut;
+  a call with no pre-tool row is reported with its cause unknown. Calls it
+  cannot judge are counted, never counted as lost. An unreadable ledger
+  fails the run and keeps the stored window.
+
+A Cue or tool-call rate is marked `*` at 10 or more events over 2%. It files
+only when it got worse than the window of equal length just before: it
+crossed 2%, or rose 5 points. Each harness and kind is judged on its own.
+`cue/timeout` and `cue/lost` file in toolboxmd/chromeria, `cue/unavailable`
+and `moment/unlogged` here.
 
 Each Issue carries a `ContextMD-Class:` line. A later window adds a comment.
 Each post is identified by machine, class and window, so the same

VERSION

@@ -1 +1 @@
-0.5.7
+0.6.0

contextmd/serve/__main__.py

@@ -52,6 +52,8 @@ def main(argv=None) -> int:
     hp.add_argument("--config", type=Path, help="health configuration (default ~/.config/contextmd/health.json)")
     hp.add_argument("--state", type=Path, help="stored window (default ~/.local/state/contextmd/health.json)")
     hp.add_argument("--gh", default="gh", help="the GitHub CLI")
+    hp.add_argument("--observer-db", type=Path, help="Agent Observer's ledger, read-only"
+                    " (default: $AGENT_OBSERVER_DB, else Observer's default path)")
     hp.add_argument("--now", help=argparse.SUPPRESS)
     cp = sub.add_parser("card")
     cp.add_argument("--base", required=True, type=Path)
@@ -145,7 +147,7 @@ def main(argv=None) -> int:
 def _health(ns, base: Path) -> int:
     from datetime import datetime, timezone
 
-    from . import health, report
+    from . import health, measures, report
     try:
         now = report.parse_since(ns.now) if ns.now else datetime.now(timezone.utc).timestamp()
         start, end = report.parse_since(ns.start), report.parse_since(ns.end)
@@ -162,6 +164,16 @@ def _health(ns, base: Path) -> int:
     if not sliced:
         start, end = health.window(state, db, now), now
     result = health.measure(db, start, end, cfg["expected_harnesses"], now)
+    if not result.unreadable:
+        try:
+            current = measures.measure(db, ns.observer_db, start, end)
+            previous = measures.measure(db, ns.observer_db, 2 * start - end, start)
+        except measures.ObserverError as exc:
+            print(health.report(result, cfg["expected_harnesses"], cfg["machine"]))
+            print(f"contextmd-serve health: Agent Observer's ledger is unreadable: {exc}")
+            return 1
+        result.measures = measures.render(current, previous)
+        result.findings += [health.Finding(cls, rows) for cls, rows in measures.findings(current, previous)]
     print(health.report(result, cfg["expected_harnesses"], cfg["machine"]))
     try:
         actions = health.file(result, health.GitHub(ns.gh, ns.dry_run), cfg["machine"], cfg["seed"])

contextmd/serve/health.py

@@ -1,7 +1,8 @@
 """The daily serving health check (#40): rate one machine's serving log and file defects at their owners.
 
 It reads only stored values, read-only: ``moments.status`` per harness, ``candidates.outcome`` per
-harness, and ``heartbeats.at``. No model, no transcript text.
+harness, and ``heartbeats.at``. No model, no transcript text. Cue speed, server time and tool calls
+with no pre-tool row (#57) are ``measures``; their rows file through the same Issues.
 
 - **Window:** from the end of the last successful run to now (first run: the last 24 hours). The end
   is stored only after a successful run, so a spike repaired between runs is still seen. An
@@ -47,6 +48,7 @@
     **{f"moment/{s}": CONTEXTMD for s in ("bad_event", "bad_request", "oversize", "unauthorized", "bad_config")},
     **{f"moment/{s}": CHROMERIA for s in ("unjoined", "ambiguous", "unmapped")},
     **{f"candidate/{o}": CHROMERIA for o in FILED_OUTCOMES},
+    "cue/timeout": CHROMERIA, "cue/lost": CHROMERIA, "cue/unavailable": CONTEXTMD, "moment/unlogged": CONTEXTMD,
 }
 CONFIG = "~/.config/contextmd/health.json"
 STATE = "~/.local/state/contextmd/health.json"
@@ -71,6 +73,7 @@ class Result:
     harnesses: dict = field(default_factory=dict)
     findings: list[Finding] = field(default_factory=list)
     unreadable: str | None = None
+    measures: list[str] = field(default_factory=list)  # the Cue and lost-call lines (#57)
 
 
 def utc(ts: float) -> str:
@@ -143,6 +146,7 @@ def report(result: Result, expected: list[str], machine: str) -> str:
         if h["outcomes"]:
             parts.append("outcomes " + ", ".join(f"{o} {n}" for o, n in sorted(h["outcomes"].items())))
         lines.append(". ".join(parts))
+    lines += result.measures
     if not result.findings:
         lines.append("Nothing to file.")
     return "\n".join(lines)
@@ -158,7 +162,10 @@ def section(finding: Finding, result: Result, machine: str) -> str:
         for r in finding.rows:
             rate = f"{r['rate']:.1%}" if "rate" in r else "n/a"
             kinds = ", ".join(f"{k} {n}" for k, n in sorted((r.get("kinds") or {}).items())) or "n/a"
-            count = (f"{r['count']} of {r['total']} moments" if finding.cls.startswith("moment/")
+            if "previous" in r:
+                kinds = f"previous window {r['previous']:.1%}"
+            count = (f"{r['count']} of {r['total']} {r['unit']}" if "unit" in r
+                     else f"{r['count']} of {r['total']} moments" if finding.cls.startswith("moment/")
                      else f"{r['count']} candidate rows, of {r['total']} moments")
             lines.append(f"| {r['harness']} | {count} | {rate} | {kinds} |")
         ids = [i for r in finding.rows for i in r["ids"]][:MAX_IDS]
@@ -178,8 +185,8 @@ def window_mark(finding: Finding, result: Result, machine: str) -> str:
 def body(finding: Finding, result: Result, machine: str) -> str:
     return "\n".join([
         f"**What:** The serving health check found `{finding.cls}` on {machine} over its threshold.",
-        "**Why:** The daily check (toolboxmd/contextmd#40) rates the serving log's stored statuses and outcomes;"
-        " this class is a defect this repository owns.",
+        "**Why:** The daily check (toolboxmd/contextmd#40, #57) rates the serving log's stored statuses, outcomes"
+        " and Cue results, and tool calls Agent Observer saw; this class is a defect this repository owns.",
         "**So what:** An agent finds the cause and fixes it here; the check comments with each later window"
         " over the threshold.",
         "",