Skip to content

Verify: hook capture loses or merges Observations #466

Closed

Description

@possiblyneal

Part of #448

Question

Do hooks exit before the observe request is sent, do repeated prompts collide on id, and does import-jsonl mis-time or double-import? Is the 'recover captures while the daemon is down' gap real for the Operator? Upstream: rohitg00#1456, rohitg00#1468, rohitg00#1526, rohitg00#1436, rohitg00#1437, rohitg00#1438.

Activity

  1. self-assigned this
    on Nov 18, 2026
  2. possiblyneal commented on Nov 18, 2026

    @possiblyneal
    OwnerAuthor

    Verdict #466: hook capture loses or merges Observations

    Worktree: tmp/wt/verify-466 (branch verify/466-hooks), base fdc3d45. Nothing committed.
    Hooks were not changed and build was not run; a hook change reaches plugin/scripts only through npm run build.

    Claims

    rohitg00#1456 (hooks exit before the request is sent) - REJECT

    • Repro attempt: tmp/verify/466/hook-exit.mjs spawns src/hooks/post-tool-use.ts via tsx against a stub server (HOME under the worktree). A fast server, a blocked-loop server and a slow-response server (reply at 647 ms) each received the full 5270-byte body; the hook exited about 500 ms after fetch(), code 0. The 500 ms cap cuts only the wait for the response.
    • Residual risk: a process stall over 500 ms between fetch() and the socket write. Not reproduced.
    • Fix: none. Operator reach: see gap measurement (cause unattributed).
    • Class 2: no.
    • Retry if: the gap measurement shows loss with the daemon up and attributes it to hook exit, or a stall repro appears.

    rohitg00#1468 (repeated prompts collide) - REJECT (by design)

    • Observation ids are random (generateId). The 5-minute DedupMap dropping an identical prompt_submit is deliberate and pinned by test/observe-dedup-prompt.test.ts ("still dedups an identical prompt_submit within the TTL window").
    • Fix: none. Class 2: no.
    • Retry if: an eval shows repeated prompts bear on Recall.

    rohitg00#1526 (import-jsonl timestamps and duplicates) - APPLY (timestamp half), duplicate half already fixed

    • Repro: test/jsonl-timestampless.test.ts, 2 tests, fail 3/3 on base, pass with fix. A line without timestamp took wall-clock now, which dragged lastTs and session end time to the import moment.
    • Fix: src/replay/jsonl-parser.ts only advances first/last timestamps from real entry timestamps; a timestampless line inherits the last real one (fallback now only when none seen).
    • Re-import duplicates: already idempotent (fingerprint ids from position:line, fresh filtered by stored rows; test/replay-reimport.test.ts).
    • Operator reach: nil. 0 sessions in the live copy are tagged jsonl-import.
    • Class 2: yes in the sense that it changes stored import timestamps and session times (storage content); run eval:suite before shipping, though replay-based evals build through this parser.
    • Regression: test/replay*.test.ts and test/import-jsonl-result.test.ts pass (63 tests); typecheck 0 errors.

    rohitg00#1436 / rohitg00#1437 / rohitg00#1438 (durable capture, acceptance semantics, identity across restart) - NOT APPLIED (by design)

    • No hook spool exists by design: CLAUDE.md "never on the critical path". ObserveRetryQueue is in memory and full-disk only; the dedup map is in memory. api::observe already answers 201 stored or 202 queued. docs/upstream-issue-triage.yaml marks them by-design.
    • Fix: none. Class 2: no.
    • Retry if: a durable hook spool is adopted, or the gap below is attributed to daemon-down windows.

    Operator reach: capture gap measurement (upper bound)

    Read-only on tmp/verify/live-copy.sqlite vs ~/.claude/projects transcripts (counts only). Window: from the first stored session (2026-08-21), transcripts under 4 MB, ending at least 1 h before the copy, at least 5 tool calls (scripts gaps.mjs, gaps2.mjs).

    • 430 transcripts qualify; 280 have Observations.
    • 150 have none: 140 with no session row, 11 with a session row but 0 Observations (overlap in counts due to separate runs).
    • Of the 150 (gaps2), 141 fall in 10-minute windows where other sessions did store Observations, so the daemon was up; only 9 sit in quiet windows. By month: Nov 139 (124 active/7 quiet on 131), Oct 17, Sep 2.
    • Observations-per-tool-call ratio, p10/p50/p90: 0.16/0.74/1.15 (n=280).
    • Missing sessions hold about 4514 of 19140 tool calls (about 24%).
    • Caveats: transcripts include sessions from non-hook sources (other tools, projects without the plugin, subagent-only files); files over 4 MB excluded. The cause of the 141 is not attributed, so this does not support fix(hooks): stop exiting before the observe request is sent rohitg00/agentmemory#1456 specifically. Cheapest next step: split the 150 by project directory to see whether the gap is concentrated in projects without the plugin.

    Capture gap, attributed (coordinator, gaps3–6.mjs)

    Of the 150 storeless transcripts (≥5 tool calls), 114 never loaded the plugin (eval sandboxes under devex/kelpie/tmp/441-popt-*, manatee/tmp/bench-pr420). Of the 36 that did:

    Bar

    • typecheck 0 errors; npm test 3,118 passed / 1 skipped; eval:gate and eval:gate:rules pass; eval:suite fast vs base-fdc3d459: every metric Δ 0.000 (15 of 15).
    • Branch verify/466-hooks, uncommitted, worktree tmp/wt/verify-466. Only src/replay/jsonl-parser.ts changes; no hook source changed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions