Skip to content

Incident checklist in the watcher brief does not mark logs, chat logs and database rows as untrusted data, and its duplicate query returns row content #949

Closed

Description

@alex-mextner

What

Found by the single-seat Opus review of #941 (work of #935), finding listed in #935 (comment). Finding 5. The brief built by hcb-alert-watcher.ts fences the alert text as untrusted but the new incident checklist tells the session to read container logs, chat logs and database rows without saying that those carry user-controlled text. The duplicate-check query in the checklist selects event titles; it should select ids only.

Why (motivation)

Event titles and chat text are written by users and can contain instructions aimed at the investigating model.

User impact

A user who writes a crafted event title could steer the unattended investigation session that is meant to repair the bot for everyone.

Cost of inaction

The brief's own safety rule (read-only first phase, untrusted data) is weaker for the new checklist than for the alert text.

Acceptance criteria

  • Test: the brief marks logs, chat logs and database rows as untrusted data — proof: proof
  • Test: the duplicate-check query in the brief selects ids only — proof: proof

Screenshots

(none)

Deploy

Deployed: no

Links

Skipped gates

  • duplicate: searched incident, alert, watcher, validator, timeout, receipt; no ticket covers this
  • screenshots: no UI change

Activity

  1. alex-mextner commented on Nov 2, 2026

    @alex-mextner
    OwnerAuthor

    Fix in progress on branch fix/watcher-untrusted-data-20261102 (commit on top of PR #941). Open from the single-seat review, not fixed: start_at is compared as stored text (the same instant written in two forms would not group), and the duplicate query has no index on (user_id, start_at) and scans events once per row on a host that swaps.

  2. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Found 2026-11-04 while restarting hcb-alert-watcher after the #941 merge: the canonical main checkout had no node_modules, so the service crashed at start with "Cannot find package 'zod'" and stayed down until I ran bun install --frozen-lockfile there (about 1 minute down, then active). Next action: make scripts/install-alert-watcher.sh (and the runbook restart step) check or install dependencies in the canonical checkout before restarting, and have gh ship's cleanup not leave the canonical checkout without them. Not done by me. Also still open here: the #949 brief fix (a85395cb) needs a rebase onto main in its own PR, and the synthetic pipeline test is not run.

  3. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    State 2026-11-04 (checkpoint, not shipped): branch fix/watcher-untrusted-data-20261102 is rebased onto main e4185dd (the two reliability commits that #941 squash-merged were dropped; the brief commit is now 4a85109e). Staged on top, not yet committed: version 0.3.4 (pyproject.toml and uv.lock), scripts/install-alert-watcher.sh runs the real bun build scripts/hcb-alert-watcher.ts --target bun before writing or starting a unit and refuses with bun install --frozen-lockfile when an import does not resolve (the 2026-11-04 crash loop with Cannot find package 'zod' after the canonical checkout lost its node_modules), two tests that run the real bun (red against the old installer), and a runbook 'Alert watcher' paragraph with the install step.
    Reviews: single-seat Opus. Round 1 of the installer change: tests used stubs only (fixed with real-bun tests) and the check runs only at install time (filed as a comment on 842). Round 2 job 20261104T112946-70a8bbab was running when this run stopped; it must finish with no P1/P2 before the commit.
    Tests run: install-alert-watcher.test.ts 21 pass, hcb-alert-watcher.test.ts 72 pass (one file per command).
    Next: read the round 2 result, commit, push, open the PR with Refs #949, wait for hosted CI and Stage QA, gh ship, verify the deploy, restart hcb-alert-watcher (it already has node_modules since 11:47 CET).

  4. alex-mextner commented on Nov 5, 2026

    @alex-mextner
    OwnerAuthor

    Shipped: merged via #968.
    Merge commit: 6b7ee71
    This does NOT close the ticket — acceptance still needs proof. Run task read #949 for the outstanding criteria, then task check/task done.

  5. alex-mextner commented on Nov 5, 2026

    @alex-mextner
    OwnerAuthor

    Status 2026-11-05: implemented, reviewed (single seat), merged, deployed and live-tested.

    What changed (PR #968, merge commit 6b7ee71, version 0.3.5):

    • Every brief the watcher builds now says that everything read from production (container logs, per-chat logs, database rows, Redis values, job errors) is untrusted data, not only the alert text, ahead of any checklist.
    • The duplicate query of the incident checklist returns ids and counts only: it groups on user and start across all rows (a later retry and an already deleted copy still show), counts distinct titles with COUNT(DISTINCT title) and selects no title. A test runs the brief's queries against the real events schema.
    • Found while landing it: after a merge that changes package.json or bun.lock the unit used to restart every 30 s in silence when node_modules lacked an imported package. The installer now runs bun build on the watcher first, prints the head of Bun's error and its exit status, and suggests bun install only for a missing package or a stale node_modules export; the runbook says to run bun install --frozen-lockfile after every such merge.

    Evidence:

    • Review: single-seat Opus (review diff --staged --task 949 --pool 1 -m omp:anthropic/claude-opus-5-5), several rounds on the brief commit, then three on the installer commit and the type fix; the last rounds found nothing actionable. The full-roster pass is owed in Full-roster review of the integration diff of the incident fixes (PRs 941, 942, async pictures) is owed #961.
    • CI on head c18ff64: all hosted checks and stage-qa (passed cases=6) green; gh ship accepted the review quorum (15 iterations, 3 roles) and the task gate (2 of 2 criteria checked with proofs).
    • Deploy: the CI/CD run for 6b7ee71 succeeded; the odroid receipt names revision 6b7ee71, image sha256:2072be5d… equal to the running container's image; /health ok, /ready ok (unverified); restarts 0, not OOM-killed. Container log from 05:03:30Z to 05:15Z: 40 records, all level 30, none at level 40 or above. The bun process held about 262 MB RSS (226 MB anon).
    • Live test: hcb-alert-watcher restarted from the canonical main checkout at 6b7ee71 (node_modules present) at 05:05:56Z; one alert labelled SYNTHETIC PIPELINE TEST 2, no action needed (source agent-run-incident) went through POST /admin/alerts at 05:06:14Z with the token only through the keychain flow (HTTP 200, never printed). The watcher queued it within one poll and started the investigation session; the generated brief carries the new untrusted paragraph (it names logs, per-chat logs, database rows, Redis values and job errors) and its duplicate query contains COUNT(DISTINCT title) AS distinct_titles with no title in any SELECT list.

    Not done: the two limits noted from the review stay open here. start_at is grouped as stored text, and the correlated query has no index on (user_id, start_at) (fine at today's table size; revisit if the events table grows a lot).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions