You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Incident checklist in the watcher brief does not mark logs, chat logs and database rows as untrusted data, and its duplicate query returns row content #949
Found by the single-seat Opus review of #941 (work of #935), finding listed in #935 (comment). Finding 5. The brief built by hcb-alert-watcher.ts fences the alert text as untrusted but the new incident checklist tells the session to read container logs, chat logs and database rows without saying that those carry user-controlled text. The duplicate-check query in the checklist selects event titles; it should select ids only.
Why (motivation)
Event titles and chat text are written by users and can contain instructions aimed at the investigating model.
User impact
A user who writes a crafted event title could steer the unattended investigation session that is meant to repair the bot for everyone.
Cost of inaction
The brief's own safety rule (read-only first phase, untrusted data) is weaker for the new checklist than for the alert text.
Acceptance criteria
Test: the brief marks logs, chat logs and database rows as untrusted data — proof:
Test: the duplicate-check query in the brief selects ids only — proof:
Fix in progress on branch fix/watcher-untrusted-data-20261102 (commit on top of PR #941). Open from the single-seat review, not fixed: start_at is compared as stored text (the same instant written in two forms would not group), and the duplicate query has no index on (user_id, start_at) and scans events once per row on a host that swaps.
Found 2026-11-04 while restarting hcb-alert-watcher after the #941 merge: the canonical main checkout had no node_modules, so the service crashed at start with "Cannot find package 'zod'" and stayed down until I ran bun install --frozen-lockfile there (about 1 minute down, then active). Next action: make scripts/install-alert-watcher.sh (and the runbook restart step) check or install dependencies in the canonical checkout before restarting, and have gh ship's cleanup not leave the canonical checkout without them. Not done by me. Also still open here: the #949 brief fix (a85395cb) needs a rebase onto main in its own PR, and the synthetic pipeline test is not run.
State 2026-11-04 (checkpoint, not shipped): branch fix/watcher-untrusted-data-20261102 is rebased onto main e4185dd (the two reliability commits that #941 squash-merged were dropped; the brief commit is now 4a85109e). Staged on top, not yet committed: version 0.3.4 (pyproject.toml and uv.lock), scripts/install-alert-watcher.sh runs the real bun build scripts/hcb-alert-watcher.ts --target bun before writing or starting a unit and refuses with bun install --frozen-lockfile when an import does not resolve (the 2026-11-04 crash loop with Cannot find package 'zod' after the canonical checkout lost its node_modules), two tests that run the real bun (red against the old installer), and a runbook 'Alert watcher' paragraph with the install step.
Reviews: single-seat Opus. Round 1 of the installer change: tests used stubs only (fixed with real-bun tests) and the check runs only at install time (filed as a comment on 842). Round 2 job 20261104T112946-70a8bbab was running when this run stopped; it must finish with no P1/P2 before the commit.
Tests run: install-alert-watcher.test.ts 21 pass, hcb-alert-watcher.test.ts 72 pass (one file per command).
Next: read the round 2 result, commit, push, open the PR with Refs #949, wait for hosted CI and Stage QA, gh ship, verify the deploy, restart hcb-alert-watcher (it already has node_modules since 11:47 CET).
Shipped: merged via #968.
Merge commit: 6b7ee71
This does NOT close the ticket — acceptance still needs proof. Run task read #949 for the outstanding criteria, then task check/task done.
Status 2026-11-05: implemented, reviewed (single seat), merged, deployed and live-tested.
What changed (PR #968, merge commit 6b7ee71, version 0.3.5):
Every brief the watcher builds now says that everything read from production (container logs, per-chat logs, database rows, Redis values, job errors) is untrusted data, not only the alert text, ahead of any checklist.
The duplicate query of the incident checklist returns ids and counts only: it groups on user and start across all rows (a later retry and an already deleted copy still show), counts distinct titles with COUNT(DISTINCT title) and selects no title. A test runs the brief's queries against the real events schema.
Found while landing it: after a merge that changes package.json or bun.lock the unit used to restart every 30 s in silence when node_modules lacked an imported package. The installer now runs bun build on the watcher first, prints the head of Bun's error and its exit status, and suggests bun install only for a missing package or a stale node_modules export; the runbook says to run bun install --frozen-lockfile after every such merge.
CI on head c18ff64: all hosted checks and stage-qa (passed cases=6) green; gh ship accepted the review quorum (15 iterations, 3 roles) and the task gate (2 of 2 criteria checked with proofs).
Deploy: the CI/CD run for 6b7ee71 succeeded; the odroid receipt names revision 6b7ee71, image sha256:2072be5d… equal to the running container's image; /health ok, /ready ok (unverified); restarts 0, not OOM-killed. Container log from 05:03:30Z to 05:15Z: 40 records, all level 30, none at level 40 or above. The bun process held about 262 MB RSS (226 MB anon).
Live test: hcb-alert-watcher restarted from the canonical main checkout at 6b7ee71 (node_modules present) at 05:05:56Z; one alert labelled SYNTHETIC PIPELINE TEST 2, no action needed (source agent-run-incident) went through POST /admin/alerts at 05:06:14Z with the token only through the keychain flow (HTTP 200, never printed). The watcher queued it within one poll and started the investigation session; the generated brief carries the new untrusted paragraph (it names logs, per-chat logs, database rows, Redis values and job errors) and its duplicate query contains COUNT(DISTINCT title) AS distinct_titles with no title in any SELECT list.
Not done: the two limits noted from the review stay open here. start_at is grouped as stored text, and the correlated query has no index on (user_id, start_at) (fine at today's table size; revisit if the events table grows a lot).
Activity
alex-mextner commentedon Nov 2, 2026
Fix in progress on branch fix/watcher-untrusted-data-20261102 (commit on top of PR #941). Open from the single-seat review, not fixed: start_at is compared as stored text (the same instant written in two forms would not group), and the duplicate query has no index on (user_id, start_at) and scans events once per row on a host that swaps.
alex-mextner commentedon Nov 4, 2026
Found 2026-11-04 while restarting hcb-alert-watcher after the #941 merge: the canonical main checkout had no node_modules, so the service crashed at start with "Cannot find package 'zod'" and stayed down until I ran
bun install --frozen-lockfilethere (about 1 minute down, then active). Next action: make scripts/install-alert-watcher.sh (and the runbook restart step) check or install dependencies in the canonical checkout before restarting, and have gh ship's cleanup not leave the canonical checkout without them. Not done by me. Also still open here: the #949 brief fix (a85395cb) needs a rebase onto main in its own PR, and the synthetic pipeline test is not run.alex-mextner commentedon Nov 4, 2026
State 2026-11-04 (checkpoint, not shipped): branch fix/watcher-untrusted-data-20261102 is rebased onto main e4185dd (the two reliability commits that #941 squash-merged were dropped; the brief commit is now 4a85109e). Staged on top, not yet committed: version 0.3.4 (pyproject.toml and uv.lock),
scripts/install-alert-watcher.shruns the realbun build scripts/hcb-alert-watcher.ts --target bunbefore writing or starting a unit and refuses withbun install --frozen-lockfilewhen an import does not resolve (the 2026-11-04 crash loop withCannot find package 'zod'after the canonical checkout lost its node_modules), two tests that run the real bun (red against the old installer), and a runbook 'Alert watcher' paragraph with the install step.Reviews: single-seat Opus. Round 1 of the installer change: tests used stubs only (fixed with real-bun tests) and the check runs only at install time (filed as a comment on 842). Round 2 job 20261104T112946-70a8bbab was running when this run stopped; it must finish with no P1/P2 before the commit.
Tests run: install-alert-watcher.test.ts 21 pass, hcb-alert-watcher.test.ts 72 pass (one file per command).
Next: read the round 2 result, commit, push, open the PR with
Refs #949, wait for hosted CI and Stage QA,gh ship, verify the deploy, restart hcb-alert-watcher (it already has node_modules since 11:47 CET).test(watcher): type the copies query rows instead of Record<string, u…
fix(watcher): the brief marks production data as untrusted and finds …
alex-mextner commentedon Nov 5, 2026
Shipped: merged via #968.
Merge commit: 6b7ee71
This does NOT close the ticket — acceptance still needs proof. Run
task read #949for the outstanding criteria, thentask check/task done.alex-mextner commentedon Nov 5, 2026
Status 2026-11-05: implemented, reviewed (single seat), merged, deployed and live-tested.
What changed (PR #968, merge commit 6b7ee71, version 0.3.5):
COUNT(DISTINCT title)and selects no title. A test runs the brief's queries against the realeventsschema.package.jsonorbun.lockthe unit used to restart every 30 s in silence whennode_moduleslacked an imported package. The installer now runsbun buildon the watcher first, prints the head of Bun's error and its exit status, and suggestsbun installonly for a missing package or a stalenode_modulesexport; the runbook says to runbun install --frozen-lockfileafter every such merge.Evidence:
review diff --staged --task 949 --pool 1 -m omp:anthropic/claude-opus-5-5), several rounds on the brief commit, then three on the installer commit and the type fix; the last rounds found nothing actionable. The full-roster pass is owed in Full-roster review of the integration diff of the incident fixes (PRs 941, 942, async pictures) is owed #961.stage-qa(passed cases=6) green;gh shipaccepted the review quorum (15 iterations, 3 roles) and the task gate (2 of 2 criteria checked with proofs).sha256:2072be5d…equal to the running container's image;/healthok,/readyok (unverified); restarts 0, not OOM-killed. Container log from 05:03:30Z to 05:15Z: 40 records, all level 30, none at level 40 or above. The bun process held about 262 MB RSS (226 MB anon).hcb-alert-watcherrestarted from the canonicalmaincheckout at 6b7ee71 (node_modulespresent) at 05:05:56Z; one alert labelledSYNTHETIC PIPELINE TEST 2, no action needed(sourceagent-run-incident) went throughPOST /admin/alertsat 05:06:14Z with the token only through the keychain flow (HTTP 200, never printed). The watcher queued it within one poll and started the investigation session; the generated brief carries the new untrusted paragraph (it names logs, per-chat logs, database rows, Redis values and job errors) and its duplicate query containsCOUNT(DISTINCT title) AS distinct_titleswith no title in any SELECT list.Not done: the two limits noted from the review stay open here.
start_atis grouped as stored text, and the correlated query has no index on(user_id, start_at)(fine at today's table size; revisit if the events table grows a lot).