fix: readiness check, no-op graph publish, NUL-stranded runs, and an invalidated test

[BeLazy167]

Fixes from a production incident on 2026-09-03, ported to the open-source tree. All four are merged and deployed in the private tree (#284–#287); this brings the public repo in line.

What happened

The argus-db Postgres volume filled and Postgres crash-looped every ~2 seconds for three days:

PANIC: could not write to file "pg_logical/replorigin_checkpoint.tmp": No space left on device
LOG: checkpointer process was terminated by signal 6: Aborted

93% of the database was one table — graph._sync_log, the pgGraph CDC log, at 10.2M rows / 5.9 GB.

1. /readyz health check (backend/fly.toml)

/healthz returns a hardcoded {"status":"ok"} and never touches the store (internal/api/server.go:302). It was the only check wired into [checks], so the platform reported the app healthy for three days while every data request failed. The static Next.js shell served 200s on top, which is why it surfaced as "the site loads but nothing works".

/readyz already existed, pings the pool (server.go:308), and correctly returned 503 throughout. Nothing was watching it. Added as a second check; /healthz keeps liveness semantics.

2. Skip no-op graph publish swaps (internal/graph/fullindex.go, migration 087)

publishGraphGeneration deletes every code_node for a repo and re-inserts the staged snapshot in one transaction. That is deliberate — fullindex.go:553 calls it "the visibility boundary" — and this keeps it. What it need not do is run when the staged generation reproduces the graph already live. Measured over 22 days:

	Logged inserts	Live rows	Whole-graph rewrites
code_nodes	2,279,287	210,623	10.8
code_edges	2,841,573	264,365	10.7

Migration 087 adds graph_index_generations.content_hash; the swap is skipped when the staged payloads hash to the published generation. Bookkeeping still runs on both paths via the extracted finishGraphGenerationPublish.

Narrow by design: ConfirmGraphDefaultHead already short-circuits an unchanged head, so this only fires when the head moved but no indexed symbol changed. The skip also requires that nothing wrote to code_nodes/code_edges since that publish, because the PR indexer mutates both between full publishes.

3. NUL in suppression keys stranding runs forever (internal/pipeline/suppression.go)

suppressionKey joined path/line/body with NUL. Those are map keys on PipelineRun.SuppressedKeys, and persistState marshals the run into the jsonb pipeline_states.payload column. jsonb cannot represent a NUL:

failed to recover run
error: "persisting state: upserting pipeline_states:
        ERROR: unsupported Unicode escape sequence (SQLSTATE 22P05)"

The payload is how a run resumes, so an affected run could never persist and never recover — the sweep claimed it, failed identically, damped it 30 minutes, and repeated. Seven production runs stranded in synthesizing for up to 13 days.

It presented as a healthy system throughout: 8 ms recovery scans reporting error_present: false (correct — the runs were inside their backoff), and an empty pipeline_states.error column. The reason existed only in a log line emitted once per 30 minutes per run.

Separator moved to U+001F, which keeps the "no path or review body contains this" property and is representable in jsonb.

Confirmed in production: after deploying this fix, a stranded run moved synthesizing → posting on its own within seconds, with no intervention.

4. Mirror ownership test that migration 086 invalidated

TestMirrorWorkerDeleteKeepsMemoryOwnedByDuplicatePattern built its setup from two co-existing patterns sharing one memory_custom_id. Migration 086's unique index makes that impossible, so the test deleted the only owner and asserted a memory nothing owned would survive — it could never pass. A test left behind by a schema change, not a product defect.

Note this tree's CreatePattern raw-inserts, so it fails with duplicate key value violates unique constraint; where CreatePattern upserts, it silently returns the first row instead. Same dead end, different symptom — the comment covers both.

Now covers three branches of patternMirrorDeleteAuthorized:

Test	Rows	Branch
…OwnedByDuplicatePattern	two rows, memory_doc_id set, custom_id NULL	explicitlyOwned COALESCE
…OwnedByLivePattern	live row + superseded row's late delete	explicitlyOwned, sequential
…OwnedByLegacyDuplicatePattern (pre-existing)	two rows, both columns NULL	legacyOwner derivation

086's index is partial (WHERE memory_custom_id IS NOT NULL), so legacy memory_doc_id rows are unconstrained and the duplicate-owner case is still reachable — which is why the first branch is restored rather than dropped.

Note for self-hosters running pgGraph

The root cause of the disk exhaustion was not in this code. internal/store/graph_index.go:826 documents that the projection "is registered with sync_mode = 'manual'", and RebuildCodeGraphProjection correctly calls graph.build() after each index. But graph.sync_mode was never set on the database, so pgGraph used its default of trigger, installed 8 CDC triggers, and logged every graph write forever with no consumer draining it.

If you run Argus against Postgres with the pgGraph extension, set it explicitly:

ALTER DATABASE <db> SET graph.sync_mode TO manual;

Without it, graph._sync_log grows unbounded — pgGraph will not prune a log no consumer has read. Per its docs the prune floor is the minimum of the projection watermark and the lowest applied_sync_id across backend heartbeats, so with no consumer no floor is ever safe.

Verification

Full suite against the CI Postgres image (PG 19beta2 + pinned pgvector), go test -race -count=1 ./...:

2875 passed, 0 failed

All three ownership tests verified by mutation — stubbing patternMirrorDeleteAuthorized to always authorise fails all three. TestSuppressionKeyIsRepresentableInJSONB verified by reverting the separator to NUL. The separator guard rejects printable characters and whitespace controls alike (|, A, \n all rejected).

fly config validate passes.

[BeLazy167]

Added a third fix to this branch: NUL in suppression keys stranding pipeline runs forever (4d37f04).

suppressionKey joined path/line/body with NUL. Those are map keys on PipelineRun.SuppressedKeys, and persistState marshals the run into the jsonb pipeline_states.payload column. jsonb cannot represent a NUL, so Postgres rejects the whole upsert with SQLSTATE 22P05.

Because that payload is how a run resumes, an affected run can never persist and never recover — the sweep claims it, fails identically, damps it 30 minutes, and repeats forever. Six production runs sat stranded in synthesizing for up to 13 days, while the recovery sweep reported 8 ms scans with error_present: false and an empty pipeline_states.error column.

Separator moved to U+001F, which keeps the "no path or review body contains this" property and is representable in jsonb. Regression test verified to fail when the separator is reverted to NUL.

Full analysis in the corresponding private PR.