[tnunamak]

What

When a startup lexical or semantic search-index backfill fails for a connector, startup now queues a durable retry instead of only logging. For each affected connector instance it marks every extant stream dirty in search_index_dirty; the existing reconcile loop (maintenance sweep and read-time self-heal) retries with its backoff. Before this, an idle source stayed behind until the next ingest or restart.

Two more paths leave a ticket, so no failed or skipped startup pass is left unrecorded:

• If the ticket write itself throws (for example a transient read or write fault while the scopes are found), that manifest stays in a local list for the run and is retried with capped exponential backoff (250 ms to 30 s) until a mark commits or shutdown aborts the run. The list is not stored.
• If a semantic backend is configured but not available at startup (for example still warming) and the connector declares semantic_fields or still holds semantic index state (meta, backfill progress or vectors), the semantic pass is skipped and a ticket is queued. Retained state counts because removed-field cleanup also needs the backend. With no backend configured, or neither fields nor retained state, nothing is queued, so a deployment without a backend does not dirty every connector on every boot.

Why

Root cause: the startup backfill is the only repair path that runs without a prior write, and its failure left no persisted state. The dirty table is the existing durable retry ledger; startup failure just never wrote to it. No new table or column is added. The change is 4 small SQL files, one 54-line module (server/startup-search-index-retry.ts), three store functions, and one retry loop in the startup task (the backoff above). Existing databases need no migration, and old and new binaries share the same search_index_dirty rows.

State model

One row per (connector_instance_id, stream) in search_index_dirty. Existing states, unchanged:

• clean: dirty=0
• dirty/eligible: dirty=1, next_attempt_at null or past
• dirty/backoff: dirty=1, attempts>0, next_attempt_at in the future
• parked: operator_parked=1 (operator lever; rearm returns it to eligible)

Transitions:

• startup failure -> markStartupSearchIndexFailure: clean becomes dirty/eligible with attempts=0; dirty or parked keeps attempts, next_attempt_at, marked_at, operator_parked; revision+1 in all cases.
• reconcile success -> clear, compare-and-set on revision.
• reconcile failure -> attempts+1 and backoff, compare-and-set on revision.

Crash analysis:

• Crash after the backfill fails but before the mark commits: no state written; the next startup runs the backfill again and queues on failure.
• Crash between two instances' marks: each instance is one transaction (PostgreSQL transaction, SQLite writeTransaction) under the instance deletion fence, so each instance is fully marked or not marked. The next startup repeats the backfill.
• Crash during a reconcile: the row stays dirty; the repair is idempotent.
• Startup mark racing an in-flight reconcile: the mark bumps revision, so the reconcile's clear is a no-op and the scope is retried.
• Two retries of one scope at once: scopesInFlight in search-index-reconcile.ts allows one per scope per process; the backfills take the per-instance fences.
• Deleted instance: the insert is skipped for tombstoned instances and for instances owned by another connector.
• Storage close or swap mid-mark: assertCurrentStorageGeneration before and inside each transaction; the work is dropped.

Both additions reuse existing transitions. One in-process state is added: queue-pending, a manifest whose ticket write failed. It moves to done (a mark commits) or is dropped on shutdown. A crash while queue-pending loses it, and the next startup repeats the backfill, which fails or succeeds again from the same persisted state. A backend that is unavailable at startup gets the startup-failure transition; reconcile keeps the scope dirty until a backend can prove the semantic half.

Discovery: a ticket is written for every instance that holds records or search-index state for the connector (one store query), or for the pinned instance when the manifest names one. An instance with neither has nothing to repair.

PostgreSQL: the mark reads retained semantic scope keys as text and decodes them in code. A key that is not a JSON array with a string stream names no stream and is skipped, so one malformed key cannot stop the ticket; the repair still fails closed on it.

Defect found and fixed here: the loop over instances stopped at the first instance whose mark threw, so siblings got no ticket until the next restart. It now attempts every instance and throws an AggregateError afterwards, one error per failed instance naming its connectorInstanceId with the original error as cause (new test, below). The startup caller logs that error and continues; a ticket write that keeps failing never fails startup (test below).

Other changes

• search-semantic.ts: the "rebuild completed" log now follows the final transaction instead of preceding it, and the non-participating-stream log says "checking cleanup" because it can run when nothing is dropped.
• startServer option testOnlyAlreadyAdmittedPostgresChildAttachment: test-only pass-through to the existing initPostgresStorage option so a restart test can reopen a populated test database. It is inert in production: initPostgresStorage reads it only when the test-database guard is active (PDPP_TEST_POSTGRES_URL or PDPP_REQUIRE_TEST_DATABASE=1), the capability is single-use and minted inside the target database, and no environment variable or CLI flag maps to it. No smaller shape exists: the restart reopens a database that already holds owner data, which admission refuses without the capability. Eight existing test fixtures already pass the same capability to initPostgresStorage.
• closeHttpServer is exported from the test helper.
• postgres-template-eligibility.ts and run-tests-resource-policy.test.ts: the new test is listed as cold-required (count 37 to 38) because it starts the real server over retained state.
• Not changed: server/retrieval-startup-backfill.ts has no importer; the live path is the copy in server/index.ts.

Tests

cd reference-implementation && node --import tsx --test test/startup-backfill-durable-retry.test.ts scripts/run-tests-resource-policy.test.ts: 21 pass, 0 fail (memory/SQLite). npm run typecheck passes.

PostgreSQL 18 with pgvector, private cluster via scripts/test-sandboxed.sh --pg-env ... scripts/test-postgres-private.sh, one file per run: startup-backfill-durable-retry.test.ts 11 pass, 0 fail; semantic-startup-backfill-catches-crash-abandoned-record.test.ts 70 pass, 0 fail. (Run together in one process, the files share the cluster and fail; run them one at a time.)

The new tests fail without the fix (checked on memory and on PostgreSQL):

• real startServer with an embedder that times out: with the queue call removed, no ticket exists; with it, one maintenance round converges with no ingest.
• sibling isolation: a trigger rejects one instance's ticket; with the per-instance loop reverted to the old early exit, the healthy sibling gets no ticket. Each failure also names its instance.
• ticket write fails twice at startup: with a single retry, no ticket exists; with backoff retry, a ticket exists with no restart. The fault is a trigger that rejects the first two writes (a sequence on PostgreSQL, a trigger dropped on the second logged failure on SQLite). ESM exports cannot be stubbed, so the fault is injected at the database.
• ticket write fails on every attempt: startup is not failed, the server answers, and closing the server aborts the backoff sleep (a loop that ignored the abort hangs the test).
• configured but unavailable semantic backend with semantic_fields: without the new branch, no ticket exists.
• manifest drops its last semantic field while semantic blob state is retained and the backend is unavailable: without the retained-state check, no ticket exists (red on memory).
• malformed retained semantic scope keys (not json, null, {"a":1}, [1]) on PostgreSQL: without in-code decoding, the mark fails and no ticket exists (red on PostgreSQL; SQLite guards the JSON in SQL).
• an instance that has only records, with no binding or index state, still gets a ticket (outcome only; no lookup is named).
• negative cases pass before and after: no backend configured, and a connector with no semantic_fields and no retained state, queue nothing.

Also covered: history preserved on dirty/parked scopes, clean scopes reset, tombstoned scope not resurrected, stale generation rejected while another writer holds the fence. The external embedder is a stub; no real service is touched.

Not run

scripts/quality-ratchet/check-mass-ratchet.ts needs the workspace Biome binary, which this checkout does not have. npm run typecheck passes. Repo lint reports format differences in files this PR edits (server/index.ts and others) that predate it.

Open risks

• While a ticket write keeps failing, startupBackfillDone stays pending (the retry is its tail). Only the awaitStartupBackfill option waits on it, and only a test fixture sets that; the listeners and the periodic sweep do not wait on it.
• Retry is a hint: every extant stream of a failed connector instance is marked, including in-sync ones; reconcile no-ops on those.
• Two server processes on one PostgreSQL database are not coordinated beyond the existing per-instance fences.

Assisted-by: AI