testnet-canary: restore foreground catch-up backpressure recovery

[branarakic]

Canary purpose

Restores the foreground catch-up backpressure behavior from #1895 on the current testnet-canary tree for live acceptance. This is the canary counterpart of #1934.

The original #1895 merge was removed by #1903 so that its test candidate exactly matched main plus #1877 and #1898. This forward-port reapplies the behavior after that isolated candidate was accepted.

Behavior

• explicit CLI and subscribe catch-up uses typed foreground mode
• foreground work receives elevated scheduler priority
• local backpressure is retried with bounded delays of 100 ms, 250 ms, and 500 ms
• durable VM catch-up completes before dependent SWM starts
• an SWM-only deferral retries SWM without refetching durable data
• automatic background catch-up remains best-effort and does not wait
• foreground and background runs have distinct single-flight identities

Validation on current canary head

• full 17-package CLI dependency build passed, including Solidity compilation, agent type tests, package-root verification, and CLI TypeScript build
• agent policy and single-flight tests: 25 passed
• CLI worker catch-up tests: 12 passed
• diff check passed

Live acceptance after merge

Run an explicit foreground catch-up while sync capacity is occupied and confirm transient deferral recovers, durable precedes SWM, persistent pressure remains bounded, and the node remains healthy.

[Jurij89]

Review — does #1937 faithfully restore #1895?

Reviewed specifically as a forward-port of #1895 onto the restructured canary tree (worth doing carefully, since #1895's retry helper sync/catchup-backpressure-retry.ts no longer exists on canary — the logic had to be rehomed, which is exactly where a behavior can silently drop). Verified behavior-by-behavior against the post-merge canary source (merge fb46af287), the #1895/#1937 diffs, and the tests, and cross-checked for conflicts with the intervening #1877 / #1898 / #1934 changes.

Verdict: faithful and correct. All five #1895 behaviors are restored, and the port integrates cleanly with the current tree.

#	#1895 behavior	#1937
B1	Explicit CLI/subscribe catch-up → typed foreground, elevated priority (can't be starved by background)	✅ restored
B2	Bounded-backoff retry (100/250/500 ms), only the deferred plane	✅ restored
B3	VM before SWM; SWM-only deferral retries SWM without refetching VM	✅ restored
B4	Foreground vs background have distinct single-flight identities	✅ restored
B5	Background catch-up stays best-effort (no wait/retry)	✅ restored

Details verified sound:

• The standalone retryCatchupPlaneOnBackpressure (in the now-deleted catchup-backpressure-retry.ts) is replaced by a single canonical orchestrator runCatchupPlanesWithPolicy in sync/catchup-policy.ts, shared by both the agent (runCatchupOverPeers) and the CLI worker — cleaner than fix(sync): retry foreground catchup under backpressure #1895's inline wiring.
• fix(sync): retry foreground catchup under backpressure #1895's paired priority + retryDeferredBackpressure flags become a typed CatchupMode ('foreground' | 'background'); catchupPriorityForMode('foreground') = 2_000, background undefined.
• Constants byte-identical: FOREGROUND_CATCHUP_SYNC_PRIORITY = 2_000 (outranks exact-VM-repair 1000 > background 0), CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS = [100, 250, 500]; retry = 1 initial + ≤3 retries, break-on-clear.
• The elevated priority is load-bearing and threaded end-to-end to the admission gate: syncContextGraphFromConnectedPeers(mode) → runCatchupPlanesWithPolicy → syncFromPeerDetailed({priority}) → runChangelogLane(…, priority) → runContextGraphSyncWithBackpressure(priorityOverride).
• No conflict with the intervening tree: priorityOverride pre-existed on canary (from the exact-VM-repair path at priority 1000) and testnet-canary: restore foreground catch-up backpressure recovery #1937 correctly reuses it; fix(sync): preserve incomplete durable catchup progress #1898 (HTTP-503 classification) and fix(storage): re-probe maintainIndex on concurrent generation drift instead of demoting to a full-store rebuild #1877 (graph-set-index-store) don't overlap. Single-flight keys are genuinely distinct (foreground 2000 vs background null), transport/authz/timeout failures are not retried (their results carry deferredBackpressure: 0), and the new shared: null case (durable deferred → SWM skipped) is guarded at every consumer — no NPE, no miscount.

One follow-up worth doing — LOW, test-only, not a blocker

The admission-boundary unit coverage was dropped in the port. #1895 shipped three assertions in packages/agent/test/sync-requester-priority.test.ts proving the foreground priority 2_000 actually reaches runContextGraphSyncWithBackpressure's priorityOverride. #1937 doesn't touch that file, and the canary copy has none of those assertions. #1937's own priority tests stop at a mocked syncFromPeerDetailed / worker invoke boundary — so the live syncFromPeerDetailed → runChangelogLane → runContextGraphSyncWithBackpressure(…, priority) segment (the actual "foreground elevates at the scheduler gate" property, which is the whole point of the fix) is exercised by no end-to-end test. It's correct by inspection today, but a future refactor dropping the priority arg from that admission call would pass CI silently.

Recommend porting #1895's three sync-requester-priority.test.ts assertions, adapted to mode: 'foreground' ⇒ expected priorityOverride === 2_000, across runLegacyDurableSync / runChangelogLane / the SWM-deferral path.

(Trivial: the rationale docstrings on both constants — the 2000 > 1000 > 0 ranking, and "transport/authz/timeout/integrity deliberately not retried" — were lost in the move; the values themselves are unchanged.)

Net: the behavior is fully and correctly restored; the only real gap is a dropped admission-layer regression test.

---

Reviewed with multi-agent cross-verification (faithfulness / adversarial / test-coverage lenses), all checked against post-merge canary source rather than gh pr diff (which was stale for this branch).