Calibration eRPC 429 storms: deal flow resolveByProviderId fans out O(active data sets) eth_calls per deal

[SgtPooki]

Problem

dealbot deal jobs call synapse.storage.createContext({ providerId }) → synapse-sdk StorageContext.resolveByProviderId, which fires getActivePieceCount + getAllDataSetMetadata per active data set (batch size ≥50) to metadata-select a set to reuse. On calibration dealbot has up to ~196 active data sets with a single provider (see #605), so one resolve fires ~132 eth_calls, ×2–3 per deal, with all SPs phase-locked to the same scheduler tick → 300–500 eth_call/s bursts at the top of the minute. These drain chain.love's calib token (currently 1000 req/min) in ~2–3s → eRPC ErrUpstreamsExhausted 429 storms (~9,748 chain.love 429s / 20h, ~12k Ankr-free budget trips). Mainnet is clean today (lower volume) but runs the same armed code path.

Evidence

• Decoded burst selectors (exhausted eth_calls, 6h): getActivePieceCount 2,219, getAllDataSetMetadata 656 — standalone, no Multicall3 (matches resolveByProviderId, not the piece-cleanup aggregate3 path).
• Per-second 429s: up to 337 in a single second; job-window correlation deal-flow 217 vs piece-cleanup 2.
• getActivePieceCount is O(n)-in-pieces and OOG-reverts on large sets (getActivePieceCount fails with out-of-gas for large datasets pdp#265, getPdpDataSet fails on large data sets because getActivePieceCount reverts inside multicall synapse-sdk#851), so it dominates the failure sample.

Fix plan

• synapse-sdk (root cause): tracked in resolveByProviderId: replace batch enrichment with p-queue sliding window synapse-sdk#631 — metadata-filter first, replace O(n) getActivePieceCount > 0 with getActivePiecesByCursor(limit=1) (#848), bounded concurrency with real early-exit, Promise.allSettled. Validate with the rpc-tracer against a ~196-data-set fixture, then patch-package into dealbot.
• infra eRPC guardrail: disable rateLimitAutoTune, set static budget ≈ chain.love cap, exclude 429 from failsafe.retry.
• dealbot hygiene: bound active data-set accumulation (Dealbot accumulates many active data sets per provider (calib: 196 on provider 23) — no reuse/cleanup #605).

Related

FilOzone/synapse-sdk#631, #851, #848; FilOzone/pdp#265; #603 (RPC timeout).

Done criteria

• Calibration deal-flow eth_calls per deal bounded (not O(active data sets)).
• Calibration ErrUpstreamsExhausted/429 storms eliminated; mainnet stays clean after v1.9.0.

[SgtPooki]

Note that I looked into this deeply and left some notes at FilOzone/synapse-sdk#631 (comment) that might help others taking this on gain some more context.