Dealbot accumulates many active data sets per provider (calib: 196 on provider 23) — no reuse/cleanup

[SgtPooki]

Problem

On calibration, dealbot's wallet (0x305025…dca0b) holds a large and growing number of active (un-terminated, pdpEndEpoch=0n) data sets per provider, instead of reusing or cleaning them up:

provider	active dealbot data sets
23 (🚀, 0x6c8E…d796)	196
4, 6	25
2, 9	24
5, 8, 15, 22, 27	15
26	7
24 (sp-playground)	0 (terminates all → contrast)

Provider 23's 196: 182 share identical metadata {"source":"dealbot","withIPFSIndexing":""} (data set ids 13115–13751), 14 are unique dealbotDS:N indexed slots; all have pieces, zero terminated. (Source: Foc-observer fwss_data_set_created LEFT JOIN fwss_service_terminated.)

Why it matters

StorageContext.resolveByProviderId (synapse-sdk) scans every active data set for the provider on each deal to metadata-select one. So this count is the per-resolve RPC fan-out width — provider 23 alone forces ~132 eth_calls per resolve and it grows daily. See the RPC-burst tracking issue. (Also wastes storage/payment rails.)

Open question (to resolve)

Are the 182 identical-metadata sets a reuse failure (resolveByProviderId should return the existing metadata-matching set, but dealbot creates a new one anyway) or intended fresh-create by data_set_creation without cleanup? Provider 24 terminates everything; provider 23 terminates nothing — so cleanup is at least inconsistent.

Next steps

• Determine the create-vs-reuse path for the base-metadata sets (instrument createContext/createDataSetWithPiece).
• Decide policy: reuse the existing match, or terminate old test data sets (and why provider 23 never terminates while 24 does).
• Sweep the existing accumulated sets on calib.

Done criteria

• Active dealbot data-set count per provider stays bounded on calib.
• Root of the 182-set accumulation identified and fixed (reuse or cleanup).

[beck-8]

23 is my problem. Node 23 uses glif's RPC, and now it encounters the too many pending messages for actor issue (unable to send too many messages). Waiting for Lotus to release the release, and the glif to upgrade. after I don't think I have this problem.

[SgtPooki]

thanks for the info @beck-8

I think there is also work here to ensure that dealbot is not leaving a bunch of data-sets in created state. dealbot should be able to see this and terminate some data-sets if they're accumulating (I need to get #548 over the line)