Support checks from multiple regions

[BigLep]

Done Criteria

Our core checks (data storage, retrieval) are:

1. Run from multiple regions
2. Have their metrics attributed with the "testing-from" region in our metric datastore
3. Any UI of our metrics make clear where the check was run from

Why Important

Dealbot gives a truer sample of performance from wherever it is running. It's currently running from the EU, which biases its results for EU-based users.

Open questions

1. What geographic identifier(s) will we use when attributing the "testing-from" region?

• Hetzner location names (e.g., fsn1, sin)?
• IATA airport codes (e.g., LAX, LHR)?
• AWS regions (e.g., us-east1)
• ISO 3166 country codes (USA, CAN)

2. Do all SPs get tested by all dealbot instances or do they get divided up amongst the dealbot instances?

Notes

1. This is a placeholder for now. Given it's not a GA priority, it hasn't been scoped out more.
2. I assume we'll still have one global database instance that all dealbot instances use

[BigLep]

Infrastructure deployments to multiple regions is happening in https://github.com/FilOzone/infra/pull/69

[BigLep]

Dropping down some thoughts on this given its priority goes up if we rely on it as part of #427

Geographic identifiers

For public metrics, I think we should is IATA airport codes because:

1. City-level granular - Country is not specific enough, at least for places like the US.
2. Infrastructure-agnostic - We're not encoding the names of a given infra provider like Hetzner / AWS.
3. Globally standardized
4. (According to Claude) Already the de facto convention used by Cloudflare, Fastly, and other major infrastructure providers for exactly this purpose. (I know internally that AWS uses these codes.)
5. Human-readable

I think it's also good for metrics we view internally for their to be extra details about the infra provider and infra region (e.g., aws:us-east-1, hetzner:fsn1)

Regional Distribution of Work

I think it's most useful to have "NA/EU/APAC" SPs get tested by infrastructure in "NA/EU/APAC" (i.e., each SP gets tested from one region, not all regions).

Why?

1. That will give most representative performance numbers, assuming a client uses an SP in their region. (This isn't a valid default assumption though per below)
2. Cleaner dealbot succes signal - We expect it should reduce some of the intermittent failures we have seeing going cross region which has also prompted increasing timeouts (example).
3. Cleaner dealbot latency/bandwidth signal - Rather than mixing latencies/bandwidths from multiple regions which will inherently have have different performance profiles, we get a read from a single more proximate region.

Why not do this?

1. Synapse (per private copilot chat pasted below) doesn't currently do any work to try an select a geographically close SP. Thus dealbot will make things look better than they actually are in the wild.

I am still in favor of doing this though for moving us in the right direction. If we do, the key things we'll need to adjust in dealbot are:

1. Knowing which region to map an SP. This should be hard though given the SP registry has the location: https://filecoin.cloud/service-providers (It's just that the location isn't an IATA airport code - but that's fine for now, the mapping to region isn't hard)
2. Ensuring scheduler/workers are region aware (I assume meaning we'll have region-specific queues?)
3. Handle work that is (or can be) truly global (e.g., dataset creation, data retention checks)
4. Tag metrics accordingly

copilot response on whether Synapse uses geographic proximity

**No, there is no geographic proximity or latency-based selection.** The ping is used purely as a **health/reachability check**, not for ranking providers by latency.

Here's what the SP selection logic actually does, in order:

selectProviders (core ranking logic — select-providers.ts)

1. Endorsed providers first — if requireEndorsedPrimary=true, the first slot is restricted to the endorsed provider list from the chain.
2. Prefer providers with an existing matching dataset — reuses an existing payment rail rather than creating a new dataset.
3. Within matching datasets: prefers datasets with pieces already in them, then falls back to older datasets (lower ID).
4. Otherwise: picks any eligible approved provider (no geographic or latency consideration).

smartSelect (ping-retry loop — context.ts)

Once selectProviders returns a candidate, smartSelect calls SP.ping() on it. The ping is pass/fail only:

• ✅ Responds → use this provider.
• ❌ Fails → exclude it and re-run selectProviders with the next candidate.

The ping response time is not measured or compared between candidates. There's no sorting by RTT, no geo-IP lookup, no location metadata from the chain being compared to the client's location.

Summary

Factor	Used?
Endorsed/approved on-chain	✅ Yes
Existing dataset reuse	✅ Yes
SP is reachable (ping health check)	✅ Yes
Geographic proximity	❌ No
Ping latency/RTT comparison	❌ No

If geo-aware selection is desirable, it would need to be added — e.g., measuring ping RTT and sorting candidates by it, or using location metadata if/when SPs publish it on-chain.