New check: retrieval++

[BigLep]

Done Criteria

We have a retrieval check that does not easily reveal the checker as dealbot by:

1. Running on generic/dynamic infrastructure (rather than just coming from static dealbot k8s clusters).
2. Checking for retrieval of pieces beyond what dealbot created. Anything on chain with an ipfsRootCid is fair game.

Once a piece for checking has been determined, it should follow similar behavior to the basic retrieval check of doing CID looking in IPNI and doing /ipfs retrieval.

All metrics should have similar dimensionality to basic retrieval check (e.g., provider info, checkType). A new check type should be used to disambiguate this from the basic retrieval check.

Open questions

1. What to call this new checkType (e.g., retrievalAnon)?
2. Piece selection logic/method (e.g., query subgraph X for all pieces with ipfsRootCid on provider X)
3. How much /ipfs retrieval do we do (e.g., all blocks, n blocks, n MB)?
4. Do we make these "anonymous retrieval checks" form all locations to all SPs or be more geographically aware (e.g., check on EU SPs from the EU). We should follow suit or learn from Support checks from multiple regions #246

Why Important

This gives a more honest analysis of how an SP is doing. This methodology makes it harder for the SP to be on their best behavior because "the teacher is watching".

User/Customer

Prospective/current FOC users who want to have confidence that data stored with FOC will be truly retrievable later by anyone.

Notes

Subtasks

• Anything listed as a sub-issue
• Move the subgraph to the FOC WG goldsky team as discussed in operationalize dealbot subgraph #498
• Enable on staging
• Enable on production
• Monitoring of the subgraph per operationalize dealbot subgraph #498
• Add to https://probelab.io/filecoin/foc

[rvagg]

Things we care about:

• Piece retrieval works for whole piece and the bytes are correct
• IPFS retrieval works for all of the blocks in a CAR that we upload in a deal with withIPFSIndexing on
• IPNI has all of the CIDs in a CAR in a piece stored with withIPFSIndexing on

We also need to be careful not to blame the SP for data that isn't actually IPFS data but the client claims it is.

Here's how I propose to handle this:

1. Find pieces to fetch. They're going to want to be large enough that you get decent bandwidth metrics from them, but not so large as to exclude providers who don't happen to have big deals or you're testing the same pieces regularly. Maybe we have to be fully open about size criteria but account for it (somehow) in any speed testing.

Here's the distribution of piece size on both networks currently

But we'd also want to focus on pieces in data sets that have the withIPFSIndexing metadata set. This is what tells the SP to index the pieces if they are CARs and serve those IPLD blocks through the ipfs retrieval endpoint. Without that metadata, we can't expect a piece that's a CAR to be retrievable, we've made it opt-in (unlike classic PoRep).

Here's the piece / size distribution accounting for withIPFSIndexing

We could randomly sample from both withIPFSIndexing and non-withIPFSIndexing pools just to get full coverage (to ensure an SP isn't prioritising the CAR pieces), but we do want to make sure we have a good representation of withIPFSIndexing pieces in our pool.

So, this is probably going to need a subgraph or other indexer of FOC (I'm running https://github.com/filozone/foc-observer, but a subgraph specific to this task might be better). It would need to collect data sets, data set metadata, piece CID, piece size (contained in the Piece CID so you can decode it in the subgraph code) and provider details (you can map data set owner to provider wallet to get provider ID in the ServiceProviderRegistry).

2. Given a piece, perform a full retrieval via the /piece/{pieceCID} endpoint for the SP, measure speed, latency etc. Then hash the piece with CommP and make sure it matches the PieceCID you have. If it fails, this is a heavy black mark against the SP.

3. If the piece is in a data set with withIPFSIndexing check whether the piece is a CAR, if it's a CAR, inspect it, make sure it's valid and the block data match the CIDs and get the list of CIDs in it (go-car has both car ls and car inspect --full that can show how to do this, also possible in JS). If the piece isn't a CAR or the CAR isn't valid in any way, don't proceed, and it's not the SP's fault, this is on the client.

4. Check IPNI for all of those CIDs in the CAR, does IPNI have records that link the individual CIDs back to the provider. If not, that's a mark against the SP.

5. Perform an /ipfs/{ipfsCID} retrieval for some or all of the CIDs in the CAR against the SP and then hash the returned bytes against the CID to make sure it's correct. Any failure to serve or bytes that don't hash properly are a mark against the SP.

Things we can't assume:

• Client uploaded a CAR in a withIPFSIndexing data set
• Client uploaded a valid CAR (SP will try to index but bail if it's not valid)
• CAR is a coherent DAG of blocks with a valid root; there may be no root in the CAR header and the blocks may be a random assortment with no links between them (filecoin-pin add <path> will make a complete DAG, put a root in the CAR and put ipfsRootCID on chain metadata for the piece, but this isn't strictly required or expected)
• CAR contains blocks of certain sizes--technically there are only some basic restrictions in go-car, but even the spec itself doesn't place requirements, so you may even find a block that's a CAR with a single 1GiB IPLD block in it that's still valid.
• Piece size will be nicely bounded. There's a technical limitation on CommP at the lower end, so don't expect smaller than 127 bytes, but upper bound is not strict. In Curio we cap it to 1016MiB but we may uncap that and we also aren't necessarily just seeing SPs using Curio and our restrictions, Storacha is involved and we may see custom forks or implementations. PDP & FWSS don't strictly limit size.

[BigLep]

Fantastic write up @rvagg! I'm fully on board with this proposal. I'll get the done criteria updated 2026-04-02 to match this.

This seems like a great new check to add that give higher assurances that SPs are serving retrievals of all stored data, not just dealbot data.

[dennis-tra]

Hi everyone,

just for your information, the ProbeLab team plans to work on/help with this issue.

To recap with my own words. In this feature we want to achieve two things:

1. Geographically diverse probes
2. Probes of Non-dealbot data

Then what you @rvagg suggest consists of

1. a proposal regarding candidate pieces requirements (e.g., piece size, metadata)
2. how we can discover such pieces (a subgraph or other indexer of FOC)
3. an improvement to the probes themself (items 2.-5.)

(To be honest, I need to familarize myself with the foc-observer/PDP explorer/subgraphs/etc. because I don't quite understand that part yet)

To me, the second enumeration covers the "Non-dealbot data probes" part. Therefore, I need some more clarity around the geographic diverse infrastructure. In previous calls we discussed the idea of deploying probing infrastructure on CloudFlare workers. If we're okay with this route we would need to make a few architecture decisions. There are a few options but I'll just describe what I would do:

We'll have a backend deployment that acts as a central orchestrator plus multiple dumb workers. The existing backend selects pieces, enqueues jobs, calls out to CF Workers via HTTP, workers execute the probe and return results in the response, backend writes to the DB. Workers are stateless functions with no DB access. The backend code will need to be adjusted to account for piece selection and scheduling of workers.

Regarding the worker, I would create a new worker app inside the dealbot repository that imports the relevant probing parts of the backend app, potentially extracting parts into a common package. Alternative, I could create a new DEALBOT_RUN_MODE. Then, we would deploy multiple of these workers in specific locations where each worker gets its own URL. These URLs together with location information would be passed as configuration parameters to the backend code so that it can call the relevant endpoints. I believe that this is easier than somehow detecting workers dynamically or letting the workers somehow register themselves. Worker invocations should be limited to our backend deployment by some kind of authentication mechanism (I think a simple shared key will do it).

Open questions:

1. How do we prevent SPs from identifying worker traffic as dealbot? CF egress IPs are well-known. Is traffic to SPs commonly coming from CloudFlare? If so, our traffic could hide in that stream.
2. Where would we want to deploy the workers? How many?
3. What geographic identifier do workers report? (ties into Support checks from multiple regions #246)
4. Cloudflare Workers have CPU time and memory constraints. Can a full retrieval check (IPNI lookup + /ipfs fetch) complete within those limits? I need to research this
5. How do we monitor the health of the workers themselves (uptime, error rates, deploy status)?
6. How reliable are the timing measurements from workers? I suspect they run on shared infra which may skew numbers?
7. Separate app or new run mode?

Not all of them need to be answered right now but can be addressed as we go along. These are just things that came to my mind.

[dennis-tra]

These are the cloudflare worker limits: https://developers.cloudflare.com/workers/platform/limits/#account-plan-limits

It's possible to request adjustments to the limits.

[dennis-tra]

I'm planning to make this issue (the geographic diverse deployment) my main priority for next week. It would be great if I got a confirmation regarding the general direction until tomorrow EoD. cc @BigLep

The week after I would look into the probes of non-dealbot data.

[BigLep]

Hi @dennis-tra! Great discussion here. Like a lot of things, once I got into it and think about it more, I realize there is actually more here than I was originally appreciating...

I stewed on this quite a bit and then had a conversation with Claude to check my thinking. I'll paste that at the end.

The biggest bang for the buck is anonymous piece selection, not anonymous infrastructure.

Even if we run retrievalAnon from our existing known Hetzner IPs, we've already meaningfully raised the bar. Today an SP can be on best behavior specifically when "the teacher asks for items the teacher created." With random non-dealbot piece selection, the SP has to perform well for any retrieval request — they can't know which pieces will be checked next. That's a much stronger signal, and we get it without any infrastructure changes.

On the infrastructure architecture: let's not introduce major rearchitecture yet.

The CF Worker proposal requires dumb stateless workers because CF Workers can't make DB connections. That's the architectural forcing function behind the whole backend-as-intermediary design. AWS Lambda doesn't have that constraint — it runs full Node.js and can connect to our Postgres DB directly, so Lambda workers would be peers to our existing Hetzner workers rather than requiring a new orchestration layer. But even Lambda is more than we need right now.

Before adding Lambda, let's just deploy to more Hetzner regions. The code is identical — it's purely an ops change. When we eventually want probes from IPs outside the Hetzner ASN, Lambda won't be a big lift: same Node.js runtime, same pg-boss dequeue logic, just needs the scheduler to call lambda.invoke() when there's work in the queue for a given region.

Concrete proposal:

1. Implement retrievalAnon as @rvagg outlined, with one adjustment: rather than checking all blocks in a CAR for IPNI indexing and /ipfs retrieval, sample n blocks (suggest n=5) to keep probe duration bounded.
2. Deploy to one NA region and one APAC region on Hetzner alongside our existing EU node. @SgtPooki can help here (and has even gotten started in https://github.com/FilOzone/infra/pull/69 )
3. Label all with IATA airport codes for the public-facing location identifier (e.g. IAD, SIN, HEL) — infrastructure-agnostic so the label stays stable if we add Lambda later. Internally in ClickHouse we can store infra provider and infra-native region alongside it for debugging (e.g, hetzner: fsn1, aws:us-east1).

Once we get that solid, we take stock on where we are and evaluate about getting more anonymity by deploying through Lambda (but that needs to be weighed against the maintenance/financial cost of using Lambda and onboarding onto the AWS stack for permissions, log access, etc).

[BigLep]

Below are the output from my conversation with Claude as I was stewing on the architecture.

Notes

Current Architecture

The backend is a NestJS app with:

• Scheduler (api/both mode): tick() loop that calculates which SP+job combinations are due, enqueues them via pg-boss
• Workers (worker/both mode): pg-boss workers pull jobs, call RetrievalService.performRandomRetrievalForProvider() → IPNI check → HTTP retrieval against SP → write retrieval rows to DB
• pg-boss is tightly coupled: workers hold open DB connections, results are written in-process

---

The Proposal (Option A: Synchronous RPC to CF Workers)

pg-boss queue → backend worker dequeues → HTTP call to CF Worker → CF Worker probes SP → response body → backend writes to DB

Strengths:

• Simple: no new infrastructure beyond the CF deployment
• CF Workers remain stateless and auditable
• Auth is straightforward (shared secret)
• Maps cleanly onto existing job structure

The critical risk: long-lived HTTP connections. CF Workers have a wall-clock timeout per request (30s default, extendable for paid plans). More importantly, the backend worker must hold an open outbound HTTP connection for the duration of the probe. Downloading a piece can take minutes. If the backend worker's HTTP client times out before the CF Worker finishes, you get a lost result with no retry path unless you re-enqueue.

The CF Worker CPU time limit (30s paid) is less of a concern — piece retrieval is I/O-bound — but wall clock limits need to be verified for your plan.

---

Option B: Async Callback / Webhook (Push Results Back)

backend: enqueue job → dispatch to CF Worker (fire-and-forget POST) → CF Worker probes → CF Worker POSTs results to backend /results endpoint → backend writes to DB

Strengths over A:

• No long-lived connection held by backend worker; it dispatches and moves on
• CF Worker can work at its own pace within CF's wall-clock limits
• Naturally handles cases where backend restarts mid-probe

Weaknesses:

• More complex: need a results endpoint with auth, result routing back to the originating job
• Harder to correlate job IDs across the dispatch and callback
• CF Workers need to know the backend's public URL

---

Option C: CF Workers with DB Access (via Cloudflare Hyperdrive)

This is technically feasible, but there's a hard constraint: CF Workers cannot open TCP connections to PostgreSQL natively. They use the Fetch API. Cloudflare Hyperdrive is their product that proxies TCP to CF Workers via an HTTP tunnel. You'd also need to expose pg-boss-compatible job locking or replace it with a different mechanism.

Why this is more expensive than it looks:

• You'd pay for Hyperdrive ($0.15/million queries on paid plans)
• You lose pg-boss's job locking/deduplication unless you replicate that logic
• CF Workers could create DB connection pool exhaustion (they're serverless and can fan out)
• The existing TypeORM/pg-boss schema is all PostgreSQL; you'd need to expose it safely to an external compute plane

For these reasons, DB access in CF Workers adds meaningful complexity and cost without a clear benefit — the backend is already doing the DB work, and keeping that centralized is easier to reason about.

---

Option D: Queue-based Decoupling (CF Queues or SQS)

backend → CF Queue (job spec) → CF Worker pulls → probes SP → CF Queue (result) → backend consumer writes to DB

This is the most robust pattern but adds infrastructure (CF Queues, or an external queue). It's worth flagging as a future evolution path if reliability becomes a major concern, but it's over-engineered for a first pass.

---

Recommendation

The proposal (Option A) is the right starting point, with one important modification: treat CF Worker calls as bounded-time operations with explicit timeouts, not open-ended streaming downloads.

Concretely:

1. Scope the CF Worker's job carefully. For this new retrievalAnon check, you don't need to download entire pieces — as rvagg notes, a partial download (first block, first N MB, or just IPNI verification + metadata check) is sufficient to mark an SP healthy. Keep CF Worker wall-clock time under 25s to stay safe.
2. Set explicit HTTP timeouts on the backend side when calling CF Workers. If a CF Worker times out, re-enqueue with pg-boss's retry mechanism.
3. For full-piece retrieval verification (rvagg's step 2 — downloading the whole piece and CommP-hashing it), that work should stay in the existing backend workers running in your k8s cluster, not CF Workers. CF Workers are better suited to quick-latency-and-availability checks, not bulk data transfer.

The separation this suggests:

• CF Workers → geo-diverse, fast availability checks: IPNI lookup, TTFB, first-block fetch, small piece retrieval
• Backend k8s workers → thorough checks: full piece download + CommP verification, CAR validation, bulk IPFS block fetching

This is also a cleaner answer to dennis-tra's open question #4 ("Can a full retrieval check complete within CF limits?") — the answer is: partially, if you scope the check right. Don't try to do everything in CF Workers.

---

Option E: Hetzner Regional Nodes

Since dealbot already runs on Hetzner infrastructure, deploying worker instances in each Hetzner region is a strong alternative to CF Workers for geographic diversity.

Hetzner's 6 regions: Nuremberg, Falkenstein, Helsinki (EU cluster), Ashburn VA, Hillsboro OR (US), Singapore (APAC).

Why Hetzner Nodes Are Actually a Better Fit

No architectural compromise. The same backend code runs everywhere. Workers connect directly to pg-boss/PostgreSQL — no HTTP intermediary, no result serialization, no timeout management across two runtimes. The CF Worker proposal introduced an extra hop specifically to cross the "no DB from serverless" boundary. Hetzner nodes don't have that constraint.

Full check capability without limits. CF Workers have wall-clock and CPU time constraints that force you to scope down the probe (no full CommP verification, no large piece downloads). A Hetzner node can do rvagg's full 5-step check — piece download, CommP hash, CAR inspection, IPNI verification, per-block /ipfs retrieval — with no artificial limits.

More reliable latency measurements. CF Workers run on shared serverless infrastructure; timing measurements are noisier. Hetzner dedicated/cloud nodes give you consistent baselines, which matters when throughput and TTFB are core metrics you're reporting on SPs.

Already familiar ops. Your team already runs Hetzner. Deploying a worker instance per region is a known quantity. CF Workers introduce a new runtime (Workerd, not Node.js), a different deployment pipeline, and new operational surface area.

Where CF Workers Still Win Over Hetzner

Geographic reach beyond Hetzner's 6 locations. If you care about checking from South America, Africa, Australia, or dozens of additional PoPs, Hetzner can't help you there. CF Workers have 200+ PoPs globally.

IP pool anonymization. If SPs have already fingerprinted your existing Hetzner ASN ranges as dealbot infrastructure, new nodes in the same Hetzner ASN may get the same treatment. CF Worker egress IPs are shared with enormous volumes of legitimate traffic, which provides more cover — though the bigger anonymization win is the piece selection change (checking non-dealbot data), not the egress IP.

Revised Recommendation

For a first iteration, Hetzner regional nodes are the better call:

• 6 regions covers EU, US-East, US-West, APAC — the major SP concentrations
• Zero architectural compromise vs the CF Worker approach
• Full check capability immediately (no scoping down to fit CF limits)
• No new runtime, no timeout wrestling, no results serialization across an HTTP boundary

The CF Worker approach made more sense if you assumed you needed exotic geographic diversity or had no existing geo-distributed infrastructure. But with Hetzner already in use, and its regions covering the major SP clusters, the simpler path is to extend what you have.

CF Workers could be added later as a complement if you want to probe from regions Hetzner doesn't cover, or if you specifically want traffic blending into CF's IP pool for anonymization.

---

Option F: Geographic Proxy Services

Since Hetzner nodes come from a known ASN that SPs could fingerprint as dealbot infrastructure, routing probes through a proxy service is worth considering for anonymization.

The Core Tension: Anonymization vs Measurement Fidelity

Any proxy sits in the network path between your checker and the SP. That means:

• SP sees the proxy's IP → anonymization goal achieved
• Your TTFB and throughput measurements include proxy latency/bandwidth → measurement quality degrades

This is fundamental. You can't fully anonymize and get clean performance measurements from the same probe. This suggests anonymous checks should be scoped as availability/reachability signals, not throughput benchmarks.

Proxy Options

Residential Proxy Networks (Bright Data, Oxylabs, Smartproxy)

• Traffic exits from real residential ISP IPs — essentially indistinguishable from an ordinary user
• Geo-targeting lets you say "exit from Singapore" or "exit from Germany" without deploying infrastructure there
• Most anonymous option by far
• Cons: Expensive (~$8–15/GB for residential), latency is unpredictable and will inflate TTFB measurements, proxy bandwidth may cap throughput readings

Commercial Datacenter Proxy Rotators

• Large rotating pools of datacenter IPs across many providers/ASNs
• Cheaper than residential, harder to allowlist/block than a known ASN like Hetzner
• Cons: Savvy SPs may still detect these as proxy traffic; less geographically precise

Cloud Serverless Functions as Proxies (Lambda, GCP Cloud Run, Azure Functions)

• Deploy a thin HTTP proxy function in AWS/GCP/Azure regions: eu-west-1, us-east-1, ap-southeast-1, etc.
• Each invocation gets a fresh IP from the cloud provider's massive shared pool — identical to traffic from S3, CDNs, etc.
• Much broader geographic coverage than Hetzner (20+ regions across AWS alone)
• Effectively the same concept as CF Workers but with less restrictive time limits and more regions
• Cons: IPs are still known cloud provider ranges (AWS/GCP/Azure are fingerprint-able as cloud infra, though not as dealbot specifically)

Tor

• Free, maximally anonymous
• Not suitable here: Tor exit nodes are commonly blocklisted, and the latency makes throughput/TTFB measurements meaningless as SP performance signals

Mullvad / Commercial VPN with SOCKS5

• Cheap, many exit nodes, privacy-focused, works at TCP level so protocol-agnostic
• Cons: VPN IP ranges are well-known and often blocklisted; not truly residential

Practical Recommendation

A two-tier check design fits best here:

Tier 1 — Thorough checks from known infrastructure (Hetzner nodes):
Full piece download, CommP verification, CAR validation, IPFS block fetching. Accurate throughput and latency metrics. Accept that SPs may identify this traffic.

Tier 2 — Anonymous spot checks via rotating proxies:
IPNI lookup, TTFB, first-block fetch only. Use residential proxies (Bright Data/Oxylabs) or cloud serverless functions. Report these as availability signals, not throughput benchmarks. The point is to catch SPs that serve dealbot probes well but serve everyone else poorly.

For Tier 2, cloud serverless functions (Lambda/GCP) are the pragmatic starting point — cheaper than residential proxies, more geographic coverage than Hetzner, no persistent infra to manage, and IPs are indistinguishable from the enormous volume of legitimate cloud traffic.

CF Workers vs AWS Lambda for Tier 2

The existing codebase is TypeScript/Node.js. CF Workers run on Workerd, which is not Node.js — it's a Web API subset with no Buffer, no Node.js HTTP clients, no native modules. The retrieval code would need a meaningful rewrite to run on CF Workers. AWS Lambda with the Node.js runtime is much closer to what already exists and could reuse the probe logic with minimal adaptation.

	CF Workers	AWS Lambda
Runtime	Workerd (not Node.js)	Node.js — matches existing code
Geographic control	Rough regional hints	Precise per-region (ap-southeast-1, etc.)
IP pool	CF CDN ranges — blends as end-user traffic	AWS cloud ranges — large/rotating
Time limits	Shorter, needs careful scoping	15 min — no concern
Cold starts	None	~100–500ms (fine for periodic probes)
Geographic reach	200+ PoPs	~30 regions
Deployment complexity	Simple (wrangler)	More setup (Function URLs / API Gateway, IAM)

CF Workers' main advantage — 200+ PoPs — is less meaningful for Tier 2 than it sounds. You don't need 200 locations; you need a handful of well-chosen regions that represent where real users of the data are. AWS's ~30 regions cover that adequately, and you get precise control over which region each probe reports from, which matters for the location dimension in your metrics.

Recommendation: AWS Lambda for Tier 2. Less runtime friction, reuses existing TypeScript probe code, precise geographic attribution. If probes from locations AWS doesn't cover are needed later, CF Workers can be added as a supplement.

Lambda DB Access and Architecture

Since the database is on Supabase, Lambda can connect to PostgreSQL using standard username/password auth — no IP allowlisting needed. Lambda runs in a full Node.js environment with standard TCP networking, so it connects to Supabase exactly the same way the existing backend does.

This means Lambda can be a full peer worker — not just a probe executor that needs the backend to handle DB reads and writes:

Hetzner backend (scheduler/api mode) → enqueues jobs into pg-boss
Lambda per region (worker mode)      → dequeues from pg-boss → probes SP → writes results to Supabase

pg-boss uses PostgreSQL's SKIP LOCKED for job dequeuing — pure SQL that works from any client, anywhere. The Lambda runs the existing worker code with minimal adaptation.

Connection pooling: Each Lambda invocation opens a DB connection. With low concurrency (a handful of regional probes at a time) this is manageable. If connection exhaustion becomes a concern, adding PgBouncer in front of Supabase resolves it.

Lambda runtime limits:

Limit	Value
Max timeout	15 minutes
Memory	128 MB – 10 GB
CPU	Scales with memory (1 vCPU at ~1.8 GB)
Ephemeral storage (/tmp)	512 MB – 10 GB
Default concurrent executions	1,000 per region (soft limit)
Cold start (Node.js)	~100–500ms

15 minutes is more than enough for any probe scope, including full piece downloads.

Triggering Lambda

Three options, in order of preference:

Option 1 — Backend invokes Lambda directly (async) [Recommended]

When the Hetzner scheduler determines a regional probe is due, it calls lambda.invoke() via the AWS SDK with InvocationType: 'Event' (fire-and-forget). Lambda wakes up, dequeues from pg-boss, runs the probe, writes results to Supabase. The backend returns immediately — no open connection, no waiting.

Hetzner scheduler → lambda.invoke({ InvocationType: 'Event', region: 'ap-southeast-1', ... })
                         ↓ (async, backend returns immediately)
                    Lambda dequeues pg-boss job → probes SP → writes to Supabase

This is the cleanest fit. The backend remains the single orchestrator (it already knows when each SP should be checked), Lambda is just a remote worker. Since Lambda writes to the DB directly, the synchronous HTTP result-passing problem from the original CF Worker proposal is eliminated entirely.

Option 2 — EventBridge Scheduler (cron per region)

Each regional Lambda gets an EventBridge rule that fires on an interval (e.g., every 5 minutes). Lambda wakes up, polls pg-boss for due work tagged to its region, runs it, writes results.

More decoupled — the backend doesn't need AWS credentials or knowledge of Lambda ARNs — but introduces polling overhead and means Lambda sometimes wakes up with nothing to do. Also requires pg-boss jobs to carry a region tag so each Lambda only picks up its own work.

Option 3 — SQS trigger per region

Backend pushes job specs to an SQS queue per region. Lambda is triggered by SQS events. Reliable, handles backpressure well, retries automatically on failure. Adds SQS as infrastructure and partially duplicates what pg-boss already does — worth considering if probe volume grows large, but over-engineered for a start.

The More Important Anonymization Lever

Worth saying plainly: piece selection is a bigger win than IP anonymization. An SP that's prioritizing dealbot traffic would need to know not just the source IP but also which pieces to prioritize. If you're checking random pieces from non-dealbot datasets (the whole point of the retrievalAnon check), the SP has no way to know which pieces will be checked next — even if they recognize the IP as a checker. Checking unknown pieces from an unknown IP is belt-and-suspenders. Checking unknown pieces from a known IP is still a significant improvement over the current state.

---

Recommended Incremental Path

The architecture across all options converges on the same Node.js worker code. That portability is the key insight: there is no need to solve geographic diversity and anonymous probing at the same time.

Step 1 — More Hetzner regions (now)
Deploy the existing worker to 2–3 additional Hetzner regions (e.g. Ashburn + Singapore on top of the current EU node). Pure ops work, zero code changes. Gets you geographic diversity immediately across EU, US, and APAC.

Step 2 — Non-dealbot piece selection (next)
Implement the piece selection logic for retrievalAnon checks — querying the FOC subgraph/indexer for pieces with ipfsRootCid across providers. This is the bigger anonymization win regardless of where probes run from.

Step 3 — Lambda for anonymous spot checks (later)
When you want Tier 2 anonymous checks from IPs outside the Hetzner ASN, deploy the same worker code to Lambda in the equivalent regions. The only delta is the Lambda entry point and deployment config. The backend scheduler invokes Lambda functions asynchronously via the AWS SDK — Lambda dequeues from pg-boss, probes, writes results to Supabase, no HTTP intermediary needed.

Each step is independently valuable and doesn't block the next.

---

Geographic Labeling of Metrics (issue #246)

The Core Requirement

Every metric and DB row needs a stable location label that answers "where did this probe run from?" in a way that is meaningful to end users and infrastructure-agnostic (works for Hetzner nodes, Lambda functions, and residential proxies alike).

Evaluation of the Options from issue #246

Hetzner location codes (fsn1, sin, ash)

• Too infrastructure-specific. Once Lambda or proxies are added, you'd have a mix of fsn1 and ap-southeast-1 in the same label — confusing and hard to compare.

AWS region names (us-east-1, ap-southeast-1)

• Same problem in reverse. Meaningless for Hetzner nodes. Locks metric vocabulary to AWS naming.

ISO 3166 country codes (USA, DEU, SGP)

• Not granular enough. The US has nodes on both coasts — "USA" loses that distinction, which matters for latency comparisons to US-East vs US-West SPs.

IATA airport codes (IAD, HEL, SIN, PDX)

• The right choice. City-level granular, infrastructure-agnostic, globally standardized, and already the de facto convention used by Cloudflare, Fastly, and other major infrastructure providers for exactly this purpose. Human-readable to anyone who travels.

Recommendation: IATA codes

Use two labels on every metric and DB row:

location — IATA code of the nearest major airport:

Deployment	Location	IATA
Hetzner Falkenstein	Frankfurt region	FRA
Hetzner Nuremberg	Nuremberg	NUE
Hetzner Helsinki	Helsinki	HEL
Hetzner Ashburn	Washington DC / Ashburn	IAD
Hetzner Hillsboro	Portland / Hillsboro	PDX
Hetzner Singapore	Singapore	SIN
Lambda eu-west-1	Dublin	DUB
Lambda us-east-1	Ashburn	IAD
Lambda ap-southeast-1	Singapore	SIN

Bandwidth and Compute Cost Comparison

The intuition that Lambda is more expensive for "fetching content" needs a small correction: for this workload, data flows from the SP into Lambda (dealbot is the downloader). AWS charges for egress (data leaving AWS to the internet), not ingress — so piece data flowing SP → Lambda is free on the AWS side. The egress from Lambda is negligible: the outbound HTTP request headers plus a few KB of results written to Supabase.

The real cost concern with Lambda is compute time, not bandwidth.

Lambda pricing (us-east-1 at 512 MB memory):

Probe scope	Duration	Cost per probe
First-block fetch (Tier 2)	~5s	~$0.00004
Full piece download, small piece	~60s	~$0.0005
Full piece download, large piece	~300s	~$0.0025

At 1,000 Tier 2 probes/day: ~$1.20/month. At 1,000 full-piece probes/day: ~$15–75/month depending on piece size.

Hetzner pricing:

A CX22 (2 vCPU, 4 GB RAM) costs ~€4.85/month and includes 20 TB of traffic. It runs continuously and can process probes back-to-back with no per-invocation cost. For high-frequency or long-duration probes, Hetzner is dramatically cheaper.

Conclusion: Lambda compute costs are acceptable for Tier 2 (short first-block checks). For Tier 1 thorough checks involving full piece downloads, Hetzner's flat pricing wins decisively. This reinforces the two-tier split: Hetzner for thorough checks, Lambda for lightweight anonymous spot checks.