docs: provider scoring methodology

docs/provider-scoring.md

@@ -0,0 +1,123 @@
+# FOC Provider Scoring
+
+This document describes a default strategy for how storage providers (SPs) can be ranked. No code in this repo currently uses this methodology; it is intended as a guide for dashboards that want to give a sort order for SPs (e.g., https://probelab.io/filecoin/foc/).
+
+## Overview
+
+Providers are sorted by three criteria evaluated left-to-right. The first criterion that differs between two providers determines their relative position while later criteria only matter when earlier ones are tied.
+
+| Priority | Criterion | Direction | What it means |
+|---|---|---|---|
+| 1 | Approved (yes/no) | Approved first | Providers marked as approved in the on-chain SP Registry appear above those that are not |
+| 2 | Bayesian score (0–100) | Higher first | Confidence-adjusted weighted quality signal; missing check data lowers it (see below) |
+| 3 | Provider ID | Lower first | Deterministic tiebreaker when everything else is equal |
+
+Approval itself is an on-chain decision. The dealbot check data and the [approval criteria](checks/production-configuration-and-approval-methodology.md) guide that decision. The [scenarios below](#concrete-scenarios) note the check signals that would typically inform whether a provider is approved.
+
+## Bayesian Score
+
+### Why raw percentages aren't enough
+
+A raw success rate treats 97 % from 200 checks identically to 97 % from 2 000 checks. A newly onboarded provider that happened to have a perfect first two weeks looks identical to an established one with years of data. This conflates certainty with quality.
+
+The Bayesian score instead asks: *given what has actually been observed, what success rate can we be 95 % confident the provider truly sustains?* The answer is always lower than the observed rate but increasingly less so as the sample count grows.
+
+### How the score is calculated
+
+For each check type, the score is the **95 % Bayesian credible lower bound** on the provider's true success rate. The statistical model assumes no prior knowledge of the provider (a uniform prior), then updates based on the observed successes and failures:
+
+```
+lower bound = 5th percentile of Beta(1 + successes, 1 + failures) × 100
+```
+
+In plain terms: given the observed data, there is a 95 % probability that the provider's true long-run success rate is at least this high. The bound is conservative by design. It represents a floor we can be confident in and not the most likely value.
+
+### Example Calculations
+
+| Observed rate | Sample count | Bayesian lower bound |
+|---|---|---|
+| 97.0 % | 200 | ~94.2 % |
+| 97.0 % | 2 000 | ~96.3 % |
+| 99.0 % | 200 | ~96.9 % |
+| 99.0 % | 2 000 | ~98.6 % |
+| 100.0 % | 200 | ~98.5 % |
+| 100.0 % | 2 000 | ~99.9 % |
+
+Notice that the 97 % provider with 2 000 checks (lower bound ~96.3 %) and the 99 % provider with 200 checks (lower bound ~96.9 %) score nearly identically — the scoring correctly recognizes them as about equally trustworthy despite the different observed rates.
+
+### Check weights
+
+The three active check types are combined as a weighted average of their individual lower bounds:
+
+| Check | Weight | Rationale |
+|---|---|---|
+| Data retention | 40 % | Data loss is the worst outcome for a storage service; ongoing proof-of-custody is hard to fake and has no client-side recovery path |
+| Data retrieval | 35 % | Inability to serve stored data when a client needs it is an immediately user-visible failure with no retry on the client side |
+| Data storage | 25 % | Upload failures are retryable; important as the end-to-end ingest signal but lower stakes than the read path |
+
+When a check type has no data at all (zero samples), its lower bound is treated as zero and it still contributes its full weight to the average. A provider missing data on a check therefore scores lower than an otherwise-equal provider with full coverage. This keeps the ranking simple as a single score handles both quality and data completeness.
+
+### Checks not yet in the score
+
+These additional check types already run in dealbot and could be incorporated into the score:
+
+* [Pull](checks/pull-check.md)
+* [Sampled Retrieval](checks/sampled-retrievals.md)
+
+They aren't included currently because they aren't part of the [SP approval criteria](checks/production-configuration-and-approval-methodology.md).
+
+## Missing Check Data
+
+Because a missing check contributes zero to the Bayesian score (see [Check weights](#check-weights)), data gaps are handled by the score itself rather than a separate sort criterion. A provider that genuinely has no data on a check, for example, a newly onboarded provider that has not yet accumulated enough retention periods, scores lower than peers with full coverage.
+
+When dealbot itself encounters a probing outage, the gap typically affects all providers simultaneously. Every provider loses the same check contribution, so relative ordering within each group is unchanged even though absolute scores drop.
+
+## Concrete Scenarios
+
+### 1. Established approved provider
+
+- Retention: 0 faults / 3 000 periods → lower bound ~99.8 %
+- Retrieval: 2 910 / 3 000 checks → lower bound ~96.2 %
+- Storage: 2 910 / 3 000 checks → lower bound ~96.2 %
+- **Bayesian score**: 0.40 × 99.8 + 0.35 × 96.2 + 0.25 × 96.2 ≈ **97.9**
+
+### 2. New provider, good rates, low sample count
+
+- Retention: 0 faults / 520 periods → lower bound ~98.5 %
+- Retrieval: 194 / 200 checks → lower bound ~92.8 %
+- Storage: 194 / 200 checks → lower bound ~92.8 %
+- **Bayesian score**: 0.40 × 98.5 + 0.35 × 92.8 + 0.25 × 92.8 ≈ **95.6**
+
+### 3. Borderline provider, exactly at the approval thresholds
+
+- Retention: 1 fault / 500 periods → lower bound ~97.9 % on the non-fault side
+- Retrieval: 194 / 200 checks (97 %) → lower bound ~92.8 %
+- Storage: 194 / 200 checks (97 %) → lower bound ~92.8 %
+- **Bayesian score**: 0.40 × 97.9 + 0.35 × 92.8 + 0.25 × 92.8 ≈ **94.9**
+
+### 4. Dealbot probing outage (storage checks unavailable for all providers)
+
+When dealbot cannot reach any provider for storage checks:
+
+- The storage check contributes zero for every provider (its 25 % weight is applied to a zero lower bound), so every score drops by up to 25 points
+- Because the gap affects all providers equally, no one gains or loses ground relative to peers
+- The Bayesian score still reflects retention and retrieval performance
+- Relative ordering within each group remains unchanged
+
+### 5. Poor retrieval performance, strong retention
+
+A provider has reliable storage and retention but struggles to serve data back to clients.
+
+- Retention: 0 faults / 2 000 periods → lower bound ~99.7 %
+- Retrieval: 1 600 / 2 000 checks (80 %) → lower bound ~78.3 %
+- Storage: 1 960 / 2 000 checks (98 %) → lower bound ~97.1 %
+- **Bayesian score**: 0.40 × 99.7 + 0.35 × 78.3 + 0.25 × 97.1 ≈ **91.2**
+
+### 6. High fault rate, ample samples
+
+A provider has accumulated many retention periods but consistently loses data.
+
+- Retention: 30 faults / 1 000 periods (3 % fault rate) → lower bound on non-fault side ~95.6 %
+- Retrieval: 940 / 1 000 checks (94 %) → lower bound ~92.3 %
+- Storage: 960 / 1 000 checks (96 %) → lower bound ~94.6 %
+- **Bayesian score**: 0.40 × 95.6 + 0.35 × 92.3 + 0.25 × 94.6 ≈ **94.2**