From 67058e51915acb6e7573838d293d557a2a673f78 Mon Sep 17 00:00:00 2001 From: Dennis Trautwein Date: Mon, 29 Jun 2026 22:15:37 +0200 Subject: [PATCH 01/10] docs: provider scoring methodology --- docs/provider-scoring.md | 131 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 docs/provider-scoring.md diff --git a/docs/provider-scoring.md b/docs/provider-scoring.md new file mode 100644 index 00000000..ce942dcf --- /dev/null +++ b/docs/provider-scoring.md @@ -0,0 +1,131 @@ +# FOC Provider Scoring + +This document explains how storage providers (SPs) are ranked. + +## Overview + +Providers are sorted by four criteria evaluated left-to-right. The first criterion that differs between two providers determines their relative position while later criteria only matter when earlier ones are tied. + +| Priority | Criterion | Direction | What it means | +|---|---|---|---| +| 1 | Approved (yes/no) | Approved first | Providers that meet all approval criteria always appear above those that do not | +| 2 | Complete data (yes/no) | Complete first | Within each approval group, providers with data on every check rank above those with coverage gaps | +| 3 | Bayesian score (0–100) | Higher first | Confidence-adjusted weighted quality signal | +| 4 | Provider ID | Lower first | Deterministic tiebreaker when everything else is equal | + +## Bayesian Score + +### Why raw percentages aren't enough + +A raw success rate treats 97 % from 200 checks identically to 97 % from 2 000 checks. A newly onboarded provider that happened to have a perfect first two weeks looks identical to an established one with years of data. This conflates certainty with quality. + +The Bayesian score instead asks: *given what has actually been observed, what success rate can we be 95 % confident the provider truly sustains?* The answer is always lower than the observed rate but increasingly less so as the sample count grows. + +### How the score is calculated + +For each check type, the score is the **95 % Bayesian credible lower bound** on the provider's true success rate. The statistical model assumes no prior knowledge of the provider (a uniform prior), then updates based on the observed successes and failures: + +``` +lower bound = 5th percentile of Beta(1 + successes, 1 + failures) × 100 +``` + +In plain terms: given the observed data, there is a 95 % probability that the provider's true long-run success rate is at least this high. The bound is conservative by design. It represents a floor we can be confident in and not the most likely value. + +### Example Calculations + +| Observed rate | Sample count | Bayesian lower bound | +|---|---|---| +| 97.0 % | 200 | ~94.2 % | +| 97.0 % | 2 000 | ~96.3 % | +| 99.0 % | 200 | ~96.9 % | +| 99.0 % | 2 000 | ~98.6 % | +| 100.0 % | 200 | ~98.5 % | +| 100.0 % | 2 000 | ~99.9 % | + +Notice that the 97 % provider with 2 000 checks (lower bound ~96.3 %) and the 99 % provider with 200 checks (lower bound ~96.9 %) score nearly identically — the scoring correctly recognises them as about equally trustworthy despite the different observed rates. + +### Check weights + +The three active check types are combined as a weighted average of their individual lower bounds: + +| Check | Weight | Rationale | +|---|---|---| +| Data retention | 40 % | Data loss is the worst outcome for a storage service; ongoing proof-of-custody is hard to fake and has no client-side recovery path | +| Data retrieval | 35 % | Inability to serve stored data when a client needs it is an immediately user-visible failure with no retry on the client side | +| Data storage | 25 % | Upload failures are retryable; important as the end-to-end ingest signal but lower stakes than the read path | + +When a check type has no data at all (zero samples), its weight is redistributed proportionally to the remaining checks so the score stays meaningful and within the 0–100 range. + +### Planned future checks + +Three additional check types are planned for dealbot and will be incorporated into the score when they launch: + +| Check | Planned weight | Purpose | +|---|----------------|-------------------------------------------------------------------------------| +| Sampled retrieval | TBD | Retrieves real FWSS pieces held by the provider; not synthetic dealbot corpus | +| Pull pathway | TBD | Tests the SP pull workflow | + +## Data Gap Criterion + +The second sort criterion groups providers that are missing data on any active check below those with full coverage, within the same approval tier. + +Important context when dealbot has probing issues: when dealbot itself encounters a probing outage, the gap typically affects all providers simultaneously. This means no single provider is penalized relative to its peers. The approval status and Bayesian score already reflect the reduced data, and the coverage gap marker fires equally for everyone, leaving relative ordering unchanged. + +This criterion primarily matters when one provider genuinely has no data on a check while others do; For example, a newly onboarded provider that has not yet accumulated enough retention periods. + +One structural note: a provider with a data gap on any check cannot be approved (no samples means the sample-count criterion fails, which blocks approval). So the state "approved and has a data gap" cannot occur. The data gap criterion only differentiates providers within the non-approved group. + +## Concrete Scenarios + +### 1. Established approved provider + +- Retention: 0 faults / 3 000 periods → lower bound ~99.8 % +- Retrieval: 2 910 / 3 000 checks → lower bound ~96.2 % +- Storage: 2 910 / 3 000 checks → lower bound ~96.2 % +- **Bayesian score**: 0.40 × 99.8 + 0.35 × 96.2 + 0.25 × 96.2 ≈ **97.9** +- Sort position: approved · complete data · score 97.9 + +### 2. New provider, good rates, low sample count + +- Retention: 0 faults / 520 periods → lower bound ~98.5 % +- Retrieval: 194 / 200 checks → lower bound ~92.8 % +- Storage: 194 / 200 checks → lower bound ~92.8 % +- **Bayesian score**: 0.40 × 98.5 + 0.35 × 92.8 + 0.25 × 92.8 ≈ **95.6** +- Sort position: approved · complete data · score 95.6 — approved, but ranked below the established provider despite the same observed retrieval rate, because the lower sample count means we are less certain + +### 3. Borderline provider, exactly at the approval thresholds + +- Retention: 1 fault / 500 periods → lower bound ~97.9 % on the non-fault side +- Retrieval: 194 / 200 checks (97 %) → lower bound ~92.8 % +- Storage: 194 / 200 checks (97 %) → lower bound ~92.8 % +- **Bayesian score**: 0.40 × 97.9 + 0.35 × 92.8 + 0.25 × 92.8 ≈ **94.9** +- Sort position: approved · complete data · score 94.9 — approved but ranks last among approved providers + +### 4. Dealbot probing outage (storage checks unavailable for all providers) + +When dealbot cannot reach any provider for storage checks: + +- Storage weight (25 %) redistributes to retention and retrieval for every provider +- The coverage gap criterion fires equally for all providers — no one gains or loses ground relative to peers +- The Bayesian score still reflects retention and retrieval performance +- Relative ordering within each group remains unchanged + +### 5. Poor retrieval performance, strong retention + +A provider has reliable storage and retention but struggles to serve data back to clients. + +- Retention: 0 faults / 2 000 periods → lower bound ~99.7 % +- Retrieval: 1 600 / 2 000 checks (80 %) → lower bound ~78.3 % +- Storage: 1 960 / 2 000 checks (98 %) → lower bound ~97.1 % +- **Bayesian score**: 0.40 × 99.7 + 0.35 × 78.3 + 0.25 × 97.1 ≈ **91.2** +- Sort position: not approved (retrieval below threshold) · complete data · score 91.2 — the 35 % retrieval weight pulls the score down substantially despite the near-perfect retention + +### 6. High fault rate, ample samples + +A provider has accumulated many retention periods but consistently loses data. + +- Retention: 30 faults / 1 000 periods (3 % fault rate) → lower bound on non-fault side ~95.6 % +- Retrieval: 940 / 1 000 checks (94 %) → lower bound ~92.3 % +- Storage: 960 / 1 000 checks (96 %) → lower bound ~94.6 % +- **Bayesian score**: 0.40 × 95.6 + 0.35 × 92.3 + 0.25 × 94.6 ≈ **94.2** +- Sort position: not approved (fault rate well above 0.2 % threshold) · complete data · score 94.2 — a respectable-looking Bayesian score but blocked from approval by the hard retention fault gate; the score reflects the ample sample count narrowing the interval around a genuinely poor observed rate From ca66258bde2dac6754a45f480ead6927fcc4d4dd Mon Sep 17 00:00:00 2001 From: Dennis Trautwein Date: Thu, 2 Jul 2026 11:11:37 +0200 Subject: [PATCH 02/10] docs(provider-scoring): frame as a default strategy, not implemented code Clarify that the ranking methodology is a proposed default and is not currently used by any code in this repo. It is intended as a guide for dashboards that want to give a sort order for SPs. Addresses review feedback from @BigLep on PR #628. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/provider-scoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/provider-scoring.md b/docs/provider-scoring.md index ce942dcf..c4f02dd4 100644 --- a/docs/provider-scoring.md +++ b/docs/provider-scoring.md @@ -1,6 +1,6 @@ # FOC Provider Scoring -This document explains how storage providers (SPs) are ranked. +This document describes a default strategy for how storage providers (SPs) can be ranked. No code in this repo currently uses this methodology; it is intended as a guide for dashboards that want to give a sort order for SPs (e.g., https://probelab.io/filecoin/foc/). ## Overview From 815372137e3ab0ec800121e5ade9daaee81748fe Mon Sep 17 00:00:00 2001 From: Dennis Trautwein Date: Thu, 2 Jul 2026 11:14:22 +0200 Subject: [PATCH 03/10] docs(provider-scoring): rank approval by on-chain SP Registry status Base the top-level "approved" sort criterion on the SP's on-chain approved status in the SP Registry, rather than on whether dealbot observes them currently meeting the approval acceptance criteria. Addresses review feedback from @BigLep on PR #628. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/provider-scoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/provider-scoring.md b/docs/provider-scoring.md index c4f02dd4..1c3832ce 100644 --- a/docs/provider-scoring.md +++ b/docs/provider-scoring.md @@ -8,7 +8,7 @@ Providers are sorted by four criteria evaluated left-to-right. The first criteri | Priority | Criterion | Direction | What it means | |---|---|---|---| -| 1 | Approved (yes/no) | Approved first | Providers that meet all approval criteria always appear above those that do not | +| 1 | Approved (yes/no) | Approved first | Providers marked as approved in the on-chain SP Registry appear above those that are not | | 2 | Complete data (yes/no) | Complete first | Within each approval group, providers with data on every check rank above those with coverage gaps | | 3 | Bayesian score (0–100) | Higher first | Confidence-adjusted weighted quality signal | | 4 | Provider ID | Lower first | Deterministic tiebreaker when everything else is equal | From c5eba97d10be853d844f38931c50eadcc56451c2 Mon Sep 17 00:00:00 2001 From: Dennis Trautwein Date: Thu, 2 Jul 2026 11:16:24 +0200 Subject: [PATCH 04/10] docs(provider-scoring): fold data completeness into the Bayesian score Drop the separate "complete data" sort tier. A missing check now contributes a zero lower bound to the weighted Bayesian score, so providers with coverage gaps rank below fully-covered peers through the score itself rather than a dedicated criterion. This simplifies the sort to three tiers (approved, score, provider ID) and removes the standalone "Data Gap Criterion" section, whose premise no longer holds. Worked examples and the probing-outage scenario are updated accordingly. Also removes the semicolon/capitalization issue noted by Copilot, as that sentence lived in the deleted section. Addresses review feedback from @BigLep and @Copilot on PR #628. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/provider-scoring.md | 33 ++++++++++++++------------------- 1 file changed, 14 insertions(+), 19 deletions(-) diff --git a/docs/provider-scoring.md b/docs/provider-scoring.md index 1c3832ce..169772ad 100644 --- a/docs/provider-scoring.md +++ b/docs/provider-scoring.md @@ -4,14 +4,13 @@ This document describes a default strategy for how storage providers (SPs) can b ## Overview -Providers are sorted by four criteria evaluated left-to-right. The first criterion that differs between two providers determines their relative position while later criteria only matter when earlier ones are tied. +Providers are sorted by three criteria evaluated left-to-right. The first criterion that differs between two providers determines their relative position while later criteria only matter when earlier ones are tied. | Priority | Criterion | Direction | What it means | |---|---|---|---| | 1 | Approved (yes/no) | Approved first | Providers marked as approved in the on-chain SP Registry appear above those that are not | -| 2 | Complete data (yes/no) | Complete first | Within each approval group, providers with data on every check rank above those with coverage gaps | -| 3 | Bayesian score (0–100) | Higher first | Confidence-adjusted weighted quality signal | -| 4 | Provider ID | Lower first | Deterministic tiebreaker when everything else is equal | +| 2 | Bayesian score (0–100) | Higher first | Confidence-adjusted weighted quality signal; missing check data lowers it (see below) | +| 3 | Provider ID | Lower first | Deterministic tiebreaker when everything else is equal | ## Bayesian Score @@ -54,7 +53,7 @@ The three active check types are combined as a weighted average of their individ | Data retrieval | 35 % | Inability to serve stored data when a client needs it is an immediately user-visible failure with no retry on the client side | | Data storage | 25 % | Upload failures are retryable; important as the end-to-end ingest signal but lower stakes than the read path | -When a check type has no data at all (zero samples), its weight is redistributed proportionally to the remaining checks so the score stays meaningful and within the 0–100 range. +When a check type has no data at all (zero samples), its lower bound is treated as zero and it still contributes its full weight to the average. A provider missing data on a check therefore scores lower than an otherwise-equal provider with full coverage. This keeps the ranking simple — a single score handles both quality and data completeness, with no separate sort tier for coverage gaps. ### Planned future checks @@ -65,15 +64,11 @@ Three additional check types are planned for dealbot and will be incorporated in | Sampled retrieval | TBD | Retrieves real FWSS pieces held by the provider; not synthetic dealbot corpus | | Pull pathway | TBD | Tests the SP pull workflow | -## Data Gap Criterion +## Missing Check Data -The second sort criterion groups providers that are missing data on any active check below those with full coverage, within the same approval tier. +Because a missing check contributes zero to the Bayesian score (see [Check weights](#check-weights)), data gaps are handled by the score itself rather than a separate sort criterion. A provider that genuinely has no data on a check — for example, a newly onboarded provider that has not yet accumulated enough retention periods — scores lower than peers with full coverage. -Important context when dealbot has probing issues: when dealbot itself encounters a probing outage, the gap typically affects all providers simultaneously. This means no single provider is penalized relative to its peers. The approval status and Bayesian score already reflect the reduced data, and the coverage gap marker fires equally for everyone, leaving relative ordering unchanged. - -This criterion primarily matters when one provider genuinely has no data on a check while others do; For example, a newly onboarded provider that has not yet accumulated enough retention periods. - -One structural note: a provider with a data gap on any check cannot be approved (no samples means the sample-count criterion fails, which blocks approval). So the state "approved and has a data gap" cannot occur. The data gap criterion only differentiates providers within the non-approved group. +When dealbot itself encounters a probing outage, the gap typically affects all providers simultaneously. Every provider loses the same check contribution, so relative ordering within each group is unchanged even though absolute scores drop. ## Concrete Scenarios @@ -83,7 +78,7 @@ One structural note: a provider with a data gap on any check cannot be approved - Retrieval: 2 910 / 3 000 checks → lower bound ~96.2 % - Storage: 2 910 / 3 000 checks → lower bound ~96.2 % - **Bayesian score**: 0.40 × 99.8 + 0.35 × 96.2 + 0.25 × 96.2 ≈ **97.9** -- Sort position: approved · complete data · score 97.9 +- Sort position: approved · score 97.9 ### 2. New provider, good rates, low sample count @@ -91,7 +86,7 @@ One structural note: a provider with a data gap on any check cannot be approved - Retrieval: 194 / 200 checks → lower bound ~92.8 % - Storage: 194 / 200 checks → lower bound ~92.8 % - **Bayesian score**: 0.40 × 98.5 + 0.35 × 92.8 + 0.25 × 92.8 ≈ **95.6** -- Sort position: approved · complete data · score 95.6 — approved, but ranked below the established provider despite the same observed retrieval rate, because the lower sample count means we are less certain +- Sort position: approved · score 95.6 — approved, but ranked below the established provider despite the same observed retrieval rate, because the lower sample count means we are less certain ### 3. Borderline provider, exactly at the approval thresholds @@ -99,14 +94,14 @@ One structural note: a provider with a data gap on any check cannot be approved - Retrieval: 194 / 200 checks (97 %) → lower bound ~92.8 % - Storage: 194 / 200 checks (97 %) → lower bound ~92.8 % - **Bayesian score**: 0.40 × 97.9 + 0.35 × 92.8 + 0.25 × 92.8 ≈ **94.9** -- Sort position: approved · complete data · score 94.9 — approved but ranks last among approved providers +- Sort position: approved · score 94.9 — approved but ranks last among approved providers ### 4. Dealbot probing outage (storage checks unavailable for all providers) When dealbot cannot reach any provider for storage checks: -- Storage weight (25 %) redistributes to retention and retrieval for every provider -- The coverage gap criterion fires equally for all providers — no one gains or loses ground relative to peers +- The storage check contributes zero for every provider (its 25 % weight is applied to a zero lower bound), so every score drops by up to 25 points +- Because the gap affects all providers equally, no one gains or loses ground relative to peers - The Bayesian score still reflects retention and retrieval performance - Relative ordering within each group remains unchanged @@ -118,7 +113,7 @@ A provider has reliable storage and retention but struggles to serve data back t - Retrieval: 1 600 / 2 000 checks (80 %) → lower bound ~78.3 % - Storage: 1 960 / 2 000 checks (98 %) → lower bound ~97.1 % - **Bayesian score**: 0.40 × 99.7 + 0.35 × 78.3 + 0.25 × 97.1 ≈ **91.2** -- Sort position: not approved (retrieval below threshold) · complete data · score 91.2 — the 35 % retrieval weight pulls the score down substantially despite the near-perfect retention +- Sort position: not approved (retrieval below threshold) · score 91.2 — the 35 % retrieval weight pulls the score down substantially despite the near-perfect retention ### 6. High fault rate, ample samples @@ -128,4 +123,4 @@ A provider has accumulated many retention periods but consistently loses data. - Retrieval: 940 / 1 000 checks (94 %) → lower bound ~92.3 % - Storage: 960 / 1 000 checks (96 %) → lower bound ~94.6 % - **Bayesian score**: 0.40 × 95.6 + 0.35 × 92.3 + 0.25 × 94.6 ≈ **94.2** -- Sort position: not approved (fault rate well above 0.2 % threshold) · complete data · score 94.2 — a respectable-looking Bayesian score but blocked from approval by the hard retention fault gate; the score reflects the ample sample count narrowing the interval around a genuinely poor observed rate +- Sort position: not approved (fault rate well above 0.2 % threshold) · score 94.2 — a respectable-looking Bayesian score but blocked from approval by the hard retention fault gate; the score reflects the ample sample count narrowing the interval around a genuinely poor observed rate From 4b6c126352e21d316c0444ae1add78e564eedc9f Mon Sep 17 00:00:00 2001 From: Dennis Trautwein Date: Thu, 2 Jul 2026 11:18:37 +0200 Subject: [PATCH 05/10] docs(provider-scoring): correct the "not yet scored" checks section The section claimed three planned checks but listed two, and described them as future/unlaunched. Pull and Sampled Retrieval already run in dealbot, so link to their docs and explain the real reason they are not scored yet: they are not part of the SP approval criteria. Addresses review feedback from @BigLep and @Copilot on PR #628. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/provider-scoring.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/provider-scoring.md b/docs/provider-scoring.md index 169772ad..bcd74469 100644 --- a/docs/provider-scoring.md +++ b/docs/provider-scoring.md @@ -55,14 +55,14 @@ The three active check types are combined as a weighted average of their individ When a check type has no data at all (zero samples), its lower bound is treated as zero and it still contributes its full weight to the average. A provider missing data on a check therefore scores lower than an otherwise-equal provider with full coverage. This keeps the ranking simple — a single score handles both quality and data completeness, with no separate sort tier for coverage gaps. -### Planned future checks +### Checks not yet in the score -Three additional check types are planned for dealbot and will be incorporated into the score when they launch: +These additional check types already run in dealbot and could be incorporated into the score: -| Check | Planned weight | Purpose | -|---|----------------|-------------------------------------------------------------------------------| -| Sampled retrieval | TBD | Retrieves real FWSS pieces held by the provider; not synthetic dealbot corpus | -| Pull pathway | TBD | Tests the SP pull workflow | +* [Pull](checks/pull-check.md) +* [Sampled Retrieval](checks/sampled-retrievals.md) + +They aren't included currently because they aren't part of the [SP approval criteria](checks/production-configuration-and-approval-methodology.md). ## Missing Check Data From 90ea83f1458157ee51b73c99a50fbe3a5b4ea544 Mon Sep 17 00:00:00 2001 From: Dennis Trautwein Date: Thu, 2 Jul 2026 11:18:51 +0200 Subject: [PATCH 06/10] docs(provider-scoring): use American spelling consistently Change "recognises" to "recognizes" to match the American spelling used throughout the rest of the docs (e.g. "behavior", "optimize"). Addresses review feedback from @Copilot on PR #628. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/provider-scoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/provider-scoring.md b/docs/provider-scoring.md index bcd74469..a5c91433 100644 --- a/docs/provider-scoring.md +++ b/docs/provider-scoring.md @@ -41,7 +41,7 @@ In plain terms: given the observed data, there is a 95 % probability that the pr | 100.0 % | 200 | ~98.5 % | | 100.0 % | 2 000 | ~99.9 % | -Notice that the 97 % provider with 2 000 checks (lower bound ~96.3 %) and the 99 % provider with 200 checks (lower bound ~96.9 %) score nearly identically — the scoring correctly recognises them as about equally trustworthy despite the different observed rates. +Notice that the 97 % provider with 2 000 checks (lower bound ~96.3 %) and the 99 % provider with 200 checks (lower bound ~96.9 %) score nearly identically — the scoring correctly recognizes them as about equally trustworthy despite the different observed rates. ### Check weights From 83d96ee0d896d836b230af8838ed1ffc2272c59b Mon Sep 17 00:00:00 2001 From: Dennis Trautwein Date: Thu, 2 Jul 2026 11:35:45 +0200 Subject: [PATCH 07/10] docs(provider-scoring): custom wording changes --- docs/provider-scoring.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/provider-scoring.md b/docs/provider-scoring.md index a5c91433..08cedd14 100644 --- a/docs/provider-scoring.md +++ b/docs/provider-scoring.md @@ -53,7 +53,7 @@ The three active check types are combined as a weighted average of their individ | Data retrieval | 35 % | Inability to serve stored data when a client needs it is an immediately user-visible failure with no retry on the client side | | Data storage | 25 % | Upload failures are retryable; important as the end-to-end ingest signal but lower stakes than the read path | -When a check type has no data at all (zero samples), its lower bound is treated as zero and it still contributes its full weight to the average. A provider missing data on a check therefore scores lower than an otherwise-equal provider with full coverage. This keeps the ranking simple — a single score handles both quality and data completeness, with no separate sort tier for coverage gaps. +When a check type has no data at all (zero samples), its lower bound is treated as zero and it still contributes its full weight to the average. A provider missing data on a check therefore scores lower than an otherwise-equal provider with full coverage. This keeps the ranking simple as a single score handles both quality and data completeness. ### Checks not yet in the score @@ -66,7 +66,7 @@ They aren't included currently because they aren't part of the [SP approval crit ## Missing Check Data -Because a missing check contributes zero to the Bayesian score (see [Check weights](#check-weights)), data gaps are handled by the score itself rather than a separate sort criterion. A provider that genuinely has no data on a check — for example, a newly onboarded provider that has not yet accumulated enough retention periods — scores lower than peers with full coverage. +Because a missing check contributes zero to the Bayesian score (see [Check weights](#check-weights)), data gaps are handled by the score itself rather than a separate sort criterion. A provider that genuinely has no data on a check, for example, a newly onboarded provider that has not yet accumulated enough retention periods, scores lower than peers with full coverage. When dealbot itself encounters a probing outage, the gap typically affects all providers simultaneously. Every provider loses the same check contribution, so relative ordering within each group is unchanged even though absolute scores drop. From 1d426548e8f10c318e1ff98b87dd8b859f5a01ed Mon Sep 17 00:00:00 2001 From: Dennis Trautwein Date: Thu, 2 Jul 2026 11:41:27 +0200 Subject: [PATCH 08/10] docs(provider-scoring): clarify approval is an on-chain decision Approval is decided by an on-chain process; the dealbot check data guides that decision rather than setting it. Make this explicit under the criteria table and soften scenario 6, which implied dealbot itself "gates"/"blocks" approval. Per follow-up from @dennis-tra on PR #628. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/provider-scoring.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/provider-scoring.md b/docs/provider-scoring.md index 08cedd14..ebb1e970 100644 --- a/docs/provider-scoring.md +++ b/docs/provider-scoring.md @@ -12,6 +12,8 @@ Providers are sorted by three criteria evaluated left-to-right. The first criter | 2 | Bayesian score (0–100) | Higher first | Confidence-adjusted weighted quality signal; missing check data lowers it (see below) | | 3 | Provider ID | Lower first | Deterministic tiebreaker when everything else is equal | +Approval itself is an on-chain decision. The dealbot check data — and the [approval criteria](checks/production-configuration-and-approval-methodology.md) built on it — guides that decision rather than setting it directly; dealbot does not flip the approved flag. The scenarios below note the check signals that would typically inform whether a provider is approved. + ## Bayesian Score ### Why raw percentages aren't enough @@ -123,4 +125,4 @@ A provider has accumulated many retention periods but consistently loses data. - Retrieval: 940 / 1 000 checks (94 %) → lower bound ~92.3 % - Storage: 960 / 1 000 checks (96 %) → lower bound ~94.6 % - **Bayesian score**: 0.40 × 95.6 + 0.35 × 92.3 + 0.25 × 94.6 ≈ **94.2** -- Sort position: not approved (fault rate well above 0.2 % threshold) · score 94.2 — a respectable-looking Bayesian score but blocked from approval by the hard retention fault gate; the score reflects the ample sample count narrowing the interval around a genuinely poor observed rate +- Sort position: not approved (fault rate well above the 0.2 % approval threshold) · score 94.2 — a respectable-looking Bayesian score, but the fault rate is far outside the range that would guide an on-chain approval; the score reflects the ample sample count narrowing the interval around a genuinely poor observed rate From 7207053d2e1d33bf31f0b10fae2e4c0f7ff69965 Mon Sep 17 00:00:00 2001 From: Dennis Trautwein Date: Thu, 2 Jul 2026 11:43:19 +0200 Subject: [PATCH 09/10] docs(provider-scoring): remove sort position statement I don't think it adds much value. --- docs/provider-scoring.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/docs/provider-scoring.md b/docs/provider-scoring.md index ebb1e970..46017cb5 100644 --- a/docs/provider-scoring.md +++ b/docs/provider-scoring.md @@ -80,7 +80,6 @@ When dealbot itself encounters a probing outage, the gap typically affects all p - Retrieval: 2 910 / 3 000 checks → lower bound ~96.2 % - Storage: 2 910 / 3 000 checks → lower bound ~96.2 % - **Bayesian score**: 0.40 × 99.8 + 0.35 × 96.2 + 0.25 × 96.2 ≈ **97.9** -- Sort position: approved · score 97.9 ### 2. New provider, good rates, low sample count @@ -88,7 +87,6 @@ When dealbot itself encounters a probing outage, the gap typically affects all p - Retrieval: 194 / 200 checks → lower bound ~92.8 % - Storage: 194 / 200 checks → lower bound ~92.8 % - **Bayesian score**: 0.40 × 98.5 + 0.35 × 92.8 + 0.25 × 92.8 ≈ **95.6** -- Sort position: approved · score 95.6 — approved, but ranked below the established provider despite the same observed retrieval rate, because the lower sample count means we are less certain ### 3. Borderline provider, exactly at the approval thresholds @@ -96,7 +94,6 @@ When dealbot itself encounters a probing outage, the gap typically affects all p - Retrieval: 194 / 200 checks (97 %) → lower bound ~92.8 % - Storage: 194 / 200 checks (97 %) → lower bound ~92.8 % - **Bayesian score**: 0.40 × 97.9 + 0.35 × 92.8 + 0.25 × 92.8 ≈ **94.9** -- Sort position: approved · score 94.9 — approved but ranks last among approved providers ### 4. Dealbot probing outage (storage checks unavailable for all providers) @@ -115,7 +112,6 @@ A provider has reliable storage and retention but struggles to serve data back t - Retrieval: 1 600 / 2 000 checks (80 %) → lower bound ~78.3 % - Storage: 1 960 / 2 000 checks (98 %) → lower bound ~97.1 % - **Bayesian score**: 0.40 × 99.7 + 0.35 × 78.3 + 0.25 × 97.1 ≈ **91.2** -- Sort position: not approved (retrieval below threshold) · score 91.2 — the 35 % retrieval weight pulls the score down substantially despite the near-perfect retention ### 6. High fault rate, ample samples @@ -125,4 +121,3 @@ A provider has accumulated many retention periods but consistently loses data. - Retrieval: 940 / 1 000 checks (94 %) → lower bound ~92.3 % - Storage: 960 / 1 000 checks (96 %) → lower bound ~94.6 % - **Bayesian score**: 0.40 × 95.6 + 0.35 × 92.3 + 0.25 × 94.6 ≈ **94.2** -- Sort position: not approved (fault rate well above the 0.2 % approval threshold) · score 94.2 — a respectable-looking Bayesian score, but the fault rate is far outside the range that would guide an on-chain approval; the score reflects the ample sample count narrowing the interval around a genuinely poor observed rate From a59874c3f4299e03cb270580bf8496e92b8b312f Mon Sep 17 00:00:00 2001 From: Dennis Trautwein Date: Thu, 2 Jul 2026 11:47:34 +0200 Subject: [PATCH 10/10] docs(provider-scoring): tighten onchain approval wording --- docs/provider-scoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/provider-scoring.md b/docs/provider-scoring.md index 46017cb5..299c2e11 100644 --- a/docs/provider-scoring.md +++ b/docs/provider-scoring.md @@ -12,7 +12,7 @@ Providers are sorted by three criteria evaluated left-to-right. The first criter | 2 | Bayesian score (0–100) | Higher first | Confidence-adjusted weighted quality signal; missing check data lowers it (see below) | | 3 | Provider ID | Lower first | Deterministic tiebreaker when everything else is equal | -Approval itself is an on-chain decision. The dealbot check data — and the [approval criteria](checks/production-configuration-and-approval-methodology.md) built on it — guides that decision rather than setting it directly; dealbot does not flip the approved flag. The scenarios below note the check signals that would typically inform whether a provider is approved. +Approval itself is an on-chain decision. The dealbot check data and the [approval criteria](checks/production-configuration-and-approval-methodology.md) guide that decision. The [scenarios below](#concrete-scenarios) note the check signals that would typically inform whether a provider is approved. ## Bayesian Score