docs: provider scoring methodology#628
Open
dennis-tra wants to merge 10 commits into
Open
Conversation
Contributor
Author
|
That's how that currently looks like (tbd if we want to keep the score there): 
I think the numbers look reasonable. GTE a score of 99 it's colored in green. Even if we're not showing the score in the table we may want to use it for sorting. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new documentation page describing how Dealbot ranks storage providers (SPs) for FOC, including the sort tiers and the Bayesian credible-lower-bound scoring approach.
Changes:
- Introduces
docs/provider-scoring.mdexplaining the four-tier ranking order (approval, data completeness, Bayesian score, provider ID). - Documents the Bayesian score calculation (Beta posterior 5th percentile) and the check weightings used to combine check types.
- Provides worked scoring scenarios to illustrate how rankings change under common provider performance patterns.
BigLep
requested changes
Jul 1, 2026
BigLep
left a comment
BigLep
left a comment
Contributor
There was a problem hiding this comment.
Thanks for giving critical thought here!
My comments are below. Generally was trying to think about ways to make it a bit simpler.
| | Priority | Criterion | Direction | What it means | | ||
| |---|---|---|---| | ||
| | 1 | Approved (yes/no) | Approved first | Providers that meet all approval criteria always appear above those that do not | | ||
| | 2 | Complete data (yes/no) | Complete first | Within each approval group, providers with data on every check rank above those with coverage gaps | |
Contributor
There was a problem hiding this comment.
I wonder how important this is. I think we can keep it simple and just use the Bayesian score, and let is be zero for checks where data is missing.
|
|
||
| One structural note: a provider with a data gap on any check cannot be approved (no samples means the sample-count criterion fails, which blocks approval). So the state "approved and has a data gap" cannot occur. The data gap criterion only differentiates providers within the non-approved group. | ||
|
|
||
| ## Concrete Scenarios |
Contributor
There was a problem hiding this comment.
This will need to update if we make "approved" based on onchain state and get rid of "complete" data.
github-project-automation
Bot
moved this from 🔎 Awaiting review
to ⌨️ In Progress
in FOC
…code Clarify that the ranking methodology is a proposed default and is not currently used by any code in this repo. It is intended as a guide for dashboards that want to give a sort order for SPs. Addresses review feedback from @BigLep on PR FilOzone#628. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Base the top-level "approved" sort criterion on the SP's on-chain approved status in the SP Registry, rather than on whether dealbot observes them currently meeting the approval acceptance criteria. Addresses review feedback from @BigLep on PR FilOzone#628. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the separate "complete data" sort tier. A missing check now contributes a zero lower bound to the weighted Bayesian score, so providers with coverage gaps rank below fully-covered peers through the score itself rather than a dedicated criterion. This simplifies the sort to three tiers (approved, score, provider ID) and removes the standalone "Data Gap Criterion" section, whose premise no longer holds. Worked examples and the probing-outage scenario are updated accordingly. Also removes the semicolon/capitalization issue noted by Copilot, as that sentence lived in the deleted section. Addresses review feedback from @BigLep and @Copilot on PR FilOzone#628. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The section claimed three planned checks but listed two, and described them as future/unlaunched. Pull and Sampled Retrieval already run in dealbot, so link to their docs and explain the real reason they are not scored yet: they are not part of the SP approval criteria. Addresses review feedback from @BigLep and @Copilot on PR FilOzone#628. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Change "recognises" to "recognizes" to match the American spelling used throughout the rest of the docs (e.g. "behavior", "optimize"). Addresses review feedback from @Copilot on PR FilOzone#628. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Approval is decided by an on-chain process; the dealbot check data guides that decision rather than setting it. Make this explicit under the criteria table and soften scenario 6, which implied dealbot itself "gates"/"blocks" approval. Per follow-up from @dennis-tra on PR FilOzone#628. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
I don't think it adds much value.
dennis-tra
force-pushed
the
provider-score
branch
from
e427ae0 to
a59874c
Compare
Contributor
Author
|
Thanks @BigLep for the review. I've incorporated your remarks. The diff to your previous review is here: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/provider-scoring.mddocumenting a default strategy for how storage providers can be ranked in FOC (not yet used by code in this repo; intended as a guide for dashboards)