Default Replication

[wjmelements]

We would like for storage contexts to upload to multiple storage providers by default. It should still be possible to select only one storage provider, so this will be configurable.

Without proof of replica, it is possible for replicated service providers to cheat proof of data possession. Therefore we plan to implement an endorsement system in the short term to certify availability. It should be possible to notice by latency if a service provider only has data possession by proxy.

My current plan to implement this is to have a replica count configuration parameter for createContext, and a wrapper replicated storage context that uses Promise.all on child context operations. This would need to handle individual SP failures gracefully. However because there is not a deadline on operations like AddPieces, it is still possible that a SP may succeed after we have determined that it failed, so simply replacing an apparently bad SP can result in paying for extra replicas.

I also want to make prior replica data sets somewhat discoverable in the same way that we discover your prior data sets. There are ways this can fail though. For example, a prior replicated data set may have been terminated or created with a different number of replicas. I don't have a good solution for this but I'll fallback to large existing relationships and use the new storage context preferences for new pieces.

Followups to this changeset:

• helper to identify user's piece CID duplicates and their statuses
• helper to boost (or repair) replication for previously uploaded pieces or data sets
• upload via URL for faster uploads (requires Curio change)
• prefer to use at least one endorsed SP by default

[rvagg]

@wjmelements please background here: #282

StorageContext was intentionally introduced to support multiple of them doing operations together. A context is supposed to be identify a singular relationship with a provider+dataset and perform operations on that singular thing. But the SDK should be able to handle multiple of them at the same time. So you set up multiple contexts and then use them together to get stuff done across providers/datasets. synapse.upload(bytes, contexts) kind of thing.

Documentation switches to tell you to const contexts = await synapse.storage.createContexts(); await synapse.storage.upload(blob, { contexts }) (by default you get two) and we soft deprecate createContext(), or even complain-deprecate it with a console.warn.

[timfong888]

https://github.com/FilOzone/fs-pm/issues/250

[timfong888]

Related to user story: FilOzone/fs-pm#250

[wjmelements]

createContext is fairly complex.

Current context creation selects the SP this way:

1. Use existing default context if exists and has matching metadata
2. elseif dataSetId is provided, use that data set
3. elseif providerId is provided, use that provider
4. elseif providerAddress is provided, use that provider
5. elseif user has any active data sets, use the one, among the existing datasets with matching metadata whose provider can pass a ping test, with the most pieces that has the least data set id
6. else pick a random provider who can pass a ping test

It may also create a data set, unless a suitable existing one was found. Data set resolution is conveyed via StorageContext.resolveProviderAndDataSet ProviderSelectionResult.dataSetId

Also, StorageCreationCallbacks may be called

[wjmelements]

I would like to avoid duplicating ProviderResolver.getApprovedProviders

[wjmelements]

because there is not a deadline on operations like AddPieces, it is still possible that a SP may succeed after we have determined that it failed, so simply replacing an apparently bad SP can result in paying for extra replicas.

We will use callbacks to signal progress to the app so that the user can abandon bad SPs themselves.

[rvagg]

@wjmelements the provider selection is the hardest part, here's two possibilities:

1. Just do it outside of createContext and then pass providerId in to the new context, then you have full control over which context gets which provider and no duplicates and all of that. Potentially we could then just rip provider selection out of StorageContext#create entirely and make it a separate action. I know Hugo wants to do this and I'd also like to move to a place where provider selection is pluggable so we can insert other processes.
2. Create contexts sequentially and then exclude selected providers from the next context create, have an excludeProvidersIds list that can be passed in so subsequent creates will go through the same selection routine but exclude ones that have already been selected for previous contexts already created. You could speed this up by caching the approved prover details to select from.

I think the first option might be the right path here. We're just remodelling the scope of a StorageContext's responsibilities - it's not the thing that chooses a provider, random selection happens elsewhere, a StorageContext is just an encapsulation of a relationship between client, provider & dataset.

Re callbacks, it may be simpler if we move to a single callback, multiple event-type model like Russel has been using in Filecoin Pin: https://github.com/filecoin-project/filecoin-pin/blob/ebaabd6951bad7329d004dfc5498eb2f2e97dcdc/src/core/upload/index.ts#L22-L41 (see further down in that file for use, things like onProgress?.({ type: 'configuring-allowances' }) but we can also decorate with other properties in the event.