StorageManager: Recover from permanent upload failure

[wjmelements]

It is possible for upload to fail repeatedly. This can happen if a SP runs out of space. Ideally the SP notices before this happens and expands their capacity, but it can still happen. More generally, if we fail in the upload step, the SDK's default path (StorageManager) would not succeed on retry.

Even if the user refreshes the webpage or restarts the app, there are a few reasons our createContext(s) function might reassign the same SP and then fail again, one being that selection prefers SPs with an existing dataset.

One possible workaround could be for the storage manager to treat this as a permanent failure and make a new StorageContext, excluding the full SP.

[wjmelements]

Seems that, after #446, it is harder to distinguish failures. I see there is a TODO to put provider info into the callbacks. Maybe that should happen first, since we previously discussed conveying failure through callbacks.

[wjmelements]

I also see that we don't convey failure through callbacks yet.

[wjmelements]

An additional difficulty is that our error handling is not as precise as I expected.

The possibly-permanent errors I had in mind are caught and wrapped here:

throw createError('StorageContext', 'uploadPiece', 'Failed to upload piece to service provider', error)

and then caught and wrapped again:

const finalError = createError('StorageContext', 'addPieces', 'Failed to add piece to data set', error)

[BigLep]

@rvagg : do you have a comment on the priority of this issue?

[rjan90]

Rod has not read through it fully yet, but believe this will be fixed by the new upload flow. So putting this as TBD for now