Better clean up of blobs

[julianstirling]

Blobs currently live for a fixed time on the server. This means that any blob that creates a sizeable chunk of data that is run often could waste a fair amount of space.

Proposal

Delete the blob when it is read with GET, unless the data attached to the specific GET call specifies that the blob should remain.

Effects

The HTTP API tends to be used for two use cases - The UI, programming clients. It is worth considering these separately.

From a Programming client

I cannot think of many cases where a programming client would want to get the data multiple times (or multiple clients get the data). I don't deny there may be a specific use case, but the ability to request that the blob persists allows someone writing a client to do this if there edge case allows.

UI

I think it is worth considering short and long running tasks separately.

• Short tasks - (For example: Capturing an image)
  • If I have a client open and capture an image I don't expect a different client to be able to access it
  • If I select capture and then close the client before it finishes, I don't expect I will ever get that image (It will in this case sit on the server for 5 mins)
  • If I have 2 clients open, or open a second one mid-capture I don't expect the second client should receive the image as well
• Long-running task - (For example: Sample scanning) - Returning a blob for such a long running task is a bad idea in the first place as there is a high chance that no client is still open to receive the blob on completion. We will also always want the UI to reconnect to long-running actions, which means that the chance of multiple clients being sent a blob is high, which always seems like it is confusing.

All of these cases imply that the blob being deleted on first GET is fine.

For the short task we probably (In OpenFlexure Microscope) will need to make sure that short tasks don't auto-connect to ongoing actions as a second client could receive the image of captured by another client. This is already weird behaviour, but would be doubly bad if this issue is implemented as the client that captured may not actually receive the requested image.

[rwb27]

It is worth being clear here that the retention mechanism is via invocations: the lifetime is associated with that, and not with the .Blob itself.

If an action is started over HTTP and returns a .Blob, it will be stored as part of the Invocation's output property. By default, invocation objects persist for 5 minutes after they have finished. This is, in part, intended to allow the client time to poll the invocation and download any blobs. A fixed timeout is clearly not great memory management and coming up with something better would be nice. This is especially important when the output contains Blob data and may therefore be quite large.

Because it's the action's output, and not the individual Blob, that is retained, I'm not wild on deleting individual Blobs as it would then mean we have actions with broken output properties. I'd strongly prefer to delete the Invocation instead.

The easiest way to implement this would be to provide a DELETE endpoint that allows clients to explicitly discard it early. Clearly that minimises the likelihood of anyone being surprised by something disappearing, but leaves open the possibility that clients don't clean up after themselves and leave data hanging around.

Setting a limit on the number, or total size, of invocation outputs held in memory would be safer. The obvious way to do this would be a FIFO arrangement where older outputs get deleted to make room for newer ones. We could also do something like not starting new actions when there are too many outputs retained. That would provide some "back pressure" to encourage clients to delete outputs that are no longer needed - but I suspect that's more likely to cause annoyance than to help...

A safer take on this might be to use a much shorter timeout, but provide some sort of keep-alive mechanism. That would preserve the output as long as a client asks for it, but allow it to be deleted as soon as it's no longer needed. A keep-alive mechanism might also be nice if we ever properly support Blob data as inputs. I think my favourite option would be to have client-side Blob objects keep the server-side Blob data alive until they go out of scope or download the data, at which point either an explicit DELETE or a short timeout would get rid of it. That is yet more HTTP overhead though, and makes things even more complicated.

For what it's worth I'm not sure either of the above options is supported by the Web of Things standard, but then it's not like Blob is a part of that standard either. It is, however, very much the case that you need something like Blon to return binary data: the standard pretty much only deals with JSON and requires you to link out to binary data.

One option we used to have, but I never got around to implementing in the rewrite, was returning binary data directly. This is appropriate for short actions that return exactly one binary output (like taking a picture). If the accept header indicates a response of the correct media type would be OK, then instead of returning 201 and requiring the client to poll for completion, the response to the POST request invoking the action would be a 200 response where the content is the binary data directly. In that case, there would be no need to retain the invocation object at all. Clearly this is very simple and efficient when it works, but it's not universal (which is why I implemented the polling version first). The standard requires that you can always request a JSON response.

I know it's unlikely we'll get two GET requests for one Blob, but nonetheless it feels strange to delete stuff as a result of a GET. It's especially tricky for actions that output multiple Blob objects, because then we'd need to coordinate between them before deleting the invocation object to ensure they've all been downloaded. The Blob also doesn't know what Invocation it belongs to, though I'm sure that could be figured out.

Perhaps the answer is that we shouldn't be returning blob objects from actions we expect people to call most of the time, and anything that needs to return binary data should implement a way of managing that data (like the gallery in microscope v2). Much of the mechanism of Blob can still be useful even if the data is managed in files in a gallery folder, rather than temp files or bytes objects. My experience with the v2 API was that that got very tedious very quickly when people were doing lots of automated stuff from client scripts, and that the direct method left less junk on the server but was rather less reliable. I'm not sure what the solution looks like - probably more discussion needed...

[rwb27]

To the problem of multiple clients: there's probably a case to make that they shouldn't be able to see each other's actions. That is in a large part a security model question, and I suspect neither of us has the energy for that in the near future. We may come back to it if we end up with locks that can be held by a client, though.

[rwb27]

I'll also mention websockets here for compleness. That seems to be a direction that might be helpful for our UI in the future, and it also means there's a long-lived connection that would be ideal for keeping stuff alive while the client wants it and deleting it afterwards. I don't immediately see an obvious solution, but it feels like a continuous connection should make this easier.

[julianstirling]

To the problem of multiple clients: there's probably a case to make that they shouldn't be able to see each other's actions. That is in a large part a security model question, and I suspect neither of us has the energy for that in the near future. We may come back to it if we end up with locks that can be held by a client, though.

If I connect to a microscope that is already scanning, or I am connected to one already and someone starts scanning, then I need to be able to see it is scanning even if it was started in a different client. I think security has to be a layer before that. I either can't get to the microscope because I am not authorised, or I can and I can see what it is doing. I think we need to plan it more carefully than that.

As to the more detailed explanation above. That is a lot of useful context. I hadn't realised it is the invocation holding the blob not the blob-manager. (I see blob manager is no-more in one of the MRs). This clarifies quite a bit. I think explicitly deleting the invocation is an option, but as you say it requires people to remember to delete it after they have read it. Would it be possible to have a POST request for any completed invocation which is the standard way to get the output and cleans it up perhaps

POST /invocations/{id}/collect-and-delete

We could have a specific GET request for the blob, which doesn't delete everything but try to strongly encourage using the collect-and-delete POST request?