Currently FSBucket.put_object_stream() performs the read of the next buffer only after the previous buffer was read, which leads to a sequential processing.
If the input stream is another IO-bounded input, that in lot of the cases will be S3 bucket, this can be sensibly improved by using producer-consumer pattern or buffered I/O with a queue.
Usage of asyncio (as oposite to an additional thread) might be slightly more efficient since it doesn't perform expensive thread context-switches. I bet GPT would do this easy.
Currently
FSBucket.put_object_stream()performs the read of the next buffer only after the previous buffer was read, which leads to a sequential processing.If the input stream is another IO-bounded input, that in lot of the cases will be S3 bucket, this can be sensibly improved by using producer-consumer pattern or buffered I/O with a queue.
Usage of
asyncio(as oposite to an additional thread) might be slightly more efficient since it doesn't perform expensive thread context-switches. I bet GPT would do this easy.