Discuss potential pitfalls of concurrent upload (currently set to 3)

[skjnldsv]

[skjnldsv]

From nextcloud/server#43337

at the moment, uploading a large file is split into chunks, which get pushed to nextcloud server 3 at a time; i've found that 3 connections are not enough for me to max out my network line and think it would be great if this value is configurable via occ

@realies I'm really interested in this discussion! I chose 3 kind of randomly.
Do you have some informations or insights on which amount should be better?

[realies]

@skjnldsv, I would like to try with 10 concurrent chunks. What can I patch to try it out?

[skjnldsv]

You cannot, why 10 ? Is it just randomly chosen?
What uplink are you using? 1Gbps?

[realies]

why 'cannot'? 10, because iperf3 -c x.x.x.x -P 10 gave me optimal results; the link is 1Gbps

[skjnldsv]

Because this is compiled libraries, you cannot patch our server bundles

[MrRinkana]

If different uplinks perform optimally with different amount of chunks, shouldn't that number be dynamic rather than a compiled in constant?

[susnux]

You can try this with setting different values to:
https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/config_sample_php_parameters.html#files-chunked-upload-max-parallel-count

[MrRinkana]

Now I'm a bit confused how this works:

Is the files.chunked_upload.max_parallel_count how many chunks per file being uploaded? Such that with 3 files being uploaded in parallel its technically 15 chunks being uploaded (using default values)?

What are the benefits of increasing one vs the other (chunked upload vs # of files)?

The chunked_upload setting is a maximum value, how is the used value determined? I can know the uplink of the server but not the clients, will clients choose depending on their uplink speed, reliability (degree of packet loss) and latency? That would be the optimal as it has an large effect on the "efficiency" of the upload - but maybe most (all?) of such things are handled by the reverse proxy anyway? (relevant http2 discussion)

Does the files.chunked_upload.max_parallel_count even affect anything beyond the reverse proxy?

[susnux]

Is the files.chunked_upload.max_parallel_count how many chunks per file being uploaded? Such that with 3 files being uploaded in parallel its technically 15 chunks being uploaded (using default values)?

No it is how many parallel requests for uploads are allowed. If you set it to 3 and you do not have chunked uploads but 3 simple uploads then those 3 run in parallel.
If you have 2 upload with 2 chunks each then 3 of those 4 total chunks are uploaded.

Does the files.chunked_upload.max_parallel_count even affect anything beyond the reverse proxy?

This has nothing to do with a reverse proxy. It defined how many requests are spawned on the frontend.
And then it depends on how you configured your reverse proxy and your application server (how many handlers etc).

[MrRinkana]

Ah I see my mistake, thanks for clarifying!

During a single file chunked upload, are the chunks uploaded strictly sequentially (on the nc side)?
Such that any parallelism is only handled by the number of files uploading (3 as seen above), and protocol level thinks such as the http2/3 stuff?

[susnux]

No the chunks can be received in any order