Multiple parallell transfers?

[varenius]

Problem: For many destinations it seems transfers are limited per UDP/TCP stream. For example: OSO-->VIEN hits 1.4 Gbps, OSO-->GSI hits around 0.5 Gbps.

Reflection: However, running two etc-commands in parallell, I get 2 times 1.4 or 0.5 in these cases. This suggests that a conceptually easy way to improve the total transfer rate is to run multiple streams in parallell. In principle this is easy, because we normally transfers many files, so it's easy to parallellise by assigning one file per stream (up to som maximum). In practice, however, it is not practical to manually divide a filelist into different groups and start one etc-instance per group.

Idea: What if we could enable a number of parallell senders for each etc instance? If so, one could give a "--max-streams" argument, defaulting to 1, but which could be e.g. 5. In this case, if there are multiple files, etc would read the input file list and put them in an internal queue (as happens now, I assume). But, the queue-list would be eaten not by one but multiple sender processes. So, if there are 10 files and 5 streams allowed, the first 5 files would start in parallell. The first stream to finish would then start with file 6 etc.

Would this be possible? It would be a simple way to significantly increase the transfer speed in practice.