Phil Dreizen

I have several hundred gigabytes worth of very small files I'm transferring from my laptop to my NAS server via an NFS mount over wifi. You'd think copying some (okay, a lot) of files around would be trivial, but it turned out not to be so easy.

My first attempt was perhaps naive. I simply used mv:

> mv $src_dir $dir_on_nas

But it was taking forever. Many hours later I gave up and decided to try again with a tar pipe. I tried something like:

> cd $src_dirtar cf - . | tar xf - -C $dir_on_nas

I ran it overnight and it did not get very far. In hindsight, I'm not sure why I thought this would be much different then using mv.

In any case, my next step was to take a look at what my transfer speed was. TIL about the pv which can measure the progress of data through a unix pipe. If you provide it the size of the data being transferred via the -s argument, it can also provide an ETA. And so the above tar pipe was modified to:

tar cf - . | pv -s $(du -sb . | awk '{print $1}') | tar xf - -C $dir_on_nas

pv displayed a paltry transfer rate between 1-2 MiB/s. This is just frustratingly slow, and explained why everything was taking so long.

My next step was to measure the network speed between my laptop and the NAS. TIL about iperf3, which sets up a socket that listens on one machine; other machines connect to the socket via iperf3 to measure the connection speed. (I should note, my NAS is a Synology, so installing iperf3 required jumping through a few hoops itself.).

#on the NAS
> iperf3 -s

#on my laptop...xxx is a stand in for the NAS ip
> iper3 -c 192.168.x.xxx

My transfer speed is not good: ~25MiB/s, but a lot better than ~1. Clearly, I need to troubleshoot the LAN itself, but for now, I'm going to leave it alone. At this point, I'd be happy with that kind of speed.

From the beginning I did suspect that the problem could be that I'm transferring a lot of small files. But I didn't think it would have this much of an impact. Cursory searches confirm that this is a pretty big problem, and I've found plenty of advice about various configuration changes I could make to the mount command to make the transfer go faster. But, if the issue really is just the number of files, I'm satisfied to transfer them over as a tarball and extract it later.

Before trying the single file tar, I decided to use dd to measure write speed to the NAS. Something like:

dd if=/dev/zero of=$dir_on_nas/test.dat count=100000

This showed a transfer speed of about 23MB/s, which is much more in line with iperf3. That really does suggest that I should be able to achieve that kind of speed if I packed up my smaller files into just 1 file.

So, this time, I tried transferring the files without extracting them (and notice I'm compressing here as well):

tar -zcf - . | pv -s $(du -sb $s | awk '{print $1}') > "$dir_on_nas/file.tar.gz" 

And yes pv is showing speeds of 22-23 MiB/s! Yay!

Except I should mention: I also tried this w/o compressing the data (leaving off the -z flag). Without compressing, pv reports the same transfer speed, but seemingly the transfer would "freeze" every few seconds. I still can't quite explain it. It has the "feel" of the pipe getting filled and the tar process getting blocked until the pipe empties, but wouldn't pv report a lower transfer rate as as result? Maybe not? I'm not sure.

In any case, now, with a single compressed file, the transfer speed is much faster now which is great.