What is the most efficient way to copy a bunch of files nowadays?

What is the most efficient way to copy a bunch of files nowadays?

[Olivier Galibert]

After having read-only mucked with the headers of a bunch of files[1]
I've selected a subset of 2500 or so with sizes going from 2K to 85M
which I want to copy to another directory in the same local
filesystem.  What is the "best" (CPU usage, fragmentation, wall clock
time, system responsiveness during and after the copy) way to copy
these files?  read+write, mmap+write, read+mmap, mmap+mmap+memcpy,
something else?  That's with recent kernels, of course.

  OG.

[1] Say rpms from a random distribution and its updates for instance

Re: What is the most efficient way to copy a bunch of files nowadays?

[Jan Engelhardt]

>After having read-only mucked with the headers of a bunch of files[1]
>I've selected a subset of 2500 or so with sizes going from 2K to 85M
>which I want to copy to another directory in the same local
>filesystem.  What is the "best" (

>CPU usage

nice

>fragmentation

What? You can't really influence it.

>wall clock time

Using a filesystem with aggressive caching.

>system responsiveness during and after the copy)

See one above. Set /proc/sys/vm/dirty_ratio high to have the flush taking 
place late, or low to have it "early+often".

>way to copy these files?  read+write, mmap+write, read+mmap, 
>mmap+mmap+memcpy,

It all boils down to the same: read the file and write it back. So all 
your methods cost equally much. With splice() instead of read/write 
however, you may save some cycles.

>something else?  That's with recent kernels, of course.


Jan Engelhardt
--