[BBLISA] Fwd: Moving 100 GB and 1.3 million files

[Dean Anderson]

I notice that the ganglia report shows a _very_ high cpu load on
macintell4 during the previous 20 or so hours. I think that is a clue to
the problem, and points to the the protocol (and perhaps implmentation)  
as the culprit. I suggest transfering the files by do something similar
to:

   tar czBf - . | ssh someuser at macintell4 '(cd somewhere; tar xzBpf -)'

or cpio if you prefer. This will only create one process on the remote 
system. 

While all the suggestions I've seen so far are good, it still seems to
be too slow to be explained by slow disk and synchonous writes. Another
thing that comes to mind is to ask if all the files are in one
directory. Directories are linearly searched for file name slots. When
you create the file on the remote side, the directory is searched for a
free slot (and extended), meaning the whole thing has to be read.  The
bigger they are, the slower they get. Imagine having to read a megabyte+
file for each file transferred. And remember that directories are files,
too.  But a million+ files in a single directory won't create a high
system load as ganglia reports. You'll just see high disk i/o rates and
seemingly nothing getting done.

Good luck.

		--Dean

On Thu, 22 Jul 2010, Ian Stokes-Rees wrote:

> 
> 
>  I have a question regarding expectations for file movement between
> disks on adjacent servers.
> 
> Due to a sub-optimal file system layout, I regularly have to move lots
> of files between file systems.  The servers are in the same rack, or at
> least in racks next to each other, and I am fairly certain they are all
> connected to the same GB switch.
> 
> Moving blocks of ~300k files totaling about 5-10 GB takes hours to
> complete.  Yesterday afternoon I started a move of 1.3 million files
> totaling about 94 GB.  20 hours later the transfer seems to be less than
> half done.
> 
> Does this surprise anyone?  Any hints as to what might be wrong or what
> might speed it up?  I'm at a loss to know where to start looking.
> 
> Regards,
> 
> Ian
> 
> 
> 
> More details, for those who are interested:
> 
> The files at the origin are on RAID1 SATA disks (1 TB, ext3, Seagate
> Barracuda 7200 RPM), and I have a ganglia snapshot of the 24 hour status
> (you can see the start of the transfer about 20 hours ago):
> 
> http://dl.dropbox.com/u/1561496/shared/abitibi-origin.pdf
> 
> The destination is an Apple X-RAID array (4TB) connected to an Apple
> XServe.  The corresponding ganglia snapshot is here:
> 
> http://dl.dropbox.com/u/1561496/shared/macintel-destination.pdf
> 
> 

-- 
Av8 Internet   Prepared to pay a premium for better service?
www.av8.net         faster, more reliable, better service
617 256 5494

<