[BBLISA] Fwd: Moving 100 GB and 1.3 million files

[Dean Anderson]

On Thu, 29 Jul 2010, Ian Stokes-Rees wrote:

> > I think consensus was that things will be slow as long as you're creating a
> > million small files.  Because the small sync writes will destroy
> > performance.  But you can minimize that impact by enabling the writeback
> > cache.
> 
> I have not done exhaustive testing of the various suggestions, however
> by dividing the 250k files into sub-directories, rather than have them
> all in a single directory, improved the tar creation rate by a factor of
> about 200 (from ~1200s to 7s, for 250k files, 570 MB of data, and taking
> up 1.2 GB of disk space).

This is what I expected. :-)

> Interestingly, on the RAID 5 system tar file creation takes 4.5
> minutes. Anyone want to hazard a guess why this would be 40x slower?

Possibly the disks that are part of the raid system are impressively
slow with no cache. (software or h/w raid system?)  RAID systems aren't
always faster and can be often be slower, but 40x seems unusually slow.  
It could also be that there is something wrong.

		--Dean
 

-- 
Av8 Internet   Prepared to pay a premium for better service?
www.av8.net         faster, more reliable, better service
617 256 5494