[BBLISA] Fwd: Moving 100 GB and 1.3 million files

[Jeff Wasilko]

On Thu, Jul 22, 2010 at 02:43:50PM -0400, Ian Stokes-Rees wrote:
> 
> 
> On 7/22/10 2:34 PM, Rudie, Tony wrote:
> > As a couple of people have said, it's not the gear that's the
> > problem, it's all those little files.  But following in your
> > footsteps, doing the same calculation based on 1.3 million files, we
> > get:
> > 
> > 50% done = 650K files, in 20 * 3600 seconds = 9 files per second.
> > That seems low as well.  I just unpacked a tar file with 1000 files
> > in it in 3 seconds.
> 
> I should have included those numbers in my first email.  Yes, I've
> looked at those, and they seem stupidly bad, like the system is running
> orders of magnitude slower than it should.
> 
> What keeps frustrating me is that no one can describe to me or point me
> at a web page that describes the tools and techniques required to figure
> out what is going on.

to investigate, look at iostat. Raid5 on ATA disks is not going to be
fast.

If your array card supports write cache and it's disabled for some
reason, that would hurt. I'd check that.