[BBLISA] Fwd: Moving 100 GB and 1.3 million files

[David Miller]

Another factor is the number of files in a folder.  As the number of files
per folder increase the time required to create a new file in that directory
can slow down significantly.

Also what protocol are you using to send the files from one server to the
other?
--
David

On Thu, Jul 22, 2010 at 2:43 PM, Ian Stokes-Rees <
ijstokes at crystal.harvard.edu> wrote:

>
>
> On 7/22/10 2:34 PM, Rudie, Tony wrote:
> > As a couple of people have said, it's not the gear that's the
> > problem, it's all those little files.  But following in your
> > footsteps, doing the same calculation based on 1.3 million files, we
> > get:
> >
> > 50% done = 650K files, in 20 * 3600 seconds = 9 files per second.
> > That seems low as well.  I just unpacked a tar file with 1000 files
> > in it in 3 seconds.
>
> I should have included those numbers in my first email.  Yes, I've
> looked at those, and they seem stupidly bad, like the system is running
> orders of magnitude slower than it should.
>
> What keeps frustrating me is that no one can describe to me or point me
> at a web page that describes the tools and techniques required to figure
> out what is going on.
>
> Ian
>
> _______________________________________________
> bblisa mailing list
> bblisa at bblisa.org
> http://www.bblisa.org/mailman/listinfo/bblisa
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.bblisa.org/pipermail/bblisa/attachments/20100722/c4e0d4f3/attachment.htm