[BBLISA] Fileserver opinion

[Bill Bogstad]

On Wed, Aug 11, 2010 at 1:55 PM, Ian Stokes-Rees
<ijstokes at crystal.harvard.edu> wrote:
>
> Diligent readers will recall the thread a few weeks ago on slow disk
> performance with a PATA XRaid system from Apple (HFS, RAID5).  Having
> evaluated the situation, we're looking to get a new file server that
> combines some fast disk with some bulk storage.  We have a busy web
> server that is mostly occupied with serving static content (read only
> access), some dynamic content (Django portal with mod_python/httpd), and
> then scientific compute users who do lots of writes (including a 100
> core cluster).
>...
>[LOTS of details about hardware ideas, etc.]
>...
>
> 4. How can we estimate our IOPs and throughput requirements?

I think this is THE most important question.   All the other answers
are completely dependent on this one.  You need to attach specific
numbers (with error bars) to current usage as well as estimate future
changes.  "busy web server", "some dynamic content", "lots of writes"
are based on experience/context.

I wish I could give you specific suggestions on tools to gather this
information, but that's going to be very dependent on your situation.
 One generic thing, I would suggest is to analyze the log files
for your web server.   You want to get an idea on what the "working
set" size is for the web site.  If the number is small enough you
should consider memory caching or possibly SSDs in the web server
itself rather then doing something on the file server.

Good Luck,
Bill Bogstad