[BBLISA] Fwd: Re: Fileserver opinion

[John Stoffel]

Ian> On 8/26/10 1:19 PM, Rudie, Tony wrote:
>> I would recommend an NFS appliance in this case, (and actually in every
>> case, mine included) either from NetApp or Sun or someone else.  Simpler
>> than dealing with a general purpose OS.   You still have to deal with a
>> new platform for monitoring and backups, but everything else will be a
>> little easier than building it yourself. 


Ian> We have a $10k budget, so I don't think we can afford any
Ian> "appliance" solution, unless someone knows otherwise.  For $10k
Ian> it looks like we can get:

Ian> 2x4core 2.4 GHz Intel Nehalems

Why are you going for extra cores?  Or have you looked at Opterons/AMD
solutions as well?  You don't need killer CPU performance from a file
server, but lots of bandwidth to the disks/network.  

Ian> 3U 16 bay chassis w/ SAS2 extender for expansion storage shelves

What kind of power supplies are you getting here?  And are they
redundant and hot-swappable by any chance?  

Ian> 24 GB RAM

Seems like overkill for a file server, because you don't want to cache
that much of your data in RAM unless you have a rock solid UPS, etc.
And RAM isn't cheap. 4 to 8 would be a better fit in my book.

Ian> 4 TB RAID10 storage + hot swap (SATA 6 GB/s, 10k RPM drives)
Ian> 1.2 TB RAID10 storage + hot swap (SAS2 6 GB/s, 15k RPM drives)
Ian> 2 port TOE Intel GigE NIC

Which interface will the GigE ports be on?  Remember, that's 120MB/sec
per port MAXIMUM bandwidth.  Even if you bond them together, you'll be
pushed to get more than 200MB/sec through them.  Which isn't chicken
feed mind you!  But I suspect alot of your traffic will be NFS
get_attr calls and such, so being able to keep alot of the inode cache
in RAM might be helpful there for lookups like that.  This does go
against my recomendation above for RAM, but not really since the inode
cache isn't all that big.

Ian> Adaptec MaxIQ SAS2 RAID controller with 64 GB SSD read cache and BBU

What interface is it using?  PCIe 4x, 8x or what?  And can you afford
a second one (instead of extra RAM) for better redundancy and
performance.  You want to spread your IO load across as many
controllers/spindles that you can afford. 

Ian> 600 GB SATA 10k RPM system drive

Why not a pair (or three with one hot spare) of mirrored smaller
drives for the OS?  

You also haven't mentioned which OS you want to run.  Have you looked
at FreeNAS?  It has ZFS builtin on a FreeBSD core.  Seems solid,
though a little annoying to administer to actually create volumes
until you wrap your brain around their setup.

And in that case, I'd just get a pair of CF cards and boot the system
off Flash, which reduces where things can go wrong.  

Also, how will user accounts be mapped on this sucker?  NIS, LDAP?
There's another area to think about.  With a box this beefy, I'd make
it an NIS slace or an LDAP replica at the drop of a hat.  It's got
tons of spare performance.

As for filesystems, I'm comfortable with ext3, thinking about ext4 or
xfs (leaning more this way) on linux and ZFS on FreeBSD.  

John