[BBLISA] Fwd: Re: Fileserver opinion

[Ian Stokes-Rees]

> Ian> 2x4core 2.4 GHz Intel Nehalems
> 
> Why are you going for extra cores?  Or have you looked at Opterons/AMD
> solutions as well?  You don't need killer CPU performance from a file
> server, but lots of bandwidth to the disks/network.  


It doesn't add that much to the price (a few hundred), and gives us the
option of running VMs on it if we find network latency is hurting us,
and accessing disk local to the VM host through the VM guest is better.


> Ian> 3U 16 bay chassis w/ SAS2 extender for expansion storage shelves
> 
> What kind of power supplies are you getting here?  And are they
> redundant and hot-swappable by any chance?  

1200W dual, hot-swappable.

> Ian> 24 GB RAM
> 
> Seems like overkill for a file server, because you don't want to cache
> that much of your data in RAM unless you have a rock solid UPS, etc.
> And RAM isn't cheap. 4 to 8 would be a better fit in my book.


We've been told by people we trust that more is better.  Also see
comment on VMs above.


> Ian> 4 TB RAID10 storage + hot swap (SATA 6 GB/s, 10k RPM drives)
> Ian> 1.2 TB RAID10 storage + hot swap (SAS2 6 GB/s, 15k RPM drives)
> Ian> 2 port TOE Intel GigE NIC
> 
> Which interface will the GigE ports be on?  Remember, that's 120MB/sec
> per port MAXIMUM bandwidth.  Even if you bond them together, you'll be
> pushed to get more than 200MB/sec through them.  Which isn't chicken
> feed mind you!  But I suspect alot of your traffic will be NFS
> get_attr calls and such, so being able to keep alot of the inode cache
> in RAM might be helpful there for lookups like that.  This does go
> against my recomendation above for RAM, but not really since the inode
> cache isn't all that big.

This is easy to upgrade if 2xGigE is not sufficient.  IB has also been
suggested, but I think this is unlikely.

> Ian> Adaptec MaxIQ SAS2 RAID controller with 64 GB SSD read cache and BBU
> 
> What interface is it using?  PCIe 4x, 8x or what?  And can you afford
> a second one (instead of extra RAM) for better redundancy and
> performance.  You want to spread your IO load across as many
> controllers/spindles that you can afford. 

PCIe 2.0 8x.  No, we can't afford a second one.

> Ian> 600 GB SATA 10k RPM system drive
> 
> Why not a pair (or three with one hot spare) of mirrored smaller
> drives for the OS?  

Possibly.

> You also haven't mentioned which OS you want to run.  Have you looked
> at FreeNAS?  It has ZFS builtin on a FreeBSD core.  Seems solid,
> though a little annoying to administer to actually create volumes
> until you wrap your brain around their setup.

Probably CentOS + ext3 or maybe xfs, since we have the most experience
with this.  Others have suggested Solaris variants and to go with zfs.

> And in that case, I'd just get a pair of CF cards and boot the system
> off Flash, which reduces where things can go wrong.  
> 
> Also, how will user accounts be mapped on this sucker?  NIS, LDAP?
> There's another area to think about.  With a box this beefy, I'd make
> it an NIS slace or an LDAP replica at the drop of a hat.  It's got
> tons of spare performance.

LDAP.


> As for filesystems, I'm comfortable with ext3, thinking about ext4 or
> xfs (leaning more this way) on linux and ZFS on FreeBSD.  


See above.

Thanks for your comments.  They are all appreciated.

Ian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ijstokes.vcf
Type: text/x-vcard
Size: 383 bytes
Desc: not available
Url : http://www.bblisa.org/pipermail/bblisa/attachments/20100826/8950c0b4/attachment-0001.vcf