[BBLISA] ZFS Anyone?

[Dewey Sasser]

Hello all,

I am in need of more, FAST storage. 



The Brief Background:

I have 6 beefy VMWare ESX hosts (DL380, 2xQuad, 32GB), soon to expand to
9 (and all to go up to 48GB), on which I'm running VMWare ESX 3.5
managed by LabManager (current capacity 180 virtual machines, expanding
to around 400 with host and RAM expansion).  My storage so far has
managed to usually keep up but occasionally it falls off a cliff.  I've
3 months of IOStat numbers but don't believe I have a very clear idea of
where said cliff edge really is.

I currently have a 20 disk HP DL380 running RedHat EL 5.3 and have some
stability problems which significantly interfere with performance tuning
(right now my heavy load is on a single 6 disk RAID 10 and every time I
try to resize the partition the volume hangs.)

I'm supporting development and test machines for a 40 person development
group.  This system is *NOT* customer facing but is important for
engineering productivity.

I've been talking to various VARs about NetApp, EquiLogic and Left Hand
storage systems and the price/GB seems...high.  It's also clear that my
existing system performance is pretty darn good for the base tech (990
IOps average in 50% write load on 7.2k SAS drives).

We've recently started thinking about deploying a Sun ZFS based storage
system, either on a "build it ourselves" or "buy a 7000 series storage
box" basis.  I've read through Sun's "this is the best thing since the
transistor" slide presentation and a fair amount of the admin manual,
and it looks fairly cool and I can see some reasons it might be better
than LVM2 on RAID.



The Questions:

    * Does anyone have experience with ZFS?  What has your experience been?
    * Is it stable/reliable/etc.  Honestly, before this experience LVM2
      has been rock solid for me across many deployments.
    * What kind of performance should I expect?
    * What are the worst/most annoying parts of dealing with ZFS?
    * Why should I not use it?
    * Why should I use it?

Thanks,

--
Dewey