[BBLISA] Faster than 1G Ether ... ESX to ZFS

[wayne blair]

There are other costs/considerations to ESX nic bonding.
1) Traffic over the bonded links is not bi-directional until you do more
switch and vkernel configuation.
2) You will have differe behavior to consider depending upon if
your're using mac or src/destination IPs

Yes, it stinks that the cost to boldly go beyond 10+ year old technology is
so high but other hassels and complexities go away when you do not need to
resort to convuluted configurations.
I think the perceived costs to upgrade are not being offset by the many
other costs that disappear (such as the ESX details mentioned above and
others).

But beware, your bottle neck might move away from your 1gb ethernet to some
place else, perhaps the spindles of your nifty ZFS storage system (a nice
problem to have....)

-w.b.



On Fri, Nov 12, 2010 at 12:00 PM, <bblisa-request at bblisa.org> wrote:

> Send bblisa mailing list submissions to
>        bblisa at bblisa.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>        http://www.bblisa.org/mailman/listinfo/bblisa
> or, via email, send a message with subject or body 'help' to
>        bblisa-request at bblisa.org
>
> You can reach the person managing the list at
>        bblisa-owner at bblisa.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of bblisa digest..."
>
>
> Today's Topics:
>
>   1. Faster than 1G Ether ... ESX to ZFS (Edward Ned Harvey)
>   2. Re: Faster than 1G Ether ... ESX to ZFS (Tim Winter)
>   3. Re: Faster than 1G Ether ... ESX to ZFS (John Orthoefer)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 12 Nov 2010 09:21:26 -0500
> From: Edward Ned Harvey <bblisa3 at nedharvey.com>
> Subject: [BBLISA] Faster than 1G Ether ... ESX to ZFS
> To: <bblisa at bblisa.org>
> Message-ID: <000501cb8274$e75b7920$b6126b60$@nedharvey.com>
> Content-Type: text/plain; charset="us-ascii"
>
> Since combining ZFS storage backend, via nfs or iscsi, with ESXi heads, I'm
> in love.  But for one thing.  The interconnect between the head & storage.
>
>
>
> 1G Ether is so cheap, but not as fast as desired.  10G ether is fast
> enough,
> but it's overkill and why is it so bloody expensive?  Why is there nothing
> in between?  Is there something in between?  Is there a better option?  I
> mean . sata is cheap, and it's 3g or 6g, but it's not suitable for this
> purpose.  But the point remains, there isn't a fundamental limitation that
> *requires* 10G to be expensive, or *requires* a leap directly from 1G to
> 10G.  I would very much like to find a solution which is a good fit. to
> attach ZFS storage to vmware.
>
>
>
> Any suggestions?
>
>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://www.bblisa.org/pipermail/bblisa/attachments/20101112/54ebe2a9/attachment.html
>
> ------------------------------
>
> Message: 2
> Date: Fri, 12 Nov 2010 10:30:21 -0500
> From: Tim Winter <tim at gigageek.net>
> Subject: Re: [BBLISA] Faster than 1G Ether ... ESX to ZFS
> To: Edward Ned Harvey <bblisa3 at nedharvey.com>
> Cc: bblisa at bblisa.org
> Message-ID:
>        <AANLkTikB+EB+4o-O3upZxOVygWMkEButuaiP=Qxnwvox at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> What about bonding multiple 1G interfaces?
>
>
> On Fri, Nov 12, 2010 at 9:21 AM, Edward Ned Harvey <bblisa3 at nedharvey.com
> >wrote:
>
> > Since combining ZFS storage backend, via nfs or iscsi, with ESXi heads,
> I?m
> > in love.  But for one thing.  The interconnect between the head &
> storage.
> >
> >
> >
> > 1G Ether is so cheap, but not as fast as desired.  10G ether is fast
> > enough, but it?s overkill and why is it so bloody expensive?  Why is
> there
> > nothing in between?  Is there something in between?  Is there a better
> > option?  I mean ? sata is cheap, and it?s 3g or 6g, but it?s not suitable
> > for this purpose.  But the point remains, there isn?t a fundamental
> > limitation that **requires** 10G to be expensive, or **requires** a leap
> > directly from 1G to 10G.  I would very much like to find a solution which
> is
> > a good fit? to attach ZFS storage to vmware.
> >
> >
> >
> > Any suggestions?
> >
> >
> >
> > _______________________________________________
> > bblisa mailing list
> > bblisa at bblisa.org
> > http://www.bblisa.org/mailman/listinfo/bblisa
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://www.bblisa.org/pipermail/bblisa/attachments/20101112/3cfcd49e/attachment-0001.htm
>
> ------------------------------
>
> Message: 3
> Date: Fri, 12 Nov 2010 10:39:39 -0500
> From: John Orthoefer <jco at direwolf.com>
> Subject: Re: [BBLISA] Faster than 1G Ether ... ESX to ZFS
> To: Back Bay LISA <bblisa at bblisa.org>
> Message-ID: <0F4F520B-B17A-42C6-A5D5-E669AE02C5F7 at direwolf.com>
> Content-Type: text/plain; charset=windows-1252
>
> Between the two is link bonding/etherchannel.   Which is gets expensive in
> it's self, since you are using multiple switch ports and card slots to
> increase your link speed.   A quad GigE card is about $500 and uses a PCIe
> port, then 4 ports on your switch is going to be say 60$/port, so to get 4G
> out of a machine you are looking at $750.   The card and a 10G port cost
> about $700 and then a switch with ports you are looking at about $1200/port
> for a switch to plug it into, so call it $2k/10G.
>
> In general terms you 3xdollars buys you 10xspeed, without all the side
> arguments about rated speed and bus rates etc.
>
> My basic though is that in general if 1G isn't enough goto 10G UNLESS you
> know you'll only need say 2-4G uplink it will save you from revisiting
> things in a year or two.
>
> You can also look at what you might be able to do to squeeze a bit more
> performance out of 1G till the 10G comes down more, if the money really is a
> show stopper.  Look at using Jumbo packets, and using a a P-t-P link (plug a
> straight though cable into each device without a switch, which BTW you can
> do with 10G too) for your traffic, make sure nothing but file access are
> going over your link etc...
>
> johno
>
> On Nov 12, 2010, at 9:21 AM, Edward Ned Harvey wrote:
>
> > Since combining ZFS storage backend, via nfs or iscsi, with ESXi heads,
> I?m in love.  But for one thing.  The interconnect between the head &
> storage.
> >
> > 1G Ether is so cheap, but not as fast as desired.  10G ether is fast
> enough, but it?s overkill and why is it so bloody expensive?  Why is there
> nothing in between?  Is there something in between?  Is there a better
> option?  I mean ? sata is cheap, and it?s 3g or 6g, but it?s not suitable
> for this purpose.  But the point remains, there isn?t a fundamental
> limitation that *requires* 10G to be expensive, or *requires* a leap
> directly from 1G to 10G.  I would very much like to find a solution which is
> a good fit? to attach ZFS storage to vmware.
> >
> > Any suggestions?
> >
> > _______________________________________________
> > bblisa mailing list
> > bblisa at bblisa.org
> > http://www.bblisa.org/mailman/listinfo/bblisa
>
>
>
> ------------------------------
>
> _______________________________________________
> bblisa mailing list
> bblisa at bblisa.org
> http://www.bblisa.org/mailman/listinfo/bblisa
>
> End of bblisa Digest, Vol 84, Issue 6
> *************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.bblisa.org/pipermail/bblisa/attachments/20101112/1364f76e/attachment.htm