[BBLISA] Faster than 1G Ether ... ESX to ZFS

[Bill Bogstad]

On Tue, Nov 16, 2010 at 8:52 PM, Edward Ned Harvey
<bblisa4 at nedharvey.com> wrote:
>> From: Sean Lutner [mailto:slutner at rentul.net]
>...
>
> I just can't bear the idea of optimizing all my 6Gbit disks for performance,
> only to funnel them all across a 1Gbit bottleneck.

Your message yesterday about point-to-point bandwidth limitations got
me started thinking (and googling) about this.   It seems like most of
the methods for utilizing multiple network links for more then just
redundancy use some kind of hash function to determine which link a
particular packet is sent over.   This works fairly well if you have
many-to-many network flows.  Unfortunately that's not what you have.
With a single destination address, you are going to end up using a
single link being used due to most hashing schemes just looking at the
addresses of the packets.  What you want is a simple round robin
scheme to spray outgoing packets across both interfaces.  BTW, the
reason must systems don't do round robin is that network protocols
prefer to receive packets in order and with consistent latency.   With
multiple active links per network flow, you get out of order and
variable latency in the general case.  This can screwup video
streaming or VOIP pretty badly.  You are in a single datacenter and
aren't using those protocols, so we can hope it won't matter to you.
(Be sure to test this. :-)

If this was pure Linux-to-Linux, you could look into equal cost
multi-path routing.  Here's a random link that I found

http://etutorials.org/networking/integrated%2Bcisco%2Band%2Bunix%2Bnetwork%2Barchitectures/chapter%2B8.%2Bstatic%2Brouting%2Bconcepts/equal-cost%2Bmulti-path%2Becmp%2Brouting/)

If both servers are setup with machine local subnets, you should be
able to use multipath to force use of the redundant paths equally for
routing packets between the two machine local subnets.   Note: This
could still fail due to issues with Linux ARPing on a per machine
rather then per interface for its IP addresses.   This can screw you
up by having all of your per-link IP addresses associated with a
single Ethernet address and Ethernet spanning tree is only going to
use one incoming link on the destination as a result.    Manually
setting up the ARP tables on the machines is probably the easiest way
to deal with this.   Dedicated wiring/separate VLANs/distinct subnets
per link could all be useful/needed as well.   Feel free to burn lots
of 10.*.*.* non-routable subnets to get this to work.

I also tried looking into iSCSI multipath and haven't figured out if
that would be useful or not.  Still it bears more investigation.

If no single VM on the compute server needs more then 1Gbit you might
be able to associate each one with a separate IP address and give your
file server multiple IPs as well.

In general, I would suggest starting by thinking about what goes
across the wire(s) and working your way up through the
hardware/protocol stack and see at what point you can introduce a
mapping between a single higher level endpoint to multiple lower level
endpoints.  Then check to make sure nothing merges the lower level
streams until you want it.  (i.e. Linux's weird ARPing.)

Finally, all of the above is based on "reading knowledge".  I haven't
tried it myself.   (I also don't know about Solaris/ESXi
capabilities.)

Good Luck,
Bill Bogstad