[BBLISA] Faster than 1G Ether ... ESX to ZFS

[Bill Bogstad]

On Wed, Nov 17, 2010 at 11:38 AM, Daniel Hagerty <hag at linnaean.org> wrote:
> Bill Bogstad <bogstad at pobox.com> writes:
>
>> variable latency in the general case.  This can screwup video
>> streaming or VOIP pretty badly.  You are in a single datacenter and
>> aren't using those protocols, so we can hope it won't matter to you.
>> (Be sure to test this. :-)
>
>    Out of order delivery is pretty easy to get with or without
> variable speed of light delays.  Queueing and packet sizing are each
> happy to provide on their own.
>
>    TCP doesn't deal well with it, at least if you're mentioning it in
> the same breath as performance.  This is probably so for anything that
> has to restitch something that has been fragemented and scrambled.

Good point.  I hadn't considered the fact that unsynchronized queues
are likely to generate out of order reception.   In a datacenter,
direct host port to host port cabling will eliminate any switch
queuing issues.   Light speed issues on cables running next to each
other are probably not an issue. :-)   Unfortunately, even if all host
ports involved are dedicated to a single bi-directional netflow
between the machines, host OS queueing and the priority of interrupt
processing will probably result in the network layer on the
destination machine seeing packets out of order.   Many of the things
that have been done to improve the performance of OS networks stacks
is going to make seeing packets out of order more likely:  processing
multiple packets per interrupt on reception, allowing multiple CPUs to
process incoming packets, queuing multiple outgoing packets to an
interface per operation.   At this point, I wouldn't even try to guess
what performance will be.   One possibility could even be lower
performance when using multiple links; you could end up with lock
contention on the receiver as multiple CPUs try to insert packets into
the correct stream.  Lots of fun....

Bill Bogstad