[BBLISA] 10GBE NICS

[Daniel Feenberg]

We want to PXE boot a dozen compute and file servers with over 10GBE 
ethernet. All of them boot fine with the motherboard NICs. We have
a Brocade Ironport switch and a dozen direct attach cables. We also
have  samples of 3 brands of 10GBE NICs to test.

1) The HP card boots correctly when PXE is enabled on the NIC.

2) Chelsio N320E gives a "PXE-E61 - Media Failure" error and
    the NIC link light never comes on.

3) Brocade 1010 - says "Adaptor 1/0 link initialization failed.
    Disabling BIOS" or "No target devices or link down or init
    failed" depending on the NIC BIOS setting. Again, the NIC link
    light does not come on.

We discount a bad cable (it works with the HP, and we have tried
several) or a motherboard incompatibility (if we boot RHEL from
a local drive and enable eth2 we can use the Chelsio or Brocade
cards). Is there some configuration issue we are missing? Chelsio
support did not offer a solution, we haven't contacted Brocade yet.

The motherboard is a Gigabyte GA-P55M-UB2. All the cards have the
latest firmware. Since the failure occurs before any packets are
sent it can't be a dhcpd or tftp problem. Is the problem that some
cards offer less than full support for direct attach? Or is direct
attach not fully standardized? Should be try fiber optic cables? The
documentation for all the cards suggests that their primary purpose
is ethernet SANs. Perhaps the vendors don't care about other uses?

We only need a single port per server, while the HP offers 2. Because
of heat, power and cost reasons, we would prefer a single port card.

Any wisdom greatly appreciated. This is our first experience with 10GBE.
We could boot over 1Gb and then switch to 10GBE for file service, but
we wish to reduce the amount of cabling.

Two odd details - even booting from the local drive the Chelsio
card fails on intensive use if IRQPOLL is enabled. An additional
advantage of the HP card is that brief network interuptions do
not affect it, while the Chelsio card will hang the computer if
the switch reboots or a cable is moved from one port to another.

Daniel Feenberg
NBER