[BBLISA] BGP and multicast (thread renamed)

[Bill Bogstad]

On Tue, Jul 20, 2010 at 10:56 PM, Dean Anderson <dean at av8.com> wrote:
> ISC isn't that big. $2million doesn't really go that far. Especially
> when the CEO take 1/8th off the top.  The 5 top employees all make
> around 80k or so.  So, you can see by the financials on the IRS 990,
> that its a small operation.

For reference purposes, $250K is about what the Executive Director for
USENIX makes
(vaguely BBLISA relevant).  Their budget is 2-3x that.  OTOH, the
USENIX Executive Director is (as far as I know) not technically
trained.

>...
> You have seized on another myth of anycast proponents: Route's have to
> flap to have Anycast instability. Not so.  In fact, routes _don't_ need
> to be "flapping" for TCP anycast to be unstable.  Some years ago, it was
> common that routes were removed from cache every 60 seconds (by
> default).  (remember that's true at at every hop, so divide 60seconds by
> number of hops)

You are making a big assumption here.  Just because router A sees two
equal cost routes (X & Y) to destination D does not mean that router B
(the next hop router on route X) will see two distinct but equal cost
routes as well.  If you think of route announcements as propagating
outward from the anycast locations through the network graph, it's
only those locations which are precisely equidistant between two
different anycast locations which will ever see this problem.  If you
think of expanding balloons, it is only where the surface of the
balloons touch that this can happen.  Most people will probably be
inside a balloon and this "problem" won't matter to them.

>  But it gets worse: As the route table grows, routes are
> forced out of cache more frequently than 60 seconds. On replacement, any
> equal cost path can be inserted on a subsequent packet.  The more equal
> cost anycast paths you have anywhere in the path, the more likely
> subsequent packets will go to different servers.

If what you are saying is true, you don't even need to be doing
anycast to have route flapping based on this cache flushing issue.
All you need for this to happen is to be somewhere on the network
which has two equal length paths back to the same host.   Since it is
the same destination host, you might think that this wouldn't be
visible.  However, it will be visible in at least two different ways:

1. Equal length is unlikely to mean equal time.  This will result in
the destination seeing packets out of order.   UDP based applications
are likely to be confused and TCP itself will probably get confused as
well.  The symptoms are likely to be similar to what happens when you
have a split route (the route used on outgoing packets is different
from return packets).

2.  Multiple traceroutes will show two different routes depending on
which equal cost route was picked.  Most noticeable if you use
something like mtr (Matt's traceroute) which does continuous
traceroutes in a gui/screen oriented display.

I admit that I've never looked very hard either of these signs, but I
can't recall ever running across this myself or seeing any discussion
of it.  I vaguely recall that BGP sessions generally try to avoid
route flapping so I kind of wonder if maybe this cache flushing
doesn't really happen as much as you think it does.

>
> In our case, the public are deceived and lucky not to be using TCP DNS.
> But things are changing, and actually, it appears that fewer and fewer
> people think this is acceptable.  And in our case, the proponents are
> well aware that they are pushing
> hokum.

Is it hokum if it actually works (for years).  (Even if it violates
RFCs.)   I'm aware of situations where TCP DNS service  wasn't even
allowed for some services for quite some time.  Until the (very
recent) advent of actual DNSSEC deployment, judicious care with
selection of hostnames made it possible to always be able to respond
to DNS HOST queries with a single UDP response packet.  In that case,
anycast/route flapping is irrelevant.

Things change and other things stop working.  Most people won't pay
for "perfect solutions", they just want things to work NOW.  That's
life...

Bill Bogstad