[BBLISA] BGP and multicast (thread renamed)

[Dean Anderson]

On Wed, 21 Jul 2010, Bill Bogstad wrote:

> On Tue, Jul 20, 2010 at 10:56 PM, Dean Anderson <dean at av8.com> wrote:
> > ISC isn't that big. $2million doesn't really go that far. Especially
> > when the CEO take 1/8th off the top.  The 5 top employees all make
> > around 80k or so.  So, you can see by the financials on the IRS 990,
> > that its a small operation.
> 
> For reference purposes, $250K is about what the Executive Director for
> USENIX makes
> (vaguely BBLISA relevant).  Their budget is 2-3x that.  OTOH, the
> USENIX Executive Director is (as far as I know) not technically
> trained.

Indeed, according to Guidestar, Ellie Young took home ~$256,000 in
compensation, on 3.7million in revenue.  This is also scandolously high
for a charity---And its sort of like defending Bernie Ebers by saying
Madoff stole more; that's a pretty weak defense.  However, there is key
a difference: As far as I can tell and remember, Ellie Young is not a
founder of Usenix.  The law prohibits _founders_ from recieving a
benefit.  So the scandal is on the judgement of the Usenix Board of
Directors.  But the law objective is to prevent someone from founding a
non-profit to benefit themsselves with money from charity.  Paul Vixie
is a founder of ISC.

As a point of reference, the highest paid FSF employee is Peter Brown
(controller), at ~$78,000.  Charities usually pay less (often much less)
than comparable jobs at for-profit companies.  But even a for-profit
company don't give 1/8 of its gross revenue as compensation to the CEO.  
Managing a few people and a $2million budget just isn't that tough.

> You are making a big assumption here.  Just because router A sees two
> equal cost routes (X & Y) to destination D does not mean that router B
> (the next hop router on route X) will see two distinct but equal cost
> routes as well.

Of course not. And I didn't say A's view implied B's view. You assert a
fallacy about my statements.

However, both A and B _can_ have equal cost routes to the D. The more
anycast instances of D you have, the more likely that A and B will both
have equal cost routes to D


> if you think of route announcements as propagating outward from the
> anycast locations through the network graph, it's only those locations
> which are precisely equidistant between two different anycast
> locations which will ever see this problem.  If you think of expanding
> balloons, it is only where the surface of the balloons touch that this
> can happen.  Most people will probably be inside a balloon and this
> "problem" won't matter to them.

Err no. Every hop sees every possible balloon. To give an example, if
you distributed a hundred anycast instances in the internet, pretty soon
everyone is 6 hops away from several.  2 hops from two, 3 hops from 8, 4
hops from 16, 5 hops from 33.  Repeat this at every router. What Vixie
has been trying to do recently is ensure one ISP can only see one
anycast instance.

[Of course, the Anycast instance itself can still send packets to the
world, so if you run the DNSSEC 100x amplification attack on the anycast
roots from a reaonably randomly distrubuted botnet, all 100+ instances
see the packets coming in from the botnet (which is sending to only 1 IP
address), and all 100 instances respond to the queries. Those responses
still reach the unicast target address. So now you have 100 high
bandwidth servers on 10Gig connections sending 100x amplification. Of
course this is a different problem from TCP stability, but an anycast
problem nonetheless and one that can't be filtered by target or by root
server. The only solution is to track down the botnet, and that's
usually hard.


> >  But it gets worse: As the route table grows, routes are forced out
> > of cache more frequently than 60 seconds. On replacement, any equal
> > cost path can be inserted on a subsequent packet.  The more equal
> > cost anycast paths you have anywhere in the path, the more likely
> > subsequent packets will go to different servers.
> 
> If what you are saying is true, you don't even need to be doing
> anycast to have route flapping based on this cache flushing issue. All
> you need for this to happen is to be somewhere on the network which
> has two equal length paths back to the same host.  Since it is the
> same destination host, you might think that this wouldn't be visible.  
> However, it will be visible in at least two different ways:

If they go to the same unicast host, there is no problem.  At most,
packets can arrive out of order. (and yes, that does happen)

> 1. Equal length is unlikely to mean equal time.  This will result in
> the destination seeing packets out of order.  UDP based applications
> are likely to be confused and TCP itself will probably get confused as
> well.  The symptoms are likely to be similar to what happens when you
> have a split route (the route used on outgoing packets is different
> from return packets).
> 
> 2.  Multiple traceroutes will show two different routes depending on
> which equal cost route was picked.  Most noticeable if you use
> something like mtr (Matt's traceroute) which does continuous
> traceroutes in a gui/screen oriented display.

Yes, that happens.

> Is it hokum if it actually works (for years). 

You make a counterfactual claim.  TCP DNS Anycast hasn't ever worked.  
People almost exclusively use UDP for DNS, with stateless transaction
sizes. Its only DNSSEC, and anti-spoofing techniques that is leading to
more TCP use. When ordinary admins talk about anycasting DNS, they are
anycasting stateless UDP with 512 byte UDP packets. There is nothing
wrong with stateless anycasting.

Almost every con has a bait-and-switch. You were baited with stateless
UDP DNS anycast. There is nothing wrong with anycasting stateless UDP.  
Indeed, some nameservers didn't support TCP at all until recently.  But
TCP isn't stateless. And root server operators _have_ to support TCP

		--Dean


-- 
Av8 Internet   Prepared to pay a premium for better service?
www.av8.net         faster, more reliable, better service
617 256 5494