Subject: Re: [dnsop] Re: Root Anycast (fwd)

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 1 Oct 2004 10:49:29 +0200

On 1-okt-04, at 2:48, Dean Anderson wrote:

>> Note though that it's *very* hard to create a setup where packets are
>> delivered to different multicast instances, as it's hard to imagine 
>> how
>> any real-world anycast setup could match the criteria in

> Its quite easy for anycast: (real names used, but not real 
> relationships)

>                 Av8
>               /   \
>           sprint   att
>               \   /
>               F-root

> If Av8 turns on PPLB, traffic to F-root will go through both sprint and
> att on a per-packet basis.

No, that won't happen because in order for BGP to install multiple 
paths in the routing table, the following conditions must be met:

"To be candidates for multipath, paths to the same destination need to 
have the following characteristics equal to the best path's 
characteristics:

[...]

• 	One of the following:
	◦ 	Neighboring AS or sub-AS (before the  eiBGP  Multipath feature was 
added)
	◦ 	AS-PATH (after the  eiBGP  Multipath feature was  added)"

(From the URL I cited in my previous message.)

In this case either of the sub-conditions won't be met.

Per packet load balancing is inappropriate in cases where the entire 
path is different, as the limitations of the inferior path are imposed 
on the full set. BGP has enough knobs to distribute the load without 
this anyway. FIB-based load balancing is only appropriate when a small 
part of a single path is populated with more than one physical link. 
And in most of those cases, load balancing mechanisms that make sure 
one flow runs over one link are more desirable. Only in cases where 
this leads to a significantly unbalanced traffic distribution, per 
packet load balancing is a good idea. (But even then most people don't 
enable it as they are (too, IMO) afraid of packet reordering.)

.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Dean Anderson <[email-masked]>
Date: Fri, 1 Oct 2004 15:02:20 -0400 (EDT)

> paths in the routing table, the following conditions must be met:

If you use bgp multipath only, that is true. If you turn on PPLB, as 
described in my URL, then that isn't true.

-- 
Av8 Internet   Prepared to pay a premium for better service?
www.av8.net         faster, more reliable, better service
617 344 9000   

.

---

From: "Joe Shen" <[email-masked]>
Date: Sat, 2 Oct 2004 19:48:40 +0800

>described in my URL, then that isn't true.

Yup,  I think  PPLB  proper only in situation that multiple physical
links exist between two routers. 
And, PPLB is not good for ISPs who tries balancing traffic between
peers.

Joe

.

.

---

From: "Joe Shen" <[email-masked]>
Date: Sat, 2 Oct 2004 19:45:45 +0800

Is there situation that multiple root servers installed behine multiple routers within one AS?
e.g. 
                                                               / ------ Router 1 ---- Root Server-F-1
       AV8-----  ASBR-------Router-P  /
                                                             \
                                                               \  ----- Router 2 ---- Root Server - F-2

If router-P enables PPLB, would there be some problem with TCP based DNS requests? 

Joe

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Iljitsch van Beijnum
Sent: Friday, October 01, 2004 4:49 PM
To: Dean Anderson
Cc: [email-masked]; [email-masked]
Subject: Re: [dnsop] Re: Root Anycast (fwd)

On 1-okt-04, at 2:48, Dean Anderson wrote:

>> how any real-world anycast setup could match the criteria in

> If Av8 turns on PPLB, traffic to F-root will go through both sprint 
> and att on a per-packet basis.

[...]

• 	One of the following:

.

.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sat, 2 Oct 2004 20:42:17 +0100

On Thu, Sep 30, 2004 at 08:48:37PM -0400,
 Dean Anderson <[email-masked]> wrote 

> and att on a per-packet basis.

Troll Bot <[email-masked]> keeps mentioning PPLB. May be some people
more knowledgeable about BGP than I am will explain to me why PPLB is
such a new issue for anycasting?

Even without PPLB, the simple and normal (though infrequent) change of
the routes by BGP may disturb existing TCP sessions if the target is
anycasted. This is why anycast is currently deployed only on
mostly-UDP services like the DNS.

So, it seems there is nothing new coming from the PPLB thing.
.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 3 Oct 2004 23:01:42 +0200

> such a new issue for anycasting?

I have no idea how new this is, but I have to admit I'm slightly 
worried. Not to the degree Dean seems to be, though.

It is true that if you turn on load balancing over multiple paths in 
BGP and then per packet load balancing between several links, packets 
belonging to one session can end up on different anycast instances. 
(This would be harmful in the case of TCP, but TCP will probably 
recover by retransmitting. It would be quite deadly in the case of 
fragmented UDP packets.)

What can happen is this:

     A
    / \
   B1  B2
   |    |
   C    D
   |    |
   E1  E2

AS A connects to two different routers in AS B, and each of these 
routers prefers a different external path towards different anycast 
instances of AS E. In order for this to happen the paths from B to both 
anycast instances E1 and E2 must be completely identical, except that 
for one router in B one path is preferred and for another router the 
other. This will only happen if these routers connect to ASes C and D 
themselves, or if one sees a better IGP metric towards the router 
connecting to C and another sees a better IGP metric towards the router 
connecting to D.

Now the part that worries me is what's happening in .org. They only use 
two addresses in the delegation from the root, and both are heavily 
anycasted. This makes no sense at all as it effectively hides all but 
two of the .org TLD servers while there are no reasons at all for not 
making at least have a dozen others visible. End-user impacting issues 
with this have been reported (but have predictably been almost 
impossible to reproduce) but the situation persists.

Fortunately, the root operators have more sense (or inherited a better 
situation). Still, I'm not entirely comfortable with the fact that each 
of them seems to make anycasting decisions on their own. Anycast has 
many things going for it as it allows root servers to be installed in 
many more places than could be done otherwise, but it's also risky as 
more and more root servers seem to be in the same place from any given 
viewpoint, and especially from not so well connected viewpoints. 
Problems such as congestion and BGP blackholes or (temporary) BGP 
instability can then impact most or even all of the root servers.  
(Only for some places connected to the net, though.) So I feel it's 
very important to have a reasonable number of root servers that are NOT 
anycast. Preferably, those should be in locations that are far apart.

.

---

From: Paul Vixie <[email-masked]>
Date: 04 Oct 2004 05:22:06 +0000

> belonging to one session can end up on different anycast instances. 

Actually, no.  That would be a disaster.  Noone would ever be able to turn
PPLB on under those conditions, since anycast and multihoming both produce
multiple candidate paths.  BGP's default (as I said in my earlier reply) is
to only install the best path, not all of them.  (At least on C and J gear.)

Doing multipath BGP "safely" would require end to end metrics beyond just
aspath-length.  That's why multipath BGP is not the default.  Given the state
of the BGP technical community, I think the next likely evolution will be
some kind of flow setup/teardown negotiation and a lot more state in routers,
rather than research into the kind of end-to-end metrics that multipath BGP
would require.

> ...

If you can make a strong and cohesive argument about this, ICANN would listen.
(I know that UltraDNS would be happy implementing whatever the DNS community
felt was the strongest configuration, and I think ICANN would say the same.)

> of them seems to make anycasting decisions on their own.

Then you can rest easier, because I'm here to tell you, we didn't.

> Anycast has many things going for it as it allows root servers to be
> installed in many more places than could be done otherwise, but it's also
> risky as more and more root servers seem to be in the same place from any
> given viewpoint, and especially from not so well connected viewpoints.

The rootops don't have a motto, and that's too bad.  If it were allowed, the
motto I would urge to be taken up is "strength through diversity."  The fact
that each rootop goes only where they're invited (as in F-root's case) or
they can afford to go (as in I-root, J-root, M-root, and K-root's cases) is
*wonderful* in terms of root server system survivability.

In the cases where this means a bunch of root servers all anycast in the
same metro, two *great* things happen: (1) attacks and failures against
a single root "letter" don't affect all root servers anycasting in a metro;
and (2) attacks against the entire root server system (all letters) have to
be much stronger in order to upstream-congest every root server instance in
any given metro.

In the cases where this means only one root server anycasts from a given
metro, it's still a lot better than no anycast in that metro at all, and it
may reflect the best use of available resources at a particular point in time.

> instability can then impact most or even all of the root servers.  (Only
> for some places connected to the net, though.)  So I feel it's very
> important to have a reasonable number of root servers that are NOT

Without speaking for any of them, and without telling you who they are, I
will say that several rootops feel as you describe, and the likelihood is
very high that your preferences will be followed in this matter.
-- 
Paul Vixie
.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Paul Vixie <[email-masked]>
Date: 04 Oct 2004 05:05:16 +0000

> > and att on a per-packet basis.

Um.

I just don't think that's true.  The default BGP configuration on vendor-C
and vendor-J and presumably every other vendor is that only the best path
is copied from the RIB to the FIB for any given prefix.  So, in order for
Av8 (or anyone else) to see per-packet load balancing toward an f-root (or
any other anycasted service), the following conditions would have to obtain:

1. override the default on BGP's movement of routes from the RIB to the FIB;
2. see the same prefix aspath-length and MED from multiple transits/peers;
3. override the default to change per-destination to per-packet LB.

It's worth noting that since this is not an example of link-bundling (for
which PPLB was put into IOS), the demerits noted in UniverCD about PPLB will
apply -- mainly that video and voice quality will be poor.  Also, TCP
performance overall will be lower.

It's also worth noting that from Av8's (or anyone's) perspective, anycast is
indistinguishable from multihoming.  That means that during the years f-root
was connected to 100++ BGP peers at PAIX, the world saw 100++ paths to f-root
and would have had exactly the same problems with PPLB.

Finally, it's worth noting that even if somehow DNS were the only service
affected (and not voip or video streams or etc), and even if somehow a remote
BGP listener were able to distinguish between multihoming and anycast (which
it can't), then only TCP would be injured, and DNS is sigma-6 UDP.

> such a new issue for anycasting?

It isn't.  The troll in question has only negative objectives, and too much
time on his hands, and this thread is just his latest attempt to try to make
enough smoke that somebody somewhere will worry that there's a fire burning.

> mostly-UDP services like the DNS.

Yes.

> So, it seems there is nothing new coming from the PPLB thing.

It's much worse than that, but at a minimum, yes.
-- 
Paul Vixie
.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 4 Oct 2004 12:16:07 +0200

> Actually, no.  That would be a disaster.

Whether it's possible and whether it's a disaster are two different 
issues.  :-)

I think the explanation in my previous message shows that it is 
actually possible for this to happen.

The case where a customer connects to a singe upstream AS over 
different links and these links are terminated on different routers 
within this upstream AS isn't a big stretch: this way, the customer is 
protected against failure of one of the routers they're connected to. 
(Note that in this case incoming traffic can't be balanced, but if both 
links terminate on the same router on the customer side, outgoing 
balancing is still possible.)

The second requirement is that these different routers select different 
outgoing paths to an anycast instance. This is fairly unlikely, but it 
could happen in a network design where all components exist in pairs, 
and the links between routers in the same "half" of the network are 
preferred over links between both halves, or it takes more links to get 
to a place connected to the other half.

> Noone would ever be able to turn
> multiple candidate paths.

I don't follow your reasoning. For multihoming this isn't much of an 
issue since all packets end up in the same place regardless of how they 
get there. And if you have two expensive links, one of which sees 99% 
utilization and the other 10%, you really want that per packet load 
balancing.

(Often times people are reluctant to enable per packet load balancing 
as it may introduce reordering. However, this is only the case when the 
entire path towards the load balanced links is the same speed as those 
links, unless there is serious congestion. So when load balancing over 
two GE links a host connected over 100 Mbps won't see any reordering as 
its packets arrive at the bundled link with more time between them than 
the different propagation times for different size packets or random 
differences in queuing delays can make up for.)

> BGP's default (as I said in my earlier reply) is

Yes, and by default load balancing is per destination rather than per 
packet. But people have been known to override defaults...

> aspath-length.

That's why it's required that either the next hop AS (C old behavior) 
or the entire AS path (C new behavior) are identical.

[just two anycasted addresses are authorative for .org]

>> End-user impacting issues

> felt was the strongest configuration, and I think ICANN would say the 

This has been discussed fairly extensively on NANOG in the past. If you 
are positive ICANN/Ultradns are willing to listen, I'll see if I can 
find the time to write something down. (I don't have any .org domains 
so it may take me a while to get around to this.)

> be much stronger in order to upstream-congest every root server 

That's great. But this is also the case where unpleasant anycast side 
effects crop up most easily, as AS paths and metrics towards borders 
are more easily identical. And if an attacker gets to saturate a 
network in that metro, all the anycast instances there suffer. It's 
harder to saturate many links in many places. And from a performance 
viewpoint, just a few "local" roots are all you need, as good resolvers 
home in on servers with low RTTs.

> may reflect the best use of available resources at a particular point 

Not sure what you mean here.

>> (Only

> very high that your preferences will be followed in this matter.

Excellent. Please tell your anonymous friends that a statement to this 
effect from them would be highly appreciated.

.

---

From: Dean Anderson <[email-masked]>
Date: Mon, 4 Oct 2004 13:45:50 -0400 (EDT)

> worried. Not to the degree Dean seems to be, though.

I'm not saying that the sky is falling.  

I'm just saying that if we ignore forseeable problems, then bad things
will happen, and that since they are foreseeable, we should take some
action to avoid those problems before we start to experience them in
practice.

And I am rather annoyed that I'm being called names by children who admit
they don't even have the technical background to argue one way or the 
other.

And I am rather concerned that fairly obvious foreseeable problems
(Anycast is known not to mix with TCP, DNS is known to have TCP and known
to use it more in the future) were not discussed or foreseen earlier when
alternative solutions could have been proposed an implemented.  This is a
valid criticism of those who deployed anycast on the roots.

And I am rather concerned that significant infrastructure is in the hands
of children who can't make better argument than name-calling. Not only is
that immature, it shows that they are not very creative or intelligient.  
Children grow out of that behavior because they become more intelligent
and can articulate themselves. I think it is a bad thing for the people
placed in charge of important servers to have deficits in maturity,
creativity, and intelligence (and perhaps professional ethics). 

Which problem is worse, the staffing problem, or the anycast-TCP problem
is anyone's guess.  Probably the former, since it is harder to fix.

		--Dean

.