Subject: Anycast root metrics and analysis

From: [email-masked]
Date: Tue, 31 Oct 2000 13:32:58 -0800 (PST)

I've attached the data for the aroot reachability times from here for
10/30/2000.  As you'll note, there are long stretches where things are
fine, but also lots of 10 second + spikes which indicate that the
service was essentially unreachable.

Looking over the past few sets of data, that is a pretty typical graph
for us.  Thinking about that, I've realized that one of things that
worries me about this anycast methodology is that I'm not sure how
well it plays with traditional criteria for selecting among the roots.
The BIND selection system uses round trip time as the fundamental
measurement for determining which root server will be used.  This
isn't a single measurement, of course, but a race run and re-run based
on the number of requests coming in.  The use of RTT and periodic
checks on server response time recognizes the importance of quick
responses for the DNS system; it also recognizes that the selection of
which root server to use should be something that requires very little
intervention on the part of DNS operators.

What worries me is that the anycast system described in Masataka's
draft may end up requiring a fair amount of maintenance from the BGP
community to work optimally according to the DNS's native metrics.
The way things are put together now, this system would increase the
number of servers acting as roots but would not increase the number of
servers available for selection to any one location on the network.
Without any additional tweaking, an individual network will get the
anycast server which is reachable via the shortest AS path-length.
That may or may not map onto the server instance that would generate
the best performance.  Through our Verio link, for example, we reach
the embratel instance, but the best performance would be reached to
the mbone-cern aroot (as tracerouting its other interface
demonstrates).  As Masataka's draft notes, this is fixable, but it
isn't trivial.

As an example, I am AS 14609.

The AS used in this experiment is 4128.  Imagine that 559 and 1 are
trusted peers I want to hear it from. I want to reject everyone else.

Of course, it is best if I peer directly with 559 or 1.  If not, I
need to make sure my upstream will announce routes to the Anycast AS
thru 559 or 1, and not some less-preferred provider.  This is not
impossible, and is typical of today's inter-provider coordination, but
just another thing that could get lost thru the cracks.

# Cisco IOS snippet

# as-path route filter: only accept routes to AS 4128
# when it is seen immediately behind AS 559 or 1.
ip as-path access-list 75 permit _559_4128$
ip as-path access-list 75 permit _1_4128$
ip as-path access-list 75 deny _4128_

# apply the as-path filter via a route map
route-map EXT-PEER permit 4128
 match as-path 75

router bgp 14609
 # set up a peer-group policy that uses our route map
 # (among other typical policies)
 neighbor ext-peers peer-group
 neighbor ext-peers next-hop-self
 neighbor ext-peers send-community
 neighbor ext-peers remove-private-AS
 neighbor ext-peers version 4
 neighbor ext-peers soft-reconfiguration inbound
 # distribute list 100 has the bogon filters: RFC1918, etc.
 neighbor ext-peers distribute-list 100 in
 neighbor ext-peers distribute-list 100 out
 neighbor ext-peers route-map EXT-PEER out

 # apply the peer group to each peer BGP session
 neighbor 10.2.2.2 remote-as 559
 neighbor 10.2.2.2 peer-group ext-peers
 neighbor 10.2.2.2 description "peering with anycast-clueful AS 559"

 neighbor 10.2.2.4 remote-as 1
 neighbor 10.2.2.4 peer-group ext-peers
 neighbor 10.2.2.4 description "peering with anycast-clueful AS 1"

The access lists in the route filter would need to be updated whenever
a different instance of the anycast server is notably better.  Of
course, none of this would be necessary if there were a perfect match
between AS Path length and RTT performance, but that match isn't
perfect.  That means that maintaining the best access to the full set
of available roots requires separate testing and interdomain routing
updates rather than simply periodically checking the results of a race
condition.

In Masataka's draft, one of the arguments put forward is that there is
a parallel between locally scoped unicast addresses and the use of
anycast addresses (section 2, "Suggested Operation").  In the case
that an anycast system is so well distributed that every AS runs its
own copy of an anycast AS and root server, the parallel would indeed
be strong.  In the current case, though, the very small number of
participants in the system seems to demonstrate that the BGP policy of
the adjacent and transit ASes is going to dominate the selection of
anycast root server instances.  That produces a conflict of metrics
and suboptimal results.

You can, of course, argue that the increase in topological diversity
is worth this and that the network will move toward optimal results as
increasing numbers of ASes offer transit to their local copies of the
root server.  I would like to suggest, though, that there is
compromise possibility here.  That would be take one of the existing
root servers and assign it and its network to a new AS.  Any network
which wanted to run a local copy of the root could do so using that AS
and network, but they would be forbidden from offering transit to that
AS.  The current provider could maintain the existing service by
offering transit through their current AS to the new AS.  This would
mean that the BGP community would need to filter out any other
announcements which might happen to leak, but it could become a
standard part of the "bogon" filters and would not need a great deal
of continuing support.  That seems to offer both increased topological
diversity, a limited operational impact to the BGP community, and
some clarification of the "whose neck to wring" problem (since the
answer is always either your own or the existing provider)

What do others think of this analysis?
				regards,
					Ted Hardie

---

From: [email-masked]
Date: Tue, 31 Oct 2000 13:47:57 -0800 (PST)

> 10/30/2000.  

Okay, I ran on so long on the original message that I forgot the
data.  My apologies, here it is.
				regards,
					Ted Hardie

---

From: Robert Elz <[email-masked]>
Date: Wed, 01 Nov 2000 15:54:02 +0700

Date:        Tue, 31 Oct 2000 13:32:58 -0800 (PST)
    From:        [email-masked]
    Message-ID:  <[email-masked]>

  | on the number of requests coming in.  The use of RTT and periodic
  | checks on server response time recognizes the importance of quick
  | responses for the DNS system;

Yes, of course, faster is always better - but there is nothing in
the DNS that requires a response from the quickest server, any one
will do.

  | Without any additional tweaking, an individual network will get the
  | anycast server which is reachable via the shortest AS path-length.

That's fine.  Basically the point is to add additional servers and
help spread the load.  As much as that also results in increased
performance to the end sites, that's great.

  | That may or may not map onto the server instance that would generate
  | the best performance.

True - but nor does any algorithm, the one bind uses now bases expected
response time upon past performance - which also may, or may not, map
onto the server that would generate the best performance.

Each additional anycast group of root servers (ideally there would
be several of them, at different addresses, living in different ASs)
will give you an extra choice of root server to use.  Further, as the
concept becomes proven, it may be that there will be many more root
servers participating in each anycast group - it is entirely possible
that each fairly major AS will have one local.   That ought mean that
a server qith very good response time could be found from just about
anywhere.

  | The access lists in the route filter would need to be updated whenever
  | a different instance of the anycast server is notably better.

I don't think doing manually operated performance based routing is
a scheme that will ever work...

  | In the case
  | that an anycast system is so well distributed that every AS runs its
  | own copy of an anycast AS and root server, the parallel would indeed
  | be strong.  In the current case, though, the very small number of
  | participants in the system seems to demonstrate that the BGP policy of
  | the adjacent and transit ASes is going to dominate the selection of
  | anycast root server instances.  That produces a conflict of metrics
  | and suboptimal results.

The current setup is intended as a proof of concept - that is, that it
can be made to work at all.   I don't think it was ever intended to be
optimal.

  | What do others think of this analysis?

I suspect that you're worrying too much over nothing.

The spikes in reachability to the servers that your data collection
shows could do with more analysis though - do you have any idea what
is causing them?   Is something dropping off the net?

kre

---

From: Frederico A C Neves <[email-masked]>
Date: Wed, 1 Nov 2000 10:37:22 -0200

> is causing them?   Is something dropping off the net?

some time ago i sent an email to the list telling that we have a very
congested 12 Mbits link to the US, it still are, and will be until the
OC3 get up and running. To all the ASs that peer directly with us here
in Brazil, our server is the best one and appears to solve the problem
very well to regions of the net not so well connected to the core like
ours.

This is our verification of one of the auth servers for COM.BR located
outsided of Brazil, just for time comparing.

http://registro.br/stat/ns/nsvixcom.html

bash> traceroute ns-ext.vix.com
traceroute to ns-ext.vix.com (204.152.184.64), 30 hops max, 40 byte packets
 1  gw.REGISTRO.BR (143.108.23.1)  0.159 ms  0.132 ms  0.126 ms
 2  fe010-core.spo.ansp.br (143.108.13.65)  0.546 ms  0.404 ms  0.401 ms
 3  166.49.29.201 (166.49.29.201)  576.024 ms *  727.498 ms
 4  core2.WestOrange.cw.net (204.70.4.185)  792.518 ms  743.608 ms  650.876 ms
 5  cw-ip-eng-interconnects.WestOrange.cw.net (204.70.1.38)  888.656 ms  883.417 ms  937.981 ms
 6  nynap.above.net (192.157.69.64)  751.384 ms  935.749 ms  1543.965 ms
 7  iad-penn-oc3.iad.above.net (209.249.0.49)  798.519 ms  741.973 ms  747.353 ms
 8  * core1-core3-oc48.iad.above.net (209.249.203.34)  814.138 ms  694.633 ms
 9  sjc2-iad-oc48.sjc2.above.net (216.200.127.26)  877.041 ms  846.111 ms  726.771 ms
10  mibh-above-1.mibh.net (208.184.102.10)  847.176 ms  443.224 ms  825.496 ms
11  head7.pla.mibh.net (128.177.255.21)  1109.885 ms  1411.469 ms  661.921 ms
12  pla1b.feed1.pla.mibh.net (204.152.184.211)  616.937 ms  711.377 ms  584.922 ms
13  ifw2-core.pa.vix.com (204.152.184.5)  778.297 ms  725.023 ms  856.954 ms
14  * ns-ext.vix.com (204.152.184.64)  802.943 ms  863.011 ms

Fred

-- 
 Frederico A C Neves              Registro .br - R.Pio XI, 1500
 +55 11 3838-4130             São Paulo, SP, Brazil - 05468-901

---

From: [email-masked]
Date: Wed, 1 Nov 2000 09:58:14 -0800 (PST)

> the DNS that requires a response from the quickest server, any one

It's not a DNS requirement that the quickest server be used, it's
an optimization that helps the applications which depend on the DNS.

> performance to the end sites, that's great.

I actually believe the point is to distribute the data to multiple
places in the network topology in order to improve performance for
those who have been under served.  The load can be handled where it
is, just not in a way that satisfies the performance preferences of
all the users.  If the current plan resulted in worse performance to
end sites, I believe it to have failed.  I also believe that there
might be other consequences, such as concentration of load on a small
number of the extant roots.

> a server qith very good response time could be found from just about

I agree that having one local to an AS would be useful, and that is why
I suggested splitting things so that any network could have a local copy
but only one network could offer transit to the AS/address.  

> a scheme that will ever work...

On this point, we are in violent agreement.

> can be made to work at all.   I don't think it was ever intended to be

How, then, can we test that the system it is proving can be optimal?

					regards,
						Ted

---

From: Robert Elz <[email-masked]>
Date: Thu, 02 Nov 2000 22:09:16 +0700

Date:        Wed, 1 Nov 2000 09:58:14 -0800 (PST)
    From:        [email-masked]
    Message-ID:  <[email-masked]>

  | It's not a DNS requirement that the quickest server be used, it's
  | an optimization that helps the applications which depend on the DNS.

Yes, so we desire it to be "fast enough" - we don't require that there
not be something even faster that we're missing out on.

  | I actually believe the point is to distribute the data to multiple
  | places in the network topology in order to improve performance for
  | those who have been under served.

Yes, it should have that effect.

  | If the current plan resulted in worse performance to
  | end sites, I believe it to have failed.

Absolutely.   But there is a difference between "worse performance
than now" and "worse performance than could theoretically be obtained".
The former is what we need to guard against, not the latter.

  | I also believe that there
  | might be other consequences, such as concentration of load on a small
  | number of the extant roots.

That would not be good - but is it likely that it would be worse
than it now is?  In fact, is it even really possible (unless someone
deliberately set out to achieve that).

  | How, then, can we test that the system it is proving can be optimal?

I don't think we need to - all that's needed (long term) is that it
be better than it is now.   Then by placing root servers at more
locations, it should be able to be improved even more - measurements
from various sites can help determine that - if not, the new server
that made it worse should be (re-)moved.   I don't think we should
consider anything to have failed just because it could be even better still.

kre

---

From: [email-masked]
Date: Fri, 03 Nov 2000 08:50:24 +1100

As the one who tuned BIND's RTT break points I agree with
	kre that it just needs to be better not best.  BIND's
	algorithm sets the first break point at 32 ms, then goes
	up in steps of 64 ms.  The primary purpose was to keep
	traffic within the continent, or closest continent in my
	case. Secondary to that it was to keep traffic in the local
	area of the continent.

	The real problem, which exists today, is to ensure that the
	infrastructure is capable to taking the traffic load without
	large queuing delays and without large packet losses.

	The root server operators can easily measure the losses at the
	server (if any).  It's much harder for them to see the losses
	in the network servicing the root servers.

	Mark
--
Mark Andrews, Nominum Inc.
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: [email-masked]
Date: Thu, 2 Nov 2000 17:52:07 -0800 (PST)

kre quotes me, then writes
> The former is what we need to guard against, not the latter.

We are examining a change to the operation of a pretty fundamental
part of the Internet infrastructure.  I agree that we need to guard
against "worse performance than now", but I also believe that we need
to ensure that the fix we employ is one which gives the best bang
(performance) for the buck (operational pain).  It is mighty unlikely
that we will get apply more than one fix to this particular problem.
To me, that means a test of how well we can get this system to perform
is important.  There is also the tricky question of performance for
whom, and a fair likelihood that the results of that will not be an
even distribution.

In my first email, I proposed a change to Masatka's proposal which
would allow as many local copies of a specific anycast root as were
desired, but restricted transit to them to a single well-connected
provider (as is the current case).  I have heard you say that you
don't believe that the problem I identified is important enough to
warrant further testing and that you believe that the increased
distribution of anycast root server instances would solve the problem
anyway.  Without further data, I doubt we will agree on the first
claim.  Any chance that you can run an aroot instance from where you
are so that we can gather more data to prove your second point?

				regards,
					Ted

---

From: Robert Elz <[email-masked]>
Date: Fri, 03 Nov 2000 13:20:04 +0700

Date:        Thu, 2 Nov 2000 17:52:07 -0800 (PST)
    From:        [email-masked]
    Message-ID:  <[email-masked]>

  | Any chance that you can run an aroot instance from where you
  | are so that we can gather more data to prove your second point?

I can run a server easily enough.   But I am not close enough these
days to the routing system to be able to impose ASs or routes to the
magic anycast address (nd certainly not to verify that it is done
properly).   So, I doubt that I can really help a lot in any useful way.

Is there no-one in Australia participating in the test now?   (Not that
I am in Australia right now in any case).

On the "anyone can run their own server"  .. with the anycast scheme
they can't really be stopped, and other than by filters in other providers
access lists, they can't really be prevented from offering transit either
should they want to.   But there's more to the stability of the DNS system
than just getting the answers back as quickly as possible - the database
needs to be properly maintained and updated.   What's more the comparative
stability of the root zone actually makes that harder (it is easy to set up
and test update procedures for a database that changes every day, much
harder for one that only changes once a year or so).  More servers spread
around improves access, but decreases trustworthiness of the data.  The
two need to be balanced.

kre

ps: where I am now I am getting 1.8second RTTs to home (at least 1.2s to
any root server) and approx 30% packet loss ... I know what benefits there
are to be obtained from this.

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 03 Nov 2000 11:11:23 +0100

>         in the network servicing the root servers.
OTOH, measuring this loss is easy for the end users.
Here's losses measured from a site in Norway for the .in-addr.arpa server set:

1271 samples
Server               Samp      %  Min  Ave  Max   0 1.0 2.0 3.0 4.0 5.0 6.0
A.ROOT-SERVERS.NET   1238  97.40  138  311 5256 
1204                  34
B.ROOT-SERVERS.NET   1241  97.64  205  669 5659 
1148   2              91
C.ROOT-SERVERS.NET   1170  92.05  145  917 5621 
1032                 138
D.ROOT-SERVERS.NET   1262  99.29  138  217 5360 
1255                   7
E.ROOT-SERVERS.NET    119   9.36  212  224  248 119
F.ROOT-SERVERS.NET   1230  96.77  195  255 5274 
1223                   7
G.ROOT-SERVERS.NET   1129  88.83  177 1344 5924 888   2             239
H.ROOT-SERVERS.NET   1116  87.80  164 1413 5946 896                 220
I.ROOT-SERVERS.NET   1228  96.62   31  120 5509 
1207   2   1          18

The stuff at +5 seconds is the part that needed a retransmission.

And here are the numbers for the .com servers:

1558 samples
Server               Samp      %  Min  Ave  Max   0 1.0 2.0 3.0 4.0 5.0 6.0
A.GTLD-SERVERS.NET   1493  95.83  137  190 5232 
1486   1               6
A.ROOT-SERVERS.NET   1504  96.53  137  261 5443 
1476   1              27
B.GTLD-SERVERS.NET   1503  96.47  200  372 5532 
1488       1          14
C.GTLD-SERVERS.NET   1471  94.42  138  191 5999 
1463   2               6
D.GTLD-SERVERS.NET    890  57.12  203  340 5552 874   1              15
E.GTLD-SERVERS.NET   1463  93.90  194  243 5220 
1457                   6
F.GTLD-SERVERS.NET   1476  94.74  196  259 5340 
1468                   8
F.ROOT-SERVERS.NET    594  38.13  195  239 5212 592                   2
G.GTLD-SERVERS.NET   1496  96.02  137  199 5220 
1486                  10
I.GTLD-SERVERS.NET   1380  88.58   22  430 5216 
1286                  94
J.GTLD-SERVERS.NET   1483  95.19  137  267 6519 
1475                   7   1
K.GTLD-SERVERS.NET   1480  94.99   57  553 5616 
1338   9             133
M.GTLD-SERVERS.NET   1453  93.26  358  518 7698 
1418   6   2   1      25       1

The reason for the abysmal response here on f.root-servers.net is that it
was dropped from the .com server set while the test was running.
e.root-servers.net gets "host unreachable" from Stockholm; I don't know why.

             harald

--
Harald Tveit Alvestrand, [email-masked]
+47 41 44 29 94
Personal email: [email-masked]