Subject: grow: WGLC for Operation of Anycast Services (draft-ietf-grow-anycast-04.txt)

From: Hitesh Ballani <[email-masked]>
Date: Sun, 17 Sep 2006 18:40:16 -0400

Since this is my first time here, I thought I'd introduce myself -- I am
Hitesh Ballani, a graduate student at Cornell University working with
Paul Francis. In the recent past, I have worked on IP Anycast and as
part of this, we deployed a small inter-domain anycast service
(http://pias.gforge.cis.cornell.edu/deployment.php). I realize that the
deadline for the last call on the draft has passed but I just got my
hands on this draft and thought I'd chime in with my two cents. 

While I realize that the draft is intended to serve as a BCP for IP
Anycast in general, most of my comments pertain to "inter-domain" IP
Anycast with the anycast nodes advertising the covering prefix into BGP.
Overall, I think that the draft does a very good job of laying down the
advantages and the pitfalls of using IP Anycast and so, would serve as a
good BCP. Anyway, some of my high-level comments follow (I'll probably
send the low-level comment on the nitty-gritty of the document
later):

- Page 5, Section 3.1 
>    Distribution of load between
>    nodes for the purposes of reliability, and coarse-grained
>    distribution of load for the purposes of making popular services
>    scalable can often be achieved, however.

Y'all probaby already have a lot of experience with this but I thought
I'd point out that our measurement work supports this remark. We have
used AS path-prepending at individual anycast nodes to manipulate the
amount of traffic being delivered to them and found that, if
intelligently used, this does provide the operator with coarse control
over load across the nodes. For those of you interested, these results
are described in section 8 of [1]. Further, we are working on trying to
measure the impact of specific advertisement using BGP
community-attributes offered by ISPs as a more fine-grained load control
knob.

- Page 6, Section 3.2 Goals

I found it sort of surprising that the list of common objectives for
using Anycast does not include "Replication of a service for
scalability, robustness, etc. in a fashion that is transparent to the
service users.". This *seems* like a pretty basic reason to use anycast.

- Page 6, Section 3.2
>        Topological nearness within the
>        routing system does not, in general, correlate to round-trip
>        performance across a network; in some cases response times may
>        see no reduction, and may increase.

Given the general nature of the document, I understand the need for
having this remark. As a matter of fact, we have found that in many of
the existing IP Anycast deployments (for example, F-Root, J-Root, AS112,
etc.), the use of Anycast does not correlate well with round-trip
performance (not that these deployments care a lot about this but
anyway!). However, we have found that it is indeed possible for an
operator to choose the anycast node locations so as to ensure that in a
majority of the cases, IP Anycast routes clients to the anycast node
closest to it (Section 5 of [1]). In short, the idea is to restrict the
deployment to a single globally-spread ISP and cover the ISP well with
geographically-spread anycast nodes.

- Page 7, Section 4.1

>    This document deliberately avoids prescribing rules as to which
>    protocols or services are suitable for distribution by anycast; to
>    attempt to do so would be presumptuous.

Again, the use of IP Anycast for stateful services has probably been
passionately debated on this and other forums and many past studies have
presented a different picture of this. FWIW, we found that IP Anycast
does offer a very good substrate for most stateful services (Section 7
of [1]). We did find cases where clients were very frequently being
routed to different anycast nodes, but such cases were rare and were
caused by load-balancing at the client end-site (as pointed out in
section 4.4.3 of this draft). That being said, I agree that it is
probably safer to avoid prescribing any rules. Or, maybe it could be
possible to say that past experience suggests that, in most cases, IP
Anycast can be used for stateful services but there are some caveats
that do need to be addressed.

- Page 8, Section 4.2

>    In general node placement decisions should be made withSince this
is my first time here, I thought I'd introduce myself -- I am
Hitesh Ballani, a graduate student at Cornell University working with
Paul Francis. In the recent past, I have worked on IP Anycast and as
part of this, we deployed a small inter-domain anycast service
(http://pias.gforge.cis.cornell.edu/deployment.php). I realize that the
deadline for the last call on the draft has passed but I just got my
hands on this draft and thought I'd chime in with my two cents. 

While I realize that the draft is intended to serve as a BCP for IP
Anycast in general, most of my comments pertain to "inter-domain" IP
Anycast with the anycast nodes advertising the covering prefix into BGP.
Overall, I think that the draft does a very good job of laying down the
advantages and the pitfalls of using IP Anycast and so, would serve as a
good BCP. Anyway, some of my high-level comments follow (I'll probably
send the low-level comment on the nitty-gritty of the document
later):

- Page 5, Section 3.1 
>    Distribution of load between
>    nodes for the purposes of reliability, and coarse-grained
>    distribution of load for the purposes of making popular services
>    scalable can often be achieved, however.

Y'all probaby already have a lot of experience with this but I thought
I'd point out that our measurement work supports this remark. We have
used AS path-prepending at individual anycast nodes to manipulate the
amount of traffic being delivered to them and found that, if
intelligently used, this does provide the operator with coarse control
over load across the nodes. For those of you interested, these results
are described in section 8 of [1]. Further, we are working on trying to
measure the impact of specific advertisement using BGP
community-attributes offered by ISPs as a more fine-grained load control
knob.

- Page 6, Section 3.2 Goals

I found it sort of surprising that the list of common objectives for
using Anycast does not include "Replication of a service for
scalability, robustness, etc. in a fashion that is transparent to the
service users.". This *seems* like a pretty basic reason to use anycast.

- Page 6, Section 3.2
>        Topological nearness within the
>        routing system does not, in general, correlate to round-trip
>        performance across a network; in some cases response times may
>        see no reduction, and may increase.

Given the general nature of the document, I understand the need for
having this remark. As a matter of fact, we have found that in many of
the existing IP Anycast deployments (for example, F-Root, J-Root, AS112,
etc.), the use of Anycast does not correlate well with round-trip
performance (not that these deployments care a lot about this but
anyway!). However, we have found that it is indeed possible for an
operator to choose the anycast node locations so as to ensure that in a
majority of the cases, IP Anycast routes clients to the anycast node
closest to it (Section 5 of [1]). In short, the idea is to restrict the
deployment to a single globally-spread ISP and cover the ISP well with
geographically-spread anycast nodes.

- Page 7, Section 4.1

>    This document deliberately avoids prescribing rules as to which
>    protocols or services are suitable for distribution by anycast; to
>    attempt to do so would be presumptuous.

Again, the use of IP Anycast for stateful services has probably been
passionately debated on this and other forums and many past studies have
presented a different picture of this. FWIW, we found that IP Anycast
does offer a very good substrate for most stateful services (Section 7
of [1]). We did find cases where clients were very frequently being
routed to different anycast nodes, but such cases were rare and were
caused by load-balancing at the client end-site (as pointed out in
section 4.4.3 of this draft). That being said, I agree that it is
probably safer to avoid prescribing any rules. Or, maybe it could be
possible to say that past experience suggests that, in most cases, IP
Anycast can be used for stateful services but there are some caveats
that do need to be addressed.

- Page 8, Section 4.2

>    In general node placement decisions should be made with
consideration
>    of likely traffic requirements, the potential for flash crowds or
>    denial-of-service traffic, the stability of the local routing
system
>    and the failure modes with respect to node failure, or local
routing
>    system failure.

Page 12, Section 4.4.4

>    care should be taken to
>    arrange that the AS_PATH attributes on routes from different nodes
>    are as diverse as possible.  For example, Anycast Nodes should use
>    the same origin AS for their advertisements, but might have
different
>    upstream ASes.
 
I realize that anycast node placement decisions are guided mainly by
business/practical concerns. On the technical side, there is also the
question of node placement to ensure good performance (i.e. clients
being routed to close-by anycast nodes, ensuring fast failover when an
anycast node fails etc.) which is not mentioned in the BCP at all.
Actually, there is an interesting trade-off here: For things like
protection against DoS attacks and route-flap dampening, anycast nodes
should have different upstream ASs. On the other hand, for good
performance, anycast nodes should have the same upstream AS (as
mentioned earlier and discussed in section 5,6 of [1]).

- Page 20, Section 6.2

>    The potential benefit of being able to take compromised servers
off-
>    line without compromising the service can only be realised if there
>    are working procedures to do so quickly and reliably.

I found it surprising that the document had no mention of failover-rate 
considerations, i.e. when an anycast node goes offline (planned or
unplanned), how soon are clients using that node re-routed to some other
anycast node. This, I presume, would be important for service
availability and is probably one of the reasons why most IP Anycast
deployments use a clustered deployment model. And as I mentioned, we
studied the failover rate for our anycast deployment (section 6 of [1])
and found that smart node placement can ensure that anycast convergence
is not impacted by BGP convergence (which can be slow in extreme cases).

Reference:

[1] H. Ballani, P. Francis and S. Ratnasamy. "A Measurement-based
Deployment Proposal for IP Anycast," in Proceedings of Internet
Measurement Conference, Rio de Janeiro, Brazil, Oct 2006.
(http://pias.gforge.cis.cornell.edu/publications.php)

Cheers.
-- 
hitesh

_________________________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/grow.html
web archive:        http://darkwing.uoregon.edu/~llynch/grow/

---

From: Geoff Huston <[email-masked]>
Date: Mon, 18 Sep 2006 10:28:11 +1000

Hi,

Procedurally, this document has completed IETF Last Call and is now 
awaiting the go-ahead from the Area Director to pass the document to 
the IESG for their review:

https://datatracker.ietf.org/public/pidtracker.cgi?command=search_list&search_button=SEARCH&sub_state_id=6&search_filename=draft%2dietf%2dgrow%2danycast%2d04%2etxt)

I _don't_ gather from your comments that any parts of the the 
document is inherently incorrect or misleading, and that, as far as I 
can see, while you may have a somewhat different perspective than 
that offered in the document on a small set of matters, these appear 
(to me at any rate) to be points of difference of opinion rather than 
factual errors in the document. Using this interpretation of your 
note, I do not see that your note raises sufficient grounds for 
halting the publication process and pushing the document back to the 
GROW Working Group for a further re-spin.

I would be obliged if you could inform the Working Group and the Area 
Director if you hold a different view than that I have interpreted 
from your note, and I would request you to do so _immediately_, as 
the document is well advanced in the publication process.

Geoff Huston
GROW WG Chair

---

From: Hitesh Ballani <[email-masked]>
Date: Mon, 18 Sep 2006 00:37:31 -0400

Hi,

> document is inherently incorrect or misleading, 
Yes, I don't think that any parts of the existing draft are incorrect or
misleading .. as I mentioned earlier, I think that the draft (in its
current form) would be a very useful BCP.

> and that, as far as I 
> factual errors in the document. 
Yes.

> Using this interpretation of your 
Yes, my comments were not intended to halt the publication process. I
realize that I had missed the last call deadline but decided to send in
the comments anyway in case the authors were thinking of a final
revision-pass or something. 

> the document is well advanced in the publication process.
No, your interpretation of my note is correct and I have *no problem* in
the existing draft being published as a BCP. Hope that suffices.

Cheers.
-- 
hitesh

---

From: Dean Anderson <[email-masked]>
Date: Tue, 19 Sep 2006 15:07:22 -0400 (EDT)

Hitesh, do you have any data to backup your assertions about stateful
anycast?  A Previous report (by Daniel Karrenberg) asserting the
stability of stateful anycast were found to be falsified.

Karrenberg didn.t perform stateful Anycast testing; he didn.t reveal 
this relevant fact, yet he still misrepresented his data as being 
relevant to stateful Anycast and refuting claims that stateful Anycast 
was unreliable.  While Karrenberg didn't explicitly say that he made 
stateful TCP queries (indeed Karrenberg didn't say anything at all about 
this very important detail--this is a dishonest omission),  Karrenberg 
did say that his results refuted the "false rumor" that Anycast was 
somehow unsafe--this is a dishonest statement.  An honest statement 
would have said that his data had no bearing on the question of whether 
stateful Anycast was safe. 

Thanks,

Dean Anderson
Av8 Internet, Inc

-- 
Av8 Internet   Prepared to pay a premium for better service?
www.av8.net         faster, more reliable, better service
617 344 9000

---

From: Hitesh Ballani <[email-masked]>
Date: Tue, 19 Sep 2006 15:20:22 -0400

Dean,

> anycast?  
The data collected by us as part of our IP anycast measurement study is
publicly available at http://pias.gforge.cis.cornell.edu/measure.php
(the "Affinity Data-set"). An explanation of how this data was
collected, the analysis and its implications are discussed in Section 7
of our measurement paper
http://pias.gforge.cis.cornell.edu/pub/anymeasure-imc-camera-final.pdf .
Please take a look. I'd be more than happy to help out in case you want
more details about the analysis. And I hope you read the part in my
earlier mail about a small fraction of clients which see poor affinity
(seemingly due to load-balancing at or close to the client-site) and
hence, the need for "caveats" when talking about stateful anycast.

Cheers.
-- 
hitesh

---

From: Geoff Huston <[email-masked]>
Date: Wed, 20 Sep 2006 08:25:54 +1000

Mr Anderson,

While I am perfectly aware that you have some strong convictions 
about anycast, that does not provide any form of license to accuse 
others of being dishonest. At the very least I expect from you a 
retraction of this accusation and I also believe that an apology to 
Mr Karrenberg is in order.

As Working Group Chair I also find it my duty to inform you that this 
is a final warning to you, and that any further postings from 
yourself that include similar forms of personal attacks or attempts 
to disrupt the working group's activities will result in the 
immediate suspension of your posting rights to the GROW mailing list.

Geoff Huston

    GROW WG Chair

---

From: Dean Anderson <[email-masked]>
Date: Tue, 19 Sep 2006 19:44:44 -0400 (EDT)

On Tue, 19 Sep 2006, Hitesh Ballani wrote:

> Please take a look. 

I'm looking now.

> I'd be more than happy to help out in case you want

Yes, I originated the analysis of load-balancing as a factor of
instability in stateful Anycast.  Since fine-grained load balancing is a 
feature of RFC1812, it can't be defined away. 

It is the lack of discussion "caveats" is what makes the
ietf-grow-anycast-draft misleading.  

I also notice that much of your analysis follows mine, yet I am not
cited. Why is that?

I notice that you do not cite the Karrenberg paper in your (to be
published?) October 2006 paper.  Is this an oversight, or it is because
of the fraudulence?

Also BTW, I notice that you appear to have been misled by the draft: If
anycast works for reliably any stateful protocol, it works reliably for
_all_ stateful protocols.  If this were true, there would be no need to
consider the details of the specific protocols, anymore than say,
Gigabit Ethernet needs to be tested with specific protocol.  On the
other hand, if anycast can't reliably keep state, it doesn't work
reliably for any stateful protocol.  Suppose one created BrokenBit
Ethernet, where 3% of the packets were delivered to the wrong host. It
wouldn't work reliably for any stateful protocol. This is fundamental:
If anycast reliably keeps state, it can be used to reliably implement a
finite automaton or a turing machine.  If it doesn't reliably keep
state, it cannot reliably implement either a finite automaton or a
turing machine.

Many years ago, while working for Kendall Square Research, a company
than built supercomputers, the hardware team was working on a bug that
happened only every once in a while. I coined a joke about the
statistical supercomputer, that statistically gave the right answer if
you repeated the calculation enough times.  While the joke was obviously
never told to customers---and the hardware group only laughed nervously
---I have no doubt that customers would have been very unhappy if it
weren't found and fixed.  No one wants a computer that works 96% of the
time.  That's a pretty significant "caveat" to forget to mention.  Most
people, I think, would consider such an omission misleading.

So, I'd ask you again, is the draft misleading?

		--Dean

---

From: Bill Woodcock <[email-masked]>
Date: Tue, 19 Sep 2006 17:37:33 -0700 (PDT)

On Tue, 19 Sep 2006, Dean Anderson wrote:
    > If anycast works for reliably any stateful protocol...

...and if pigs had wings, the Internet would be reliable.  But they don't, 
so the Internet has redundancy and resiliency, rather than reliability.  
It's what makes us more small and furry and warm-blooded and cuddly than 
the bell-heads.

Now, stop trolling with these straw-man arguments.  Take your hands off 
the keyboard and back away slowly.  Go get some fresh air and sunshine.

I can't believe I'm replying to a troll.  It must be September.

                                -Bill

---

From: Dean Anderson <[email-masked]>
Date: Wed, 20 Sep 2006 06:42:32 -0400 (EDT)

> It's what makes us more small and furry and warm-blooded and cuddly than 

In the case of failed stateful anycast, redundancy and resiliency is of
no benefit and does not save the situation. 

> I can't believe I'm replying to a troll.  It must be September.

I can't believe that I'm replying to a troll.

		--Dean

---

From: Dean Anderson <[email-masked]>
Date: Wed, 20 Sep 2006 07:21:26 -0400 (EDT)

On Sun, 17 Sep 2006, Hitesh Ballani wrote:
> of [1]).

I have just read your paper, "A Measurement-based Deployment Proposal
for IP Anycast" and the paper does not find that "IP Anycast does offer
a very good substrate for most stateful services" in either the abstract
or in Section 7.  There seems to be no basis for this statement in the
paper, except through mistake due to misleading discussion contained in
the paper.

The paper discusses some of the issues that are related to stateful
anycast---Issues that are unrelated to stateLESS anycast, and so
therefore one may be misled into assumptions about relevance to stateful
Anycast.  The discussion of those issues is incomplete and fragmentary.
For example, in Section 7, you mention that a route "flap" could cause a
TCP connection to break. This is true, but so can fine grained load
balancing, which you omit.  Elsewhere, you mention load balancing at the
client, but omit discussion of load balancing at other points between
the client and the anycast server.  Load balancing at any point is
equally deadly to stateful Anycast.

Your paper does cite other papers which state that stateful Anycast
(TCP) is a problem area, and you seem to be aware of the controversy
involving stateful Anycast, but you omit any detailed analysis of
stateful topic, and don't make any specific claims of stateful testing.  
This seems misleading, especially given your comment quoted above. Since
you are aware of the controversy, there is no excuse for any lack of
clarity or ambiguity regarding stateful and stateless anycast.  Yet
there is no clear statement regarding the relevancy of the findings of
your paper on each subject.

Your paper also cites other papers call for most testing as a motivation
for your study. I note that there is no call for stateless studies, but
rather for stateFUL testing.  Something I'm starting to suspect, but
isn't explicitly stated in the paper, is that your study is only
relevant to stateless Anycast.

Before I slog though your data, is any of your data stateful?  Have you
done any stateful testing?

I believe that until your paper is adequately clarified, your paper
should be withdrawn or withheld.

---

From: "Paul Francis" <[email-masked]>
Date: Wed, 20 Sep 2006 01:28:33 -0400

Dean,

If you had followed the link Hitesh provided and looked around a bit, you'd
see that he actually posted his data sets.  So you are free to look and see
for yourself.  And I'd sure appreciate not hearing back from you until you've
done that.

PF

-----Original Message-----
From: Dean Anderson [mailto:[email-masked]]
Sent: Tuesday, September 19, 2006 3:07 PM
To: Hitesh Ballani
Cc: [email-masked]; Paul Francis; [email-masked]; [email-masked];
[email-masked]
Subject: Re: grow: WGLC for Operation of Anycast Services
(draft-ietf-grow-anycast-04.txt)

Hitesh, do you have any data to backup your assertions about stateful
anycast?  A Previous report (by Daniel Karrenberg) asserting the stability of
stateful anycast were found to be falsified.

Karrenberg didn.t perform stateful Anycast testing; he didn.t reveal this
relevant fact, yet he still misrepresented his data as being relevant to
stateful Anycast and refuting claims that stateful Anycast was unreliable.
While Karrenberg didn't explicitly say that he made stateful TCP queries
(indeed Karrenberg didn't say anything at all about this very important
detail--this is a dishonest omission),  Karrenberg did say that his results
refuted the "false rumor" that Anycast was somehow unsafe--this is a
dishonest statement.  An honest statement would have said that his data had
no bearing on the question of whether stateful Anycast was safe.

Thanks,

Dean Anderson
Av8 Internet, Inc

On Sun, 17 Sep 2006, Hitesh Ballani wrote:

> Since this is my first time here, I thought I'd introduce myself -- I
> am Hitesh Ballani, a graduate student at Cornell University working
> with Paul Francis. In the recent past, I have worked on IP Anycast and
> as part of this, we deployed a small inter-domain anycast service
> (http://pias.gforge.cis.cornell.edu/deployment.php). I realize that
> the deadline for the last call on the draft has passed but I just got
> my hands on this draft and thought I'd chime in with my two cents.
>
> Overall, I think that the draft does a very good job of laying down
> the advantages and the pitfalls of using IP Anycast and so, would
> serve as a good BCP. Anyway, some of my high-level comments follow
> (I'll probably send the low-level comment on the nitty-gritty of the
> document
> later):
>
>
> - Page 5, Section 3.1
> are described in section 8 of [1]. Further, we are working on trying
> to measure the impact of specific advertisement using BGP
> community-attributes offered by ISPs as a more fine-grained load
> control knob.
>
> the existing IP Anycast deployments (for example, F-Root, J-Root,
> AS112, etc.), the use of Anycast does not correlate well with
> round-trip performance (not that these deployments care a lot about
> this but anyway!). However, we have found that it is indeed possible
> for an operator to choose the anycast node locations so as to ensure
> that in a majority of the cases, IP Anycast routes clients to the
> anycast node closest to it (Section 5 of [1]). In short, the idea is
> to restrict the deployment to a single globally-spread ISP and cover
> the ISP well with geographically-spread anycast nodes.
>
> - Page 7, Section 4.1
>
> passionately debated on this and other forums and many past studies
> have presented a different picture of this. FWIW, we found that IP
> Anycast does offer a very good substrate for most stateful services
> (Section 7 of [1]). We did find cases where clients were very
> frequently being routed to different anycast nodes, but such cases
> were rare and were caused by load-balancing at the client end-site (as
> pointed out in section 4.4.3 of this draft). That being said, I agree
> that it is probably safer to avoid prescribing any rules. Or, maybe it
> could be possible to say that past experience suggests that, in most
> cases, IP Anycast can be used for stateful services but there are some
> caveats that do need to be addressed.
>
> - Page 8, Section 4.2
>
> is my first time here, I thought I'd introduce myself -- I am Hitesh
> Ballani, a graduate student at Cornell University working with Paul
> Francis. In the recent past, I have worked on IP Anycast and as part
> of this, we deployed a small inter-domain anycast service
> (http://pias.gforge.cis.cornell.edu/deployment.php). I realize that
> the deadline for the last call on the draft has passed but I just got
> my hands on this draft and thought I'd chime in with my two cents.
>
> Overall, I think that the draft does a very good job of laying down
> the advantages and the pitfalls of using IP Anycast and so, would
> serve as a good BCP. Anyway, some of my high-level comments follow
> (I'll probably send the low-level comment on the nitty-gritty of the
> document
> later):
>
>
> - Page 5, Section 3.1
> are described in section 8 of [1]. Further, we are working on trying
> to measure the impact of specific advertisement using BGP
> community-attributes offered by ISPs as a more fine-grained load
> control knob.
>
> the existing IP Anycast deployments (for example, F-Root, J-Root,
> AS112, etc.), the use of Anycast does not correlate well with
> round-trip performance (not that these deployments care a lot about
> this but anyway!). However, we have found that it is indeed possible
> for an operator to choose the anycast node locations so as to ensure
> that in a majority of the cases, IP Anycast routes clients to the
> anycast node closest to it (Section 5 of [1]). In short, the idea is
> to restrict the deployment to a single globally-spread ISP and cover
> the ISP well with geographically-spread anycast nodes.
>
> - Page 7, Section 4.1
>
> passionately debated on this and other forums and many past studies
> have presented a different picture of this. FWIW, we found that IP
> Anycast does offer a very good substrate for most stateful services
> (Section 7 of [1]). We did find cases where clients were very
> frequently being routed to different anycast nodes, but such cases
> were rare and were caused by load-balancing at the client end-site (as
> pointed out in section 4.4.3 of this draft). That being said, I agree
> that it is probably safer to avoid prescribing any rules. Or, maybe it
> could be possible to say that past experience suggests that, in most
> cases, IP Anycast can be used for stateful services but there are some
> caveats that do need to be addressed.
>
> - Page 8, Section 4.2
>
> I found it surprising that the document had no mention of
> failover-rate considerations, i.e. when an anycast node goes offline
> (planned or unplanned), how soon are clients using that node re-routed
> to some other anycast node. This, I presume, would be important for
> service availability and is probably one of the reasons why most IP
> Anycast deployments use a clustered deployment model. And as I
> mentioned, we studied the failover rate for our anycast deployment
> (section 6 of [1]) and found that smart node placement can ensure that
> anycast convergence is not impacted by BGP convergence (which can be slow
in extreme cases).

---

From: Dean Anderson <[email-masked]>
Date: Wed, 20 Sep 2006 09:56:54 -0400 (EDT)

On Wed, 20 Sep 2006, Paul Francis wrote:

> 
> Awww heck, I'm in a rascally mood...
> 
> What do you mean by "stateful testing"?  By which I mean we tested
> "affinity", which is the extent to which packets from the same host go to the
> same anycast server over a long period of time.  From this we can make
> statements about how often TCP connections would break (due to a flap).  I
> consider this to be stateful testing.  
> 
> PF 

I mean, "did you perform TCP testing?"

Your analysis of "affinity" is very limited. You state incorrectly that
"none of the aforementioned studies have attempted to delve into reasons
behind routing flaps".  Which the qualification of "routing flaps",
makes the statement literally correct, it is actually misleading.  
Several people including myself have offered analysis and references to
sources of fine grained load balancing, both intentional, and
unintentional, both at the client and within the network.

Your data indicates that 92% of the "flaps" occurred with %1 of the
clients. This suggests that perhaps something may be wrong with
selection of clients. You don't explain this, and you don't report the
number of times per day the %1 most affected, experienced "flaps". (I
calculate 270 times per day for 17 days).  One might wonder whether
these 58 are actually experiencing a problem or simply have fine-grained
load-balancing.  

I note also that you state that "anecdotal evidence from the anycasted J
root-server" is also a mischaracterization of the reference. Kosters et 
al reported statistics they collected that showed that 3.6% of IP 
addresses showed up at multiple anycast servers. This isn't "anecdotal", 
so your characterization of that evidence is misleading.

The data from J-root indicating 3.6% hosts at multiple anycast sites
compares with your %1 or 58 hosts that apppear at multiple anycast
sites.  Your statement that "99% of the clients observe less than 10
flaps per day" is perhaps literally correct, but misleading: it obscures
the fact that 92% of your flaps appear to be more than can be explained
by network failure, as your perjorative and misleading use of the term
"flap" implies.  This spike in "flap" data should be the focus of any
analysis of stateful anycast.  Instead, your paper just dismisses this
and misleadingly claims that there is "good" affinity.  A reasonable
person would not conclude that 58 hosts would continuously flap on
average 270 times a day for 17 days without correction.  There is no
reason to think there is anything wrong with these 58 hosts. There is
also no reason to think that if your set of clients were larger, that
your data wouldn't converge to something closer to Verisign's
statistics.  Yet, your paper, rather misleadingly, seems to indicate
otherwise.

BTW, Root DNS servers aren't operated for 99% of the people. They are
operated for 100% of the people who comply with RFC1812 and the DNS
RFCs.  Fine grained load balancing is a proper behavior under RFC1812.
Your data, honestly interpreted, actually supports Verisign's data. Yet,
misleadingly, you claim the opposite.

> Sent: Wednesday, September 20, 2006 7:21 AM
> To: Hitesh Ballani
> > (Section 7 of [1]).
> I have just read your paper, "A Measurement-based Deployment Proposal for IP
> Anycast" and the paper does not find that "IP Anycast does offer a very good
> substrate for most stateful services" in either the abstract or in Section 7.
> There seems to be no basis for this statement in the paper, except through
> mistake due to misleading discussion contained in the paper.
> 
> anycast---Issues that are unrelated to stateLESS anycast, and so therefore
> one may be misled into assumptions about relevance to stateful Anycast.  The
> discussion of those issues is incomplete and fragmentary.
> For example, in Section 7, you mention that a route "flap" could cause a TCP
> connection to break. This is true, but so can fine grained load balancing,
> which you omit.  Elsewhere, you mention load balancing at the client, but
> omit discussion of load balancing at other points between the client and the
> anycast server.  Load balancing at any point is equally deadly to stateful
> Anycast.
> 
> involving stateful Anycast, but you omit any detailed analysis of stateful
> topic, and don't make any specific claims of stateful testing.  
> This seems misleading, especially given your comment quoted above. Since you
> are aware of the controversy, there is no excuse for any lack of clarity or
> ambiguity regarding stateful and stateless anycast.  Yet there is no clear
> statement regarding the relevancy of the findings of your paper on each
> subject.
> 
> Your paper also cites other papers call for most testing as a motivation for
> your study. I note that there is no call for stateless studies, but rather
> for stateFUL testing.  Something I'm starting to suspect, but isn't
> explicitly stated in the paper, is that your study is only relevant to
> stateless Anycast.
> 
> Before I slog though your data, is any of your data stateful?  Have you done
> any stateful testing?
> 
> I believe that until your paper is adequately clarified, your paper should be
> withdrawn or withheld.
> 
> 
>

---

From: "Paul Francis" <[email-masked]>
Date: Wed, 20 Sep 2006 07:36:14 -0400

PF

-----Original Message-----
From: Dean Anderson [mailto:[email-masked]]
Sent: Wednesday, September 20, 2006 7:21 AM
To: Hitesh Ballani
Cc: [email-masked]; Paul Francis; [email-masked]; [email-masked];
[email-masked]

---

From: "Paul Francis" <[email-masked]>
Date: Wed, 20 Sep 2006 11:09:33 -0400

We tested with a series of packets.  It was not TCP per se.  I don't think it
has to be TCP per se to test affinity.

Where did you publish your analysis?

PF

-----Original Message-----
From: Dean Anderson [mailto:[email-masked]]
Sent: Wednesday, September 20, 2006 9:57 AM
To: Paul Francis
Cc: Hitesh Ballani; [email-masked]; [email-masked]; [email-masked];
[email-masked]
Subject: RE: grow: WGLC for Operation of Anycast Services
(draft-ietf-grow-anycast-04.txt)

On Wed, 20 Sep 2006, Paul Francis wrote:

> "affinity", which is the extent to which packets from the same host go
> to the same anycast server over a long period of time.  From this we
> can make statements about how often TCP connections would break (due
> to a flap).  I consider this to be stateful testing.
>
> PF

Your analysis of "affinity" is very limited. You state incorrectly that "none
of the aforementioned studies have attempted to delve into reasons behind
routing flaps".  Which the qualification of "routing flaps", makes the
statement literally correct, it is actually misleading.
Several people including myself have offered analysis and references to
sources of fine grained load balancing, both intentional, and unintentional,
both at the client and within the network.

Your data indicates that 92% of the "flaps" occurred with %1 of the clients.
This suggests that perhaps something may be wrong with selection of clients.
You don't explain this, and you don't report the number of times per day the
%1 most affected, experienced "flaps". (I calculate 270 times per day for 17
days).  One might wonder whether these 58 are actually experiencing a problem
or simply have fine-grained load-balancing.

I note also that you state that "anecdotal evidence from the anycasted J
root-server" is also a mischaracterization of the reference. Kosters et al
reported statistics they collected that showed that 3.6% of IP addresses
showed up at multiple anycast servers. This isn't "anecdotal", so your
characterization of that evidence is misleading.

The data from J-root indicating 3.6% hosts at multiple anycast sites compares
with your %1 or 58 hosts that apppear at multiple anycast sites.  Your
statement that "99% of the clients observe less than 10 flaps per day" is
perhaps literally correct, but misleading: it obscures the fact that 92% of
your flaps appear to be more than can be explained by network failure, as
your perjorative and misleading use of the term "flap" implies.  This spike
in "flap" data should be the focus of any analysis of stateful anycast.
Instead, your paper just dismisses this and misleadingly claims that there is
"good" affinity.  A reasonable person would not conclude that 58 hosts would
continuously flap on average 270 times a day for 17 days without correction.
There is no reason to think there is anything wrong with these 58 hosts.
There is also no reason to think that if your set of clients were larger,
that your data wouldn't converge to something closer to Verisign's
statistics.  Yet, your paper, rather misleadingly, seems to indicate
otherwise.

BTW, Root DNS servers aren't operated for 99% of the people. They are
operated for 100% of the people who comply with RFC1812 and the DNS RFCs.
Fine grained load balancing is a proper behavior under RFC1812.
Your data, honestly interpreted, actually supports Verisign's data. Yet,
misleadingly, you claim the opposite.

> for IP Anycast" and the paper does not find that "IP Anycast does
> offer a very good substrate for most stateful services" in either the
abstract or in Section 7.
> There seems to be no basis for this statement in the paper, except
> through mistake due to misleading discussion contained in the paper.
>
> therefore one may be misled into assumptions about relevance to
> stateful Anycast.  The discussion of those issues is incomplete and
fragmentary.
> For example, in Section 7, you mention that a route "flap" could cause
> a TCP connection to break. This is true, but so can fine grained load
> balancing, which you omit.  Elsewhere, you mention load balancing at
> the client, but omit discussion of load balancing at other points
> between the client and the anycast server.  Load balancing at any
> point is equally deadly to stateful Anycast.
>
> This seems misleading, especially given your comment quoted above.
> Since you are aware of the controversy, there is no excuse for any
> lack of clarity or ambiguity regarding stateful and stateless anycast.
> Yet there is no clear statement regarding the relevancy of the
> findings of your paper on each subject.
>
> Your paper also cites other papers call for most testing as a
> motivation for your study. I note that there is no call for stateless
> studies, but rather for stateFUL testing.  Something I'm starting to
> suspect, but isn't explicitly stated in the paper, is that your study
> is only relevant to stateless Anycast.
>
> Before I slog though your data, is any of your data stateful?  Have
> you done any stateful testing?
>

---

From: Dean Anderson <[email-masked]>
Date: Wed, 20 Sep 2006 13:14:00 -0400 (EDT)

On Wed, 20 Sep 2006, Paul Francis wrote:

> We tested with a series of packets.  It was not TCP per se.  I don't
> think it has to be TCP per se to test affinity.

An indirect answer.  Was it perhaps stateless UDP?  I agree that there 
are other stateful protocols, but none seem to fit your methodology.

Your use of the word "affinity" is itself a misleading. It suggests a
hidden mechanism that doesn't actually exist.  The client has no control
of Anycast server selection. The Anycast server has no control over
client selection. Every router inbetween client and server, is capable
of an independent route selection.  There is no "affinity", as the word
is defined and people would interpret the word.  Instead, you have only
performed a test of roughly whether load balancing or some other
mechanism such route changes (both intentional and otherwise), exists in
the path or not. The only thing your data reveals is whether a client
can see multiple anycast servers, and roughly how often. This does not
imply "affinity". Collecting statistics on the number of clients that
are serviced with load balancing or some other fast switching mechanism,
even if the number were zero, would not demonstrate "affinity" between
client and server.  Your result of "affinity" does not follow from your
testing.

What you've done is rather like asserting an affinity to processor 0
using data from a lot of uniprocessor machines, and a few
multiproccessors, when there is no affinity mechanism in the operating
system, as evidenced in the large degree of random behavior of the small
number of multiprocessors.  I think most people would find that to be an
misleading result.

And given that you are aware of load balancing as a cause of stateful
Anycast problems, it seems misleading to describe DNS TXT queries as
"route flap" indicators. The existance of a "route flap" is a conclusion
that has to be infered from other facts besides the change of Anycast
server.

> Where did you publish your analysis?   

Discussions on DNSOP and DNSEXT Working Groups 2002-present, Nanog, and
the main IETF list. These are proceedings of a professional society,
from which ideas were apparently taken, but not properly credited.

		--Dean

---

From: "Paul Francis" <[email-masked]>
Date: Wed, 20 Sep 2006 13:24:22 -0400

My mistake...I didn't actually follow the discussions on all of those mailing
lists.

Would it be possible for you to summarize your contributions to those
discussions?

PF 

-----Original Message-----
From: Dean Anderson [mailto:[email-masked]] 
Sent: Wednesday, September 20, 2006 1:14 PM
To: Paul Francis
Cc: Hitesh Ballani; [email-masked]; [email-masked]; [email-masked];
[email-masked]

> think it has to be TCP per se to test affinity.

An indirect answer.  Was it perhaps stateless UDP?  I agree that there are
other stateful protocols, but none seem to fit your methodology.

Your use of the word "affinity" is itself a misleading. It suggests a hidden
mechanism that doesn't actually exist.  The client has no control of Anycast
server selection. The Anycast server has no control over client selection.
Every router inbetween client and server, is capable of an independent route
selection.  There is no "affinity", as the word is defined and people would
interpret the word.  Instead, you have only performed a test of roughly
whether load balancing or some other mechanism such route changes (both
intentional and otherwise), exists in the path or not. The only thing your
data reveals is whether a client can see multiple anycast servers, and
roughly how often. This does not imply "affinity". Collecting statistics on
the number of clients that are serviced with load balancing or some other
fast switching mechanism, even if the number were zero, would not demonstrate
"affinity" between client and server.  Your result of "affinity" does not
follow from your testing.

What you've done is rather like asserting an affinity to processor 0 using
data from a lot of uniprocessor machines, and a few multiproccessors, when
there is no affinity mechanism in the operating system, as evidenced in the
large degree of random behavior of the small number of multiprocessors.  I
think most people would find that to be an misleading result.

And given that you are aware of load balancing as a cause of stateful Anycast
problems, it seems misleading to describe DNS TXT queries as "route flap"
indicators. The existance of a "route flap" is a conclusion that has to be
infered from other facts besides the change of Anycast server.

> Where did you publish your analysis?   

Discussions on DNSOP and DNSEXT Working Groups 2002-present, Nanog, and the
main IETF list. These are proceedings of a professional society, from which
ideas were apparently taken, but not properly credited.

		--Dean