Subject: grow: draft-ietf-grow-anycast

From: Joe Abley <[email-masked]>
Date: Thu, 11 Aug 2005 09:09:07 -0400

I made some small changes to draft-ietf-grow-anycast-01 in response  
to Pekka's most recent suggestions (see expurgated diff below). If  
there are no other comments on -01, I'll submit this as -02. The  
revised draft will be a candidate for a WGLC.

Joe

@@ -790,9 +790,11 @@
     that address might be signalled using a (32-bit) host route.
     For IPv6-numbered services, Anycast Addresses are not scoped
-   differently from unicast addresses [RFC3513].  As such the  
guidelines
-   presented for IPv4 with respect to address suitability follow for
-   IPv6.
+   differently from unicast addresses.  As such the guidelines  
presented
+   for IPv4 with respect to address suitability follow for IPv6.  Note
+   that historical prohibitions on anycast distribution of services  
over
+   IPv6 have been removed from the IPv6 addressing specification in
+   [I-D.ietf-ipv6-addr-arch-v4].
4.6  Data Synchronisation
@@ -1237,6 +1237,11 @@
10.1  Normative References
+   [I-D.ietf-ipv6-addr-arch-v4]
+              Hinden, R. and S. Deering, "IP Version 6 Addressing
+              Architecture", draft-ietf-ipv6-addr-arch-v4-04 (work in
+              progress), May 2005.
+
     [RFC0793]  Postel, J., "Transmission Control Protocol", STD 7,
                RFC 793, September 1981.
@@ -1257,9 +1262,6 @@
                Defeating Denial of Service Attacks which employ IP  
Source
                Address Spoofing", BCP 38, RFC 2827, May 2000.
-   [RFC3513]  Hinden, R. and S. Deering, "Internet Protocol Version 6
-              (IPv6) Addressing Architecture", RFC 3513, April 2003.
-
     [RFC3704]  Baker, F. and P. Savola, "Ingress Filtering for  
Multihomed
                Networks", BCP 84, RFC 3704, March 2004.
@@ -1413,11 +1413,15 @@
        some structural reorganisation; general wordsmithing.  Document
        discussed at IETF 62.
-   draft-ietd-grow-anycast-01:  This appendix added; acknowledgements
-      section added; commentary on [RFC3513] prohibition of anycast on
+   draft-ietf-grow-anycast-01:  This appendix added; acknowledgements
+      section added; commentary on RFC3513 prohibition of anycast on
        hosts removed; minor sentence re-casting and related jiggery-
        pokery.  This revision published for discussion at IETF 63.
+   draft-ietf-grow-anycast-02:  Normative reference to [I-D.ietf-ipv6-
+      addr-arch-v4] added (in the RFC editor's queue at the time of
+      writing; reference should be updated to an RFC number when
+      available).

---

From: Dean Anderson <[email-masked]>
Date: Thu, 11 Aug 2005 19:10:37 -0400 (EDT)

I have some concerns about this draft.

First, the issue of Anycast DNS has been extensively discussed on the IETF
DNSOP WG, and is a very controversial notion.  Your draft should at least
acknowledge that Anycast is inappropriate for protocols that 1) use TCP,
2) require state, or 3) uses packets large enough to be fragmented.  
Abley participated in the DNSOP discussion.

Second, the draft fails to consider Per-Packet-Load-Balancing (PPLB) or
other fine grained load balancing behaviors which may be introduced at any
level in the network, including by the end-user.  PPLB and Anycast will
cause insurmountable problems for any protocol that (as before)  either 
  1)  uses TCP, 
  2) requires state, or 
  3) uses packets large enough to be fragmented.  

This behavior is not exclusively limited to Cisco PPLB. Any fine grained
load balancing (or a router architecture with lack of a flow or route
cache)  will induce Anycast problems.  While the draft considers some
issues with BGP configuration, the draft does not completely or thoroughly
discuss all routing issues involving Anycast, and is particularly silent
on fine grained load balancing and related issues.  PPLB is just one
example that is currently in Cisco production software and loaded on
thousands of routers [if not configured for use yet].  It is not the only
routing method that may route successive packets over different paths.  
This is a significant caveat that is omitted.  Indeed, one rather gets the
impression that through some BGP configuration, the end-to-end routing
path can be controlled, despite one caveat in the draft to the contrary.

Third, there seems to be little discussion of what protocols (besides
DNS--presumably why ISC is interested), might be suitable for Anycast.  
Examples of appropriate and inappropriate use of Anycast would be helpful
to prevent people from getting the wrong impression of suitability for a
particular protocol from your draft.

Fourth, there is little or no discussion or warning against the rather
perniciously difficult if not impossible to identify hazards of Anycast.  
This is rather similar to problems introduced by filtering packets in an
ethernet bridge.

In summary, there seems to be little acknowlegement of RFC1546 admonitions
for Anycasting;

===============================================================
How UDP and TCP Use Anycasting

   It is important to remember that anycasting is a stateless service.
   An internetwork has no obligation to deliver two successive packets
   sent to the same anycast address to the same host.
===============================================================

The BGP gyrations described by the draft seem to be an attempt to assume
that two successive packets will be sent to the same anycast address on
the same host.  However, neither BGP configuration, nor any routing
protocol can make such a network-wide guarantee.

		--Dean

**[I am working on text of a draft to be submitted to DNSOP WG to
recommend against using anycast for DNS, and to advise that Root DNS
Anycast violates the requirements RFC 2870 section 2.6, 3.3.1, and 3.3.2]

-- 
Av8 Internet   Prepared to pay a premium for better service?
www.av8.net         faster, more reliable, better service
617 344 9000   

_________________________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/grow.html
web archive:        http://darkwing.uoregon.edu/~llynch/grow/

---

From: Joe Abley <[email-masked]>
Date: Fri, 12 Aug 2005 14:23:13 -0400

On 11-Aug-2005, at 19:10, Dean Anderson wrote:

> DNSOP WG, and is a very controversial notion.

I do not believe this is an accurate assessment. Rather, the use of  
anycast to distribute DNS authority servers has been widely  
implemented, and has been studied, and I think it is reasonable to  
conclude that it is now an established technique.

>   Your draft should at least

The draft includes extensive discussion about the use of anycast in  
protocols carried over TCP, or otherwise requiring state. See  
sections 4.1 and 4.4.3, for example. If you have specific suggestions  
as to improvements, please send text.

>   3) uses packets large enough to be fragmented.

The draft includes discussion about the impact of equal-cost paths in  
section 4.4.3. Again, please send text if you have changes to suggest.

> DNS--presumably why ISC is interested), might be suitable for Anycast.

The draft is not at all specific to the DNS; it is a general  
discussion of the use of anycast for distributing services. Where DNS  
is mentioned, it is used as an example of a service for which anycast  
distribution is a tried and tested technique.

I asked the room in the grow meeting at IETF 62 whether anybody there  
considered the draft to be DNS-centric, and the consensus was that it  
was not. You are the first person apart from me to bring this up, in  
fact.

See section 4.1 for discussion about protocol suitability.

> perniciously difficult if not impossible to identify hazards of  

The draft discusses service monitoring in section 5, and security  
considerations in section 6. If you have suggestions as to how these  
sections could be made more clear, please send text.

> In summary, there seems to be little acknowlegement of RFC1546  

Since RFC 1546 was written there has been much operational experience  
gained in the use of anycast to distribute services. It is partly  
because of this additional experience (and the extent to which some  
of the advice in RFC 1546 has been proven to be over-cautious or  
pessimistic) that draft-ietf-grow-anycast has been written.

> protocol can make such a network-wide guarantee.

In fact, most of section 4 is concerned with describing exactly these  
hazards, together with many more.

> Anycast violates the requirements RFC 2870 section 2.6, 3.3.1, and  

Good luck with that.

Joe

---

From: Dean Anderson <[email-masked]>
Date: Mon, 15 Aug 2005 17:33:51 -0400 (EDT)

Long, inline.

In summary, there are many problems with this draft, including a 
misunderstanding of the routing system as "node selector" instead of "next 
hop forwarder", and the facts outlined in RFC1546. 

This draft rather reminds me of the complaints made about effects of
out-of-order packets on TCP performance. They showed inordinate
performance degradation due to out-of-order packets. They argued that
routers should be altered to disallow out-of-order packets. But two of the
fundamental features of IP are:

	1) that packet delivery is not reliable
	2) that packets may be delivered out of order.

TCP stacks that can't efficiently handle out-of-order packets have
implementation problems. The stacks should be fixed for efficient middle
insertion.  If we could assume that the IP layer was reliable and always
in order, we wouldn't need TCP at all. The purpose of TCP is to compensate
for unreliable and out-of-order packet delivery, so an application can
have an ordered, reliable stream.

Likewise, another fundamental fact of IP is 

   "An internetwork has no obligation to deliver two successive packets 
   sent to the same anycast address to the same host."

Despite the fundmental unchangeability of this fact, the present draft
attempts to change this fundamental fact.  Apparently, it plans to do this
with a magic routing genie which will change the nature of IP routing from
forwarding to something that it will globally choose a unique Anycast node
in every case. It does this magically, with only a little router
configuration:  "ip magic-anycast-node-selection-genie on"  Don't believe
it.

More inline.

On Fri, 12 Aug 2005, Joe Abley wrote:

> > First, the issue of Anycast DNS has been extensively discussed on the
> > IETF DNSOP WG, and is a very controversial notion.
> 
> I do not believe this is an accurate assessment. 

The controversy is well documented.  Before we get into that, lets do some
disclosure. The IETF requires that conflicts of interest be disclosed.  
You should disclose the facts involved in that ISC operates the F root DNS
server, and has substantial income from Anycast Root Services and Anycast
consulting services. ISC sells an Anycast Root Server copy to Latin
America and in Asia and elsewhere.  It has a financial stake in selling
DNS Anycast. DNS support now requires TCP and large UDP packets and
fragments.  Much of the controversy is beyond the control of GROW WG, but
full disclosure is always good.

Back to the controversy: As I understand it, the controversy arose because
ISC began making its claims about Anycast in a 2002 Nanog presentation. A
major point of criticism is that the ISC did not bring this first to the
IETF DNSOP group for study before telling operators that it was safe.

As was pointed by several people on DNSOP in 2002, your claim of safety is
not substantiated, nor studied. Your current repetition of this claim is
also undermined by the time line of events.  This draft was created _after_
the controversy about Anycast erupted.  

While you've made an _explicit_ attempt to make this draft not specific to
DNS, this actually belies your knowledge of this drafts' relevance to DNS.  
And that demonstrates that this draft is ISC's attempt to justify Anycast
as safe in general and specifically safe for DNS.  

The previous paragraph needs some clarity so I'll say it again in simpler
terms:  It seems ISC is hoping that the GROW WG will be uninformed of the
DNS controversy and that the GROW WG will be relatively unconcerned about
any false assumptions in your proposal: Anycast is optional. If it doesn't
work for everyone, or even at all...well...that just happens. No sweat for
GROW WG.  So it appears that ISC hopes to be able to argue to DNSOP (and
presumably Root server operations oversight) that the GROW WG has approved
Anycast as safe in general, and so it must be safe for DNS. So if it isn't
safe, ISC has GROW WG saying it is safe in the meantime.

Why this is important: ISC rationalization of its deployment of untested
and unstudied Anycast extensions are unacceptable. The appropriate
sequence should have been for ISC to have proposed the Anycast extension
in 2000, or 2001 so that extensive study and testing could have been done
a couple years before telling operators that the extension was safe and
deploying it on F-Root. So, it seems your draft exists to try to allay the
DNS controversy. The draft is not independent of the controversy, as you
_explicitly_ have tried to make it appear.

Of course, maybe ISC is getting out of DNS and into routing, that would
certainly be news.

> Rather, the use of anycast to distribute DNS authority servers has been
> widely implemented, and has been studied, and I think it is reasonable
> to conclude that it is now an established technique.

That's quite an exaggeration.  For modern DNS, with large UDP packets and
TCP, the answer is no: your extension is not widely implemented, nor even
tested.  Further, the only testbed that I know of is F-Root.  F-Root is
not an appropriate testbed site.

Another reminder of history:  ISC told operators in 2002 that its Anycast
extension was safe and uncontroversial; that its Anycast DNS extension was
a safe practice well before it was studied or even discussed. One of the
controversial issues was precisely that ISC made the 2002 Nanog
presentation without consulting anyone on the DNSOP (or DNSEXT) list. Its 
assurances weren't substantiated then, and still aren't.

Second, a relatively recent poll of Root operators indicated that about
half of the 13 root server operators say they have deployed Anycast or are
planning to based on ISC's assurances of safety. But even so, this is not
"widely implemented", but rather implemented a small number of very
critical servers. Servers that don't yet have much TCP or large UDP
packets, but soon will.  Despite widespread objection to the practice on
DNSOP, ISC has _still_ not notified root server operators of the
controversy, and continues to assure them of its safety.  On DNSOP, only
ISC-connected participants argued that the extension was safe.

At least one operator deployed Anycast in 2003 on the basis of an
incorrect belief of "no-controversy". (from Power point presentation
justifying change). In fact, controversy first erupted in 2002 on DNSOP
after Dr. Dan Bernstein drew attention to the Nanog presentation and the
invalid assumptions it contained.

Indeed, if your Anycast extension were an "established technique", this
draft would be completely unnecessary.  The function of the draft is to
establish the technique after the fact of ISC's deployment of it.

So: What studies were conducted before 2002?  
    Who conducted them?  
    How was fine grained load balancing studied?

Who approved use of F-Root as a testbed for this activity?

And why did ISC tell people it was safe in 2002?

Why does ISC still tell people it is safe in spite of the lack of testing,
documentation, or study?

Why does the draft completely ignore the subject of fine grained load 
balancing?

> protocols carried over TCP, or otherwise requiring state. See sections
> 4.1 and 4.4.3, for example. If you have specific suggestions as to
> improvements, please send text.

Nothing in section 4 alters the fact stated in RFC1546: that one cannot
guarantee that two successive packets will be delivered to the same
Anycast host.

> > In summary, there seems to be little acknowlegement of RFC1546
> > admonitions for Anycasting;
> 

Your statement is plainly incorrect. While there is operational experience
with RFC1546, which is to say experience with anycasting stateless small
UDP packets, by contrast there is almost no experience with your Anycast
extension: anycasting TCP or large UDP packets and fragments. There is no
theoretical support for the extension to TCP and large UDP packets and
fragments. Just the opposite. In theory, (as outlined by RFC1546) you
can't assume that two packets will go to the same Anycast host.  And there
are practical examples (fine grained load balancing)  that also show it
won't work.  Your present draft is the most detailed list of assumptions
documented so far about the ISC proposal. It has hardly been analyzed.

And I think that even your (unauthorized?) testing with F-Root has not
actually resulted in extensively testing TCP and large UDP packets and UDP
fragments.  Essentially, all you have done is test with small, stateless
UDP packets. But we have no objection to RFC1546. We do have an objection
to services like DNS that in the future will need to support TCP and large
UDP packets and fragments from sites that do fine grained load balancing.

You say there is much operational experience with this extension. If so,
what non-DNS operational experience is there with you Anycast extension?

> > The BGP gyrations described by the draft seem to be an attempt to
> > assume that two successive packets will be sent to the same anycast
> > address on the same host.  However, neither BGP configuration, nor any
> > routing protocol can make such a network-wide guarantee.
> 

It doesn't mention fine grained load balancing at all. Strange. It was 
extensively discussed on DNSOP as a fatal problem.

Section 4 talks a lot about the assumptions necessary for your extension.  
Not all of those assumptions are correct. However, it doesn't change
anything fundamental about IPv4 or IPv6; Particularly, it doesn't change
the fact that in IPv4 or v6, one can't guarantee (network-wide) that two
successive packets won't be sent to different Anycast hosts.  

But the very first sentence of section 4.1 reveals one of your wrong 
assumptions:

   When a service is anycast between two or more nodes, the routing
   system makes the node selection decision on behalf of a client.

Routing systems make forwarding decisions, not node selection decisions. A
remote router has no idea what node will finally receive any particular
packet.  It cannot choose a node.  It can only make forwarding decisions.  
And these forwarding decisions may involve fine-grained load balancing
across several circuits.  Fine grained balancing may be a design goal, or
not. It may simply be a consequence of not tracking flows while doing load
balancing.  Internet Gateways are not required to track flows. There is no 
global genie.

> > Anycast violates the requirements RFC 2870 section 2.6, 3.3.1, and  

Thanks. Your draft has been _very_ helpful with these tasks.  Just in case
you don't know, RFC 2870 section 2.6 specifies that Root DNS server
operators must operate servers that respond to _any_ Internet host.  That
is, Root nameservers that only work for say, 95% of the world are not
acceptable.

   2.6 Root servers MUST answer queries from any internet host, i.e. may
       not block root name resolution from any valid IP address, except
       in the case of queries causing operational problems, in which
       case the blocking SHOULD last only as long as the problem, and be
       as specific as reasonably possible.

Your draft includes a slew of assumptions that aren't true of the entire
Internet.  Now I don't have to prove what assumptions you are making.  
You've just laid them out, indisputably. They are now stipulations and so
my task is easier.  Your effort at proving your extension could work for
_someone_, _somewhere_ plainly shows that it doesn't work for everyone,
everywhere. But that's what RFC 2870 requires of DNS Root servers.

---

From: Joe Abley <[email-masked]>
Date: Mon, 15 Aug 2005 19:05:07 -0400

> Long, inline.

For the benefit of the archive, I don't intend to address the points  
raised by Dean in this message; I believe the technical points were  
already covered (either in the draft or in my previous reply), and  
the extensive, non-technical commentary is off-topic for this list  
(and is easily clarified in any case by spending 5 minutes with Google).

Joe

---

From: Dean Anderson <[email-masked]>
Date: Tue, 16 Aug 2005 15:12:02 -0400 (EDT)

For the benefit of the archive, this is just about exactly the response
ISC has given Dr. Dan Bernstein (initial posts), Iljitsch van Beijnum (who
noted UDP fragmentation issues--it isn't just TCP problem) and others who
have contributed to these questions on DNSOP.  We were stonewalled there
too.  

In short, these (repeated) questions have never been answered by ISC.

In fact, many of the questions are more appropriately on-topic to DNSOP,
but weren't answered there either.  But I think GROW should consider the
DNS issues because they are not impossibly esoteric. Since the extension
is only implemented with DNS, DNS issues have to be considered to some
extent.  However, the routing and packet load balancing criticisms are far
more relevant to GROW than they would be to DNSOP.

ISC should substantiate the claims of testing and studying that it made to
the GROW WG. (and others).  It should answer the questions asked fully and
honestly.

But to show the, umm, Inaccuracy of Abley's latest claim of having
answered all questions, consider this:

Fine grained load balancing has been brought up by several people over
several years, (before the anycast-draft was submitted to GROW even!) and
the current draft doesn't even contain the word 'balancing' or 'balance'
except to talk about the balancing of load between anycast nodes. It
doesn't address AT ALL the impact of fine grained load balancing of
packets on routers, an issue that seems pretty well-suited to GROW.  Yet
Abley _claims_ all technical points are addressed in the draft. This issue
seems pretty technical, and is more on-topic for GROW than DNSOP.
Ironically, this is yet another unsubstantiated claim by Abley and ISC.

Well, I suppose when one is pushing something that wouldn't stand any
critical examination, probably the best thing to do is refuse to answer
questions.  But of course, critical examination and consensus building is
the role of a WG. 

So, is there a consensus on the GROW WG to give an assurance that this
Anycast extension for Anycast TCP and UDP fragments works for everyone,
everywhere with the specified configuration changes made only by the
Anycast server operators?  [with the implication being that this Anycast
extension is safe for DNS Root Server large UDP and TCP services] I rather
expect the answer is going to be no, but of course, I can't answer that 
question.

Is there a consensus that this proposed Anycast extension is 'unsafe at
any speed', as it were, for universal client use of Anycast TCP and
fragmented UDP protocols? [the implication being that Anycast DNS
operations should be immediately stopped, before clients begin using TCP
and large UDP packets]

I think the inability or unwillingness by the author to substantiate the
claims made in the draft and by the author in support of the draft should
signal that this draft is not yet ready for last call.

		--Dean

---

From: Alan Barrett <[email-masked]>
Date: Tue, 16 Aug 2005 22:21:44 +0200

On Tue, 16 Aug 2005, Dean Anderson wrote:
> the current draft doesn't even contain the word 'balancing' or
> 'balance' except to talk about the balancing of load between
> anycast nodes. It doesn't address AT ALL the impact of fine grained
> load balancing of packets on routers, an issue that seems pretty
> well-suited to GROW.

I believe that the above part of Dean's criticism is valid, and I think
that it would be worthwhile for a section on per-packet load balancing
to be added to the draft.  This section could point out the places where
PPLB is or is not likely to interact badly with anycast services that
require multiple packets.  For example:

   per-packet load balancing across multiple parallel links between the
   same pair of routers is almost always safe;

   per-packet load balancing across diverse paths within your own AS,
   but where the paths converge before exiting your own AS, is likely to
   cause performance problems traceable to packet re-ordering, but these
   problems are unlikely to be any worse with anycast services than with
   non-anycast services;

   per-packet load balancing across links to different neighbouring
   ASes (such as different upstream providers) may cause serious
   problems with anycast services that are implemented according to the
   suggestions in this draft, because the different links might lead to
   different instances of the anycast service;

I suggest that the draft could recommend "don't do the problematic types
of load balancing", and could observe that hardly anybody does the
problematic type of load balancing anyway.  This is opposed to Dean's
apparent preference for recommending "don't do anycast for services that
use TCP".

--apb (Alan Barrett)

---

From: Pedro Roque Marques <[email-masked]>
Date: Tue, 16 Aug 2005 13:44:24 -0700

> hop forwarder", and the facts outlined in RFC1546. 

Dean,
My understanding of your e-mails is that your fundamental concern
relates to anycast and load balancing.

It seems to me that the issue is whether: 
 - if an originating AS advertises a given "anycast" destination to 2
different neighboring ASes: 1 and 2 with 2 different servers mappings A,
B
 - can an AS Y downstream of 1 and 2 balance traffic arbitrarily between
servers A and B ?

i.e. who controls the distribution of traffic to different servers ? the
advertising AS or some random transit AS.

If the latter then a arbitrary transit (or traffic originating AS) can
break the service. If the former, then proper management of routing
advertisements can guarantee that the service will work correctly.

My current understanding, and this is the area which is directly
relevant to grow, is that even if an AS downstream of 1 and 2 sees 2 BGP
paths which are equal up-to the router id criteria of BGP path
selection, that such AS y is not allowed and will not load-balance
traffic between these 2 different paths. BGP load balancing requires the
neighboring AS to be equal for paths to be acceptable (otherwise
forwarding loops are possible).

As such the control of which servers are access by different sources
does is up-to the originating AS of route advertisements. Thus,
"anycast" type route advertisements are not more problematic than
non-anycast route advertisements.

  Pedro.

---

From: Pedro Roque Marques <[email-masked]>
Date: Tue, 16 Aug 2005 14:57:58 -0700

On Tue, 2005-08-16 at 22:21 +0200, Alan Barrett wrote:
> to be added to the draft.

I'd argue that it is not. That in fact load-balancing is not in play
when anycast is used. Inter-domain routing (bgp) will deterministically
pick one of the available paths.

I may be wrong. But i've yet to see any explanation of how multiple
paths would be acceptable for load-balancing.

  Pedro.

---

From: Alan Barrett <[email-masked]>
Date: Wed, 17 Aug 2005 09:33:36 +0200

On Tue, 16 Aug 2005, Pedro Roque Marques wrote:
> > I believe that the above part of Dean's criticism is valid, and I
> > think that it would be worthwhile for a section on per-packet load
> > balancing to be added to the draft.
>
> I'd argue that it is not. That in fact load-balancing is not
> in play when anycast is used. Inter-domain routing (bgp) will
> deterministically pick one of the available paths.
>

The people who want to do per-packet load balancing across multiple
paths might not be using BGP in the way you expect.  For example, they
might have multiple default routes to multiple upstream providers, and
they might try to load balance over the default routes, using some
mechanism outside BGP.  They might just have static default routes to
their upstream providers.  They might be using a private extension to
BGP that allows multiple routes to the same destination.  They might be
in the future, using some routing protocol not yet invented.

If packets addressed to the same destination IP address are
load-balanced across multiple default routes that end up going to
different neighbouring ASes, and if the destination IP address is that
of an anycast service, then different packets can end up going to
different instances of the anycast service.  This will not work.  So
people should be advised not to do it.

--apb (Alan Barrett)

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 17 Aug 2005 09:50:40 -0400

> pick one of the available paths.

While ECMP for eBGP isn't standardized, several major vendors have
implement some form of this.  Given the lack of standardization I'm
not sure how you can say that load-balancing isn't in play.

> paths would be acceptable for load-balancing.

In that, I think we're in rabid agreement.

-- 
Jeff Haas 
NextHop Technologies

---

From: Pedro Roque Marques <[email-masked]>
Date: Wed, 17 Aug 2005 10:09:37 -0700

> not sure how you can say that load-balancing isn't in play.

I disagree. There is a de-facto standard. Most of it is just direct 
consequence of how BGP works (i.e. as-path as loop detection).

Thus, i still do believe that we can infeer the behaviour of transit 
ASes regarding multiple unicast advertisements.

   Pedro.

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 17 Aug 2005 13:49:35 -0400

> ASes regarding multiple unicast advertisements.

From the document in question:
http://www.ietf.org/internet-drafts/draft-ietf-grow-anycast-01.txt

: 4.4.3  Equal-Cost Paths
: 
: [...]
: 
:    Equal cost paths are commonly supported in IGPs.  Multi-node
:    selection for a single transaction can be avoided in most cases by
:    careful consideration of IGP link metrics, or by applying equal-cost
:    multi-path (ECMP) selection algorithms which cause a single node to
:    be selected for a single multi-packet transaction.  For an example of
:    the use of hash-based ECMP selection in anycast service distribution,
:    see [ISC-TN-2004-1].
: 
:    For services which are distributed across the global Internet using
:    BGP, equal-cost paths are normally not a consideration: BGP's exit
:    selection algorithm usually selects a single, consistent exit for a
:    single destination regardless of whether multiple candidate paths
:    exist.  Implementations of BGP exist that support multi-path exit
:    selection, however, and corner cases where dual selected exits route
:    to different nodes are possible.  Analysis of the likely incidence of
:    such corner cases for particular distributions of Anycast Nodes a
:    recommended for services which involve multi-packet transactions.

Thus, your BGP ECMP algorithm must not only select multiple paths
that are good for purposes of BGP protocol correctness - lack of
standardization aside.  It must also make sure that multiple-packet
traffic for a given node is sent consistently to one of the paths.

An example of this corner case is where the same AS has more than one
instance of an anycast service.  In one vendor's BGP multipath
algorithm, if you select more than one path to this destination in
the same AS, you may still have traffic that may reach different anycast
node instances.

While this is indeed a corner case, it relies on that particular vendor's
implementation to "get it right".  Given that this is a non-standardized
feature and that I've heard several non-optimal multipath suggestions,
this document would be well served by having some sort of caveat in it.

E.g.:

"In cases where a BGP implementation supports multi-path exit selection
in such a fashion that it is possible that routes to different Anycast
Nodes may result, that implementation SHOULD support a feature to
disallow multi-path route selection for Anycast reachability."

Effectively, I think this means "the bgp routes should be almost identical
and that hash-based ECMP traffic forwarding" are requirements.  
Putting such a thing in this document is probably inappropriate. :-)

-- 
Jeff Haas 
NextHop Technologies

---

From: Pedro Roque Marques <[email-masked]>
Date: Wed, 17 Aug 2005 12:15:55 -0700

> the same AS, you may still have traffic that may reach different anycast

Jeff,
If the originating AS advertises a single mapping (prefix -> server) to
its neighbors, the downstream ASes will not override this decision. It
is up to the originator to control where the traffic gets received.

As for the issue of consistently selecting one of multiple paths... take
one unicast routing advertisement from AS X. If there is multipath
downstream and a consistent path is not selected for a given 5-tuple
(src,dst,proto,port#s), plain IP features like PATH MTU discovery may
not work correctly.

The problem exists independently of "anycast".

> feature

All depends on one's definition of standard. I'd argue that BGP
multipath "works consistently among different vendors", despite the lack
of an IETF approval stamp.

>  and that I've heard several non-optimal multipath suggestions,

There is no reason why BGP needs or should treat "anycast" different
from unicast... in fact it MUST NOT know whether there is more than one
advertisement for a prefix. This kind of text is harmful.

  Pedro.

---

From: Dean Anderson <[email-masked]>
Date: Wed, 17 Aug 2005 17:35:23 -0400 (EDT)

>    different instances of the anycast service;

Exactly right.  However, these situations are not always explicitly
chosen.  For example, Av8 Internet offers redundant service connections
via T1 and DSL. The T1 and DSL services are handled at different sites,
and outgoing packets from those sites will be sent through different
upstreams.  If customers turn on fine-grained load balancing at their CPE,
we have no knowledge, nor any control over this. Customers are _very_
interested in having this ability. Nearly every customer wants to do load
balancing.  We also use equal cost static egress routes on our interior
multiply connected routers, load balancing our exiting packets to the
several upstreams.  This is coarsely balanced with flow caches (mostly
Cisco 7500 and 3600 series equipment).  To enable fine grained load
balancing, one just configures the flow cache off. Newer routers don't
have flow caches, I don't know how they keep flows together, or if they
do.

Further, as pointed out, in principle, fine grained load balancing may
occur for reasons other than by explicit configuration goal: Simply
because there are no flow-tracking structures in the router to enforce
flow-based routing. For example, the Cisco GSR has no route/flow
cache--but I don't know what its load balancing behavior is.  I know of no
IETF document that prohibits fine grained load balancing. Just the
opposite: RFC1546 (at least) indicates that you must be prepared for the
case when two successive packets to the same source address follow
different paths.

But in principle, anyway, such a router starts fine grained load balancing
anytime it sees equal cost routes to the same destination.  It doesn't
matter how those routes were entered into the routers route table. For
example, if such a router is used as a BGP core router, then anytime a
structure exists from two or more peer AS through to a single end AS, and
the BGP costs are equal, then fine grained load balancing would begin.  
And for example, equal cost static routes would have the same effect. And
many sites use equal cost static default egress routes on their multiply
connected interior routers.  This would have to change.

> apparent preference for recommending "don't do anycast for services that

There is only one problem with this (at least for services that have
global usage such as DNS Root Server services):  such a declaration means
that you have imposed _on the entire Internet_ an additional requirement
that:

	Any two successive packets MUST be delivered to the same host over 
        exactly the same path.

        Practically, this means that all routers MUST have a flow cache or
equivalent mechanism. Or else not allow multiple routes to the same
destination prefix in the route table.  I think this is a rather draconian
demand on the entire Internet.

Making this demand on explicitly cooperating ASs might be more reasonable.
However, in that case, I suggest that, at minimum, the draft has to
indicate that this extension can ONLY be used for those protocols that use
TCP and Large UDP and fragments where use is LIMITED to use exclusively
within the explicitly cooperating AS's, and that those AS' have verified
that they don't have any possibility for fine grained load balancing over
diverse paths.  In other words, this extension can't be used for globally
used Root DNS services.

		--Dean

---

From: Joe Abley <[email-masked]>
Date: Wed, 17 Aug 2005 18:54:17 -0400

On 17 Aug 2005, at 13:49, Jeffrey Haas wrote:

> [...]

Such a caveat is there in the last sentence that you quoted. Do you 
think additional text is required?

> disallow multi-path route selection for Anycast reachability."

This sounds like text more appropriate for RFC1771's successor than for 
this draft? It's also BGP-specific, whereas I intended the text to be 
more protocol-agnostic (although I was not as explicit as apb was in 
his earlier summary).

Joe

---

From: Pedro Roque Marques <[email-masked]>
Date: Wed, 17 Aug 2005 16:02:15 -0700

On Wed, 2005-08-17 at 17:35 -0400, Dean Anderson wrote:
>  This is coarsely balanced with flow caches (mostly
> balancing, one just configures the flow cache off.

Load balancing doesn't depend on a flow cache.

Most systems enforce a deterministic choice for a given 5-tuple
(src,dst,ports) by using a hash function on that 5 tuple.

>  Newer routers don't

They do have the ability to do deterministic load balancing. Most don't
have the ability to to per-packet.

> flow-based routing.

Deterministic load balancing doesn't require flow tracking.

> For example, the Cisco GSR has no route/flow
> cache--but I don't know what its load balancing behavior is.

cisco documentation is resonably clear on this, imho.
google's says:
http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/12cgcr/switch_c/xcprt2/xccefc.htm#wp3589

cisco CEF -> default per-destination lb. per-packet not really supported
on some hardware.

so called per-packet (non-deterministic) LB can easily break plain
unicast features: example TCP PMTU discovery.

Distributing packets between paths of different MTUs is a sure way to
cause havoc.

>   I know of no

That doesn't mean it is a good idea to split traffic of same flow
between different paths.

> anytime it sees equal cost routes to the same destination.

My point is that "anycast" doesn't imply "equal cost"... you seem to
assume that it does. In fact one would use anycast to have unequal cost.

So that some traffic is attracted to different points.

The reason that anycast works well in practice is that:
a) the advertisements are intentionally unequal in cost.
b) equal cost for BGP implies same neighboring AS.
c) in practice, non-deterministic load-balancing of flows tends to cause
a lot of issues for any sort of TCP flows.

>   It doesn't

That is incorrect. That is the point i'm trying to make...
It is only acceptable for a BGP router to balance between equal cost
paths of same neighboring AS.

> connected interior routers.

What they don't usually do is non-deterministic flow load-balancing. If
they did they would find out their TCP performance would tend to suck.

> 	Any two successive packets MUST be delivered to the same host over 

That is not quite the case.
The requirement is that a transit node should make its best possible
effort to have a consistent decision between multiple available paths
for the same flow. I argue that the requirement exists today in
practice.

> equivalent mechanism.

Your assumption is incorrect here.

  Pedro.

---

From: Dean Anderson <[email-masked]>
Date: Thu, 18 Aug 2005 15:57:23 -0400 (EDT)

> (src,dst,ports) by using a hash function on that 5 tuple.

On IOS, to enable PPLB, you disable the flow cache. That's all there is to
it (well, you have to have at least two routes to the same destination).
There is a Cisco URL, if you care, I'll dig it up.

If you don't have a flow cache (or its disabled), or some similar
mechanism, and you enter multiple routes to the same destination in the
route table, and the packet forwarding decision selects among multiple
routes (ie balances), then by definition, you will get fine grained load
balancing behavior. You may not have explicitly chosen this behavior, but
you don't have any structure to prevent it.

> >  Newer routers don't have flow caches, I don't know how they keep
> > flows together, or if they do.
> 

How do they do this without a flow cache? 

And lets not get too caught up in existing routers. We are talking about
the requirements for routers. Conceptual, hypothetical routers, even.  
There is presently no requirement that routers have structures to prevent
fine grained load balancing.  So the question isn't whether it happens
very often now or whether its a good idea, its whether you have to be
prepared for that possibility.  [but in fact, PPLB does happen now, and
many people think its a good idea]

> Deterministic load balancing doesn't require flow tracking.

It requires something: random selection from multiple routes in the route
table for each packet results in fine grained balancing.  Of course, if
you don't allow multiple routes to the same destination in the route
table, you can't load balance at all.

> cisco CEF -> default per-destination lb. per-packet not really supported

You seem to have missed the link on that page that says:

"Configure Per Packet Load Balancing"

What the default is, doesn't matter to us.

> unicast features: example TCP PMTU discovery.

This is wrong. [and irrelevant, but I'll answer anyway] The PATH MTU would
become the minimum MTU over any path. That is not "broken". Any and all
paths can send the necessary ICMP messages. Nothing is broken.

> Distributing packets between paths of different MTUs is a sure way to

Nonsense.  But that isn't the question. Some people say out-of-order
packets cause havoc, too. However, the IETF RFCs say that you must be
prepared to handle out-of-order packets.  The IETF RFCs also say that you
must be able to handle two successive packets over the differnent paths.  
PPLB is allowed behavior.

> That doesn't mean it is a good idea to split traffic of same flow

"Good idea", and "must be able to handle" are different.

> assume that it does. In fact one would use anycast to have unequal cost.

I didn't say it _implied_ equal cost. But neither does it deny equal cost.  
What hasn't been denied, is possible and so must be considered. 

One cannot enforce the assumption that anycast not have equal cost routes
anywhere in the internet to different hosts.  This isn't possible with 
BGP.

Equal cost could be 1,6,12,15 (etc) hops away from 2 different anycast
hosts.  Even if you add something to your announcements, there may be
offsetting path differences so that two paths are still equal at some
remote AS.

And of course, BGP isn't necessary.  It could come from load balancing
over static default exit routes.

> a lot of issues for any sort of TCP flows.

Err, no. The reason that anycast (RFC1546)  works in practice is that
stateless small UDP datagrams are used. There is just one packet per
request. 

By contrast, TCP has not been tested in practice, except in very limited
environments where PPLB has not been used during tests.  Introducing PPLB
into test Anycast environments causes TCP failure and also failure with
large UDP packets and fragments which are also required for global DNS
Root Servers. We expect therefore that any practice involving Anycast of
TCP and large UDP packets and fragments on the global internet where we
know already that PPLB is used to some extent, will also fail [and thus
violates RFC 2870 requirements].  We have to choose between PPLB and TCP
Anycast for any global service like Root DNS Service.  [and since there
are alternatives to DNS Anycast to get more than 13 root servers, those
alternatives are better than globally denying PPLB]

> >   It doesn't matter how those routes were entered into the routers
> > route table. For example, if such a router is used as a BGP core
> > router, then anytime a structure exists from two or more peer AS
> > through to a single end AS, and the BGP costs are equal, then fine
> > grained load balancing would begin.
> 

What BGP does, doesn't matter. BGP is not the sole enterer of routes into
the route table used to route packets. Read my statement again.

But I think you have the BGP issues wrong, too, but it doesn't matter.  If 
you were right then:

E1 and E2 are anycast hosts in AS E on same IP.

 Like this:

     A
    / \
   B1  B2
   |    |
   C    D
   |    |
   E1  E2

AS A connects to two different routers in AS B, and each of these 
routers prefers a different external path towards different anycast 
instances of AS E. 

But I think that BGP can load balance this, as well:

     A
    / \
   C    D
   |    |
   E1  E2

Paths to AS E are equal cost for BGP and could both entered into the route 
table at A.

But lets not get hung up on what BGP does.  What BGP does doesn't matter.

Its what happens after there are equal costs routes in the route table
that matters. How they got there is irrelevant.  For our purposes, we can
assume they get there _somehow_ unless we can prove it is impossible for
that to ever happen. And were aren't anywhere close to that.

There is also no requirement that a router run BGP. It is possible, if
unwieldy, to peer with static routes. Or peer with some other routing 
protocol yet to be invented.  Our question is what one must be prepared to 
handle.

> they did they would find out their TCP performance would tend to suck.

Tested, and works on production Cisco with BGP, OSPF and static routes.  
But again, what is "usual" and what "you must be prepared to handle", are
quite different things.

TCP performance for out-of-order packets sucks due to TCP stack
implementation problems. Fix those, and performance is good again. But it
is still quite a different thing to say "you do not need to handle
out-of-order packets".  Likewise, we are not able to say "you do not need
to handle successive packets that take different paths".

And Root DNS servers do need to handle this situation according to RFC
2870.  Globally denying PPLB seems beyond the scope of this draft.

> That is not quite the case. The requirement is that a transit node
> should make its best possible effort to have a consistent decision
> between multiple available paths for the same flow. I argue that the
> requirement exists today in practice.

Consistent decisions at the transit node do not ensure globably unique
paths.  End nodes can do load balancing, and a router (transit node or
end-user) has no requirement to have flow caches or any equivalent
mechanism to prevent fine grained load balancing.  I gave examples that
one gets PPLB on any cisco router that usually has a flow cache, but the
flow cache has been turned off. The flow cache might be turned off
explicitly to get PPLB, or it may be turned off for some other reason, and
PPLB is just a side effect.  Or some future router may simply not have the
capability to not do PPLB: flow-cache-like structures are necessary to
avoid PPLB, and if for some reason the router doesn't have that (the GSR
doesn't have a flow cache), then PPLB is the only behavior for multiple
routes in the route table (and as I said before, I don't know what the GSR
LB behavior is).  But you haven't been reading.

> Your assumption is incorrect here.

This isn't an assumption. Its a practical restatement of the assertion 
above.   You need to explain what mechanism prevents fine grained load 
balancing anywhere in the entire Internet. And how in theory, you ensure 
that:

So that Anycast TCP and large UDP and fragments will be guaranteed to work 
for everyone, everywhere.

---

From: Pedro Roque Marques <[email-masked]>
Date: Thu, 18 Aug 2005 13:46:49 -0700

> We are talking about
> fine grained load balancing.

Lets go down to basics:

You have an unicast advertisement from originating AS 1, which it
advertises to 2 peer ASes (2, 3). In some random AS (4) where a source
is located, its systems see two "comparable" paths, one that happens to
transit through 2 and another through 3.

What are the requirements for load balancing for TCP congestion
detection / avoidance on this system ?

If AS 4, as a load balancing behaviour of sending each consecutive
packet through AS 2 and 3 paths, TCP for this unicast routing
announcement will not work correctly. TCP RTT measurements, congestion
detection / avoidance, PMTU, all assume that you do have *one* path for
a flow.

The analysis of anycast has to start from the assumption that you system
components do work correctly in unicast. Or it is completly
uninteresting.

> On IOS, to enable PPLB, you disable the flow cache. 

This is an incorrect statement. You can both do "per-destination" load
balancing with or without flow cache in most cisco boxes.
Load balancing defaults to per-destination.

So disabling "flow cache" will not automatically enable per-packet load
balancing. Which is not even supported in most platforms.

> balancing behavior.

This is incorrect.

>  You may not have explicitly chosen this behavior, but

> How do they do this without a flow cache? 

I explained this in the previous e-mail. Most systems use a
deterministic hash on the packet header to pick which of the N-possible
paths to forward traffic.

  Pedro.

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 19 Aug 2005 16:42:33 -0400

> How do they do this without a flow cache? 

In an attempt to try to shorten this particular part of the thread:

Given a prefix, P with a set of nexthops nh1-nhx, forward
traffic to a destination p in P by calculating a hash over some
data in the header that maps deterministically to a given nexthop bucket.

This data is typically going to be the same tuples one would consider
part of a flow.  I.e. src/dst address, src/dst port, etc.

As long as something like this is done, you don't need to keep any
flow state.  This does not guarantee optimal load balancing.

-- 
Jeff Haas 
NextHop Technologies

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 19 Aug 2005 17:31:26 -0400

Joe,

> think additional text is required?

To wordsmith the original section a little:

   For services which are distributed across the global Internet using
   BGP, there are implementations of BGP which support multi-path exit
   selection.  Corner cases where dual selected exits route
   to different nodes are possible.

In other words, just say that it can happen.  The question is where
it is appropriate to make a recommendation, as in the IGP section above it,
to deal with the problem.

> this draft?

I think I mentioned in the original reply that this wasn't the place
for it.  The BGP ECMP spec would be the right place if we ever agree
on one.

-- 
Jeff Haas 
NextHop Technologies

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 19 Aug 2005 17:54:04 -0400

> The problem exists independently of "anycast".

It's even simpler than that.  Mechanisms like path mtu have to deal
with a topology change where the path has a different mtu.

The scenario I had in mind is this:

     _____
    (     )
   ( a   a )
    (     )
     0___0
     |   |
     1   1
      \ /
       2

Two anycast instances exist in AS 0.  Each border router advertises
the prefix for a.  2, in the above example, is the first chance for
ecmp.  As long as 2 deterministically sends its traffic for a given
flow to the same peer, we don't have a problem.   If we don't send
them deterministically, we have a problem.

If there's just one instance of a, then the traffic iwill
consistently end up at that a regardless of the path.  Most vendors
will be sane enough to insist that the bgp paths are otherwise
identical.  You not only have to have them be identical, you must
have the flow directed deterministically.  You can skip this last
step if a wont have multiple instances.

In the cases where a is present from multiple origin ASes, I'm not aware
of a BGP ecmp implementation that will load balance the traffic.

While I'm sure you can say your implementation "does the right thing"
and the $OTHER_LARGE_VENDOR "does the right thing", it's hard to say
that everyone does the right thing.  If anyone doesn't, there's a problem.

Honestly, just given normal Internet topology changes, relying on
a given path to a given anycast instance being stable for a large
traffic flow seems somwehat unlikely.  But I'll leave it to those who
keep such statistics to speak on the subject.

> advertisement for a prefix. This kind of text is harmful.

Yet you've already suggested that people's implementations have been
written in such a way as to deal with this problem.

I would agree that bgp shouldn't differentiate unicast and anycast,
but I'd be shocked if there aren't people who have asked for per-packet
load balancing that ignores flow determinism.  I think Dean was alluding
to that.  In those cases, one needs to be aware of what you're load
balancing and why it might be a Bad Thing to do so.

-- 
Jeff Haas 
NextHop Technologies

---

From: Dean Anderson <[email-masked]>
Date: Tue, 23 Aug 2005 16:45:00 -0400 (EDT)

> data in the header that maps deterministically to a given nexthop bucket.

This would seem to fall under "some similar mechanism" that I was positing
in the paragraph above the one your replied to. I should have added the
same qualification to my question, too.  My point is that these mechanisms
are not required for an Internet Gateway and that PPLB is allowed
behavior, whatever your opinion of its performance implications.

Lets just summarize what we agree on:

No one seems to be saying that this anycast extension will work when PPLB
is used.   Lets forget about whether PPLB is a good idea or how one might 
implement course-grained load balancing.

Is there a consensus that this anycast extension will not work with PPLB?

Do we agree that a clear warning about the incompatibility of this anycast
extension with PPLB should be prominently placed in the draft?

Do we agree that for this anycast extension to work properly, one must
guarantee that 2 or more successive packets must go to the same Anycast
host?

		--Dean

---

From: Dean Anderson <[email-masked]>
Date: Tue, 30 Aug 2005 19:42:54 -0400 (EDT)

Missed this:

> detection / avoidance on this system ?

None, ( RFC 1812 )  See section on "load splitting". 

On page 73, 5.2.4.3 Next Hop Address

   It is also possible for the
   algorithm to terminate when more than one route remains in the set.
   In this case, the router may arbitrarily discard all but one of them,
   or may perform "load-splitting" by choosing whichever of the routes
   has been least recently used.

Selecting the route "Least-recently used" for the next hop address results 
in fine grained load balancing. But it gives a great deal of lattitude in 
the load-splitting algorithm:

On page 78, 5.2.4.5 Load Splitting

     At the end of the Next-hop selection process, multiple routes may
     still remain.  A router has several options when this occurs.  It
     may arbitrarily discard some of the routes.  It may reduce the
     number of candidate routes by comparing metrics of routes from
     routing domains that are not considered equivalent.  It may retain
     more than one route and employ a load-splitting mechanism to divide
     traffic among them.  Perhaps the only thing that can be said about
     the relative merits of the options is that load-splitting is useful
     in some situations but not in others, so a wise implementor who
     implements load-splitting will also provide a way for the network
     manager to disable it.

This is why RFC1546 notes that:

Particularly, you should pay attention to the second sentence:

> announcement will not work correctly. 

It works correctly, now.  Perhaps you could explain.

> TCP RTT measurements, congestion detection / avoidance, PMTU, all assume
> that you do have *one* path for a flow.

They don't assume one flow. Perhaps **you** __assume__ one flow.  But
RFC1812, etc makes no such assumption.  RFC2991 is Informational, it
expresses an opinion, but doesn't change the fundamentals.

As explained, PMTU works fine with fine grained load balancing: The PMTU
becomes the smallest MTU over any path. If PMTU discovery is stopped by
the client after the first packet, then the DF flag is cleared, and in
that case, the other paths can fragment.  I should note that most clients
that use PMTUD don't ever disable it, but it is permissible to stop PMTU
discovery after a session is established.

RTT measurements are of no consequence. There may already be a great deal 
of variation in RTT over one path. Multiple paths just (at most) add to 
the variation.

The point here, is that one should never assume there is only one path.  
If you've made such an assumption, you've made a mistake.

RFC2991 details some potential problems with multipath. It discusses PMTU
issues and TCP performance issues as reasons to avoid multipath. As I
showed, the PMTU issue was a red-herring.  It also mis-states the TCP
issue. RFC2581 describes the Fast retransmit behavior:

   "The TCP sender SHOULD use the "fast retransmit" algorithm to detect
   and repair loss, based on incoming duplicate ACKs.  The fast
   retransmit algorithm uses the arrival of 3 duplicate ACKs (4
   identical ACKs without the arrival of any other intervening packets)
   as an indication that a segment has been lost.  After receiving 3
   duplicate ACKs, TCP performs a retransmission of what appears to be
   the missing segment, without waiting for the retransmission timer to
   expire.

RFC2991 mis-states this as follows:

         When three or more packets are received before
         a "late" packet, TCP enters a mode called "fast-retransmit"

This is not the case. [However, if it were the case, it would still only
affect 6% of the packets.]  

A more thorough reading of RFC2581 reveals when an ACK should be sent:

   A TCP receiver SHOULD send an immediate duplicate ACK when an out-
   of-order segment arrives.  The purpose of this ACK is to inform the
   sender that a segment was received out-of-order and which sequence
   number is expected.  From the sender's perspective, duplicate ACKs
   can be caused by a number of network problems.  First, they can be
   caused by dropped segments.  In this case, all segments after the
   dropped segment will trigger duplicate ACKs.  Second, duplicate ACKs
   can be caused by the re-ordering of data segments by the network (not
   a rare event along some network paths [Pax97]). 

While out-of-order packets could trigger the fast retransmit, it occurs
just 3% of the time.  So just 3% of packets are unnecessarilly
retransmitted.  Not a great performance impact.

But again, at worst, this is merely a performance issue that may be more
than compensated for by the additional performance of multiple links for
more sessions and other beneficial characteristics of having multiple
paths available.  

Lets consider just a moment what those benefits are:

For example, when a path fails, it can be immediately removed from the
routers FIB, and another path can be immediately used without waiting for
routing processes to select the next best route and add it to the FIB. [no
more blackholes until next BGP scan after link failure]

VOIP RTP buffers have no such performance issues with multipath. As long
as each packet arrives before it is to be consumed, it does not matter
what order they arrive in.  PPLB would greatly improve VOIP performance
characteristics.

This is just a short couple of examples of what the benefits to 
multipath/PPLB/load-splitting/fine-grained-load-balancing are.

But aside from the benefits of multipath, and aside from the fact that one
might be able to do course-grained load-balancing, my point is that this:

==========================================================================
The Anycast extension will not work with PPLB, and that PPLB is allowed
behavior. Therefore, It should be well documented and clearly warned that
ANY use of this Anycast extension requires that no user and no
intermediate path served by this Anycast Extension use
load-splitting/multipath/PPLB/etc. [And this requirement precludes use of
this extension for Root DNS service]
==========================================================================

> Load balancing defaults to per-destination.

It can be turned off per interface on some boxes. The manual for the GSR 
Engine2 indicates that all interfaces should have the same configuration. 
IE, global.

> balancing. Which is not even supported in most platforms.

Its different on GSR versus the other platforms. On the GSR, it a
hw-module command.  The others all have you do "no route-cache".

> > routes (ie balances), then by definition, you will get fine grained load

It is certainly correct on Cisco. There are different commands on GSR, but
the job is the same. Actually, its correct on everything.  Everything else
such as the 3 methods described in RFC 2991 falls under "some similar
mechanism"

> paths to forward traffic.

As I explained, that is "[a] similar mechanism".  Its not required by 
RFC1812, which permits fine grained load balancing. 

Telling us how to do course grained load balancing is not an answer.  
There is no requirement in RFC1812 for routers to even BE ABLE to do
course grained load balancing.

		--Dean