Subject: [GROW] ISC Response to draft-ietf-grow-unique-origin-as

From: Leo Bicknell <[email-masked]>
Date: Wed, 28 Sep 2011 19:33:23 +0000

ISC, operator of the F-Root nameserver was recently made aware of
draft-ietf-grow-unique-origin-as.  After some discussion internally
on the benefits and pitfalls of this method, as well as taking our
own operational experience with the F-Root Anycast instance I drafted
our thoughts into a blog post on our web site.

https://www.isc.org/community/blog/201109/origin-asn-anycasted-services

The short version is that we don't feel this provides any added
benefit over the method we currently use, but does come with a
number of down sides.  We would find it disappointing if this method
further eroded the usefulness of inconsistent origin ASN reports,
or caused increased routing table bloat.  Further, given we don't see
any benefits it is unlikely ISC would consider changing its current
practice as a result of this draft.

Note that I am not a member of the GROW mailing list.  Feel free
to reply to me directly or keep me CC'ed if you want discussion to
remain on the GROW list.

-- 
Leo Bicknell; E-mail: [email-masked], Phone: +1 650 423 1358
INOC*DBA *3357*592; Internet Systems Consortium, Inc.  www.isc.org

---

From: Danny McPherson <[email-masked]>
Date: Wed, 28 Sep 2011 18:25:47 -0400

Leo, 
I'm not going to comment on your blog, I have cc'd you on this 
message directly and would encourage you to join the mailing list 
and share your views on topics like this into the future.  

The timing of your responses is unfortunate, given that this document 
has been with the RFC Editor for a while now, and essentially just 
completed AUTH-48 state.  That said, IMO I do not believe your issues, 
addressed individually below, would materially influence the current 
content of the document either way.

In general, and as outlined in the document several times, if this 
doesn't accommodate your current operational model then you certainly
aren't required to use it; diversity is arguably an advantage of the 
current system.  That said, I stand by the recommendations in the 
document as providing utility to network operations.  

Regarding the five issues you did bring up on your blog:

---------
1) ""Node" is not well defined.  For ISC a node is at least two servers, however if the method in the draft were to approximate the level of granularity found in the RFC 4892 method each server would need to have an ASN so it could be identified by routing.  This would require ISC to more than double the number of ASNs in use for F-Root."

In reading your comment, and the recent discussion of 
draft-anycast-diagnostics.txt on [email-masked] the last couple 
of days, I do agree that the definition of "node" is perhaps 
ambiguous.  In hindsight, and for the reasons you outline, perhaps
this is a feature, as in my mind "node" could encompass one OR
more servers (e.g., 100s), and the routing architectures associated 
with those servers is left to the devises of the operator, but I'd 
expect that in most deployment models servers which share a common 
physical location and network connectivity model should certainly 
consider using the same routing domain identifier.

BTW: the draft does NOT say "each server == node", so in your example 
you could in aggregate actually use one less AS than you appear to be 
using today ;-)

---------
2) "This method would all but render the various inconsistent-origin ASN reports available to be useless.  To remain relevant, these reports would need to white-list all Anycasted services so they do not appear on the reports which would be a significant additional burden.  These reports prove quite useful when networks with IP space, but without ASN's (or BGP) move from provider to provider."

The utility of these reports is relative; I consider them of less utility 
than you, and in fact largely because today prefixes are originated from 
the same partitioned origin AS (e.g., the F-root model you outline and 
similar models employed by many others) in multiple locations, and as a
result routing system information provides no discriminators of utility
to aid in visibility and diagnosis of routing problems based solely on 
origin AS and identifiers for various services therein -- I'd prefer active
data in the routing system providing this indication, hence the 
recommendations in the draft.

---------
3) "There are other mechanisms already in place that provide this level of data, often with increased precision.  For instance, even if ISC were to drop the use of ASN 3557 and originate the service prefix from all of the local ASN's it would be impossible to identifiy a particular server via that method.  ISC would continue to recommend using hostname.bind, the RFC 4892 mechanism, to identify F-Root servers; and would even view the RFC 5001 method as superior."

I don't see any additional "precision" in your model that's not already 
enumerated in the current document, e.g., "service level query capabilities" 
in the Introduction.  Both network and service level capabilities are useful, 
as conveyed in the current text.

---------
4) "This mechanism would likely increase the size of the DFZ routing table.  While an Anycast prefix would only take up a single routing slot entry in a BGP device, BGP devices must also consider the paths available to reach that prefix.  By creating more unique paths there will be increased memory consumption on all devices in the DFZ."

False.  Additional routes ("paths") for the prefixes in question exist 
either way in the respective Adj-RIBs-Ins throughout the routing system, 
as appropriate, and neither model should result in larger DFZ RIB or 
derived FIB sizes.  

On the other hand, as a purely academic exercise I think you could make a 
case that the extra path you add as a unique intermediate AS does consume 
additional Adj-RIB-In memory, as do any external elements that have to 
enumerate it and the common origin in AS-based routing or other 
policies  ;-)

---------
5) "This mechanism is unlikely to help end users.  Many Internet end users don't receive BGP feeds.  Should their provider run a looking glass or similar service, it may not point to the same Anycast instance as the users connection due to the way that Anycast routing works.  Thus this method would have the limited audience of people who actually run BGP speaking routers."

This isn't aimed at "end users", although ultimately, it is about adding 
more transparency to service consumers.  As discussed above, service level 
capabilities should indeed exist for "end users" and clients, I agree.  This
capability aims to help network operators and others that analyze routing 
system information to better diagnose issue with a given services and to 
ideally identify any anomalous or persistently misbehaving elements.

Regarding users and looking glass information, yes, if they query an anycasted 
prefix in a looking glass that doesn't resolve to the node with which they're 
being routed, well, there's nothing magic here to fix that.  OTOH, if it's their 
ISPs looking glass and resolves to the correct node, or if it's the ISP doing 
the diagnosis in their own routers, well, with this technique they now have 
an additional data point to inform operations.

---------
Kicker)  "In short, ISC believes this draft provides no advantages over the current methods ISC uses with F-Root.  At the same time, it has the potential to do real harm to the useful inconsistent-origin ASN services, and may be another contributing factor adding to path bloat in the DFZ.  ISC believes the RFC 4892 or RFC 5001 methods are vastly superior to identify DNS Anycast services, and believes similar functionality should be developed for any other Anycasted applications."

As indicated above, your DFZ analysis is incorrect.  Regarding RFC 4892 and 5001, 
the document already indicates that it is wholly complementary to these service 
level identifier capabilities and they are certainly not mutually exclusive.

---

From: Danny McPherson <[email-masked]>
Date: Wed, 28 Sep 2011 21:35:55 -0400

Regarding your "industry" and "uniformed" comment, I'm certain many 
folks who commented and reviewed and were involved in this effort have
as much operational experience as you and I alike, and given that 
this issue was conceived based on an operational issue observed with 
anycasted DNS operations by at least one operator with which I am 
intimately familiar, incidents that also impacted operations of 
services which you are likely intimately familiar, I suggest you 
find another card to play.

There was an operational trigger related to anycasted prefixes leaked 
outside of their intended catchment that lead to the development of 
this technique.  Obviously, the desire would be that any globally 
anycasted service resolve services universally, which was presumably 
the case at least at the origin servers, but when that is not what is
observed by the services consumer techniques to more rapidly identify 
and classify a given operational issue triggered by the routing system
issues within the routing system have real utility.

> There appears to be two kinds of identification at work here.  There
> is identifing the Anycast node that is serving a particular customer,
> and then there is identifing if the route should be originated from
> a particular node.  It is my feeling that more discussion was given
> to the second part in the previous reviews of this draft, while my
> response focused almost entirely on the first part.

No, I think the intention and discussion of this document was
precisely to that effect, and particularly to the case where 
anycasting deviates from traditional models of unicast services 
in that an autonomous system MAY be non-contiguous and MAY have 
multiple non-intuitive "autonomous" operations from many 
locations with a single origin AUTONOMOUS system number.  Providing 
a discriminator in the form of unique origin ASes at the routing 
layer to illustrate the variances is the objective here.

> With respect to detecting hijacked routes I think this proposal is
> totally backwards.  There have been efforts in the RIR community
> to track the origin ASN of a prefix
> (https://www.arin.net/policy/proposals/2006_3.html), which only
> allow a single origin ASN to be specified.  

False.

S 3.5.1: "This additional field will be used to record a list of the 
ASes that the user permits to originate address prefixes within the 
address block."

"list" == plural.

All additional discussion in the sub-bullets related to this item 
are clearly plural as well, as is all the RPKI-esque capabilities
(e.g., multiple ROAs for a given prefix) related to this topic to 
date.  

> Research has been done
> on inconsistent-origin ASN's with interesting results
> (http://www.routeviews.org/papers/nanog_origin.pdf).

Slide 4: The fact that 31% of the prefixes in that study, albeit 
a decade old, have inconsistent origins, nicely illustrates the 
lack of utility of inconsistent origin AS reporting.

Slide 6: Indictors for what's causes "Fluctuating" in the routing 
system would be far easier to identify and attribute if all anycasted 
sources were from unique origin ASes.  I.e., a discriminator in the
routing system for anycasted prefixes would aid in origin-based routing 
system analysis of anycasted prefixes.

Slide 20: Regarding flap damping, if per-path penalty models (rather 
than per-prefix) were employed unstable paths would not penalize the 
entire prefix (e.g., paths learned from stable nodes), could even 
optimize for origin only in the particular example.  Of course, per
path implementations would improve many other issues as well.

I see _nothing in this work that suggests unique origins are a bad 
idea, although I can certainly see attributes that could be further 
analyzed based on their work if unique origin ASes were used.

Can you be specific about something this technique would impair?

>  Several of
> the BGP monitoring services only allow a single origin ASN when
> monitoring a route.  The IETF's own documents recommend using a
> consistent origin ASN for troubleshooting purposes
> (http://www.ietf.org/rfc/rfc4116.txt, section 3.1.2.2).

Hence the need to discuss the technique in this document, where folks
who offer these services or build tools in house have not already 
adapted their policy and detection capabilities.  

That said, regarding: S.3.1.2.2: 

  "Operational troubleshooting is facilitated by the use of a consistent
   origin AS.  This allows import policies to be based on a route's true
   origin rather than on intermediate routing details, which may change
   (e.g., as transit providers are added and dropped by the multihomed
   site)."

Could definitely be argued inversely in that using unique upstream ASNs 
as a detection or policy trigger point (as you currently do) is more 
problematic from a policy perspective, it's certainly the case in routing 
policies and detection indicators I've attempted to enumerate.  

Simply put, if detection or policies can't be defined (discriminated) 
based on authorized origin and a single upstream adjacent AS alone then 
adjacent ASes and upstreams need to be codified.  If the origins are 
consistent in all locations and adjacent ASes are unique per location 
via the same operator, the origin, adjacent ASes, and upstream adjacent 
sets all need to be enumerated in detection or routing policies for the 
prefix, that's the only way you can find a unique discriminator in the 
set to key off of.

> The document requires N ASN's, where N is the number of "sites"
> in the Anycast cloud.  ISC's current method with F-Root requires a
> scant N+1 ASN's, using the +1 to be the consistent origin ASN.  This
> is far easier to input into many of the tools (listing one origin
> ASN, rather than hundreds), and easier to explain to other operators
> (if F isn't originated from 3557, it's wrong).

Yes, it's easier for the operators to have a consistent origin, I have 
definitely heard that.  But when diagnosing problems in the routing 
system and detection leaks or other anomalies at the routing layer, it
is certainly not easier when an anycasted prefix for a critical 
resource can appear with any set of upstreams anywhere in the system at
any time with no unique discriminators.

> However, I think the most interesting and relevant quote comes out of
> http://irl.cs.ucla.edu/papers/originChange.pdf:
> 
>  Only the origin itself (UCLA in our example) could easily and
>  accurately distinguish be- tween a legitimate origin change and a prefix
>  hijack [4].

But it can't today!  Again, the very application of a common origin 
in all locations for a given prefix, and the thought that that prefix 
might well appear in some new place with the same origin and a new 
upstream AS, or escape outside of it's intended catchment illustrates 
just the problem.  I.e., if it's leaked or hijacked how the heck would 
anyone know?

The only way you can fix this in the routing system is with both origin 
and path validation, work occurring in various places in the community.  
In the interim, we propose that a unique origin be used per location, and 
the set of plausible upstream ASes for that origin published, and then 
folks can employ how they desire for detection, policy, or other 
functions.

> My comment was not about the difference between the model proposed and
> the F-Root model, both of which use an ASN per site.  Rather it was
> about the difference between both of those models, and a model which
> uses just a single origin ASN with no per-site ASN.
> 
> I believe in particular if you think about a router with BGP
> multi-path enabled either per-site ASN configuration is likely to
> lead to increased routes in the RIB.  Since this is a general
> recommendation, more folks doing this mean larer RIBS.

Again, not true. Anycast itself may lead to more paths for a given 
prefix, but this technique has no implication on that.

> I see many of the arguments for this proposal were that it helped
> in diagnostic capabilities.  Yet in all of the GROW archives I have
> read so far, and in the RFC itself I do not see a single example
> of how this configuration leads to faster/more accurate troubleshooting
> than say the F-Root model.  

OK, re-read the draft, and read the responses above, perhaps they help
clarify.

> As I documented at the start of this mail I believe many BGP jockies
> will find this quite counter-intuitive, having been taught in classes
> and in the IETF's own other BGP documents that inconsistent origin
> ASN is bad/dangerous.  If the IETF is going to advice this document,
> than things like RFC 4116 should be updated so they don't appear
> to conflict.

Indeed, hence this document!

> But really the crux of the failing to me here is the lack of supportave
> examples.  I'd love to see an example detailed where this draft makes
> some sort of troubleshooting for Joe BGP Jockey easier, most preferably
> in the draft itself.  I sure can't see how it would make anything
> easier, but perhaps an example would make me turn around on that.  If
> this really does a lot of good, examples should be easy to produce.

Again, see example above, and re-read the draft, I hope this helps
explain the operational trigger and utility of this technique.

-danny

---

From: Leo Bicknell <[email-masked]>
Date: Wed, 28 Sep 2011 23:51:56 +0000

> content of the document either way.

I agree the timing is unfortunate, but wanted to comment to be on
record that I don't feel this documents the "Best Current Practice",
which I believe is the point of the exercise.  I feel bad when the
industry recommends to the uninformed a method that may not be the
best.

Having read all the back posts I can find, I think I see part of the
disconnect.

allow a single origin ASN to be specified.  Research has been done
(http://www.routeviews.org/papers/nanog_origin.pdf).  Several of

The fact is there is no way to detect these events without some
knowledge.  You either have to know the list of valid ASN's (likely
only the origator does), or rely on them to publish you the
information.  Given groups like ARIN are now collecting (a single)
origin ASN with a route, there is a source for this data but only
if the origin is consistent.

In terms of advice to someone new to Anycast who wants to set up their
first cloud I think this is actively harmful advice.  They will find
that inconsistent origins do not square with the history of BGP
recommendations, the tools available today, or the expectations of those
on the other end of the phone.  I see no way such a method could be
called Best Current Practice.

Now, with that out of the way we can return to the more end user facing
problem of identifing an anycast node...

[Re point #1]

> of days, I do agree that the definition of "node" is perhaps 

The question is entirely how granular the information in the routing
system should be to enable the functionality this draft seeks to
enable.  For instance an ASN may have 100 servers Anycasted internally
with OSPF, that generates a single external route from that ASN
which is Anycasted with other ASN's.  What level of detection is
required to be useful?

[Re point #2]

> recommendations in the draft.

I think before the IETF renders an entire set of reports useless with a
policy change it would be worth at least seeking out some of the users
of those reports and finding out why and how they use them.  I am
disturbed that none of the folks who commented appear to be users of
these reports, and yet are eager to make them useless.

No additional comment on point #3.

[Point #4]

[Point #5]

---

From: Leo Bicknell <[email-masked]>
Date: Thu, 29 Sep 2011 13:06:32 +0000

> prefix, that's the only way you can find a unique discriminator in the 

I think I see where your world view and mine do not match.

In a world where an Anycast service contracts with 4-8 upstream
networks for transit they can easily enumerate all of the paths
that might appear, and send it to someone.

In a world where an Anycast service peers with a large number of
networks, this is totally impractical.

If I wanted to list the full set of possible paths by which you
might see F-Root that list would be about 3,000 ASN's long right
now.  I suspect you don't notice that because we also make generous
use of no-export both directly and indirectly, meaning a lot of
these paths may not be visible to you, from your vantage point.

I find, and track down (mostly innocent) route leaks of F on a
weekly basis due to our rich peering.  I can honestly say, for us,
the approch you describe would not make things one bit easier for
me.  

> just the problem.  I.e., if it's leaked or hijacked how the heck would 

I have no idea how anyone, other than the operator of the service
would know.  In all proposed configurations the operator of the
service knows how they configured it, and is the only one qualified
to diagnose.

Simply: A third party cannot know if the route is legitimate unless the
        Anycast operator tells them.

If the operator must enumerate all paths anyway, the difference
between your proposal and say the way F does it is the +1 ASN (N
for your proposal, N+1 for F), but the list of paths is exactly the
same length, N.  That one extra ASN means inconsistent origin isn't
required.

So the operator can provide a list of N things in your proposal, or the
same N things the way F does it, except our way doesn't go against the
tradition of inconsistent origin being a "bad" thing.  Indeed, I can
write the vi transform easily:

   s/$/ 3557/

It's my hope that in the future DNSSEC, SecureBGP, RPKI, or some other
out of band (to the routing system) method will be able to verify, but
until then this proposal doesn't advance things.

> OK, re-read the draft, and read the responses above, perhaps they help

Not really.  An example would be:

router> show ......

---->>> This line has foo and bar in it, which mean a leak.

Seriously, show me an algorythm to find a leak for a Anycast service
that does not require any pre-knowledge about how the Anycast service is
configured.  I submit such a thing does not exist.

If it doesn't, this draft does not advance anything, it recommends
one method out of many with no particular advantage, and at least one
easy to document disadvantage (inconsistent origin).

---

From: Danny McPherson <[email-masked]>
Date: Thu, 29 Sep 2011 09:27:57 -0400

> configured.  I submit such a thing does not exist.

I never said "does not require any pre-knowledge".  As a matter of fact,  
what I said, and what the draft says, is that with unique origins the 
services operator _could publish in a well-known location a list of origin 
ASNs for a given prefix and the feasible adjacent upstreams for each ASN.  
With that information network operators can make informed decisions about 
the legitimacy of a new path in the routing system for a critical Internet
services prefix.

Today, a prefix and common origin can appear from anywhere and network 
operators have no indication of whether it is feasible or not (I know of 
at least two occurrences of this that have impacted consumers), and operators 
have very little to inform them as to how to detect rogue nodes or malicious 
paths, and nothing to apply policy to a path known to be undesirable, or how 
to determine if a new node is legitimate or not.

These static policies in no way provide the long-term fix, but in the 
interim they provide a routing system discriminator and visibility to 
operators.  If you don't believe a discriminator in the routing system 
for anycasted prefixes is useful, well, duly noted.

-danny

---

From: Leo Bicknell <[email-masked]>
Date: Thu, 29 Sep 2011 13:35:12 +0000

> the legitimacy of a new path in the routing system for a critical Internet

Ok, fair enough.  So let's ask the direct question:

  Would it not be even better then for them to have a unique origin
  ASN, and publish the list of paths that originate the route,
  achieving the same result without needing to have an inconsistent
  origin?

Seems like that would be a quick change to the draft...

And couldn't the entire draft thus be greatly simplifed to a single
paragraph?

  Anycast operators SHOULD publish a list of all valid AS-Paths to reach
  their Anycast service to aid in the detection of routing leaks.

I suppose bonus points if we could agree on a method for publishing
(RPSL?).

---

From: Danny McPherson <[email-masked]>
Date: Thu, 29 Sep 2011 11:22:45 -0400

>  achieving the same result without needing to have an inconsistent

With normal unicast there's a single tree for the prefix in the routing 
system, while anycasting inherently introduces a polytree for the prefix.  
However, anycasting with a common origin AS creates a 'pseudo AS' at the 
root of the graph (i.e., the illusion of a single tree).  By doing what 
you describe, you force anyone doing routing system analysis to know 'a 
priori' which prefixes employ a pseudo AS at the root of the tree - only 
so that they can step deeper into the tree, to the actual polytree roots, 
and then evaluate the adjacent nodes of those roots.  This is particularly 
problematic when new nodes are introduced to the system.

I don't see how this is simpler.  

-danny

---

From: Leo Bicknell <[email-masked]>
Date: Thu, 29 Sep 2011 19:07:06 +0000

> problematic when new nodes are introduced to the system.

I'm sure you have studied this in great detail, and so I will simply
assume you're correct that the method described is easier when
writing a program to find hijacked routes.

However, that is a small part of operating an Anycast system.  Far
more frequently than I help folks find hijacked routes (which are
fortunately rare) I help troubleshoot ordinary performance problems.
It's far easier for me to say "send me show ip bgp regex _3557$"
and get all the info I need than send a list of routes (3 in our
case) or local ASN's (about 55 in our case).  The same goes for
when I'm looking at various looking glasses to see what is going
on.

I also think, and this is totally subjective, that it is easier to
explain to folks on the phone, particularly folks with no Anycast
experience.  They can just pretend in their mind that 3557 is one
big global network that peers in a lot of places and their mental
image is complete.

So even if I spot you that it makes it easier for a third party to
programmatically detect hijacks I don't think that's enough alone
to rise to calling this method the "best current practice", which
is the path this draft is on.  It has impacts on day to day operations
(including things like the inconsistent asn reports) that also need
to be weighed.

But, I think we've both overmade our points.  I also realize I was late
to this game, you can't watch everything in every group all the time.
I think the question is if anyone else on the list cares one way or
another, because if no other opinions have been changed we might as well
stop.

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 29 Sep 2011 15:49:18 -0400

jumping in in the middle...

> when I'm looking at various looking glasses to see what is going

it sounds like for the case of (for example) Leo troubleshooting
specific ISC problems one method he has works well. (and if you see
F-root originated from anything BUT 3557 you KNOW there's a problem..
again, as an example he could have 25 asns for this, but I'm assuming
1)

For a more generic problem that Danny/authors are attempting to make
simpler, each 'anycast service' would have it's own group of asn's
which would correlate to a set of both transits/peers AND geographic
locations... So, (again as a purely hypothetical example) A-root from
AS123333 behind AS4134 seen in London is likely far, far less happy
than A-root from AS123333 behind AS9150 in London. Also, A-root from
AS15169 anywhere is just bad news... (again, making up an example of
both 'location aware' troubleshooting for performance and one for
'mis-originated')

I don't think that the subject doc was intended to force compliance in
one way or the other, just to document one possibly reasonable method
to do anycast service hosting (and show how it is considered
reasonable).

Smart operators of services likely already have lots of these guts for
themselves done, if it works don't fix it.

-chris

---

From: John Kristoff <[email-masked]>
Date: Thu, 29 Sep 2011 15:04:03 -0500

On Thu, 29 Sep 2011 19:07:06 +0000
Leo Bicknell <[email-masked]> wrote:

> time. I think the question is if anyone else on the list cares one
> way or another, because if no other opinions have been changed we
> might as well stop.

I often play contrarian or skeptic, if not an honest critic when I take
the time to review a proposal such as this.  My review posted to this
list almost a year ago resulted in no responses or changes in the
draft.  While I didn't find the initial draft as written very convincing
of BCP status, I don't feel strongly about it one way or another to
argue for changes.

It seems to me this is VeriSign trying to document a BCP that works for
them and may work for others.  So perhaps it is just the BCP status
that is disconcerting?

John

---

From: Leo Bicknell <[email-masked]>
Date: Thu, 29 Sep 2011 20:20:42 +0000

> them and may work for others.  So perhaps it is just the BCP status

That may be part of it.  At the risk of offending the group (since
many of them came out of here, I suspect) I believe many of the
"BCP" documents are more like "AWTDI" documents, A Way To Do It.

Look, anyone who's deployed a hundred nodes of Anycast, be it
Verisign or ISC, or any other, isn't going to change the deployment
without a really, really good reason.  If it's already deployed,
it's going to stay that way.

When writing a BCP, I think the target audence is folks who don't
do this at all.  The guy who's deploying the technology for the
first time, and looking for the wisdom of those with beards longer
and greyer than their own.  To that end, if there are three ways
to do it, each with pros and cons but no clear winner we should
document the three ways and say "pick one that fits you", not pick
one of them document it as "best" and then ignore the other two.

So the question to ask is, do we have a good reason to strongly
encourage all new Anycast deployments to be in this model, as opposed to
the alternative models?

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 30 Sep 2011 00:40:24 -0400

in the vein of 'new deployment folks may want to shortcut all of the
bad lessons everyone else learned'

> encourage all new Anycast deployments to be in this model, as opposed to

keep in mind that 'enterprises' may be deploying anycast services
inside their wan's, NOT publicly visible... What advice would you give
them in such a situation?

I suppose my point is/was/will-be that documenting a path that's not
insane, works, could help new folks isn't bad. I agree that there are
2-3-17 ways this is currently deployed, each has their similarities,
troublepoints and wins... I fall back on the fact that most of these
are operated by very smart folk who know the innards of their
deployment/scenario (and the larger internet in most cases) very, very
well. As you say leo, telling them 'change your ways!!' is just not
sensible.

-chris

(keeping in mind that often enterprise IT folks are not
super-bgp-ninjas like leo...)

---

From: Danny McPherson <[email-masked]>
Date: Fri, 30 Sep 2011 08:58:06 -0400

> bad lessons everyone else learned'

Indeed.. 

I understand the concern about "don't attempt to force me to do this" with a 
BCP track document- that is certainly not the intention was was attempted to
be conveyed in the document text as is - and I am completely open to documenting 
other deployment models and contrasting the benefits and trade-offs of each, 
that's the whole reason for this WG.  However, I will also note that this 
isn't just about folks that operate those services, it's also about folks who 
rely on or monitor the services they operate, as previously outlined.

That said, there were many requests for feedback and the WG and IETF last call 
process for IETF BCPs is well know -- I do see now that I didn't respond expressly 
to John's comments from last year, and although I don't think they would have 
changed my personal recommendations, and they didn't appear to sway the WG as 
provided in the mailing list archives, perhaps his concerns could have been better 
considered and accommodated in the text (of course, that's always the case in
instances like this, but I like John ;-).

Let me circle back with the chairs and see if we can find an amenable 
path forward here, and if it's not to late to at least hit the pause button
on publication of this documents as-is until we consider the broader 
feedback.

-danny

---

From: Leo Bicknell <[email-masked]>
Date: Fri, 30 Sep 2011 14:20:18 +0000

> them in such a situation?

Just to level set here, I'm assuming you mean a large enterprise
using something like a MPLS L3VPN service, and thus running BGP
with their service provider inside the VPN to communicate routing
information.  I clarify, because if it's a simpler IGP only Enterprise
I don't think much of anything we've been discussing makes any
difference, drop it in the IGP and be done.

I think, basically, most of what we've discussed does not apply.
If it really is a private network there will be no third parties
doing route monitoring.  The chance of a rogue ASN injecting the
route is near zero.  The provider of the service is also the end
user of the service, and has control of virutally every routing
device in the middle, so there are a multitude of ways they could
monitor and troubleshoot.  There will be no external services doing
things like inconsistent asn reports.

Personally I would still originate all of my Anycast from a private
ASN (a la what F does with 3557) for the sole reason that then the
announcement is consistent, thus "show bgp ipv4 unicast inconsistent-as"
actually produces useful output.  While it is far less likely to
be used in an enterprise context, it is far more likely to be useful
when it is used given the end to end control.  If the enterprise
migrates routes from one site to another (e.g. virtuals in a data
center move) the command will show if they are coming from two
different sites quickly and easily.

Honestly though, if I were providing recomendations to an enterprise
I would likely encourage them not to run Anycast.  They control the
end clients.  They can configure the clients for even better
redundancy than Anycast provides given a little thought.  Plus, as
you said, enterprises tend not to be staffed with super-bgp-ninjas,
and so giving them something outside the CCNA traning class is not
a good idea.

Anycast is best used when you don't control, or have limited control
over the end clients.  For instance root operators get to provide
1 IP address in root.hints or in the root zone, which limits the
ability to do redundancy with multiple IP adddresses...

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 30 Sep 2011 10:51:58 -0400

> with their service provider inside the VPN to communicate routing

that's one option, enterprises come in lots of flavors... I've seen
them with mpls-vpn cores and with leased line cores and with
ipsec-over-internet cores. (I'm sure you have as well)

> difference, drop it in the IGP and be done.

sure.

> route is near zero.  The provider of the service is also the end

the troubleshooting though with something like unique origin is
'simpler' for them: "Oh, as27 is NYC, as37 is WDC... why is Newark
seeing WDC as the best path?"

> things like inconsistent asn reports.

true, we hope.

> different sites quickly and easily.

you are highlighting one troubleshooting technique, yes. I highlighted
another. potato/potatoe. Again, for 'smart folks' who know their
service inside/out they'll figure this out on their own. For a new
deployment, however, some guidelines seem like a good plan.

> and so giving them something outside the CCNA traning class is not

:) but that network world article... I agree with you mostly, I've
seen some good uses of anycast internal to enterprises. Again, 'smart
people' and all of that.

> ability to do redundancy with multiple IP adddresses...

dns servers exist for desktop/vpn clients in corporations too. maybe
it's a little 'all I have is a hammer', but it does work well.

-chris

---

From: Leo Bicknell <[email-masked]>
Date: Fri, 30 Sep 2011 15:04:00 +0000

> seeing WDC as the best path?"

I'm not sure I get it.

Unique origin ASN output (simplified)

  10.1.2.3/32
        65000
        65001
*       65002 (Best)
        65003 65001

So we know it's coming from site 65002.

Using the F-Root model, with an origin of 65999 for the Anycast routes:

  10.1.2.3/32
        65000 65999
        65001 65999
*       65002 65999 (Best)
        65003 65001 65999

We still know it's coming from site 65002 by just looking.  I have a
hard time calling one "simpler" than the other, indeed I would call them
pretty darn equal.

But here's the difference, let's say I have 20 Anycasted routes for
different services, but I originate them all from 65999, I can see the
status of all 20 with:

show bgp ipv4 uni regexp _65999$

I'll get 20 lines, showing me how each virtual is routed.

With a unique origin ASN configuration, to get the same info for 20
routes I would need 20 lines like:

show bgp ipv4 uni 10.1.2.3/32
show bgp ipv4 uni 10.1.2.4/32
show bgp ipv4 uni 10.2.2.1/32

And so on, or have the discipline to keep them all in one netblock so:

show bgp ipv4 uni 10.1.2.3/24 longer-prefixes

produces useful output.

Thus I think I can make a good argument that having a consistent
origin ASN makes troubleshooting easier when you have multiple
Anycast routes, because you can select the entire set of routes by
origin ASN.