Subject: [rrg] draft-narten-radir-problem-statement-05.txt

From: Thomas Narten <[email-masked]>
Date: Wed, 17 Feb 2010 19:40:10 -0500

Hi.

With the recent thread on route scaling, now might be a good time for
folk on this list to have a fresh look at the "route scaling problem"
document below. This was put together by the Routing & Addressing
Directorate (http://wiki.tools.ietf.org/group/radir/) and is a joint
effort by all of its members.

The document has been sitting for a while, but we are now looking at
it again with the goal of getting it published as an Informational
RFC. In order to do that, we really need folk to review it and either
point out its shortcomings, or indicate that it's a useful document to
publish as is.

Thanks!

> A New Internet-Draft is available from the on-line Internet-Drafts directories.
> 
> 	Title           : On the Scalability of Internet Routing
> 	Author(s)       : T. Narten
> 	Filename        : draft-narten-radir-problem-statement-05.txt
> 	Pages           : 25
> 	Date            : 2010-02-17
> 
> There has been much discussion over the last years about the overall
> scalability of the Internet routing system.  Some have argued that
> the resources required to maintain routing tables in the core of the
> Internet are growing faster than available technology will be able to
> keep up.  Others disagree with that assessment.  This document
> attempts to describe the factors that are placing pressure on the
> routing system and the growth trends behind those factors.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 17 Feb 2010 20:16:05 -0500

Thomas, one major and some minor comments on the document.

The discussion of IPv6 routing table size in section 4.6 seems 
misleading.  The discussion seems to have no component to represent the 
present and increasing demand for PI addresses in order to support 
multi-homing and ease of renumbering.  If we do not have a routing 
architecture and system which addresses those pressures, and if we do 
not manage to keep in place an artificial barrier to such PI 
assignments, the size of the tables will grow dramatically.  Some 
stimates place the expected number of multiohomed enterprises in the 
ballpark of 10,000,000.  That would represent a significant growth in 
the control plane, the RIB, and the FIB.

Section 3.3 is titled "Alignment of Incentives".  It then begins with a 
partial description of the pressures that Multihoming and Traffic 
Engineering put on the system.
It would seem that the document would be clearer if there were a section 
describing the need to support multihoming and traffic engineering in 
general (leaving the details for section 4 probably), and the fact that 
these tend to produce growth in all the dimensions cited earlier.  That 
could then be followed by section 3.3 with something similar to the 
current second paragraph, which is actually about the Alignment of 
Incentives.
It would then seem sensible to include a paragraph about the need for 
any mechanisms for improvement to have to property that the costs and 
incentives for deploying the improvement are aligned.  (This appears in 
the summary in section 6, but needs to be explicitly stated earlier in 
the document.)

Would it make sense to include in section 4.3 (End Site Renumbering) to 
observation that this problem has also driven some NAT deployments? 
(No, that is not a route scaling problem, but it reminds the reader of 
an indirect cost of not solving these problems in an effective fashion.)

Should we at least acknowledge in section 5 that our habit of addressing 
any and all problems with BGP extensions puts pressure on the control 
plane?  (It may be that this component is manageable, but I wonder.) 
Each of these features have been put in for very good reasons, but RFC 
2547 VPNs, Flow-Routes for black-holing DDOS, and add-path routes to 
allow use of multiple parallel routes, are all examples of features wew 
have or are putting in the system that increase the pressure on the 
Control Plane.  [Note, I am not saying that these are wrong choices. 
They may well be correct choices, and BGP may well be the right place to 
address these needs.  But we should recognize that this puts pressure on 
the system.]

Yours,
Joel M. Halpern

Thomas Narten wrote:
> _______________________________________________
> rrg mailing list
> [email-masked]
> http://www.irtf.org/mailman/listinfo/rrg
>

---

From: Danny McPherson <[email-masked]>
Date: Thu, 18 Feb 2010 10:15:08 -0700

On Feb 17, 2010, at 6:16 PM, Joel M. Halpern wrote:

> Control Plane.

I've certainly echoed this sentiment many times now, so I agree 
Joel.  

I do have one clarification.  While they do introduce traditional control
and data plane overhead like any other BGP route, the DDoS countermeasures 
(i.e, BGP-based destination or uRPF/source-based blackhole routing) that 
are deployed today with BGP require no new "features" or attributes, they 
use only existing machinery - unlike flow spec and all the other stuff you 
mention above.

-danny

---

From: Robin Whittle <[email-masked]>
Date: Tue, 23 Feb 2010 18:03:13 +1100

Hi Thomas,

In a recent message:

   Scalable routing & architectural enhancements
   http://www.ietf.org/mail-archive/web/rrg/current/msg06099.html

I described my understanding of the routing scaling problem and some
other problems which I think would ideally be solved as part of the
"once in several decades" architectural enhancements to IPv4 and IPv6
which will be required to solve their routing scaling problems:

   IPv4 address shortage and need for higher rates of utilization.

   Mobility (both IPv4 and IPv6).

   Security of hosts against attacks (probably can't be solved with
   any network enhancement).

   Combating DoS and other network-based attacks.

   Path MTU discovery and packet length limitations

Here are some thoughts on the RADIR Problem Statement.

  - Robin

The history of the RADIR Problem Statement ID is:

  00  2007-07-26
  01  2007-10-09  Various changes, mainly to terms and summary.
  02  2008-10-17  No changes.
  03  2009-03-09  Minor changes, I think.
  04  2009-12-15  No changes.
  05  2010-02-17  Minor changes, I think.

I commented on version 00 on 2007-08-09:

  http://www.ietf.org/mail-archive/web/ram/current/msg01809.html

Also of potential interest is the RRG Goals ID, which has been in
version 01 since 2007-07-08, after relatively few discussions:

  http://tools.ietf.org/html/draft-irtf-rrg-design-goals-01

I commented on it on 2007-07-14:

  http://www.ietf.org/mail-archive/web/rrg/current/msg00203.html

but there have been no updates and little or no discussion of this
ID.  Tony confirmed recently that these goals did not represent
consensus and were subject to further revisions:

  http://www.ietf.org/mail-archive/web/rrg/current/msg05511.html

There is a page the RADIR folks might be interested in - my attempt
to describe the constraints we face due to the need for widespread
voluntary adoption:

  http://www.firstpr.com.au/ip/ivip/RRG-2009/constraints/

I think this may be relevant to your section 3.3 Alignment of Incentives.

Section 3.3 states "there are millions of potential end sites which
would benefit from being able to multihome".  Brian Carpenter and I
recently, independently, nominated 10 million as a rough upper limit
on the number of non-mobile network which want or need portability,
multihoming and/or TE (though perhaps Brian didn't mention
"portability".  My reasoning was roughly that with human population
of 10 billion, there will probably be only one organisation per 1000
people (mainly small companies, but also schools and other
organisations) which have sufficient need of portability or
multihoming to get "edge" space in a CES system and to get the two or
more physical links from two or more ISP's services as needed for
multihoming.

I think the Problem Statement's references to "4.3. End Site
Renumbering" and to certain extent to "4.4. Acquisitions and Mergers"
 could be improved by reference to address space "portability".  The
word "portability" does not appear in the Statement, and while I
think it makes some people's teeth itch, I think it should be mentioned.

The problems mentioned in these sections arguably only exist because
only PI space is "portable" and that when splitting even this
portable space into two separately advertised PI prefixes is
possible, this contributes to the routing scaling problem.

Core-Edge Separation architectures provide a new subset of the global
unicast address space which is portable in a scalable manner, and
which can be split up much more finely than the /24 limit which
applies to conventional PI prefixes.

So I think it would be reasonable to describe a key part of the
routing scaling problem as being due to the lack of portability for
any scalable form of address space in the current system.

  - Robin

---

From: Thomas Narten <[email-masked]>
Date: Wed, 24 Feb 2010 17:06:23 -0500

Hi Joel.

Thanks for the comments.

> the control plane, the RIB, and the FIB.

This section was intended to simply point out that dual stack adds
even more routes, in the sense that one will need both IPv4 and IPv6
route to the same destination. That only further increases the overall
number of entries a router deals with.

It goes without saying (and looking back, the document doesn't say
this) that all the  pressures that apply to IPv4 routing, apply pretty
much equally to IPv6.

Seems to me, that the best way to deal with that is to add a new
paragraph (perhaps in the introduction) that calls this out
explicitely. I.e., something like:

   Most of this document does not distinguish between IPv4 and
   IPv6. The overall routing archtecture for the two protocols is the
   same. Consequently, most of the issues in this document apply
   equally to IPv4 and IPv6. Any behavior that places pressure on IPv4
   routing is likely to also exert the same pressure on IPv6.
   Deployment of IPv6 will not lessen these pressures in most cases.

> the summary in section 6, but needs to be explicitly stated earlier in 

Let me think about this. I think I agree that the section would
benefit from some restructuring.

> an indirect cost of not solving these problems in an effective

I guess I'm on the fence on this one. The goal of this document was to
describe the problems surrounding route scaling. So far, we have tried
hard not to add stuff that isn't directly related especially if people
might have strong views on the topic. In fairness, if we were going to
mention NAT, we'd have to point out that it has also helped relieve
pressure on the routing system -- i.e., there is a benefit.

> address these needs.  But we should recognize that this puts pressure on 

Given yours and Danny's comment on this, how about something like:

      BGP is the routing control plane protocol in use today. Because
      of its ubiquity and centrality to routing, it is also a natural
      place to add features that enhance routing functionality, some
      of which may not be strictly necessary to support DFZ routing
      (e.g., RFC 2547 VPNs). The use of such extensions also place
      additional pressures on the routing control plane as they must
      be sent, received, processed, etc. along with other routing
      updates.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Wed, 24 Feb 2010 17:34:47 -0500

Hi Robin.

I've looked at some of the references and older messages, but it's not
clear to me what action item you want me to make. Do you have specific
concerns with the accuracy of our document? etc? It would be much more
helpful if you can point out specific wording or sections that you
feel are not right - and explain why.

> more physical links from two or more ISP's services as needed for

I don't know that I agree with this actually. In today's world of
DSL/Cable/soon-to-be ubiquitous wireless, I could imagine every home
using multiple ISPs. Won't those users want multihoming to work for
them?

But in any case, I don't know that  our document needs to take a stand
on any one particular number.

> Renumbering" and to certain extent to "4.4. Acquisitions and
> Mergers" could be improved by reference to address space
> "portability".  The word "portability" does not appear in the
> Statement, and while I think it makes some people's teeth itch, I
> think it should be mentioned.

IMO, there is an important and subtle difference between "portability"
and "renumbering hurts bad". They are not the same thing.

Portability is only wanted because that is the primary solution for
avoiding the pain of renumbering (other than NAT).

And portability also says (to me) something about individual prefixes
going into the DFZ, which is not really what people necessarily
actually care about (or want) when they want "portability". They want
to be able to switch providers without renumbering. So I think it is
more accurate to focus on that as being the true underlying issue.

> possible, this contributes to the routing scaling problem.

Section 4.2 on multihoming pretty clearly points out that PI vs. PA is
not the issue w.r.t. causing problems in the case of multihoming. You
can multihome with PA space, but you get the same bad properties.

> any scalable form of address space in the current system.

I am not convinced that this is an accurate statement.

Thomas

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 25 Feb 2010 15:40:49 -0500

Thank you for responding Thomas.  I don't think what you propose is 
sufficient.  (More after your comments, for context.)

>    Deployment of IPv6 will not lessen these pressures in most cases.

What you propose to add is good.  But, the text in 4.6 as written claims 
"It is possible to extrapolate what the size of the IPv6 routing table 
would be if wide spread adoption of IPv6 occurred..."  Unfortunately, 
the extrapolation that then takes place assumes that the same factors 
that currently constrain IPv4 sizes would constrain IPvb6 sizes, and 
that seems extremely unlikely.  Hence, this extrapolation is very 
optimistic, and misleading to the reader.

Yours,
Joel

---

From: Thomas Narten <[email-masked]>
Date: Thu, 25 Feb 2010 16:05:29 -0500

> would be if wide spread adoption of IPv6 occurred..."

Ah. Is your issue more narrow in the sense that this statement is
misleading?

The extrapolation is intended for "today" not for "next year" or "ten
years from now".

You must be assuming that the exising pressures that contribute to
problems will get worse (I tend to agree!). But, that doesn't really
have anything to do with IPv6. I.e., IPv6 doesn't make those pressures
stronger or lesser.  They will also get worse with IPv4.

> optimistic, and misleading to the reader.

How about if I change the statement as follows:

OLD:

   It is possible to extrapolate what the size of the IPv6 Internet
   routing table would be if widespread IPv6 adoption occurred, from the
   current IPv4 Internet routing table.  

NEW:

   It is possible to extrapolate what the size of the IPv6 Internet
   routing table might look like today, from the current IPv4 Internet
   routing table, if widespread IPv6 adoption were to occur
   "instantaneously",

Then, at the end of the paragraph add something like:

   Of course, this estimate is based on a current snapshot of IPv4
   routing activity. Unless the pressures described elsewhere in this
   document are reduced, the actual table size would be larger.

Would that address your concern?   

Thomas

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 26 Feb 2010 10:22:17 +1300

On 2010-02-26 09:40, Joel M. Halpern wrote:
...

> optimistic, and misleading to the reader.

Joel, as you know, it isn't the length of the announced prefixes
that really matters, but the number of non-aggregated prefixes.
So the question is, what constrains the growth in that number?

That is of course very debatable, and there is a lot of research
literature about it. If current trends persist, for example,
you can conclude that a network of 10 billion hosts would
generate just over one million BGP4 routes.
(That is making a scientifically unjustified extrapolation
of the graphs in my CCR paper). But if the rate of multihoming
increases by a factor ten, and if multihoming continues to
require de-aggregation, we'd get ~10 million routes in BGP.
Numbers beyond that imply a much bigger deviation from current
practice than seems likely to me, since it's clear that the
majority of mobility needs will be met using captive-customer
addresses that are intrinsically PA.

My conclusion is that we can hope for 1 million routes but
should have a solution for 10 million. I don't think there is
anything in the data or technology that tells us we will
need a solution for 100 million or more.

All IMHO of course.

>    document are reduced, the actual table size would be larger.

I don't think that quite covers the very reasonable speculation
that the Internet will grow to around ten billion publicly-
addressable hosts under IPv6, and that this is part of the
problem. So I think we need to be clear that significantly
larger numbers are in our future.

    Brian

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 25 Feb 2010 18:32:28 -0500

Yes, thank you.  That would address my concern.  I was reading 
"extrapolate ... would look like" in a broader sense than you intended.
Yours,
Joel

---

From: Robin Whittle <[email-masked]>
Date: Fri, 26 Feb 2010 15:48:55 +1100

Hi Thomas,

> feel are not right - and explain why.

OK.  I support the improvements you have been discussing with Joel.
I have my own attempt (msg06099) to describe the problem, and won't try
to alter your document to match that.  I do have some suggestions
regarding "portability" and other difficulties inherent in renumbering.

> using multiple ISPs. Won't those users want multihoming to work for

I doubt this would occur very much.  My impression is that most existing
services, such as via DSL, fibre or HFC cable are highly reliable except
for when a storm or other disastrous event damages infrastructure.
However I guess those who live in earthquake-prone areas, or where cyclones
and hurricanes are a constant threat, would have different perspectives.

If the DSL or whatever service is as reliable as the one I use - and I
only vaguely remember a short outage in almost four years - then I think
few people would want to add a router to link two services, and pay for
the extra service year-after-year.  The act of adding another box or
configuration item to the multihoming arrangement is would probably be
less reliable than the raw DSL service.

I suppose massively over-engineered, super-fast, multihomed home Internet
services could be the new form of hot-rodding, just as illuminated
sci-fi-like fluid-cooled gaming PCs with massive radiators seem to have
taken over from the hot-rodding of automobiles.

> on any one particular number.

OK.

> and "renumbering hurts bad". They are not the same thing.

Portability is a solution to the problem - the only solution - while
"renumbering hurts bad" is the problem.

So perhaps I am going a little beyond the formal scope of the document
by referring to the problem in terms of the lack of its solution.

Still, since I (and I think many others, especially those with networks)
perceive the problem to be a lack of having portable address space which
is affordable, accessible and does not overly burden the interdomain
routing system, I think it would be reasonable to use the "portability"
term.

> avoiding the pain of renumbering (other than NAT).

Yes, but since NAT gives no globally accessible hosts (unless by port
forwarding) "portability" of address space is the only solution - and
it comes at a price which is too high for many networks, and never
in scalable fashion.

> actually care about (or want) when they want "portability". 

Core-Edge Separation (CES) architectures such as those being considered by
the RRG: LISP, Ivip, TIDR and IRON-RANGER are all intended to provide
portable address space - which is also suitable for end-user networks
to multihome, and have inbound TE - which does not unreasonably burden
the interdomain routing system's control plane.

Perhaps to those outside the RRG, "portability" implies a prefix of
space being advertised in the DFZ, people who working on CES architectures
are in the business of providing large numbers of genuinely portable
global unicast prefixes which are not advertised as individual prefixes
in the DFZ.  This includes prefixes longer than /24 all the way to
individual IPv4 addresses.  Ivip is not restricted to prefixes and can
separately map "micronets" of any number of contiguous IPv4 addresses
or IPv6 /64s.

With Ivip, a "Mapped Address Block" (MAB) is advertised in the DFZ.  For
instance a /16 MAB is advertised, but covers the "edge" "Scalable PI"
(SPI) address space used by thousands of end-user networks.  In LISP,
the term "EID" refers to this scalable "edge" space and the term
"coarse prefix" may be used for the same thing as Ivip's MAB.

> They want

Yes, but the only way of having global unicast address space and
avoiding changing this when using a different ISP is for the address
space to be "portable" to any ISP.

> can multihome with PA space, but you get the same bad properties.

OK - I agree that the much the same routing scaling difficulties occur
due to advertising a PI prefix via one ISP or another as occur by
sometimes (when there is a fault) advertising a PA prefix which comes
from one ISP-A's shorter prefix as a more specific prefix via another
ISP-B.  With PA space, this does not normally add to the number of
prefixes in the DFZ, but making the switch to ISP-B adds a new prefix,
which is at least as disruptive as changing he advertisement of an
existing prefix.

> I am not convinced that this is an accurate statement.

I think this is a reasonable way of characterizing the problem.

Below are some suggested alterations to the RADIR ID.

  - Robin

4.3. End Site Renumbering

   It is generally considered painful and costly to renumber a site,
   with the cost proportional to the size and complexity of the network
   and most importantly, to the degree that addresses are stored in
   places that are difficult in practice to update.

   The addresses of hosts and of the network's prefix may be stored in
   configuration  files and other places outside the network.  These
   address configurations are beyond the network's control and may be
   unknown to the network.

   Altering the DNS zone files for a large network may be an error-prone
   and costly process.  Furthermore, if the network hosts services for
   other organizations, the addresses of its hosts may appear in the
   zone files of other organisations, including those of the network's
   customers.  So renumbering may require the network to inconvenience
   its own customers, and burden them with the error-prone task of
   changing their DNS zone files to include new addresses at the
   particular time when the new space is adopted.  This cannot be done
   without disruption to the services provided to these customers.

   When using PA space, a site must renumber when changing providers.
   Larger sites object to this cost and view the requirement to renumber
   akin to being held "hostage" to the provider from which PA space was
   obtained.  This raises questions of competition policy and reduces
   the ability of networks to take advantage of better and/or less
   expensive alternative services.

   Consequently, many sites desire PI space, since this is currently the
   only type of global unicast space which is portable to any ISP.
   Having PI space provides independence from any one provider and makes
   it easier to switch providers (for whatever reason).  However, each
   individual PI prefix must be propagated throughout the DFZ and so
   adds to the control plane load.

   It should be noted that while larger sites may also want to
   multihome, the cost of renumbering drives some sites to seek PI
   space, even though they do not multihome.