Subject: Re: [rrg] IPv4 & IPv6 routing scaling problems

From: "Dale W. Carder" <[email-masked]>
Date: Wed, 03 Feb 2010 22:18:46 -0600

> Joel, in (msg05925), you wrote:
>> 
>> The IPv4 Internet works. The routers, and the routing system, cope
>> with the current pressures. To my way of looking at things, the
>> question is how will the Internet routing system cope with growth.
>> But, definitionally, there really is not that much growth left in
>> IPv4.

As an operator, I would agree with this, but I fear it only to be 
true until prefixes start having substantial resale value.  At that
point, slicing and dicing would really begin once the money starts 
flowing.

The RIR's appear to me to be gearing up for this to some degree.

Dale

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 4 Feb 2010 22:18:35 -0500

> point, slicing and dicing would really begin once the money starts

I think it's not just 'for sale' but 'gosh /24 really isn't the limit
is it? lets start accepting and passing on /25../26../27...etc' Sure
it's not going to get to Avagadro's number of prefixes in v4, but 3B
is still way more than 2M (which is about where current vendors stop
hedging today).

In the end, ipv4 can get larger than current platforms can handle,
quickly, and future planned platforms as well. It seems that the same
problem exists regardless of protocol#.

> The RIR's appear to me to be gearing up for this to some degree.

yes.
-chris

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 04 Feb 2010 23:05:21 -0500

I may be missing something, but it seems like the driver for such 
disaggregation in v4 would be more obvious with some of the 
map-and-encaps than with the current architecture.

Once things are worth money, residential and SOHO folks be NATed.  I 
don't like it, but that simply seems to be inevitable.  Hence they will 
not be contributing disaggregated prefixes into teh pool.

In fact, as far as I can tell, anyone who is NATed can happily be 
aggregated (because the NAT will take care of renumbering / isolation 
issues for the customer.)

Folks who need to host a small number of servers might be able to get by 
with a very long prefix, but no one else could.  They either need more 
addresses, or none.
Again, if we assume economics is kicking in, it would seem that using 
hosting providers, who get benefits of scale, would be more cost 
effective than trying to host servers using very long prefixes, for 
which you pay money.

No, I can not prove any of that analysis.
And all other things being equal, I would prefer a solution taht 
addresses IPv4.
What I don't want to do is rule out good architectural solutions because 
they only address IPv6.

(One path would be to figure out what we think the right answer looks 
like.  Then figure out if it applies to IPv4.  AIf not, we can figure 
out what reasonable paths we ought to explore in case Chris is right and 
the v4 economics actually drive an explosion in disaggregated prefixes.)

Yours,
Joel

Christopher Morrow wrote:
> _______________________________________________
> rrg mailing list
> [email-masked]
> http://www.irtf.org/mailman/listinfo/rrg
>

---

From: Shane Amante <[email-masked]>
Date: Thu, 4 Feb 2010 23:29:07 -0700

Chris,

On Feb 4, 2010, at 20:18 MST, Christopher Morrow wrote:
> is it? lets start accepting and passing on /25../26../27...etc'

I have to disagree, at least with respect to anything longer than /24.  Specifically, while you're immediate upstream (whom you're paying money to) may accept longer than /24, they can't (or, actually, won't) make any guarantees that any other provider will accept that prefix, effectively leaving you with only partial (barely any?) connectivity to the Internet.  OTOH, you're correct that it's possible that much shorter IPv4 prefixes (/8 - /23) may be de-aggregated, but I would add two things:
1)  In the short run, it's possible for SP's to use well-known techniques like FIB aggregation[1] and/or proxy aggregation in an attempt to constrain FIB and RIB growth.  However, TANSTAAFL and both are likely to further stress existing control planes on existing routers, which leads to my next point ... 
2)  We don't know when/where the control plane (RIB) will break with increased pressure of de-aggregation.  The traditional _assumption_ that we'll run out of FIB space long before the control plane falls over[2]; however, what could also happen is that with larger numbers of IPv4 *routes* in the global routing table, convergence may become intolerably slow for some (or, most?) applications, thus pushing people more quickly toward IPv6 with a not yet as big routing table and/or that doesn't have some of the "optimizations" that will be necessary to the IPv4 control plane protocols to slow down convergence to keep the system stable (cf: Sprint, iMCI, etc. in the mid- to late '90's) ... 

Note, neither of the above scenarios are by any means "pretty", but I did want to add there are 'tools' that may provide some [small] amount of runway and dampen some of the hysteria that the sky is falling and we need to have a fully deployed ID-Loc solution by tomorrow ...

-shane

[1] http://tools.ietf.org/html/draft-zhang-fibaggregation-02 <-- already (was) being looked at in GROW!
[2] Let's not forget that it's not *just* routes in BGP, but just as importantly *paths* that has negative scaling implications, as well, on the control plane, (note slide 13):
http://www.ietf.org/proceedings/74/slides/grow-6.pdf
    ... and, keep in mind, there are things on the horizon (i.e.: add-paths, SIDR?) that may add _more_ state to be pushed around in BGP (should they see widespread deployment), which is scary.  OTOH, on a more optimistic note, there are some additional enhancements that can make BGP more efficient, however it's unclear how much benefit several of the enhancements may provide as not all of them have been studied.

> Sure

---

From: Robin Whittle <[email-masked]>
Date: Fri, 05 Feb 2010 20:16:02 +1100

Short version:  Joel seems to be stating that map-and-encaps (AKA
                CES) for IPv4 would lead to worse disaggregation -
                more prefixes in the DFZ.  I don't understand this
                at all, since CES architectures such as LISP and
                Ivip are intended to, and would, greatly reduce this.

Hi Joel,

Chris Morrow (msg05965) agreed with Dale Carder that once there was
money to be made reselling IPv4 address space, that the slicing and
dicing would "really begin" - which I understand means an increase
the growth rate of the number of prefixes in the DFZ.

I think this applies to any ISP and PI prefixes which are compacted
and then split in two, with the empty half going directly or
indirectly to either an ISP or an end-user network.

The CEE and CES routing scaling solutions help by enabling end-user
space to be portable and multihomable without advertising each chunk
of it in the DFZ.  So to the extent that this slicing and dicing
would greatly increase the number of PI end-user prefixes in the DFZ,
CES and CEE architectures can largely eliminate this problem.

Chris also wrote that in addition to the sale of space, another
factor in the growth of prefixes would be an end to the current
practice of not passing on any IPv4 routes in the DFZ longer than /24.

In the absence of a scalable routing solution, I think this too
could lead to more PI prefixes in the DFZ, particularly if those with
a handful of /24s sold them in a spit form, or handed them back and
they were for some reason split again.  I assume that only end-user
networks would be using these /24s and anything longer which was
universally accepted in the DFZ.

Even within the /24 limit, there's tremendous scope for growth in PI
prefixes.  Today: http://bgp.potaroo.net/as6447/ 163,261 /24s are
advertised, which is more than half of the prefixes in the DFZ.  Yet
this only covers 34.8 million IPv4 addresses, which is less than 1%
of the global unicast address space.

Chris indicated that router vendors regard 2 million prefixes as the
maximum for today's routers - and that for both IPv4 and IPv6, the
number of prefixes in the DFZ could quickly get larger than what
current and planned routers can handle.

He also agreed with Dale that the RIRs are gearing up for (in my
words) some kind of market in address space.

> map-and-encaps than with the current architecture.

By "more obvious" I guess you mean that the pressure to advertise
more prefixes in the DFZ would increase as a result of using "some of
the map-and-encaps".  Since you haven't mentioned any specifically, I
assume you mean LISP, Ivip and the Core-Edge Separation solutions in
general.

Any one of the following shows how a CES architecture will greatly
reduce the pressure for the space of end-user networks to be
advertised as individual PI prefixes in the DFZ, which is the cause
of the scaling problem and is exactly what Chris was concerned about:

  http://tools.ietf.org/html/draft-ietf-lisp-06
  http://tools.ietf.org/html/draft-whittle-ivip-arch-03
  http://www.ietf.org/mail-archive/web/rrg/current/msg05865.html

so I don't understand why you think a CES solution would make things
worse.

> don't like it, but that simply seems to be inevitable.  

I don't understand why you think this.  IP addresses would need to be
worth a large amount of money before it was more profitable to force
residential and SOHO customers onto a behind NAT address, rather than
giving them a single PI global unicast IPv4 address like they get
today.  This is in part because the single address enables the
customer to port-forward as they wish - which can't happen if
multiple such customers are behind NAT and sharing a single IP address.

For this to occur, we would need to get to a stage of very high
address utilization right across the 3.7 billion range.  As long as
any space is not widely used, it would always be cheaper and more
profitable for all concerned to compact the usage into one part of an
underutilized prefix and then to sell, rent or whatever the now empty
part to an ISP.

As long as ISPs can get space like this, they will use it as they do
now - one IPv4 address per customer.  This is because giving the
customer a NATed address, behind a NAT box which also serves other
customers, is always going to be a lower value service - and the ISP
will be competing with others who can get space on the same market,
and provide single IPv4 address DSL etc. services as are used today.

> Hence they will not be contributing disaggregated prefixes into
> the pool.

I don't understand this.  Even if ISPs did put a large number of
residential and SOHO customers behind NAT, this wouldn't affect the
general problem of disaggregation of IPv4 space: the advertisement of
more and more prefixes in the DFZ.  An ISP with a /16 could have, in
theory at least, ~64k single IPv4 customers on this prefix.  If they
converted some or all of it to NAT boxes each serving 8 customers,
then they could get ~512k customers.  But this doesn't mean there
would be any more disaggregation - any more prefixes advertised in
the DFZ.

> issues for the customer.)

I don't understand what you are saying here. As I just wrote, whether
customers get a straight IPv4 address or an address behind NAT won't
alter how ISPs advertise their prefixes in the DFZ.

> with a very long prefix, but no one else could.  They either need more

With a CES system - which is what you are discussing - those
end-users which wanted 2, 8, 32 or whatever portable, multihomable,
IPv4 addresses would get it through the CES system, without
contributing to growth in the number of DFZ prefixes.

Each IPv4 address the CES system provides - "edge" address space: EID
space in Ivip or SPI micronet space in Ivip - is still an IPv4
address. So the overall value of IPv4 space will affect the cost of
renting or "owning" this space just as it affects the cost of ISPs
getting space for their PA customers, including all those residential
and SOHO customers with a single IPv4 address.

> effective than trying to host servers using very long prefixes, for

I think most end-user networks running web-servers and the like would
continue to use hosting companies rather than run the servers on
their own premises, and pay their ISP for the traffic.

Hosting companies are end-user networks too, so they could be using
the scalable "edge" address space provided by the CES system.

> No, I can not prove any of that analysis.

I can't understand a lot of what you wrote - and the part I think I
understand, that a CES architecture would worsen the routing scaling
problem -

   "it seems like the driver for such disaggregation in v4
    would be more obvious with some of the map-and-encaps
    than with the current architecture."

seems completely contrary to what LISP, Ivip or any other effective
CES architecture which worked would do.

> And all other things being equal, I would prefer a solution that
> addresses IPv4.
>

I think it is not absolutely necessary to have the same architecture
for IPv4 and IPv6.

For instance, if for some reason CEE was decided as the best solution
for IPv6, CES may still be the best for IPv4, since no CEE solution
works for IPv4.

This is because with CEE (Locator / Identifier Separation - not to be
confused with LISP) multihoming a /20 end-user network with two ISPs
requires the use of two /20 prefixes of PI space - one from each ISP.
 There's no way this wastage of space is feasible for IPv4 due to the
shortage of address space.

> the v4 economics actually drive an explosion in disaggregated prefixes.)

OK - CEE can't work for IPv4 and so far the only non-CEE proposals are:

    LISP, Ivip, TIDR and RANGER (CES)

    hIPv4

    Aggregation with Increasing scopes

Maybe the last two will turn out to fit the architectural pattern of
CES too - I haven't read them yet.

 - Robin

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 5 Feb 2010 13:30:37 -0500

> I have to disagree, at least with respect to anything longer than /24.  Specifically, while you're immediate upstream (whom you're paying money to) may accept longer than /24, they can't (or, actua…

I should have been more clear with a few things :( (first that I would
hate to see this sort of thing happen) I was suppposing a world where
at least the following things have come to pass:
1) ipv4 freepool is empty
2) some number of providers have no more PA space to dole out to
new/existing customers
3) IP address space is available for 'purchase' (lease? whatever)
through 'not RIR' methods
4) there exist businesses with more than one 'site', which connect to
different upstream networks

In these cases it is a possible future where the enterprise in
question pays both upstreams to exchange these routes. There's the
possiblity that this becomes commonplace enough such that more global
leaking of ip prefixes longer than /24 is 'the norm'.

> 1)  In the short run, it's possible for SP's to use well-known techniques like FIB aggregation[1] and/or proxy aggregation in an attempt to constrain FIB and RIB growth.  However, TANSTAAFL and bot…

this is independent of my example future. (unless by FIB Aggregation
you mean: "Default to the left" aka Schiller's paradigm)

> http://www.ietf.org/proceedings/74/slides/grow-6.pdf

i agree that it's not just routes, it's not just paths, it's not just
speed of change (of either of the two previous items)...

>    ... and, keep in mind, there are things on the horizon (i.e.: add-paths, SIDR?) that may add _more_ state to be pushed around in BGP (should they see widespread deployment), which is scary.  OTO…

also agree with this, adding more state or more memory/cpu
requirements to existing systems is full of unknown consequences.

-chris

---

From: Danny McPherson <[email-masked]>
Date: Fri, 12 Feb 2010 07:46:15 -0700

On Feb 5, 2010, at 11:30 AM, Christopher Morrow wrote:

>>    ... and, keep in mind, there are things on the horizon (i.e.: add-paths, SIDR?) that may add _more_ state to be pushed around in BGP (should they see widespread deployment), which is scary.  OTOH, on a more optimistic note, there are some additional enhancements that can make BGP more efficient, however it's unclear how much benefit several of the enhancements may provide as not all of them have been studied
> 

Not surprisingly, I completely agree with this as well.  It's not
just the number of prefixes in the current system, it's the number
of unique routes (paths) to each prefix, the former of which is 
often 20x or more the number of unique prefixes in the system (today!), 
and any time any one of those prefixes has state change, all the paths
see some piece of churn internal to a routing domain (and often 
inadvertently trigger external artifacts like suppression from 
damping, etc.).  

Each of those unique routes for a given prefix means more everything, 
including FIB I/O.  IPv6 during long term transitional coexistence with
IPv4 is just going to compound this.  

Surprisingly, all the work in IDR is about adding more state and
paths and attributes and features, not dealing with this scaling issue.
If the goal here is scalability, it's important to factor both inter-
domain, AND intra-domain, and understand the issues with scaling in 
the current system.

-danny

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 12 Feb 2010 11:39:49 -0500

> IPv4 is just going to compound this.

I really think that the conversation about ipv4/ipv6 and route-scaling
has to understand that for the foreseeable future we're going to have
to deal with both ip protocols... and in 25-30 (maybe more) years a
third protocol.

-chris

---

From: Danny McPherson <[email-masked]>
Date: Fri, 12 Feb 2010 09:41:59 -0700

> to deal with both ip protocols... and in 25-30 (maybe more) years a

Indeed, hence my "long term transitional coexistence" phrasing :-)

-danny

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 12 Feb 2010 11:45:48 -0500

> Indeed, hence my "long term transitional coexistence" phrasing :-)

Sorry, I meant 'there is no transition, there is only coexistence'
(from my perspective at least that seems to be what'll happen, of
course no crystal balls and only 5 computers ever will be needed.)

-Chris
(and I get that you == danny get this, but for the record I think we
should be clear that ipv4 ain't going away, ever)

---

From: Tom Vest <[email-masked]>
Date: Fri, 12 Feb 2010 11:51:45 -0500

> should be clear that ipv4 ain't going away, ever)

Does "ain't going away" mean that you cannot envision any time in the future in which, say, IPv4 has the same impact on scalability concerns as RFC1918 has today?

TV

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 12 Feb 2010 12:15:47 -0500

> Does "ain't going away" mean that you cannot envision any time in the future in which, say, IPv4 has the same impact on scalability concerns as RFC1918 has today?

1918 does have scaling implications, if you carry it as internal
routes... at some ISP's internal routes are ~30% of the total table
size seen. It's not the same cost for everyone, but neither is a
single prefix in the global table :(

I also suspect (as I said before) we'll see longer than /24 routes in
the global table (not leaks/mistakes) before this is all over. I don't
like that concept, but I don't see an outcome that doesn't include
this.

-chris

---

From: Tom Vest <[email-masked]>
Date: Fri, 12 Feb 2010 12:38:18 -0500

> single prefix in the global table :(

That had not escaped me ;-)

But it seems to me that those RFC 1918 implications are sufficiently different in degree (e.g., of extra-local effects, of susceptibility to effective local mitigation) that the question made sense -- i.e., that the comparison was more realistic than either of the alternative scenarios (A: IPv4-specific scaling impacts are never going to get any better, B: IPv4-specific scaling issues are certain to be completely eliminated).

> like that concept, but I don't see an outcome that doesn't include

Well, you have company there.
Hopefully both of us will turn out to be wrong, or alternately that this development won't eliminate all detours around A.

TV