Subject: [rrg] Paul Jakma's blog article

From: Robin Whittle <[email-masked]>
Date: Sat, 13 Feb 2010 14:09:31 +1100

Short version:    Paul's article fails to properly distinguish
                  between CES and CEE architectures.  The article
                  also implies that scalable routing solutions are
                  in general based on "Locator/Identifier Separation"
                  - but this is only true of CEE architectures.

                  His suggestion of using IPv4 header options to
                  extend TCP port numbers to 32 bits, for the
                  purpose of making a NAT box better able to
                  support multiple hosts behind NAT, is
                  unfortunately impractical.  This is due to DFZ
                  and routers in general handling packets with IPv4
                  header options on the "slow path".

Hi Paul,

On your page:

   http://pjakma.wordpress.com/2010/02/12/making-the-internet-scale-through-nat/

there are a number of statements which I think repeat mistakes made
by others:

PJ>    The IRTF’s RRG has been discussing various solutions.
PJ>    The IETF have a LISP WG developing one particular solution,
PJ>    which is quite advanced.

The LISP WG is to develop LISP with the ALT global query-server based
mapping system as an experimental protocol.  Mobility is out of
scope.   After after over two years' work, there is no indication
that ALT can scale well enough for mass adoption:

  ALT structure, robustness and the long-path problem
  http://www.ietf.org/mail-archive/web/lisp/current/msg01801.html

According to Noel Chiappa, the LISP team will soon be turning to a
DNS-based global mapping resolution system:

  http://www.ietf.org/mail-archive/web/rrg/current/msg05772.html

PJ>    The essential, common idea is to split an internet address
PJ>    into 2 distinct part, the "locator" and the "ID" of the
PJ>    actual host.

This is not a common idea among all RRG proposals.  This
"Locator/Identifier Separation" approach is used by all the Core-Edge
Separation (CES) approaches:

   GSE (not an RRG proposal)
   GLI-Split
   ILNP
   Name Based Sockets

and, although I haven't read them yet, I think these two as well:

   Name Overlay (NOL)
   RANGI

"Locator/Identifier Separation" means that separate objects, in
separate namespaces are used for uniquely Identifying hosts and for
specifying their Location (either exactly, or down to the level of a
particular ISP or end-user network).

This is a complete change from the current two-level naming structure
of IPv4 and IPv6, in which the Identifier and Locator functions are
both played by the one object: an IP address:

     Role             Level           Conventional IP naming model

     Text name  <---- FQDN
     Identifier <---] IP address
     Locator    <---]

Despite LISP being an acronym for "Locator/Identifier Separation
Protocol", LISP is not a CEE architecture and does not implement
"Locator/Identifier Separation"

LISP is a Core-Edge Elimination (CES) architecture.  These are the
CES architectures. Those in brackets are not RRG proposals and are
not longer being developed:

   (map-and-encaps)
   (LISP-NERD)
    LISP-ALT
   (APT)
    Ivip
   (TRRP)
   (Six/One Router)
    TIDR
    IRON-RANGER

All CES architectures preserve the current 2 level IP naming model
and require no changes to host stacks or applications.  They work
with both IPv4 and IPv6, except for Six/One Router, which is IPv6-only.

CEE architectures are only practical for IPv6 and always require
changes to host stacks.  Most CEE architectures require changes to
applications, but GSE and GLI-Split do not.

Please see these messages for more on the distinction between CES and
CEE architectures, and on the 2008 paper which most popularised these
terms - which I think are the most important architectural
distinction between most of the scalable routing proposals.  (Not all
proposals are CEE or CES.)

   Today's "IP addr. = ID = Loc" naming model should be retained
   http://www.ietf.org/mail-archive/web/rrg/current/msg05864.html

   CES & CEE are completely different (graphs)
   http://www.ietf.org/mail-archive/web/rrg/current/msg05865.html

   CES & CEE: GLI-Split; GSE, Six/One Router; 2008 sep./elim. paper
   http://www.ietf.org/mail-archive/web/rrg/current/msg06009.html

PJ>    The core routing fabric of the internet then needs only to be
PJ>    concerned with routing to "locator" addresses.

This is true of CEE architectures.  This is because hosts are then
Identified by something different from IP addresses.  So the
host-to-host sessions survive changes in the IP addresses used by the
hosts, and renumbering a network when choosing a new ISP does not
alter the identity of the hosts.  CEE architectures multihome by
giving each host two or more IP addresses, one from each ISP.

Your statement does not apply to CES architectures.  For CES, the DFZ
routers are primarily concerned with the ISP's prefixes, which use
the "core" subset of the global unicast address space, which is what
remains after a new subset of scalable "edge" addresses have been
removed.  These "edge" addresses are supported by the CES mechanisms
and can be used by end-user networks for portability, multihoming,
inbound TE and perhaps mobility - in a highly scalable way, meaning
without much impact on the DFZ control plane or the RIBs and FIBs of
DFZ routers.

IP addresses in both the "core" and "edge" sets function both as
Identifiers and Locators - but ITRs treat packets differently when
their destination address is an "edge" address.  The look up mapping
and tunnel the packet to an ETR address, which is different from the
usual approach of forwarding the packet to a neighbouring router.

PJ>    Figuring out how to map onward to the (presumably far more
       numerous and/or less aggregatable) "ID" of the specific
       end-host to deliver to is then a question that need only
       concern a boundary layer between the core of the internet and
       the end-hosts.  There are a whole bunch of details here
       (including the thorny question of what exactly the "ID" is in
       terms of scope and semantics) which we’ll try skip over as
       much as possible for fear of getting bogged down in them.

It is not an "ID".  Your statement would be true of CES architectures
if "edge address" was used, meaning an "EID" address for LISP or an
SPI (Scalable PI) address for Ivip.

Part of the terminology problem in this field is that LISP is
incorrectly named and its EID term - "Endpoint Identifier" - is a
misnomer too.  The term "EID" is used beyond LISP.  I don't use in
Ivip.  In LISP, EIDs are "edge" addresses which are just as much
combined Identifiers and Locators as "core" addresses.  Its just that
the ITRs use a different algorithm for getting the packets towards
their destination - one which does not rely on each "EID" prefix
being advertised in the DFZ.

CES architectures generally have a mapping system by which an ITR can
find out, for any given "edge" address it finds in the destination
field of a traffic packet, the "core" address of an ETR to which it
should tunnel this packet.  This has nothing to do with "Identifiers"
and "Locators".

CEE architectures usually have mapping systems so a query for a
particular Identifier will return the one or more Locators on which
this host can be reached.

PJ>    We will note that some proposals, in order to be as
       transparent and invisible to end-host transport protocols
       as possible, use a "map and encap" approach – effectively
       tunneling packets over the core of the internet.

These are all CES architectures.  CEE architectures have no need for
tunneling.

PJ>    Some other proposals use a NAT-like approach, like Six-One or
       GSE, with routers translating from inside to outside.

"Six/One" is an earlier host-based proposal from Christian Vogt -
arguably an improved version of Shim6.  I think you are referring to
"Six/One Router", also from Christian Vogt.  This is a CES
architecture, which uses address rewriting rather than encapsulation
for tunneling traffic packets across the DFZ.  Please see (msg06009)
for why it is a CES architecture.

GSE is an earlier CEE architecture.  ILNP and GLI-Split are inspired
by GSE.  Six/One Router is not related to GSE - it is a CES
architecture.  In (msg06009) I argue that the 2008 paper which did
most to establish the use of the CES and CEE terms mistakenly
classified GSE as a CES proposal.

PJ>    Most proposals come with reasonable amounts of state, some
       proposals appear to have quite complex architectures and/or
       control planes.

Yes, but there are big differences in where the complexity is added.

CES architectures do not require any additional complexity, or any
other changes, in host stacks or applications.  They work by adding
some new elements to the routing system - and a mapping system - but
do not require alterations to most routers.

As far as I know, CEE architectures fall into two groups.  One
involves no additional router functions, but requires extensive
changes to host stacks and applications.  The other group has some
extra router functions (address rewriting of incoming and outgoing
packets) and requires stack changes, but no application changes.

These differences, and the fact the CES maintains the current IP
naming system while CEE completely changes it, means that the
distinctions between these two types of solution are highly
significant and helpful when discussing scalable routing solutions.

Both types of architecture, if successfully implemented and widely
adopted, could in principle solve the routing scaling problem.
However, there are stark differences in the relationships between
benefits and adoption levels (msg05865).

Good CES architectures provide immediate benefits to adopters, by
supporting all their traffic, and provides scaling benefits in
proportion to the adoption rate.

CEE architectures can only provide substantial benefits to adoptors
and only provide real scaling benefits, when all, or almost all,
networks adopt the new architecture.  This means moving everyone to
IPv6 and altering all host stacks to implement the particular CEE
architecture's alteration of IPv6.  Except for GLI-Split, this also
means rewriting essentially *all* applications.  (Still, for
GLI-Split, IPv6 would need to be ubiquitously adopted and many or
most applications would need to be significantly rewritten to work
with IPv6.)

PJ>    E.g. LISP in particular seems quite complex, relative to the
       "dumb" internet architecture we’re used to, as seems to try
       to solve every possible IP multi-homing and mobile-IP problem
       known to man.

LISP and Ivip both require considerable complexity.  This is not
surprising, since we are planning a once in several decades
enhancement for the IPv4 and IPv6 Internets - with the work to be
done in the network, without requiring host changes.

My proposal Ivip includes the TTR Mobility extension, which will work
for both IPv4 and IPv6 - and an alternative to encapsulation.  So it
is pretty ambitious, as I think any architectural upgrade for the
Internet should be.

   http://www.firstpr.com.au/ip/ivip/

Both LISP and Ivip involve significant complexity.  Ivip ITRs and
ETRs will be less complex than LISP ITRs and ETRs, because the
end-user controls the mapping in real-time, relieving the ITRs of the
task of reachability testing through the ETRs to the destination
networks.  So Ivip ITRs do no such testing and make no choices
between ITRs.

PJ>    Some proposals also rely on IPv6 deployment.

This statement applies to all CEE architectures - they are all for
IPv6 and not at all for IPv4.  One fundamental reason for this is
that with CEE, an end-user network with X IP addresses requires X
addresses from each of its ISPs, and at least two ISPs are required
for multihoming.  For IPv4, this would double the address
requirements - which is clearly unworkable.

The only CES architecture which only works for IPv6 is Six/One Router
- but this is no longer being developed and apparently required a
change to the IPv6 headers to use a bit for a flag.

You suggest redesigning TCP to have a 32 bit port number - since this
would help with a NAT box with a single IPv4 address being able to
handle traffic for more hosts behind NAT.  However, any such new TCP
protocol would only work between two hosts which had upgraded stacks
and suitably upgraded applications too.  I think upgrading
applications is the hardest task of all.

You mention options for the IP header.  As far as I know, IPv6
extension headers can be used.  I understand most or all routers
handle such packets without any fuss - and I guess host stacks which
did not recognise them would ignore them.

Unfortunately the same is not true of IPv4 header options.  One of
the RRG proposals (hIPv4) relied on these, but DFZ routers handle
such packets on the "slow path", making this entirely impractical:

  End-to-end measurements on performance penalties of IPv4 options
  Fransson, P.; Jonsson, A.
  Global Telecommunications Conference, 2004. GLOBECOM apos;04. IEEE
  Volume 3, Issue , 29 Nov.-3 Dec. 2004 Page(s): 1441 - 1447 Vol.3
  10.1109/GLOCOM.2004.1378221
  http://www.sm.luth.se/csee/csn/publications/end_to_end_measurements.pdf

most routers process such packets on the "slow path" with software.
The last sentence in the abstract is:

  From the analysis it can be concluded that there is a slight
  increase in delay and jitter and a severe increase in loss rate.

I think the latter part of your article is focussed on IPv4, since
your aim is to make NAT more workable.  Unfortunately, it seems that
the "slow path" handling of packets with IPv4 option headers rules
out any solution along the lines you are suggesting.

  - Robin

---

From: Paul Jakma <[email-masked]>
Date: Sat, 13 Feb 2010 10:46:29 +0000 (GMT)

Hi,

>  http://www.ietf.org/mail-archive/web/rrg/current/msg05772.html

Interesting.

I have to say, I don't really know the details of LISP. I just know a 
lot of work has gone into it, and I have no reason to doubt that it 
can be deployed, and can solve at least a few problems. I might not 
be too keen on its apparent complexity, but given the amount of 
engineering gone into it, it seems a deployable system.

> "Locator/Identifier Separation" approach is used by all the 
> Core-Edge Separation (CES) approaches:

> specifying their Location (either exactly, or down to the level of 
> a particular ISP or end-user network).

I don't follow RRG closely enough to be completely au fait with its 
somewhat extensive ontology (which, I have to say, seems to change 
significantly even from month to month - the traffic here can 
sometimes seems from mars if you fail to follow the list closely for 
even a short while). :) A sign of the breadth of different, proposed 
solutions I guess.

I didn't realise there were proposals that did not split the ID space 
in some way. I know there are some proposals that use an implicit 
space, leaving the 2 separate spaces still combined in one greater 
space, e.g. for protocol compatibility reasons, but the split is 
still there.

I.e. I don't mind if an existing space is effectively split up, or if 
2 explicitly separate spaces are used - they're both splits to me.

[snip interesting overview of proposals]

> host-to-host sessions survive changes in the IP addresses used by 
> the hosts, and renumbering a network when choosing a new ISP does 
> not alter the identity of the hosts.  CEE architectures multihome 
> by giving each host two or more IP addresses, one from each ISP.

> removed.

Right, it's still a hierarchical split. Sometimes too much 
categorisation is not useful. I.e. I see there's a distinction, but 
that distinction is not in the general concept of there being a 
split, but in whatever practical details of it.

So I recognise there are 2 different things, call them CES and CEE if 
you want, but I don't see the utility in argueing one that one has a 
split and the other does not. Anyway, doesn't matter (to me ;) ).

> These are all CES architectures.  CEE architectures have no need for

I think I'll stick with "map-and-encap" and "rewriting". I have to 
say, I sometimes wish this WG would try adopt labels that are 
semi-evocative of their meaning where possible, rather than all these 
acronyms. Very confusing! :)

> "Six/One Router", also from Christian Vogt.

Ah, perhaps I do - thanks!

> some new elements to the routing system - and a mapping system - 
> but do not require alterations to most routers.

They tend to be complex themselves though.

> significant and helpful when discussing scalable routing solutions.

Sure, but do we really need to use acronyms? :)

> proportion to the adoption rate.

If that's so, then the map-encap systems will see adoption. I'd have 
my doubts about the "immediate benefits to adopters" though. Doesn't 
CES have a path-MTU problem though?

> architecture's alteration of IPv6.

I'd disagree with this. If you read the blog entry I'm outlining a 
scenario where the immediate problem to be solved is pressure on NAT 
(not multi-homing), leading to hacks being deployed which benefit 
both customer and ISP together. These hacks then can be later, 
slowly, extended to things like multi-homing.

I think it's plausible that IPv4 space for multi-homers will continue 
to be available for long enough to stave off map-encap deployment 
(particularly if map-encap connectivity has PMTU problems). This 
could be facilitated partly by ever greater use of NAT, such that the 
*first* problem that ISPs and end-users really clamour to have solved 
(and will upgrade for) is NAT port space pressure. Some big NAT using 
networks have already noticed such pressre problems today, e.g., so 
it's not such a hand-wavy scenario.

Everything else follows logically from that. Just saying :)

> done in the network, without requiring host changes.

I have to say, I'm not a fan of complex networks. There seem to be 
good economic reasons why a dumb inter-network of smarts hosts won 
out over a plethora of smart networks of dumbish terminals (to 
varying degrees).

Not convinced myself that avoiding changing host software is a 
long-term holy grail. Particularly if the correct place economically 
and technically is to change the hosts. As Brian Carpenter has said 
here regarding IPv6, changing the host software is /not/ that hard - 
changing the /network/ is the hard part.

I can't see customers clamouring for "LISP" (what do customers care 
about how the network works) and the path of least resistance for 
ISPs is to support their hosting and multi-homing businesses by 
using NAT to reclaim address space from access customers.

However, that's all debatable, and I'll guess we'll see.

My bets are on end-host extensions like Shim (Shim6 if IPv6 becomes 
popular, something similar for v4 if not) and, possibly, extensions 
to facilitate the dumbest possible network middle-boxes - like the 
NAT extension described in my blog entry.

> extension headers can be used.

The assumption in my blog post is that IPv6 deployment continues to 
stall, and people start looking for ways to band-aid problems with 
IPv4 to allow it to be continue to used.

I show that such an approach, by solving immediate problems 
incrementally, could iterate toward a new address family that solves 
internet scaling problems while remaining somewhat backward 
compatible with IPv4 as this development process goes on. I.e. change 
the end-host software by all means, but remain compatible with the 
network.

With the hindsight available to us today, this is probably the 
approach that should have guided IPng.

> most routers process such packets on the "slow path" with software.

Yes, I mentioned that. Thanks for the references though. However, I 
also claim that if an IP option became popular that newer hardware 
would fast-path certain common cases.

We should perhaps take care to avoid making long-term architectural 
decisions based on trivialities of todays implementations.

> out any solution along the lines you are suggesting.

I think I covered this in the blog post.

a) Today: "slow connectivity" trumps "no connectivity, cause a NAT
    box in the middle is maxed out".

b) Longer-term: IP options are slow-path only cause they're uncommon.
    There is no reason why a popular option, reliably placed in the
    packet could not stay on the fast path. (E&OE, IANAEE)

regards,
-- 
Paul Jakma	[email-masked]	Key ID: 64A2FF6A
Fortune:
A meeting is an event at which the minutes are kept and the hours are lost.

---

From: Robin Whittle <[email-masked]>
Date: Sun, 14 Feb 2010 00:40:01 +1100

Hi Paul,

>>  http://www.ietf.org/mail-archive/web/rrg/current/msg05772.html

Noel just wrote that the DNS approach will be an alternative.  While I
believe that DNS would be a better approach than ALT, I can't speak for
the LISP people and I don't know what their plans are.

My point was that just because there is a WG with a bunch of people
working on LISP-ALT doesn't mean that the system will actually work in a
way which is suitable for global adoption.

> lot of work has gone into it, and I have no reason to doubt that it can
> be deployed, and can solve at least a few problems. 

Here's at least one reason:

I haven't kept this up to date, but here are some others:

  http://www.firstpr.com.au/ip/ivip/lisp-links/#critiques

> I might not be too
> keen on its apparent complexity, but given the amount of engineering
> gone into it, it seems a deployable system.

Perhaps you are assuming that if a lot of people work hard enough on
something then it will do what it is supposed to do.  I don't think this
at all.

> significantly even from month to month - the traffic here can sometimes
> seems from mars if you fail to follow the list closely for even a short
> while). :) A sign of the breadth of different, proposed solutions I guess.

I don't understand how "ontology" applies to the RRG.

The terminology is developing, and that is part of what I am writing
about - since some of the terminology used at present is misleading, and
this leads to genuine misunderstandings about the proposals.

> I didn't realise there were proposals that did not split the ID space in
> some way. I know there are some proposals that use an implicit space,
> leaving the 2 separate spaces still combined in one greater space, e.g.
> for protocol compatibility reasons, but the split is still there.
> 
> I.e. I don't mind if an existing space is effectively split up, or if 2
> explicitly separate spaces are used - they're both splits to me.

There are fundamental and important differences.

The CES architectures create an "edge" subset of the range of global
unicast addresses and add special mechanisms (such as ITRs, a mapping
system and ETRs) to make addresses in this subset suitable for scalably
providing portability, multihoming, inbound TE and perhaps mobility to
millions or billions of End-User Networks (EUNs).

All these "edge" addresses and all the "core" addresses (the rest of
those in the global unicast range) are still interpreted according to
the same namespace: the global unicast address namespace, for whichever
of IPv4 or IPv6 this is.  IP addresses from both subsets are still
performing both Identifier and Locator roles.  Its just that there is a
new mechanism for transporting packets addressed to "edge" addresses
across the DFZ (and often across parts of the edge networks of the
sending host and the ISP used by the receiving host's EUN) - tunnelling
from ITR to ETR, rather than the usual process of a router forwarding
the plain packet to a neighbour.

This is not at all "Locator/Identifier Separation".  It is Core-Edge
Separation - separation of one subset of "edge" addresses from the
remaining "core" addresses.  No new namespaces are involved.

CEE architectures involve a separation of the currently combined roles
of Identifier and Locator.  Hosts no longer use IP addresses to Identify
and Locate other hosts.  They use something else - what it is depends on
the CEE architecture - to Identify hosts.  Sessions are established by
this Identifier and are continued on this basis, no matter what IP
addresses both hosts have or change to during the session.  So this
supports multihoming.  Portability is of the Identifiers, not of the IP
addresses a network uses - so they use whatever space their one or more
ISPs give them as PA prefixes.

Core-Edge Elimination architectures always involve "Locator/Identifier
Separation".  This is not at all the separation of one subset of global
unicast IP addresses from the rest, as "edge" space, as is done with CES
architectures.

So "split" and "separation" can be used to describe both processes - but
what is being separated is completely different in the two approaches.

> categorisation is not useful. I.e. I see there's a distinction, but that
> distinction is not in the general concept of there being a split, but in
> whatever practical details of it.

The details of what is being split are absolutely crucial!

> split and the other does not. Anyway, doesn't matter (to me ;) ).

I believe that once you understood the proposals in detail, the question
of what is being split or separated would matter to you.

> I think I'll stick with "map-and-encap" and "rewriting". I have to say,
> I sometimes wish this WG would try adopt labels that are semi-evocative
> of their meaning where possible, rather than all these acronyms. Very
> confusing! :)

You can do this and so ignore what I wrote - but you haven't argued why
anyone else should do the same or why I or anyone else should respect
your choices.

> Ah, perhaps I do - thanks!

I recall that Six/One itself hasn't been discussed since around 2007,
and without knowing it was different, it would be easy to shorten
"Six/One Router" to "Six/One".

> They tend to be complex themselves though.

They certainly do.

> Sure, but do we really need to use acronyms? :)

No - you can use "Core-Edge Separation" and "Core-Edge Elimination".  I
am trying to be concise.

> If that's so, then the map-encap systems will see adoption. 

I believe that the only proposals which should be seriously contemplated
are CES.  Firstly because I think it is a mistake to think the
"Locator/Identifier Separation" naming model is superior and secondly
because there's no way the world will alter all its host stacks or (for
most CEE architectures) rewrite its applications *completely* - which is
what is required before there would be significant benefits to CEE
adoptors or significant routing scalability benefits.

> I'd have my
> doubts about the "immediate benefits to adopters" though. Doesn't CES
> have a path-MTU problem though?

Fred Templin and I have been discussing this.  It requires extra
complexity in the ITRs and ETRs, but we think it can be done.  Please
see the "IRON: SEAL summary V2" thread and my approach:

  http://www.firstpr.com.au/ip/ivip/pmtud-frag/

> (not multi-homing), leading to hacks being deployed which benefit both
> customer and ISP together. These hacks then can be later, slowly,
> extended to things like multi-homing.

I don't see how the above paragraph relates to you disagreeing with the
paragraph of mine before it.

What you are proposing is neither a CEE nor a CES architecture.  You are
proposing to extend the port range of IPv4 TCP and perhaps UDP
protocols, as a means of enabling NAT boxes to serve more hosts from a
single IPv4 address.  This doesn't solve the routing scaling problem in
any way.  It is an attempt to cope with a situation where there is such
a shortage of IPv4 space that there's not enough to run NAT boxes with
whatever limitations they have from the current 16 bit TCP/UDP port
numbering system.

> I think it's plausible that IPv4 space for multi-homers will continue to
> be available for long enough to stave off map-encap deployment
> (particularly if map-encap connectivity has PMTU problems). This could
> be facilitated partly by ever greater use of NAT, such that the *first*
> problem that ISPs and end-users really clamour to have solved (and will
> upgrade for) is NAT port space pressure. Some big NAT using networks
> have already noticed such pressre problems today, e.g., so it's not such
> a hand-wavy scenario.

NAT has nothing to do with multihoming.  I don't understand how you see
your suggestion as relating to the goals of scalable routing which are,
roughly:

  How to accommodate much larger numbers of end-user networks (10 million
  is a figure Brian Carpenter and I have independently suggested) with
  portability, multihoming and inbound TE, in a scalable manner: with
  far less impact on the DFZ control plane and the DFZ routers' RIBs
  and FIBs than the current practice of having each such end-user
  network advertise one or more PI prefixes.

> Everything else follows logically from that. Just saying :)

I don't understand your reasoning - and I can't see where you have
explained your reasoning.

>> enhancement for the IPv4 and IPv6 Internets - with the work to be done
>> in the network, without requiring host changes.
> 
> I have to say, I'm not a fan of complex networks. There seem to be good
> economic reasons why a dumb inter-network of smarts hosts won out over a
> plethora of smart networks of dumbish terminals (to varying degrees).

I think most people feel the same way.  The Internet is a far more
powerful and adaptable system than the phone system, where the network
does everything and the end-user devices are quite dumb.

I think this is a big part of the attraction of CEE architectures,
especially those such as ILNP which involve no additional things in the
network at all.

However, to do ILNP, you have to rewrite all the applications, and
change all the stacks (and get everyone off IPv4 to the ILNP-modified
IPv6 Internet).  And then you need to accept that all hosts need to be
more complex, and that there will generally be more delays in
establishing communications - with more management packets flowing back
and forth etc.  Arguments against doing this are in:

> Not convinced myself that avoiding changing host software is a long-term
> holy grail. Particularly if the correct place economically and
> technically is to change the hosts. As Brian Carpenter has said here
> regarding IPv6, changing the host software is /not/ that hard - changing
> the /network/ is the hard part.

Please take a look at the graphs here:

about how with CEE you have to change essentially all stacks and
applications (no application changes needed for GLI-Split, but that
requires GLI-gateways to handle all packets entering and leaving each
EUN), and get everyone off IPv4 and onto this modified version of IPv6,
before adopters get real benefits and before there are any real routing
scaling benefits.

If you disagree with this, then please write why.

CES involves adding some things to the network.  This is easier than
changing all the apps.  The benefits accrue directly to adopters and
progressively for routing scaling benefits according to the level of
adoption.  The end-result, I believe, is better, since it retains the
current IP naming model, which I believe is superior to the one many RRG
people want: "Locator/Identifier Separation".

> about how the network works) and the path of least resistance for ISPs
> is to support their hosting and multi-homing businesses by using NAT to
> reclaim address space from access customers.

NAT does not provide multihoming.  Multihoming is the ability to
continue communication sessions no matter which of two or more ISPs an
end-user network is using.

> However, that's all debatable, and I'll guess we'll see.

I don't think you have debated anything.  You have stated you disagreed
with some things I wrote, but not why you disagreed.  I am wondering if
you understood all that I wrote.

> popular, something similar for v4 if not) and, possibly, extensions to
> facilitate the dumbest possible network middle-boxes - like the NAT
> extension described in my blog entry.

OK - that is your belief.  But why do you believe this?

> stall, and people start looking for ways to band-aid problems with IPv4
> to allow it to be continue to used.

I think this is an entirely realistic expectation.

> internet scaling problems while remaining somewhat backward compatible
> with IPv4 as this development process goes on. I.e. change the end-host
> software by all means, but remain compatible with the network.

But increasing the number of hosts a NAT box can handle from a single
IPv4 address has nothing to do with the routing scaling problem, or
multihoming, portability, inbound TE or mobility.

> With the hindsight available to us today, this is probably the approach
> that should have guided IPng.

I think it may well have been better to plan a new addressing system,
based on the IPv4 one, which could use the IPv4 address space as a
subset - probably with a new IP version with a different header, or
perhaps with IPv4 headers with option headers.  It would take a few
years for routers to be made or upgraded to handle the new format
properly, but I think this would have had a better chance of being
adopted widely than the completely new and separate addressing system of
IPv6.  However, many of the same problems would have been faced - and I
would not claim that any arrangement would have been satisfactory in the
long-term or have been widely adopted by now.  Unfortunately, IPv4 was
not designed to have any upgrade path.

>> Unfortunately the same is not true of IPv4 header options.  One of the
>> RRG proposals (hIPv4) relied on these, but DFZ routers handle such
>> packets on the "slow path", making this entirely impractical:
> 
> Yes, I mentioned that. Thanks for the references though. However, I also
> claim that if an IP option became popular that newer hardware would
> fast-path certain common cases.

Unfortunately it couldn't become even a little popular since it is
impractical to use it through the DFZ, or probably with most routers.

> decisions based on trivialities of todays implementations.

It is not a triviality that a few billion people use and rely on the
IPv4 Internet, and that we can't force anyone to adopt anything.  Please
take a look at my attempt to list the constraints we face due to the
need for widespread voluntary adoption:

  http://www.firstpr.com.au/ip/ivip/RRG-2009/constraints/

These are not trivialities.

If you want to debate an ideal networking system without concern for the
installed base, then that's fine - you would be debating "clean slate"
designs, with no concern about what exists or how or why anyone would
adopt it.   This is not the purpose of the RRG.  The purpose of the RRG
is to solve a very real scaling problem which exists today with the IPv4
Internet, and can be expected to arise in the IPv6 Internet if and when
it becomes very widely adopted.  This has nothing to do with "clean
slate" design exercises - unless you can suggest how we can have most
people adopt, on a purely voluntary basis, your clean-slate design, and
have people invest in designing and building it first.

>    box in the middle is maxed out".

Its not just slow - it involves such packet loss rates that
communication would be impractical.  Even being slow would be enough to
ensure no-one wants it.

We are a long way from being so desperate for IPv4 address space that we
need to think of ways of making a NAT box handle 100 hosts, rather than
10, or whatever the limit is today due to the 16 bit nature of TCP and
UDP port numbers.

>    packet could not stay on the fast path. (E&OE, IANAEE)

In the long term, I agree - five, ten years or more - routers can be
upgraded or replaced.  As this debate goes on, year after year, I think
the number of routers in the DFZ which can't have their FIB
functionality significantly upgraded by firmware updates will diminish.
 There may come a point where everyone decides it is easiest to upgrade
most of the DFZ routers and to replace the few which can't be upgraded,
than to do whatever else is the alternative.

But until such upgrades are done, there's no way anyone can reliably
communicate across the IPv4 DFZ with a new kind of protocol, or with
IPv4 packets with option headers.

Once we can contemplate such upgrades to DFZ and other routers, then we
could do a CES architecture (Ivip) without encapsulation:

  http://tools.ietf.org/html/draft-whittle-ivip-etr-addr-forw-00

As you note in the "pjakma said" addition to your page:

your proposal would also depend on upgrading applications.  I agree with
your assessment of this aspect of your proposal "This is the most
obviously crack-full part of it all."!

  - Robin

---

From: Paul Jakma <[email-masked]>
Date: Sat, 13 Feb 2010 16:21:08 +0000 (GMT)

> of what is being split or separated would matter to you.

Ok, I'm not sure I'm completely happy to have my understanding 
denigrated like that, when it could perhaps just be that I understand 
but don't consider certain things important at this point. :) But 
hey...

> You can do this and so ignore what I wrote - but you haven't argued 
> why anyone else should do the same or why I or anyone else should 
> respect your choices.

I didn't ignore what you said - CEE doesnt do tunneling. I'm trying 
to connect my terminology with yours and check whether they match.

> No - you can use "Core-Edge Separation" and "Core-Edge Elimination".  I

I don't really understand those terms tbh. I have had a glance 
through the RRG wiki (the "terminology" page particularly), and your 
http://www.firstpr.com.au/ip/ page, but I don't find an explanation.

The general architecture I'm familiar with that doesn't do tunneling 
instead uses address rewriting of some form. So I tend to call it 
"rewriting".

I'll try get accustomed to your terminology, but perhaps till then 
you can try tolerate mine. :)

> because there's no way the world will alter all its host stacks or 
> (for most CEE architectures) rewrite its applications *completely*

That is a significant concern alright. The only reason such a 
suggestion isn't complete crack is cause:

- The world basically already did this for IPv6, so we know what it
   involves.
- The apps that have updated should, by and large, now be using
   address family abstracting interafaces, making future changes
   slightly less painful.

> single IPv4 address.

Correct, and so provide a basis for a CEE architecture to be built on 
it.

The NAT shortage is the initial pressing problem that gets the basic 
option in the field and deployed widely. Further protocols, offering 
further facilities, such as core-separated multi-homing, can then be 
layered on top.

> This doesn't solve the routing scaling problem in any way.  It is 
> an attempt to cope with a situation where there is such a shortage 
> of IPv4 space that there's not enough to run NAT boxes with 

I think you must have stopped reading that blog post about half way 
through, or you skimmed it :).

> explained your reasoning.

I was proposing something more than just NAT. Specifically, I was 
describing a path of least-resistance toward a "core/edge separated 
internet, using rewriting" (CEE, right?), where the first step on 
that iteration of hacks is "make NAT work for higher number of 
hosts". Further, this proposal remains backward compatible with the 
existing IPv4 network, to a greater degree than other CEE proposals 
at least.

I think perhaps you may not have read the latter parts of that blog 
post. (My writing style is dry, dull and sucks generally, so I can 
understand you'd have fallen asleep a paragraph or two in! ;) ).

>   http://www.ietf.org/mail-archive/web/rrg/current/msg05864.html

The signalling overhead of an end-host<->end-host multihoming 
proposal is a concern. However, you have the same problem with the 
map-encap proposals - the signalling has to be done somewhere. You 
can do it on x hosts, or x/y middle-boxes.

It seems there's a compromise we have to pick between:

- carrying information about peripheral^Wedge networks in the core
   routing protocol

- carrying information about peripheral networks in packets *over*
   the core routing protocol.

To my feeble mind, I don't see any way for protocols alone to 
*reduce* the amount of signalling required, for a given amount of 
information about edge networks.

The only way to reduce that information is to organise the edge 
networks some way, e.g. through an administratively imposed hierarchy 
(aggretation by geography, IX, whatever), or by some clever dynamic 
organisation. I see that as an orthogonal matter to the protocol 
question, and one best left at the door to this WG.

>   http://www.ietf.org/mail-archive/web/rrg/current/msg05865.html

Those graphs are your opinion, rather than based on data :). If 
you're right, then CES will see deployment. We don't need to have a 
speculative argument about it here.

> If you disagree with this, then please write why.

If would move access customers to NAT, and use the reclaimed 
addresses for services that need it. If I were part of a community of 
ISPs I would start looking at cross-organisation reclaiming of 
ineffectively used addresses. A free market in IPv4 addresses may 
well develop to allow organisations to put a price on the cost of 
renumbering Vs the cost of not having public IP space, and so 
facilitate reclaiming. If I were an ISP, I would do all these things 
well before deploying CES.

As I wrote earlier, it is quite plausible that such address space 
recyling will serve the needs of the IPv4 internet for quite a while 
- at least long enough for the NAT problem to become a pressing one 
to solve.

And as I wrote in blog post, solving the NAT problem can provide the 
immediate benefit to get an extension widely deployed that can then 
be used as the basis for a CEE architecture.

Small incremental steps.

> changing all the apps.

Experience with IPv6 suggests otherwise.

It's 15 odd years now, and the inter-network still does not generally 
support IPv6. Despite the lack of IPv6 network service, the end-host 
software *did*, to a significant degree, get upgraded.

Granted, CES doesn't require the fork-lift upgrade that IPv6 does. 
But still, it assumes that ISPs will choose to add relatively complex 
CES control plane - which implies administrative overheads, which 
implies increased staffing costs - over the tried, tested and simple 
NAT box + reclaiming address space.

Anyway, we can't resolve this argument without data. The market will 
provide that in time.

> But increasing the number of hosts a NAT box can handle from a 
> single IPv4 address has nothing to do with the routing scaling 
> problem, or multihoming, portability, inbound TE or mobility.

I respectfully submit you have considered only the initial part of my 
argument :)

I will of course reread yours to see what of it I have missed.

> impractical to use it through the DFZ, or probably with most 

I don't buy that. The same thing applied to IPv6 at one stage, indeed 
it was worse with IPv6 for the DFZ couldn't forward IPv6 /at all/.

IP options being slow would be resolved within about 1 life-cycle of 
routers (at least, their line cards) on the DFZ. I don't have 
anything but anecdotal evidence as to what that is, my best guess is 
circa 5 years.

> communication would be impractical.  Even being slow would be 
> enough to ensure no-one wants it.

They found significant problems with about 15% to 21% of the paths 
they tried. So 80% odd of the 210 paths they tried were fine. Plus we 
need further studies to see how peculiar these loss rates were to the 
set of paths they measured.

It's a problem, sure, but it doesn't quite seem as bad as you've 
stated it. We face similarish connectivity problems trying to recycle 
addresses when address ranges have been listed in bogon space, or 
with low address ranges that historically have not been used and 
which have ended up being used privately in places. E.g. what 
proportion of paths have PMTU problems (which would affect CES - I 
have to read your link still).

There isn't any perfect solution, it seems. This is about choosing 
between problems.

> But until such upgrades are done, there's no way anyone can 
> reliably communicate across the IPv4 DFZ with a new kind of 
> protocol, or with IPv4 packets with option headers.

It would take a similar time-scale to start to get IP extensions 
deployed in end-hosts.

> your proposal would also depend on upgrading applications.  I agree 
> with your assessment of this aspect of your proposal "This is the 
> most obviously crack-full part of it all."!

Indeed.

But again, it can be deployed in stages, and each stage can bring 
some immediate benefits, while enabling the next stage. No single 
stage solves all the problems, but after a number of iterations more 
and more problems can be solved.

I'm not saying this is /better/ than map-encap solutions, just saying 
that if for some reason map-encap doesn't gain traction either that 
there is still a possible engineering path to a rewrite-based n-layer 
routing architecture.

Just saying..

regards,
-- 
Paul Jakma	[email-masked]	Key ID: 64A2FF6A
Fortune:
Having the fewest wants, I am nearest to the gods.
 		-- Socrates

---

From: Robin Whittle <[email-masked]>
Date: Sun, 14 Feb 2010 14:43:14 +1100

Short version:    I think Paul's proposal doesn't help with routing
                  scaling at all, and might only help with the IPv4
                  address space problem if the Earth's population
                  30 billion or more.

                  His proposal is intended to let a NAT box serve
                  more hosts from a single IPv4 address, by way
                  of providing new versions of TCP and UDP (SCTP?)
                  with 32 bit port numbers.

                  Paul states that his proposal would be the basis
                  for a CEE routing scaling solution, but no CEE
                  architecture works with IPv4 - and none work with
                  hosts behind NAT.

                  How can ACLs (Access Control Lists) be implemented
                  with a CEE (Locator/Identifier Separation)
                  architecture?

Hi Paul,

Thanks for your reply, in which you wrote:

> but don't consider certain things important at this point. :) But hey...

I got the impression from your earlier statement:

PJ> So I recognise there are 2 different things, call them CES and
PJ> CEE if you want, but I don't see the utility in argueing one
PJ> that one has a split and the other does not. Anyway, doesn't
PJ> matter (to me ;) ).

that you weren't making any great claims about your level of
understanding - and I believe that if you understood the architectures
better then you would understand why I believe the distinctions between
CES and CEE architectures are important.

The word "separation" or "split" appears in "Core-Edge Separation" and
in the "Locator/Identifier Separation" concepts (which is the basis of
all "Core-Edge Elimination" architectures - but these are completely
different concepts which have often been misunderstood and used too
loosely.  I have made the same mistakes myself.  Two completely
different sets of things are being "split" or "separated", so the fact
that these two concepts both involve pairs of things being split or
separated is of little importance.

> I didn't ignore what you said - CEE doesnt do tunneling. I'm trying to
> connect my terminology with yours and check whether they match.

OK - here is the complete exchange:

PJ >>>>>   We will note that some proposals, in order to be as
PJ >>>>>   transparent and invisible to end-host transport protocols
PJ >>>>>   as possible, use a "map and encap" approach – effectively
PJ >>>>>   tunneling packets over the core of the internet.

RW >>>> These are all CES architectures.  CEE architectures have no need
PJ >>>> for tunneling.

PJ >>> I think I'll stick with "map-and-encap" and "rewriting". I have
PJ >>> to say, I sometimes wish this WG would try adopt labels that are
PJ >>> semi-evocative of their meaning where possible, rather than all
PJ >>> these acronyms. Very confusing! :)

RW >> You can do this and so ignore what I wrote - but you
RW >> haven't argued why anyone else should do the same or why I or
RW >> anyone else should respect your choices.

PJ > I didn't ignore what you said - CEE doesnt do tunneling. I'm
PJ > trying to connect my terminology with yours and check whether
PJ > they match.

You equated "CES" with "map-and-encap" - but "map-and-encap"
(encapsulated tunneling) is not the same thing as tunneling.  Ivip has
two "Modified Header Forwarding" approaches by with ITRs tunnel traffic
packets to ETRs, without encapsulation.  One approach is for IPv4 and
the other for IPv6. Both require upgrades to all DFZ and some other
routers - which in the long-term can be achieved without significant
cost.  Also, Six-One Router, which I think has more in common with the
CES architectures, uses address rewriting.

Only some CEE architectures use "rewriting" - GSE and GLI-Split are two
which do.  ILNP Name Based Sockets and I think quite a few other CEE
architectures do not involve address rewriting.

I never suggested you equate "CES" with "map-and-encap", or "CEE" with
"rewriting" - and you didn't explain why you made these connections.

> I don't really understand those terms tbh. I have had a glance through
> the RRG wiki (the "terminology" page particularly), and your

The above page of mine is just a pointer to some others.  In my message
I suggested you read:

and the closely related:

I had a go at explaining the difference in my message, but it is better
to point you to the full, prior, explanations than to repeat things on
the mailing list.

> I'll try get accustomed to your terminology, but perhaps till then you
> can try tolerate mine. :)

OK!

> That is a significant concern alright. The only reason such a suggestion
> isn't complete crack is cause:
> 

Only a subset of applications work for IPv6 as far as I know - a pretty
small subset, I think.

>   address family abstracting interafaces, making future changes

I agree, but this is only for the subset of applications which have been
upgraded in this manner, and there are many applications which by the
nature of their protocols work much more directly with IPv4 addresses.
I think that many protocols can't directly be made to work with the CEE
arrangement of "Locator/Identifier Separation", since the protocols
themselves assume that the IP address is stable per host, being both an
Identifier and a Locator.

For instance, how can you do ACLs with the "Locator/Identifier
Separation" naming model?  Today, a router or an application can allow
or disallow access and communication with hosts based on straightforward
ranges of IP addresses.  With the "Locator/Identifier Separation" naming
model, IP addresses can't be used for this purpose.  So the routers or
software would need to establish that the incoming packet's purported
Identifier was in fact valid, by looking up the Locators for this
Identifier and checking that the Locator which is in the packet is
within the allowed set.  Only then, could the packet be allowed or not
through a router, or responded to by an application.

In the case of CEE architectures such as Name Based Sockets, where the
FQDN plays the role of Identifier (or perhaps some tricky combination of
FQDN and Locators - I am awaiting guidance from Christian on this) the
Identifier is not numeric, so it gets trickier to group ranges of them
as one can with IP addresses, for the purposes of creating ACLs.

> Correct, and so provide a basis for a CEE architecture to be built on it.

Another distinguishing thing about CEE architectures, as far as I know,
is that they don't work for hosts behind NAT.  Can you explain how a CEE
architecture could work for IPv4 with your extended TCP and UDP etc.
port numbering system?

> option in the field and deployed widely. 

I don't think it is a pressing problem yet - though one day it may be.
Can you show how many IPv4 addresses are currently used for single
customer services, such as DSL, and why this is such a large and growing
number that ISPs will be so unable to obtain address space at a given
price per IPv4 address that it will be more profitable to sell customers
a less useful service behind NAT, and so without each customer being
able to do their choice of port-forwarding?

For mobile hand-held devices, I agree it makes more sense to use NAT to
conserve space, since I think people are less likely to want to run
port-forwarding applications on these hosts, or to run their own NAT box
and private network of hosts.  However, I understand that for many home
and SOHO users, port forwarding, using generally fixed ports, is an
important part of how they use their Internet service today.  Take a
look at this list of application programs which apparently use
port-forwarding:

  http://www.portforward.com/cports.htm

Any such app which requires a fixed port will be difficult or impossible
to use if multiple customers are being a single NAT box, as you are
suggesting.

> Further protocols, offering further facilities, such as
> core-separated multi-homing, can then be layered on top.

How can a customer behind a NAT box be multihomed?

>> This doesn't solve the routing scaling problem in any way.  It is an
>> attempt to cope with a situation where there is such a shortage of
>> IPv4 space that there's not enough to run NAT boxes with whatever
>> limitations they have from the current 16 bit TCP/UDP port numbering
>> system.
> 

I read the whole thing.

> internet, using rewriting" (CEE, right?), 

No!!!

I am trying to correct your too-loose use of terminology - in part so
other people don't think that this usage is meaningful or useful - and
you continue to use it so loosely as to be confusing and/or meaningless.

"Core-Edge Separation" (CES) involves separating a subset of the global
unicast address space to be "edge", leaving the rest as "core".  It is a
separation of addresses into two subsets.

CES architectures generally do not involve rewriting.  Only Six/One
Router uses rewriting, and while I think it most resembles a CES
architecture, there is an important difference from the other CES
architectures - the fact that it requires a split horizon DNS to work
with non-upgraded hosts.

CES, "rewriting" and CEE are three completely separate concepts, so your
sentence above makes no sense.

You haven't explained how your proposal to extend the ability of IPv4
NAT boxes to handle more hosts from a single IPv4 address could be used
in any way (with CEE, CES or any other approaches) to solve the routing
scaling problem.

> where the first step on that
> iteration of hacks is "make NAT work for higher number of hosts".
> Further, this proposal remains backward compatible with the existing
> IPv4 network, to a greater degree than other CEE proposals at least.

Your proposal involves using IPv4 header options.  These are
theoretically compatible with IPv4, but they can't be relied upon in
across the DFZ.  If all routers were updated, then  your system could
proceed, with upgraded host stacks and applications which were backwards
compatible with ordinary IPv4.  On their own, I agree, your changes
would be more backwards compatible with IPv4 than any CEE architecture.
 No CEE proposal can be used for scalable routing with IPv4 because each
multihomed end-user network requires at least double the address space
it actually uses.  (This is not true of CES architectures, which use
address space 100% efficiently, except Six/One Router which I am
thinking is much less like a CES architecture than all the others - a
multihomed network with two ISPs needs three times the global unicast
space.)

> understand you'd have fallen asleep a paragraph or two in! ;) ).

I read it and I have just reread it.

You haven't explained how a host behind NAT could be multihomed, or how
any CES or CEE architecture could be used with your proposal to provide
portability, multihoming or inbound TE for hosts behind NAT.

> The signalling overhead of an end-host<->end-host multihoming proposal
> is a concern. However, you have the same problem with the map-encap
> proposals - the signalling has to be done somewhere. You can do it on x
> hosts, or x/y middle-boxes.

My message 05864 includes arguments why it is much better to do it on
middle-boxes such as ITRs.  CEE (Locator/Identifier Separation) involves
*all* hosts doing their own routing and addressing management, though
some architectures do some of the work in routers.   This is
particularly a burden for mobile hosts on frequently slow, expensive and
less than 100% reliable 3G etc. wireless links.

> To my feeble mind, I don't see any way for protocols alone to *reduce*
> the amount of signalling required, for a given amount of information
> about edge networks.

Sure - the idea of CES is to build a streamlined new mechanism for this.

CEE offloads the work onto hosts, or sometimes hosts helped by
address-rewriting routers.

I think CES is superior to CEE for reasons including:

   Only CES can support IPv4.

   CES allows the continued use of today's two-layer IP naming model,
   which some argue is "architecturally impure" - but which I think
   is better and faster than if we all had to rely on hosts doing their
   own management of routing and addressing (perhaps with some help from
   routes) via the CEE arrangement of "Locator/Identifier Separation".

   CES brings immediate benefits to adopters and progressive routing
   scaling benefits, while CEE only provides both sets of benefits once
   all networks and hosts have been upgraded.

   CES does not require host stack or application changes - while
   most CEE architectures (GSE and GLI-Split are the exceptions)
   require stack and application changes to all hosts in order in order
   for all communications to have the portability, multihoming and
   inbound TE benefits.

> The only way to reduce that information is to organise the edge networks
> some way, e.g. through an administratively imposed hierarchy

All the current proposals involve some kind of new way of organising
routing and addressing for the end-user networks which require
portability, multihoming and inbound TE.  This is because no current
proposal aims to fix the scaling problem by re-engineering BGP or the
DFZ in general to cope with ten million or more PI prefixes.

> Those graphs are your opinion, rather than based on data :). If you're
> right, then CES will see deployment. We don't need to have a speculative
> argument about it here.

None of us have data about the future.

These are my opinions - but I explain the reasons I believe these things
so other people can understand why I think this, and so that anyone who
disagrees can point out where they think my reasoning is mistaken.

You haven't pointed out why you think my "benefit vs. adoption" curves
are incorrect.

> If would move access customers to NAT, and use the reclaimed addresses
> for services that need it. If I were part of a community of ISPs I would
> start looking at cross-organisation reclaiming of ineffectively used
> addresses. A free market in IPv4 addresses may well develop to allow
> organisations to put a price on the cost of renumbering Vs the cost of
> not having public IP space, and so facilitate reclaiming. 

I agree that there will be a market for IPv4 space.  IPv4 space will be
increasingly valuable and so there are pressures to use it more
efficiently.  There is bound to be money changing hands in this process
- with the payers encouraging the address suppliers to use less space
overall than they did, either by actually using less addresses, and/or
by compacting their used addresses into smaller chunks of space, which
they retain, so they can sell, return or whatever some of the space they
currently advertise.

> If I were an ISP, I would do all these things well before deploying CES.

Core-Edge Separation is primarily something which end-user networks want
to adopt - because it will be a less expensive, and most likely the
only, way they can get portability, multihoming and inbound TE.  ISPs
will probably only want to adopt CES because of their customers.  The
ISP's own hosts don't need CES, because they are already on the ISP's
own prefixes, which are not part of the routing scaling problem.

> recyling will serve the needs of the IPv4 internet for quite a while -
> at least long enough for the NAT problem to become a pressing one to solve.

I think the coming era of IPv4 address compaction will help ISPs to
obtain space so they can continue to use it one IPv4 per DSL (etc.)
customer.  I think CES will enable more and more end-user networks to
use IPv4 space more efficiently, since they can get exactly the number
of addresses they need (1, 2, 3, 4, 5 etc. with Ivip, or 1, 2, 4, 8 etc.
with the other CES architectures) rather than having to obtain space in
chunks of 256, 512, 1024 IPv4 addresses as they do now.

I guess when there are 1.5 or 2 billion DSL etc. customers each getting
their own IPv4 address, like they do today, then things will get very
tight and there will be an increased desire to put at least some of
these customers behind NAT.  Many of them will be OK with this - if they
are not using port-forwarding applications.

So I can envisage ISPs offering two classes of service - behind NAT or
not.

Your proposal only helps to the extent that it alters the number of
hosts which can be served from a single IPv4 address.

There's no fixed number for this, but let's say it is 50 hosts, and the
average customer has 5 hosts at their home, office or whatever. (Lets
leave aside whether they run their own NAT box in the home or office,
and so are doing double NAT, or whether each host gets its own address
from the ISP's NAT box.)

So with NAT as it currently exists, for a subset of non-mobile customers
and probably many or most mobile customers, they can already be served
with NAT and so 10 to 50 or whatever customers can use a single IPv4
address.

I think your proposal would only help once we ran out of IPv4 addresses
even when using many of them to serve 10 to 50 customers per IP address.
 That is not going to happen as long as we have less then 10 billion
people on the planet.  So as far as I can see, your proposal wouldn't
help unless we had 5 or 10 times the current population, all using IPv4.
  I think your proposal is about helping to get more use from a limited
number of IPv4 addresses.  I don't think it helps with the routing
scaling problem, since I don't think it provides any new approach to
portability, multihoming or inbound TE.

> immediate benefit to get an extension widely deployed that can then be
> used as the basis for a CEE architecture.

There's no CEE architecture which works with IPv4 - or with hosts behind
NAT - so I don't see how your proposal could be used as an improved
basis for CEE.

> software *did*, to a significant degree, get upgraded.

Here is what is required to make IPv6 usable for most people to the
extent that they don't need IPv4 any more.  Until then - as long as IPv4
is the only way to reach all the hosts they want to reach - few people
are going to go to much trouble to use IPv6:

  1 - There needs to be an IPv6 network and IPv6 services to customers.

  2 - Customer's CPE and host stacks need to cope with IPv6 in a
      dual-stack or perhaps IPv6-only setting without any fuss.

  3 - Their applications need to be upgraded to IPv6.

  4 - The hosts they want to communicate with must be on IPv6.

1 and 2 have only partially happened.  3 has occurred partially for some
major apps, but the vast majority of applications people use are not
ready for IPv6.

4 has hardly occurred at all.  Ben Stasiewicz's research last year:

http://listserver.internetnz.net.nz/pipermail/ipv6-techsig/2009-October/000708.html

indicates that less than 0.1% of major websites have an IPv6 server:

     The list of IPv6 web servers was derived from the Alexa Top 1
     Million Websites list [6]. Of these 1 million domains, only 627 had
     globally routable IPv6 addresses that could be connected to on port
     80 (sad eh?).

The problems faced by IPv6 are closely analogous to those faced by CEE
proposals, and yours - the benefits do not accrue substantially to any
adopters until almost everyone has adopted.  Consequently, very few
people try to adopt it - and the situation persists indefinitely.

CES architectures do not have these problems, as I explained and
provided graphs for in (msg05865).

> Granted, CES doesn't require the fork-lift upgrade that IPv6 does. But
> still, it assumes that ISPs will choose to add relatively complex CES
> control plane - which implies administrative overheads, which implies
> increased staffing costs - over the tried, tested and simple NAT box +
> reclaiming address space.

They can already use NAT to sell a second-rate behind-NAT IPv4 service.
As long as people are prepared to pay more for the current service
and/or their helpdesk costs of explaining to people on the NATed service
why they can't do things which work fine for their next-door neighbour
and/or the cost of buying more space is less than the cost of buying and
running the NAT boxes, while charging less money per customer, then
there won't be much uptake of behind-NAT IPv4 services.

I think there's no need for your major upgrade to host stacks and
applications just to allow a NAT box to serve more hosts than is
currently possible.

> Anyway, we can't resolve this argument without data. The market will

Our job is to discuss things and choose the best approach to develop -
so no-one in the future wastes time and money pursuing something which
we can foresee now is not going to work.

I have argued why I believe no CEE proposal is suitable.  That leaves
CES proposals.  TIDR doesn't solve the scaling problem and IRON-RANGER
has not been clearly enough specified for anyone to estimate how well it
would scale.  LISP has a bunch of problems:

  http://www.ietf.org/mail-archive/web/rrg/current/msg05728.html

and that leaves my Ivip proposal:

The existing critique of Ivip gives it a pretty good rap:

 http://tools.ietf.org/html/draft-irtf-rrg-recommendation-04#section-4.2

and it would be really good for people to take a greater interest in
criticising Ivip if they think it is not the best solution to the
routing scaling problem.

> I will of course reread yours to see what of it I have missed.

OK - but you haven't explained how your proposal helps on its own, or
how it could be used with any other solution.

> I don't buy that. The same thing applied to IPv6 at one stage, indeed it
> was worse with IPv6 for the DFZ couldn't forward IPv6 /at all/.

Over a long period of time, the DFZ routers could be upgraded with
little cost, as more and more of them have entirely firmware-controlled
FIB functions, and those which don't are retired.

I don't rule this out as a possibility - but if we were going to upgrade
all DFZ routers for new types of IPv4 option headers, or new types of
headers, then I think there are more important things to do regarding
routing scaling than your proposal which just enables a NAT box to
handle more hosts per IPv4 address, and then only to the extent that the
hosts have their stacks and applications upgraded.

> routers (at least, their line cards) on the DFZ. I don't have anything
> but anecdotal evidence as to what that is, my best guess is circa 5 years.

I don't have any better information.  To the extent it is true, I would
argue for upgrades to support something which I consider more directly
useful for routing scaling than your current proposal.

>> communication would be impractical.  Even being slow would be enough
>> to ensure no-one wants it.
> 
> They found significant problems with about 15% to 21% of the paths they
> tried. So 80% odd of the 210 paths they tried were fine. Plus we need
> further studies to see how peculiar these loss rates were to the set of
> paths they measured.
> 
> It's a problem, sure, but it doesn't quite seem as bad as you've stated
> it. We face similarish connectivity problems trying to recycle addresses
> when address ranges have been listed in bogon space, or with low address
> ranges that historically have not been used and which have ended up
> being used privately in places. E.g. what proportion of paths have PMTU
> problems (which would affect CES - I have to read your link still).

The techniques Fred and I developed are different, but both can cope
with there being PMTU problems between ITRs and ETRs which do not result
in PTBs being sent to the ITR.  It is complex, and ideally we wouldn't
have to go to this trouble, but both approaches could work, I think.
Neither can cope with PTBs sent by the ITR being dropped or ignored by
the sending host - but that would be a problem caused by the sending
host or the sending host's network which needs to be fixed there, since
there is no other solution.  As long as the ITR is in the sending host's
(which is not necessarily the case, at least initially, with LISP or
Ivip) network, hopefully this will not much of a problem.

> There isn't any perfect solution, it seems. This is about choosing

OK.

> It would take a similar time-scale to start to get IP extensions

Unfortunately the benefit vs adoption curves for your proposal resemble
those of CEE - substantial benefits only when everyone has adopted it -
so I can't see how you would ever get to this stage, especially since
you need to convince all application programmers to substantially
rewrite their software and change their protocols.  This is never going
to happen, I am sure.

>> with your assessment of this aspect of your proposal "This is the most
> But again, it can be deployed in stages, and each stage can bring some
> immediate benefits, while enabling the next stage. No single stage
> solves all the problems, but after a number of iterations more and more
> problems can be solved.

But please consult the curves for CEE, since the curves for your
proposal would be at least as uninviting:

            ^
         B  |        *      Core-Edge Elimination (CEE)
         e  |       .*
         n  |      ..*      .  Proportion of communications
         e  |     ...*         with the benefits.
         f  |    ....*
         i  |   .....*      *  Actual total benefits generally
         t  |  ......*         only arise when all, or almost
            | .......*         all, communications have the
            |........*         benefits.
            0--------->
              Effort ~= level of adoption

            ^
         B  |        *      Core-Edge Elimination (CEE)
         e  |        *      Routing scaling benefits.
         n  |        *
         e  |        *
         f  |        *
         i  |        *
         t  |        *
            |        *
            |        *
            0--------->
              Effort ~= level of adoption

> there is still a possible engineering path to a rewrite-based n-layer

OK - and I am arguing in detail multiple reasons why a good CES
architecture will be superior to your approach or to any CEE architecture.

Of course, if Ivip is introduced and fails miserably . . . then maybe
you will be proven to be right.  But you haven't explained why it would
do so.

 - Robin

---

From: Paul Jakma <[email-masked]>
Date: Sun, 14 Feb 2010 11:47:09 +0000 (GMT)

>                  architecture works with IPv4 - and none work with

I don't think that's a fair characterisation at all of what I was 
suggesting. Indeed, it's completely at odds with what I was 
suggesting in the blog entry (I may have started with speculating 
about transport protocol level hacks, but that's not where my 
reasoning ended up). Also, it seems you're using NAT like "NAT, as 
NAT is today" - rather than "NAT, in the general 'rewrite' sense" or 
"NAT, but 2-way", which is how it ends up in what I was considering.

The suggestion is, in a nutshell:

1. Deploy an IP option that effectively adds a secondary ID space to
    the packet

Step 1 can happen because ISPs and consumers will perceive a growing 
problem, due to pressures on port space with NAT (NAT being the only 
thing people really care about in the short-term, being the 
assumption). Presume about 5 years to get this deployed and iron out 
whatever problems with core routers (i.e. pretty much same problem 
facing IPv6..).

2. Someone recognises that with a new address family you could allow
    2-way connectivity between these secondary ID spaces.

3. Someone takes approaches from LISP and Shim protocols and adds
    protocols to make the seconday<->middle-box mapping dynamic.

It /starts/ with solving a problem /for/ NAT. It ends up with a 
system that implements a "locator scope change" boundary between the 
core internet and a peripheral address space.

Further, I don't claim this is a well-specified system or a good one.

I'm trying to illustrate what could happen if none of the 
well-engineered proposals from the IETF, IRTF, etc end up being 
adopted, and trying to think what could happen if people instead go 
for 'cheap hacks' that don't particularly try to generally fix 
routing on the internet..

What I'm saying is that even the cheap hacks (even ones to make 
something as yucky as NAT work better) could potentially form a basis 
for something that could, in time (a long time perhaps), evolve in to 
a "fix routing on the internet" system.

Take it as you will.

regards,
-- 
Paul Jakma	[email-masked]	Key ID: 64A2FF6A
Fortune:
Harrisberger's Fourth Law of the Lab:
 	Experience is directly proportional to the amount of equipment ruined.

---

From: Paul Jakma <[email-masked]>
Date: Sun, 14 Feb 2010 13:35:36 +0000 (GMT)

On Sun, 14 Feb 2010, Robin Whittle wrote:

> proceed,

Not all - some, likely a /very/ few (see below). That study shows 
that most paths are fine (Exactly how representative that study is, 
who knows - more samples might show the problem is greater or lesser 
in extent). However, aren't you advocating upgrading /all/ routers? 
:)

> The techniques Fred and I developed are different, but both can 
> cope with there being PMTU problems between ITRs and ETRs which do 
> not result in PTBs being sent to the ITR.

This is an interesting point. Which problem is worse:

1. some DFZ routers having problems with IP options
2. some edge networks doing extremely dumb things and causing PMTU
    problems

?

Personally, I'll go for 1. Cause I know those routers are 
well-managed by generally very clueful people. The anecdotal evidence 
available to me suggests PMTU problems are caused by certain strange 
ideas about the importance of blocking everything but (tcp and dst 
port 80) having become ingrained in well-meaning but less than 
clueful edge network administrators (companies selling 
firewall/network security products bear a lot of responsibility for 
instilling that idea).

I see you have a protocol to deal with 2. Am I correct in thinking 
its Inter-TR, and exists to deal with PMTU problems between the 
xTRs? If so, then doesn't it leave the problem of where the PMTU hole 
is near the remote, non CES, edge network. e.g.:

                                 remote network
host--ETR---ITR-------------------[FW----host]

Host is "CESed". "Remote network" unfortunately is behind a PMTU hole 
- i.e. FW drops incoming PTB ICMPs, so the remote-network host can 
never do PMTU properly. Hence many consumer routers clamp TCP MSS to 
local MTU - 40.

The problem is now there's another tunnel in the way, and it's eaten 
20 from the PMTU, which 'host' (or its local access router) doesn't 
know about at all, and so can't automatically adjust its MTU for, and 
that 20 is larger than the 8 bytes of PPPoE overhead that routers 
usually are adjusting for.

So this creates another PMTU problem when remote-host tries sending a 
packet to host and never receives the PTB from the ITR.

In theory, this problem seems like it should be much easier to solve 
than fixing core routers to be nicer to things like IP options. 
However, I have my doubts :).

I notice you also have a draft that gets around this by rewriting the 
IP header. Though, that requires upgrading the vast majority of 
routers on the internet.

So it seems the problems we can choose between come down to:

1. Fixing the majority of those routers that are causing the problems
    for IP options on 15 to 20% of paths.

    If each tested path was completely independent and each path was
    10 hops on average, and the problems on each path were down to one
    router, then we're talking about about something like 1% of
    routers that need to be fixed. If average was 16 hops, then we're
    talking 0.5%. It'd be interesting to see field studies to pin this
    number down.

2. Upgrading all end-host software

3. Fixing the dumb edge networks that are causing the PMTU problems.

4. Upgrading all DFZ routers to handle a modified IP header.

It's hard to think 2 is achievable. It boils down to try re-educating 
people who don't really have much interest anyway in all this. We've 
had years and years of this problem and awareness and yet it's still 
there, and there are still widely used security products shipping 
that block ICMP by default and do all kinds of stupid things, IME. 
Also, the workaround for 2 is for end-hosts to clamp down their MSS 
even further. So adding tunneling to the internet has a risk of an 
ever escalating "race to 0 MTU" (PPPoE, VPNs, CES, etc - all of which 
can stack!).

So then it boils down to 1, 3 and 4. I think 1 is hard, but 
achievable. If 1 is considered so hard as to be unachieavable, then 
surely 4 must be impossible?

For me, the above order (1,2,4,3) corresponds to least-hardest to 
most-hardest. I.e. if you have to choose between which of those 
problems your solution must overcome, then I'd choose 1 obviously.

Obviously the choice may be more complex than that. E.g. it could be 
"1 AND 2" (e.g. having end-hosts add an IP option in some kind of 
CEE, or CEEish solution) versus "3 or 4" (e.g. CES). Then it's not so 
clear which is worse.

NB: I have finally found your terminology draft, so hopefully I can 
abuse the terminology less.

regards,
-- 
Paul Jakma	[email-masked]	Key ID: 64A2FF6A
Fortune:
"I will make no bargains with terrorist hardware."
-- Peter da Silva

---

From: Robin Whittle <[email-masked]>
Date: Fri, 19 Feb 2010 11:11:20 +1100

Hi Paul,

I hope to return to our discussions later, but at present my top
priority is to read, understand and write about all the proposals.

Please take a look at Patrick Frejborg's 05 revision of his hIPv4 proposal:

  http://tools.ietf.org/html/draft-frejborg-hipv4-05

and the discussions he and I had on the list in recent days.  I think he
adds 32 bits to the left, conceptually, of the IPv4 space, giving large
numbers of separate IPv4 like address ranges.  I suggested (msg06050)
something which might be more in line with what you are aiming for,
adding bits on the right:

  I would also want to explore options for giving multiple new
  addresses behind each existing IPv4 address.

    - Robin