Subject: [rrg] Design goals: Scalability not well defined and other problems

From: Robin Whittle <[email-masked]>
Date: Sun, 21 Nov 2010 03:20:54 +1100

Briefly:    No recognition of IPv4/v6 differences.

            Insufficient detail in defining "scalability" - it has
            technical and commercial aspects.

            3.4 is a horror.  It contains an outdated and unclear
            (and so now completely incorrect) attempt to delineate CES
            architectures (now IRON, Ivip and LISP).  It says that
            Loc/ID Separation (CEE) is both "desired" and "required".
            It says that if a CES architecture is adopted, it must be
            compatible with Loc/ID Separation (CEE) - but there's no
            reason for adopting both and no obvious way they could be
            compatible.

            3.5 on mobility should be improved.

            3.6 "Simplified renumbering" should be replaced by a
            section on portability - of IP address space (for CES) or
            of Identifier space (for CEE = Loc/ID Separation).

            3.7's mention of ideally supporting QoS, implicitly across
            the DFZ, seems unrealistic, since no-one has a clue how to
            do this and since QoS arrangements in a path of routers
            (if they could somehow be established) would become
            irrelevant in the likely event the DFZ paths change.

            3.10 on "deployability" completely avoids the constraints
            we must work within, due to the need for widespread
            voluntary adoption.

            I suggest further sections:
            	
               3.11  IPv4 address space utilization

               3.12  Support for migration to other architectures

               3.13  Solutions for IPv4 and IPv6

               3.14  Potential scaling difficulties of Core-Edge
                     Separation architectures

               3.15  Potential scaling difficulties of Core-Edge
                     Elimination (Loc/ID separation)
                     architectures

Some of the criticism below was first mentioned in 2007:

  http://www.ietf.org/mail-archive/web/rrg/current/msg00203.html
  http://www.ietf.org/mail-archive/web/rrg/current/msg00733.html

The core of http://tools.ietf.org/html/draft-irtf-rrg-design-goals-04
is sections 3.1 to 3.10 - a puny 1187 words.  There's no way a
document so lacking in detail can be a suitable guide for how to
upgrade the IPv4 and IPv6 Internets to cope with the challenges of the
next few decades.

rrg-design-goals-04 almost completely ignores the differences between
IPv4 and IPv6.  The IPv4 and IPv6 Internets have substantially
different characteristics, adoption levels and problems - so the best
solution will likely involve different arrangements, rather than the
one set of principles applied equally to both systems.  IPv6 has a
vast address space and IPv4 doesn't.  IPv6 hasn't been widely
deployed, and may never be - so it can be subject to more changes than
IPv4, which must be made more scalable since it is and will continue
to be relied upon by by essentially all users.

There are different potential techniques for ITR->ETR tunneling in
IPv4 compared to IPv6.  Six weeks ago, Sampo Syreeni suggested to me
in a private message a zero overhead tunneling method which would work
beautifully for IPv6.  It involves no changes to routers.  There are
just ordinary packet headers - and no encapsulation at all.  This will
work in IPv6 and IPv4, but is impractical for IPv4 since it really
chews address space.  This prompted me to devise a different approach
for zero overhead tunneling in IPv4, which works on different
principles which are only available in IPv4.  I plan to write more
about these soon.  I intend to update Ivip to use these techniques
from the outset, rather than encapsulation with eventual migration to
"Modified Header Forwarding".  Both these tunneling arrangements - to
be called ZOTv6 and ZOTv6 - require some elaboration in the mapping
system and the management of ETRs, but I think this is well worth the
trouble to get rid of the encapsulation overhead.  There are still
some Path MTU Discovery management things in ZOTv4 which the ITRs and
ETRs need to manage - but again, I think it is well worth the trouble.

It is easy to identify the scaling problems which exist today, and set
goals which will solve these, or avoid them getting worse.  To achieve
this, one approach is to specify that hosts must do more work so they
can maintain application sessions as each host moves arbitrarily from
one IP address to another.  This is achieved with the Locator /
Identifier Separation approach: GLI-Split, ILNP, Name-Based Sockets or
RANGI.

The real challenge is to set goals which, if achieved, will avoid the
new architecture generating new scaling problems, while also achieving
all our current goals for service improvement - ideally in the most
elegant, adoptable and open-ended fashion.

I think it would be tortuous to try to define scalability in abstract
terms which avoid the actual scalability problems we can foresee.  So
I think it would be better to identify the current and foreseeable
scalability problems of the current architectures (IPv4 and IPv6) and
then identify the foreseeable scalability problems in the different
classes of architectural enhancements we are considering.

I think there are four broad classes of enhancement which have been
contemplated:

  1 - Souping up BGP routers and/or the BGP protocol.

  2 - Replacing the BGP system with something new, such as based on
      geographic routing.

  3 - Locator / Identifier Separation (Core-Edge Elimination = CEE):
      GLI-Split, ILNP, Name-Based Sockets and RANGI.

  4 - Core Edge Separation (CES): IRON, Ivip and LISP.

There's no sign that the first two could achieve the basic goals,
including especially global mobility for billions of devices - so I
think there's no need to consider them further.

CEE and CES architectures assume the continuation of the current
interdomain routing system, with BGP and the Default-Free Zone.  Both
could, in principle, solve the scaling problems which currently exist,
while providing mass-scale global mobility, multihoming and inbound
TE.  (How well, and at what cost, is a matter of debate.)

Neither CEE or CES make new demands on the BGP system.  CEE achieves
its benefits by making all hosts do more work.  CES leaves host
protocols and workloads as they are, and adds new devices (ITRs, ETRs,
mapping servers etc.) to the network.

So the debate over CEE and CES, and between particular proposed
architectures of these kinds, involves questions of their costs,
performance and adoptability - and most importantly the inherent
scaling benefits or problems of the new mechanisms which each proposal
introduces.  rrg-design-goals-04 does not adequately consider this
last matter.

I think the document should contemplate not just the scaling
difficulties of the current system, but the potential scaling
difficulties of the two classes of solution which are most seriously
being considered: CES and Loc/ID Separation (CEE) architectures.

There are at least two aspects scalability - technical and commercial:

  1 - Can the proposed mechanism actually be done, with the
      number of hosts, packets, users etc. with available
      technology - in a manner which is sufficiently reliable,
      secure and affordable?

      This question involves anticipating the highest possible
      demands on the system, such as 20 years from now - and
      considering the likely capabilities of hosts, routers and
      the Net in general at that time.

  2 - Can the proposed mechanism be installed and operated at the
      highest anticipated levels of demand, in a manner in which
      those paying for the mechanisms are either the ones who
      directly or indirectly benefit from their expenditure, or
      who are happy for their expenditure to benefit others.

      This could be done by the new mechanisms imposing no costs
      on anyone else.  Alternatively, it could be done by the costs
      on other parties being paid directly or indirectly by those who
      derive the benefits.

For instance, we could probably make BGP system work OK with one or
two million prefixes in the DFZ, but it would involve ISPs spending a
lot of money.  Most of the additional prefixes would benefit only a
small subset of end-user networks (EUNs) - since it is presumably
these networks who have added most of the prefixes as their own PI
prefixes.  Yet these small networks do not contribute to the cost of
upgrading the DFZ routers.  This passes the technical scalability test
(with some concerns about stability and convergence time) but fails
the commercial scalability test.

Section 3.1 touches on the commercial aspects of scalability:

    "undue cost burdens on network participants that do
     not necessarily get value from the increases in
     the routing table size."

but I think it would be better to provide a more explicit definition
of "scalability", as I tried to do above.

This text:

    "It is strongly desired to make the routing subsystem
     scale independently from the growth of the Internet
     user population."

has a number of shortcomings.  Firstly, our concern is not just the
growth in the user population, but the need for them to use the Net in
new ways - including global mobility for hand-held devices and
computers etc., for multihoming of far more EUNs than can currently
get multihoming and for new types and volumes of communications.

Secondly, is "the routing subsystem" the existing interdomain routing
system (BGP routers in the DFZ) or does it include potential additions
to the total routing system, such as with ITRs, ETRs etc.?

I am not sure what the next sentence refers to: "alternative routes to
support faster convergence,".

Section 3.2 looks good, but I think the challenge of traffic
engineering is limited to a multihomed EUN's desire to get certain
subsets of traffic arriving via particular ISPs.  This is "inbound"
TE.  There's no scaling problem with outbound TE, since the network
can always put the packets out any ISP link it likes (assuming the ISP
accepts packets with these source addresses) - without any cost or
technical impact on the rest of the Net.

I think some EUNs will want to do inbound TE on the basis of traffic
type, such as VoIP vs. HTTP etc.  This is different from wanting all
the packets from remote network X to arrive via ISP A.  So the inbound
TE needs and desires of EUNs are potentially quite complex.  A good
solution will enable these needs to be met, by those EUNs which want
to go to the trouble, in a way which is technically feasible and
commercially "scalable" - that is, the BGP system and the other
networks should not be burdened in a way which the recipient EUN is
not paying them for.

Section 3.3 looks fine to me.

Section 3.4 has many problems.  I think it reflects the well known
preference of Tony Li and of the proponents of Loc/ID Separation: ILNP
etc. - for not adding anything to the routing system and for solving
the scalability problems by changing host protocols so that all hosts
do extra work.

#    3.4. Decoupling location and identification
#
#       Numerous sources have noted that an IPv4 address embodies
#       both host attachment point information and identification
#       information.  [IEN1]

The reference is from 1977.  More citations are required to support
the claim of "numerous".

Just because there are numerous complaints about the current
arrangement of the IP address functioning as both host Identifier and
Locator, doesn't mean these concerns are reason to change these
arrangements.

#       This overloading has caused numerous semantic collisions
#       that have limited the flexibility of the Internet
#       architecture.

It also enables Internet communications to proceed on a highly
efficient basis, with minimal delays, compared to what would be the
case with Loc/ID Separation:

   "Overloading" of Loc & ID functions is good for hosts and should
   be maintained  RW 2010-06-22
   http://www.ietf.org/mail-archive/web/rrg/current/msg07017.html

The current arrangement involves the routing system taking
responsibility for only delivering a packet to a host which has the
Identifier specified in the destination field.

In Loc/ID Separation, the routing system can't do this - so it becomes
the sending host's responsibility to ensure this occurs in all the
circumstances where it is required, which includes many ordinary
application protocols.  This involves more state, more work, more or
longer packets and extra delays.

#       Therefore, it is desired that a solution separate the host
#       location information namespace from the identification
#       namespace.

I absolutely oppose this goal.

While Loc/ID Separation is a method of solving the current DFZ scaling
problems, the imposition of extra work on hosts which is required for
Loc/Id Separation is entirely undesirable and will not be voluntarily
adopted.   There typically needs to be an extra mapping lookup by the
respondent host before it can send a reply to the initiator.  The
extra burden of state, longer and/or more numerous packets, and
delays, is a terrible scaling problem in itself - and is even worse
for mobile hosts.  It is impractical for mobility, since each host has
to tell all its correspondent hosts as soon as it gets a new Locator -
which can occur as frequently as every few seconds.  It has to do this
securely and reliably.  It can't do this fast enough to support VoIP
sessions.

Loc/ID Separation has so many scalability problems that I do not
support it as a desirable goal in itself.   It is a potential solution
to current scaling problems, but when properly evaluated it is
inherently unsuitable for adoption for a number of reasons.

A further difficulty with this section is that the proponents of LISP
tend to state that their architecture is an instance of Loc/ID
Separation.  It is not.  Loc/ID Separation means the hosts have an
Identifier which remains stable no matter which of potentially
multiple Locators are used.  This is a host-based system, with new
host protocols.

LISP doesn't affect host protocols or behavior.  It doesn't provide
hosts with Locators.  The host's IP address remains as its Identifier
and Locator within all networks, since (with LISP's Proxy Tunnel
Routers) a packet can be sent to the host from anywhere by setting the
destination field to that IP address.

LISP - like Ivip and IRON - involves a new set of devices (ITRs, ETRs
etc. or their equivalents in IRON) which tunnel packets across the
DFZ, but the destination fields of those tunneled packets do not
contain host Locators - they contain the IP addresses of the chosen ETR.

Because some or many LISP folk insist their architecture is an
instance of Loc/ID Separation, I am concerned that the above sentence
might be interpreted as applying to LISP and potentially Ivip (and I
guess IRON), which shares with LISP the major architectural principle
of ITRs, ETRs and tunneling across the DFZ, with no change to host
protocols or responsibilities.

So I think this section is not just wrong, but is likely to be
misleading / confusing for readers.

The second part of this section could be interpreted as excluding LISP
from the definition of Loc/ID Separation - however, without
references, this is not at all clear.  I find this section hard to
interpret, and I think it would be a source of confusion to most readers:

#       Caution must be taken here to clearly distinguish the
#       decoupling of host location and identification information,

This last line means Loc/ID Separation - CEE - in the form of
GLI-Split, ILNP, Name-Based Sockets and RANGI.

#       and the decoupling of end-site addresses from globally
#       routable prefixes; the latter has been proposed as one of
#       the approaches to a scalable routing architecture.

I think this is intended to mean CES: IRON, Ivip and LISP.  This would
have made some sense when this sentence was first written, in April
2007, which was before LISP had PTRs (Proxy Tunnel Routers - the same
as Ivip's Default ITRs in the DFZ - were introduced in December 2007).
 Ivip's DITRs (by the name "anycast ITRs in the core") were introduced
on 2007-06-15 and were intended to be used by LISP too:

  http://www.ietf.org/mail-archive/web/ram/current/msg01518.html

The above sentence and the whole of the current section 3.4 is
unchanged from draft 00 of 2007-04-11 and 01 of 2007-07-08.

In LISP and Ivip, the end-site addresses are covered by globally
routable prefixes, its just that these are single prefixes which cover
a large number of smaller (less IP addresses, more mask bits) prefixes
of many EUNs.  These have no name in LISP, but are known as MABs
(Mapped Address Blocks) in Ivip.  Routing scalability is achieved by
the one MAB replacing hundreds, thousands or hundreds of thousands of
separately advertised BGP prefixes for each EUN.

Because I know the history of this field, I understand the meaning
which was intended for this sentence.  However, the sentence does not
have this meaning now, since there is no scalable routing proposal in
which EUN's hosts addresses are not covered by a globally routable
prefix in the DFZ.

#       Solutions to both problems, i.e. (1) the decoupling of
#       host location and identification information and (2) a
#       scalable global routing system (whose solution may, or
#       may not, depend on the second decoupling) are required
#       and it is required that their solutions are compatible
#       with each other.

There's a mismatch between the "desired" goal of Loc/ID Separation
mentioned earlier in this paragraph and the "required" of this sentence.

This is my understanding of the intended meaning of this paragraph:

  1 - Loc/ID Separation (CEE) is not to be confused with CES (IRON,
      Ivip and LISP) approach of reducing the number of prefixes in
      the DFZ by making some of the remaining prefixes cover large
      numbers of EUN prefixes, while using ITRs and ETRs to tunnel
      packets to the ETRs currently used by each EUN.

  2 - Loc/ID Separation is "desired" (para 1) or "required" (para 2).
      (This is contradictory.)

  3 - CES (primarily LISP, in April 2007 - but also TIDR, which was
      not so widely known) is one of the types of architecture which
      have been proposed as a solution for scalable routing.

  4 - A solution for scalable routing is required, as is Loc/ID
      Separation (CEE), and the scalable routing solution must be
      compatible with Loc/ID Separation (CEE) - so if the scalable
      routing solution is based on CES, then it must be compatible
      with CEE.

However, the attempt to define CES is difficult to interpret for those
who know the history, and is technically invalid now, since no CES
architecture matches the description:  "decoupling of end-site
addresses from globally routable prefixes".

There is no way of combining CES (IRON, Ivip or LISP) with CEE
(GLI-Split, ILNP, Name-Based Sockets and RANGI).  Even if there was,
what would be the point, since each one is intended to be a complete
solution to the routing scaling problem, and since they work on
completely different principles?

The requirement of adopting CEE is plain wrong - and precludes the
adoption of IRON, Ivip or LISP.

IRON, Ivip and LISP are the only architectures which have a chance of
being voluntarily adopted on a wide enough basis to achieve the
scalable routing goals.

They are the only ones which preserve the perfectly efficient
(although architecturally "impure") "overloaded" semantics of the IP
address functioning as the host's Identifier and Locator.

They are the only ones which have a chance of providing mobility in a
practical manner - via TTR Mobility.  Ivip incorporates this and I
understand IRON incorporates some aspects of TTR Mobility.  LISP's
current approach to mobility is impractical, since it can't work
behind NAT and the host has to tell all its correspondent hosts every
time it gets a new IP address.  However, TTR Mobility would work fine
with LISP.

Section 3.4 is a terrible combination of being wrongly intentioned,
containing incorrect and confusing attempts to describe a class of
architectures (CES) and being contradictory - with Loc/ID Separation
(CEE) being both "desired" and "required".  It also involves the
impossible constraint on CES architectures, if adopted, being
compatible with CEE.

Section 3.5 on Mobility could be improved by extending the first
sentence with:

+  in a way which supports existing sessions and which enables
+  communications with other hosts to be initiated and continue
+  without unacceptable interruption.  The prevalence of VoIP and
+  similar real-time communications means that the architecture
+  needs to be able to support session continuation within a small
+  fraction of a second when the mobile host loses one IP address
+  or locator and gains another.
+  The prevalence of HTTP and other protocols in which the
+  application is communicating with large numbers of other hosts
+  means that the session continuation mechanisms need to be either
+  very lightweight and rapid per such host, or not involve the other
+  hosts at all.

Point 1 is not truly mobility by any ordinary definition.  Any host
can get a new IP address when it connects to a new network - and we
don't call that "Mobility".

Points 2 and 3 are OK, as is the description which follows.  However,
this list doesn't include TTR Mobility:

   http://www.firstpr.com.au/ip/ivip/#mobile

where the MN makes a tunnel from each of its one or more newly
acquired IP addresses (Care-of Addresses - CoAs) to a typically nearby
TTR (Translating Tunnel Router).  This is somewhat analogous to a home
agent, except that the MN picks a typically nearby TTR, and the CES
system has its mapping changed so packets for this MN are tunnelled to
this TTR.

The list is for "existing mechanisms" - and TTR Mobility has not yet
been implemented.  So its fair enough not to mention it in such a
list.  It is, nonetheless, a valid architecture for mobility.  It is
the only one I know which enables the MN to quickly gain and use a new
CoA without having to notify all its correspondent hosts, and without
the potentially long paths required with a single home agent.

The next sentence:

#       Mechanisms that help to provide more efficient and scalable
#       mobility support are desired, especially when they can be
#       coupled with security, especially privacy, and support
#       topological changes at a high-rate.

is fine, except that I think, like the multihoming, that mobility
should be required.

#       It is required Mobility mechanisms should completely
#       decouple mobility from routing.

This sentence is fine if "routing" is understood as "the existing
interdomain routing system" or this plus all other existing routers.
This means it is OK to add ITRs, ETRs TTRs etc.   I think it would be
better to state:

+       It is required that the mobility mechanisms do not place any
+       new burdens on existing routers - BGP routers in the
+       interdomain routing system, and all other routers in existing
+       networks.

I would add:

+       It is required that the mobility mechanisms not place extra
+       burdens on other hosts.

since this is achievable, at least with TTR Mobility.

I disagree with section 3.6 - Simplified renumbering.

#       Today many of the end-sites receive their IP address
#       assignments from their Internet Service Providers (ISP).
#       When such a site changes providers, for routing to scale,
#       the site must renumber into a new address block assigned
#       by its new ISP.  This can be costly, error-prone and
#       painful.

OK so far.

#       Automated tools, once developed, are expected to provide
#       significant help in reducing the renumbering pain.

This is a misleading statement.  Some people expect this to be the
case, but I and many other people regard their optimism as
unrealistic.  There are many problems with dynamically renumbering any
operational network, not least the difficulty of testing the capacity
for all hosts and their applications to tolerate and operate correctly
with such a renumbering without actually doing it.

Even if all hosts and routers could be magically renumbered, there
remain problems with the addresses of the network being in the config
files of hosts well beyond the network itself.  For instance, ACLs and
in the DNS files of servers in other networks.  EUN AAA may have hosts
in its network rely on DNS servers run by another company.  EUN BBB
may be a hosting company and host servers for thousands of EUNs such
as XXX, YYY and ZZZ.  When BBB renumbers, it has to securely and
reliably coordinate DNS changes in the DNS servers which are
responsible for the XXX, YYY and ZZZ domains.  Those DNS servers may
be run by XXX, YYY and ZZZ, or by still other companies.

Unless the renumbering involves using two separate address spaces at
the same time, with a long transition period, then these changes need
to be made at an exact time of the renumbering, which raises questions
of cached DNS replies in various resolvers around the Net.

The problem of doing all this securely, with little cost and in a
testable manner is not amenable to "automated tools".  This is true
for the current addressing arrangements where the IP address functions
as Identifier and Locator (as is supported by CES architectures), and
for the proposed changes of Loc/Id Separation (CEE) - since in the
latter case, the DNS typically includes information about the Locators.

#       It is not expected that renumbering will be wholly automated,
#       as some manual reconfiguration is likely to be necessary for
#       changing the last-mile link.  However, the overall cost of
#       this transition should be drastically lowered.

This represents aspirations.  I and many others regard the prospect of
renumbering substantial networks (many networks with one or more
stable IP addresses and running one or more publicly available server)
without significant risk or cost as being forever unattainable.

#       In addition to being configured into hosts and routers, where
#       automated renumbering tools can help, IP addresses are also
#       often used for other purposes such as access control lists.

Let's say a university changes from one ISP to another at 2AM, and
needs to change its networks entry on various ACLs at the exact same
time.  How can automated tools securely update the ACL of an academic
journal network which restricts access to those IP addresses (or
Locators, in Loc/ID Separation) of the universities which pay for
subscription to its journals?  It will always be impossible to
efficiently and reliably automate these things which involve multiple
companies.

#       They are also sometimes hard-coded into applications used in
#       environments where failure of the DNS could be catastrophic
#       (e.g. certain remote monitoring applications).  Although
#       renumbering may be considered a mild inconvenience for some
#       sites, and guidelines have been developed for renumbering a
#       network without a flag day [RFC4192], for others, the
#       necessary changes are sufficiently difficult so as to make
#       renumbering effectively impossible.

Yes.

#       It is strongly desired that a new architecture allow
#       end-sites to renumber their network with significantly less
#       disruption,

But there are multiple reasons why this will never be possible in a
secure, reliable, fashion.

#       or, if renumbering can be eliminated, the new architecture
#       must demonstrate how the topology can be economically morphed
#       to fit the addressing.

This is misleading, because it assumes that the only way we can cope
with not renumbering EUNs when they change ISPs is to "morph
the topology" to suit the moved addressing.   CES architectures do
this by leaving the existing topology as it is - that is, the DFZ
prefixes which are the focus of the current routing scaling problem.
They add ITRs, ETRs, mapping servers (and for TTR Mobility, TTRs) to
tunnel packets appropriately.  This is not "morphing the topology".

This section continues the prevailing theme of the document,
championed by Tony Li and a few others, which is something like this:

   The only way of making things scale well into the future is
   to expect nothing more of devices in the network - existing
   routers, new devices (including routers) - or by adding
   responsibilities to existing routers.

   Therefore, the only way forward is to bind sessions to Identifiers
   which can be reached via one or more Locators.

   Therefore, host protocols need to change.  (In practice, this
   means host stacks, application code and in some cases application
   protocols.)

The trouble is that even if this could be voluntarily adopted on a
wide enough basis (which it can't), there's no way of making the CEE
architectures work for IPv4, the extra host responsibilities and
delays are a new form of scaling problem - and the CEE architectures
can't support mobility in an acceptable manner.

Since renumbering is impossible to achieve in an acceptable manner, I
think this section should be replaced by something like:

+  3.6  Portability
+
+  There are no prospects for making the renumbering of substantial
+  end-user networks a regular, low-cost, secure and reliable
+  procedure.  This is true of the current system using IP
+  addresses as both Identifiers and Locators or in a Loc/ID
+  Separation.  In both cases, ACLs and other items of information,
+  such as in DNS servers, must be changed - yet these are often
+  located outside the network and administered by separate
+  organizations.
+
+  Consequently, to enable end-user networks to choose their one or
+  more ISPs, it is a requirement that the architectural enhancements
+  provide portability of either the end-user networks IP address
+  space, or in the case of Loc/ID Separation, their Identifier space.
+
+  Portability should involve little or no cost or other impediments
+  to the end-user network - such as concerns over security,
+  reliability or disruption.  Portability should be highly scalable
+  from the point of view of the current routing system, or whatever
+  replaces it, meaning that there is either no impact on this
+  system, or that the impact is adequately paid for by the end-user
+  network which benefits.
+
+  Portability is required only for end-user networks - not for ISPs.

I think 3.7 is pretty good, except for the last part of the last
paragraph:

#       should also anticipate changes to that model e.g. for
#       multiple differentiated and/or guaranteed upon levels of
#       service in the future.

QoS across the DFZ means reserving capacity, which means someone has
to pay for this reservation.  AFAIK, no-one has a clue about how to
implement this technically or administratively.  To the extent that
there could be reserved capacity across the DFZ, I can't imagine how
that could work well with the need for the paths to change as topology
changes.   While such goals are worth a mention, I think we are so far
from being able to achieve them that they shouldn't be given a high
priority.

I think that section 3.8 is good.  However, the best approach for
mobility (TTR Mobility, in my view) is likely to involve transient or
user-chosen conditions where the stretch is worse than today.
Firstly, when a MN gets a new CoA and must use this, but is yet to
make a tunnel to a new TTR and have the mapping of its address space
changed, it will be using the previous TTR for incoming and outgoing
packets - and that TTR could be far away.  Secondly, if the user wants
to avoid their CoA being even approximately trackable (except by the
TTR company) then they may elect to keep the one TTR no matter where
they are.

I think section 3.9 is good too - but since any solution adds
complexity, it is unlikely we will be able to avoid more potential for
security and reliability failings.

Section 3.10 on Deployability doesn't go far enough.  The term
"incrementally deployable" means different things to different people,
and the definition here is a minimal one.  It is not sufficient to
ensure that the new architecture will actually be adopted widely
enough to provide the required benefits.

Since no-one has suggested a means by which the new architecture can
forced on users, or why it should be, this leaves us wholly dependent
on voluntary adoption.  Most users will only adopt a new system for
the immediate benefits they derive from it - they won't do it just for
the generalised public good of scalable routing and addressing.

Therefore, there are in fact much more stringent constraints on any
architecture than the mere "compatibility" criteria mentioned in
section 3.10.

I tried, with the help of others, to write about this in more detail:

  Constraints imposed by the need for widespread voluntary adoption
  http://www.firstpr.com.au/ip/ivip/RRG-2009/constraints/

Here is some text which I think would be better than section 3.10,
based on the above.

Something like this definitely needs to be in the Design Goals.
Without this - or with any other unrealistic notion of what is really
required to get most or all users to adopt architectural enhancements
- the whole project is doomed to fail, or falter along for a decade or
two at the glacial pace of IPv6.

+  3.10 Constraints imposed by the need for widespread voluntary
+  adoption
+
+  Since no-one is in a position to force widespread adoption of
+  architectural enhancements, our reliance on widespread voluntary
+  adoption involves the following constraints:
+
+    1 - The solution must provide immediate and
+        compelling benefits to any new or existing
+        end-user networks which adopt it, irrespective
+        of how many other networks have so far adopted
+        it.
+
+    2 - Therefore, it must provide portability, multihoming,
+        and inbound traffic engineering for the adopting
+        non-mobile networks in a manner which fully supports
+        communications with all hosts - including all hosts
+        in non-upgraded networks.  Similarly, mobility must
+        be provided in a way which works with all other hosts
+        and networks without need to change those hosts or
+        networks.

(CEE fails this test, since it only provides benefits for
communications with other upgraded hosts in upgraded networks.)

+    3 - Therefore, the solution must be compatible with
+        all hosts (stacks and applications) and routers
+        in non-upgraded networks.  This rules out attempting
+        to solve the IPv4 Internet's scaling problems by
+        requiring adopting networks to use some other system
+        such as the IPv6 Internet.

(CEE only works for IPv6.  CES can work for both IPv4 and IPv6.)

+    4 - Likewise, the solution must be compatible with all
+        existing applications in the adopting networks.

(CEE can't do this, AFAIK, despite ILNP still being claimed as being
compatible with existing IPv6 applications.  See my msg07185 2010-08-01.)

+    5 - Although in principle, some or many host stacks
+        could be upgraded in the adopting networks, the
+        difficulties this presents means that a scalable
+        routing solution will only be widely enough
+        adopted on a voluntary basis if no such upgrades
+        are required in order for the new system to
+        provide compelling benefits.
+
+        Firstly, there are difficulties in motivating
+        operating system / stack programmers to add and
+        test new facilities.  Secondly there may be
+        costs and disruption for the adopting network in
+        upgrading the stacks.  Thirdly, in many or most
+        end-user networks, some or many hosts will not
+        be amenable to stack upgrades due to them using
+        operating systems which are no longer being
+        maintained or developed.  This is especially
+        the case for devices such as printers, routers
+        etc. which do not use one of the major
+        PC operating systems.
+
+    6 - If the solution requires extra functionality in
+        interdomain routers, then it must be feasible from
+        technical and business perspectives for the relevant
+        routers to have this functionality in time for
+        deployment.
+
+    7 - The solution must be attractive for the large
+        number of smaller networks who currently cannot
+        get PI space to do multihoming etc.  Our goal is
+        not just to reduce the number of DFZ routes but
+        to enable a much larger number of end-user
+        networks to scalably gain multihoming, TE and
+        portability than is currently possible.
+
+    8 - Ideally the solution will also be attractive for
+        most or all end-user networks who currently have
+        their own PI prefixes.  This is firstly to
+        improve scaling by attracting those larger
+        end-user networks away from their current
+        conventional PI address space usage and secondly
+        to ensure that smaller networks which aspire to
+        be bigger in the future will perceive the new
+        kind of scalable end-user network address space
+        as suitable for their long-term needs.
+
+    9 - Ideally the new system should not compromise
+        security or performance compared to that of
+        current PI or PA techniques.  Any degradation in
+        security, packet path lengths, end-to-end packet
+        transit times, packet loss rates and general
+        robustness etc. needs to be more than
+        compensated for by reduced costs and/or benefits
+        such as a more flexible and responsive
+        multihoming restoration capability than is
+        possible with current BGP techniques.
+
+   10 - The solution will probably not be widely enough
+        adopted if it requires upgrades to the internal
+        routers of adopting networks - except perhaps in
+        small networks such as SOHO where one or a few
+        inexpensive routers need upgrading anyway to
+        perform ITR and ETR etc. functions.

I suggest another further sections along the lines of:

+  3.11  IPv4 address space utilization
+
+  The solution for IPv4 should not involve a significant increase
+  in amount of address space required for a given level of
+  functionality - such as a given number of hosts in an end-user
+  network.
+
+  It is desired that the solution for IPv4 generally reduce the
+  total amount of address space required to support end-user
+  networks.  This includes the ability of a solution to reduce
+  the scaling problems which can be expected to arise from the
+  more rapid growth in the next few years of separately advertised
+  DFZ prefixes, as fresh supplies of address space diminish, causing
+  existing prefixes to be divided so the space they use can be used
+  more intensively - such as by multiple end-user networks instead
+  of just the one which advertises it now.

+  3.12  Support for migration to other architectures
+
+  It is desired that the architectural enhancement be able to
+  help with the widespread voluntary adoption of any future
+  architectures which are argued to be superior in the longer-term.

+  3.13  Solutions for IPv4 and IPv6
+
+  The optimal solution for IPv4 may differ from that for IPv6 due to
+  factors such as:
+
+     Greater urgency in implementing the solution for IPv4, due to
+     greater adoption and concerns about address utilization
+     efficiency.
+
+     The address space constraints of IPv4 and differing packet
+     formats between the two systems leading to different goals and
+     different mechanisms, such as for tunneling, which can be used
+     by solutions.
+
+     Greater or lesser emphasis on backwards compatibility, and
+     therefore potentially lesser or greater prospects for changes
+     to routers, host stacks and even applications - according to
+     the currently little used nature of the IPv6 network and its
+     protocols.
+
+  Even with such differences, it is desired that the solutions which
+  are adopted have the following characteristics:
+
+     1 - Have as much in common as possible regarding their
+         architectural principles, terminology, protocol details,
+         mapping system data structures and software implementations.
+
+     2 - The two actual solutions for IPv4 and IPv6, once
+         implemented, should not be in any way interdependent.
+
+     3 - That the two solutions should facilitate to the greatest
+         extent possible interworking between IPv4 and IPv6, and
+         the adoption of of IPv6 by new users and networks and
+         those currently using IPv4 exclusively.

+  3.14 Potential scaling difficulties of Core-Edge Separation
+       architectures
+
+  There are many potential technical and commercial scalability
+  problems in a CES architecture.  Proponents of all such
+  architectures need to identify such difficulties and argue why
+  these choices should be made, along the lines of:
+
+     1 - The mechanisms are shown to be technically feasible for the
+         initial, growing and highest imaginable levels of adoption
+         - based on the likely technology available in those times.
+
+     2 - The costs of each mechanism are shown to be acceptable and
+         not excessive, with detailed arguments about how the cost
+         will either be born happily by those who operate the
+         mechanism and/or will be be paid for by commercial
+         arrangements with those who benefit from the mechanism.
+
+         While an architecture can't mandate commercial relationships
+         between users and operators, and between the various
+         operators of different parts of the system, it should be
+         shown that the architecture is able to support such
+         relationships and that such relationships and payments are
+         likely to be both acceptable to all parties and voluntarily
+         entered into.
+
+         For instance, where multiple end-user networks benefit from
+         a mapping server, router or some other device or service
+         provided by an operator of some part of the new
+         architecture, it needs to be shown that the operator can
+         identify these end-user networks and directly or indirectly
+         obtain payments from them.  This includes the possibility
+         that the information on which this charging is based is
+         collected on the basis of statistical sampling, rather than
+         on every instance of a particular type of event.  It also
+         includes the possibility of billing intermediate
+         organizations who service those end-user networks - but with
+         sufficient information that those intermediate
+         organizations can themselves charge their end-user network
+         customers, at least approximately, according to their level
+         of actual usage.
+
+     3 - CES architectures rely on ITRs tunneling packets to one or
+         more ETRs, where the choice of ETR may need to change in
+         real-time due to network outages, inbound TE requirements
+         or for some other reason.  There are potential scaling
+         difficulties in how this can be done.
+
+         For instance if each ITR is required to continually monitor
+         the reachability of many end-user networks through multiple
+         ETRs for each such network, there are scaling difficulties
+         for the ITRs and ETRs and difficult trade-offs between the
+         desire to perform this "reachability testing" on a frequent
+         basis, to facilitate rapid response to outages, and the
+         desire for reducing overall ITR and ETR workload and its
+         associated flow of packets.  Where the ITR is instructed in
+         real-time by the mapping system to send packets to a single
+         ETR, no such problems arise, since the reachability testing
+         is done elsewhere - but new scaling and reliability
+         problems arise in whatever mechanisms are used to convey
+         this information to all the ITRs which need it.
+
+     4 - CES architectures rely on some kind of mapping system
+         and there are potential scaling problems - again due to
+         technical feasibility and the potentially unfair
+         distribution of costs if there is no mechanism for
+         charging those who benefit.
+
+     5 - Depending on its design, a CES system may introduce
+         difficulties in maintaining existing security measures
+         for address space which is not covered by the CES scheme.
+         For instance, the ability of an ISP or other network
+         operator to reject packets arriving from the DFZ and
+         being de-tunneled by an ETR in their network, with
+         inner packet source addresses spoofing the ISP's own
+         internal hosts may be made impossible, or inordinately
+         complex and expensive, depending on the nature of the ITR
+         to ETR tunneling system.
+
+         While not a scaling problem, it should be noted that CES
+         architectures are also likely to lead to diminished
+         possibilities for rejecting source address spoofed packets
+         for addresses within the prefixes they manage.  This is
+         because the purpose of the CES system is that these
+         addresses are portable.
+
+     6 - Mobility involves potential scaling and performance
+         challenges concerning what the mobile node must do
+         in order to maintain its communication sessions and
+         enable the establishment of fresh sessions every time
+         it gains a new address, especially if this event is
+         accompanied or preceded by the loss of its only other
+         address.
+
+  CES architectures bring about their own set of potential
+  problems with security, robustness and privacy.

+ 3.15 Potential scaling difficulties of Core-Edge Elimination
+      (Loc/ID separation) architectures
+
+  CEE architectures have potential scaling problems with their
+  mapping systems, since these typically provide benefits to
+  many end-user networks, but must be operated by some other
+  entity than the network itself, if only to avoid circular
+  dependencies.  Since the mapping information cannot typically
+  be cached, the queries must be answered by one of a presumably
+  limited number of authoritative servers, which are therefore
+  likely to be at a considerable distance.  Finding this
+  authoritative server or obtaining its response may involve the
+  querying host and/or some other server, sending and receiving
+  multiple packets.
+
+  The delays, reliability problems and query / reply volumes of
+  these systems involve scaling problems driven in part by the
+  frequency with which the mapping system must be used.
+
+  For instance, if the mobility arrangements fail due to both
+  hosts changing their Locators at about the same time, there
+  are scaling problems with the efforts the hosts must make to
+  determine this is the case, and the queries they must make to
+  securely use the mapping system to find the other host's current
+  Locator(s).   Similarly, there are potential scaling problems
+  resulting from the frequency with which mobile hosts must securely
+  update the mapping system and/or DNS every time they change
+  Locators.  The same scaling problems occur in a CEE mobility
+  architecture in which the mobile host needs to securely update
+  the mapping system when it gains a new IP address.
+
+  CEE architectures involve a number of scaling problems which are
+  unique to their reliance on new host protocols and the
+  responsibilities and packet exchanges which these involve.
+
+  Initial and perhaps all packets may require extra information to
+  inform the recipient host of the sending host's Identifier and/or
+  Locator.
+
+  In many application protocols, a responding host cannot send a
+  reply to a host at the Locator specified in the packet received
+  from the initiating host, since it has no way of knowing that the
+  host with the Identifier specified in the source field of the
+  initial packet currently has this Locator.  Therefore, the
+  recipient host must perform an Identifier to Locator(s) lookup
+  before sending a reply packet.
+
+  The state which hosts must store regarding the Locators of each
+  host they have recently (however defined) communicated with may
+  raise scaling problems, particularly in busy servers.
+
+  The delay inherent in this arrangement is not a scaling problem as
+  such, but the number of request and response packets may be.  This
+  is especially the case for devices relying on high latency,
+  potentially unreliable, wireless or satellite links.
+
+  Since a small initial packet could cause the recipient host to
+  send one or more query packets (as may be required to send the
+  query to an authoritative server) and to receive one or more
+  packets, this also represents a potential DoS vulnerability.

 - Robin