Subject: Re: [rrg] [lisp] [Int-area] Please respond: Questions from the IESG as to whether a WG forming BOF is necessary for LISP

From: Robin Whittle <[email-masked]>
Date: Fri, 30 Jan 2009 15:57:22 +1100

Short version:   Timeline of the LISP-ALT/CONS proposal with
                 respect to APT and Ivip.

                 LISP-ALT development looks rather hasty so far
                 and could proceed quite well without a WG.

                 But maybe a WG is the best way - develop
                 LISP-ALT ASAP and see how attractive it is to
                 the great majority of end-user networks.

                 We need most end-user networks, of all sizes,
                 to want to adopt the solution for their own
                 immediate reasons - otherwise, the solution
                 won't cover enough networks to solve the
                 scaling problem.

                 There is a LISP-ALT snowball.  Forming a
                 LISP WG would push the snowball faster.

                 The more prominence and IETF-given
                 credibility LISP gets, the more this might
                 detract attention from the alternative
                 approaches, including APT and Ivip so far.

                 I think these are more sound from technical
                 and business perspectives than LISP-ALT.
                 These are developed by smaller teams with
                 less resources and experience than the LISP
                 team - but maybe these efforts will result in
                 a better architecture.

Hi Noel,

You wrote:

>> The LISP team should do detailed critiques of APT and Ivip as
>> part of convincing IETF folks of the merits of LISP-ALT.
>
> Perhaps I'm confused, but I don't think the people working on
> LISP are trying to "convince" the IETF as a whole of the
> "merits of LISP".

I don't know the criteria is for creating a WG, but I guess the
IETF doesn't create them except for projects which have
sufficient merit in general, as well as meeting formal criteria.

> If some people out there in the Internet technical community
> think it looks good, and want to work on it, run it, etc,
> that's great; but if other people reach a contrary conclusion,
> and don't want to work on it, run it, etc (as is their
> choice), that's fine too.

I am not completely opposed to there being a LISP-WG.  I am just
saying that I think the LISP development has been hurried in a
number of respects, that it doesn't necessarily need a WG and
that a WG could detract attention from more promising other
approaches.

I think the LISP team could best facilitate progress towards
scalable routing solutions by taking more interest in other
proposals and by discussing in public why they think their
LISP-ALT approach is better than the alternatives.

Their goal is 10 billion EIDs.  Assuming this is not an
unrealistic scaling goal, they need to show why ALT could
work well with this, while APT and Ivip could never do so.

The history, as reflected in Internet Drafts, is something like
this (ignoring LISP-NERD, which doesn't scale due to the need
for every ITR to have a copy of the full mapping database):

2007-01-17 draft-farinacci-lisp-00

           This is a conceptual overview which assumes it is
           impossible to get real-time mapping to the ITRs.
           Therefore, ITRs get multiple ETR addresses and have
           to do reachability testing and ETR choice on their
           own.

           There is detailed description of a useful mapping
           system.

           There is no support for packets sent by hosts in
           networks without ITRs.

2007-07-02 draft-jen-apt-00

           APT is the first core-edge separation scheme to
           document a scalable mapping distribution system in
           in Internet Draft.  The full mapping database is
           distributed (slowly) to local query servers -
           Default Mappers.  As with LISP, ITRs - or rather
           Default Mappers - do their own reachability testing
           and make their own ETR choices.

           It later emerged that APT can support packets sent
           from hosts in non-APT networks, but only, for IPv4,
           if the EID prefix is no longer than the in-practice
           limit of the DFZ: 24 bits.  This limit would
           disappear if APT networks connected not only by
           direct router links, but by tunnels, so there was
           only one global "APT Island".

2007-07-03 draft-meyer-lisp-cons-00

           CONS is LISP's first mapping system which is intended
           to be practical in a scalable routing solution.  It
           is a global query server system which involves delays
           for initial packets.  CONS uses a a novel network
           structure comprised of special router-like devices
           called CARs and CDRs.

2007-07-15 draft-whittle-ivip-arch-00

           Fast (effectively real-time) push of all mapping info
           to local query servers.  I developed this
           independently of APT, but the two proposals involve
           some commonalities: the local query server means that
           there is no significant delay for initial (or any)
           packets.

           Ivip includes Open ITRs in the DFZ (then known,
           incorrectly, as "Anycast ITRs in the Core").  These
           support multihoming, TE etc. for packets sent by
           hosts in networks without ITRs.  A business plan for
           OITRDs appeared in 2008-04-18:

             http://psg.com/lists/rrg/2008/msg01158.html

2007-11-11 draft-fuller-lisp-alt-00

           LISP-ALT replaces CONS as the preferred mapping
           distribution system.  Like CONS, it is a global
           query server system, so it has the problem of
           delaying initial packets.  The ALT routers and
           the overall ALT network are created from pre-
           existing components: BGP, RIB, FIB and GRE.

2007-12-04 RRG message that there will be a LISP BOF regarding
           forming a LISP WG.

2007-12-05 draft-lewis-lisp-interworking-00

           LISP Proxy Tunnel Routers provide support for packets
           sent by hosts in non-upgraded networks.  AFAICT, PTRs
           are functionally identical to Ivip's OITRDs.

           However there was no business plan for PTRs or the
           related questions of how EID space is allocated.
           Dino's reply to my point 6:

             http://www.irtf.org/pipermail/rrg/2009-January/000864.html

           indicates there still is no such business plan.

In the view of many people, the delay of initial packets remains
a serious problem for LISP-ALT.  Part if this delay is due to the
global nature of the query server system.  This is made worse by
the long-path problem where the highly aggregated structure of the
ALT network leads to geographically long tunnels between the ALT
routers, which will often be physically located without much regard
for their place in the ALT topology.

In a recent message:

  http://www.irtf.org/pipermail/rrg/2009-January/000900.html

Dino confirmed the problem of initial packet delay and indicated
there is no alternative to a fully distributed query server system
(as opposed to having the full mapping data in hundreds of
thousands of local query servers, as with APT and Ivip) if the
number of EIDs, micronets etc. is to scale to 10 billion.

Yet in the same message he states (presumably about the
long-term future with 10^10 EIDs) that:

   I envision the number of hops a Map-Request takes is 4
   ALT-router hops.

In LISP-ALT, map queries (or initial traffic packets which are
implicitly map queries) arise in ITRs and need to ascend the
highly aggregated ALT hierarchy until they can cross over to the
part of the hierarchy which matches the destination address.
Then they descend, and reach the authoritative query server -
the ETR which is currently advertising this EID in the ALT
network.  (The map reply from the ETR goes to the ITR directly
via the ordinary Internet.)

There's no way that the ALT network could handle 10 billion
separate EID prefixes and avoid large numbers of ALT routers
- and therefore large numbers of hops up and down the ALT
hierarchy for the typical map request.

A 10 billion EID system could only materialise when most EIDs
are for "end-user networks" are for mobile devices which
everyone and their dog has at least one of.

There will be some number of ITRs and some number of ETRs
- I guess a million or so of each, perhaps often combined
into the one device.

The ETRs collectively advertise 10 billion separate EIDs to
the first level (L1) of ALT router.  How numerous are these?

Each such L1 ALT router needs a GRE tunnel (AFAIK) to every
ETR which is one of those currently active for a EID in the
address range the L1 router is aggregating.

The whole idea of a core-edge separation scheme is that the
EIDs (micronets for Ivip) can be freely used with any ISP
for Internet access - and that each EID can be used with
two or more arbitrarily located ISPs for the purpose of
multihoming.

Ignoring multihoming for a moment, there is still the need
for this first level (L1) of ALT routers to collectively
handle 10 billion EID prefixes.  For any one of these L1
ALT routers, there will sometimes be two or more such EIDs
advertised by the same ETR.  Those two routes share the one
ETR to ALT router GRE tunnel.  (AFAIK, GRE tunnels are used
to link ETRs to ALT routers.)

That is a lot of tunnels - and a lot of L1 routers, such as
a hundred thousand or a million or so.

We don't know how much aggregation will be achieved from
ETR to L1, or from L1 to L2 ALT routers.  Many packets
from ITRs will need to ascend to some high level in the
ALT hierarchy before there is a route to the branch of
the hierarchy they need to descend in order to reach
the ETR.

So I don't think it is reasonable for LISP people to say
their system scales to 10^10 EIDs while any other
system definitely can't.

See also a recent RRG discussion "Economic issues of long-
path / stretched routing" where Scott Brim discusses each ETR
announcing its EID prefix to ALT routers at multiple levels
above the first level.  Each such announcement involves a
GRE tunnel (AFAIK) and perhaps constitutes a BGP neighbour
for the ALT router.  This sort of thing is needed to make the
ALT network robust against failure, but it also makes it much
more complex and less able to be flattened into a few levels
- as would be required to reduce the total path length from
ITRs to ETRs.

I mention these difficulties as an example of the mismatch
between the expectations the LISP developers have - for LISP
and therefore, implicitly for APT, Ivip etc. - and the
confidence some LISP developers express in solving the
fundamental problems which have been identified with ALT.

> Sure, the people working on LISP would like to convince _some_
> people that it's a good idea, so they get i) some help working
> on it (as much remains to be done), and ii) some places to
> deploy it (so some practical experience can be gained, in
> seeing how well it works, what problems crop up in actual
> service, etc), but that's a very different kettle of fish.

I fully support all this.  As far as I know an IETF WG is not
required to do any of this.  Maybe a WG would help.  Maybe
the imprimatur of the IETF in deciding that LISP in its current
state is worthy of a WG will encourage more people to help with
LISP.

However, if a WG results in LISP gaining still more prominence,
then I think there could be a serious downside: making it
harder for other proposals to attract the same sort of interest,
attention and involvement.

I believe the initial packet delay problems and the general
fragility of any global query server system such as LISP-ALT
will ultimately sink any such proposal.  Maybe it won't sink in
the IETF, but it will fail to gain the widespread, ideally
ubiquitous,support from end-user networks which will be required
for any routing scaling solution to succeed.

I am sure that the only practical solution is of the core-edge
separation class (LISP, APT, Ivip, TRRP and maybe RANGER - I
still have to read it).

The only scalable way to avoid initial packet delays is to use
local query servers.  I think a system such as APT of Ivip would
be scalable, practical and attractive - and that something like
LISP-ALT will never be attractive enough to be widely accepted.

By the time we get to 10 billion EIDs, micronets or whatever -
which may be never - RAM and CPU power for local query servers
will be way cheaper and more plentiful than today.

Local query servers based on COTS servers today would be fine
with 100 million EIDs in RAM.  For IPv4, for Ivip, each
micronet's mapping information is only 12 bytes and could
probably be stored in less.  IPv6 mapping is more bloated, but
still, there is even more time before that is widely adopted -
more time for RAM to get more plentiful still.

The rate of change of mapping is well within practical limits
for sending to hundreds of thousands of query servers -
especially using a purpose built, streamlined, crosslinked
fan-out system as I propose with Ivip.  Even with mobility,
using the TTR approach to mobility does not result in high
rates of change of mapping.  For instance a mapping change
is only typically required when the MN moves more than about
1000km.

LISP-ALT is the most frequently discussed solution to the routing
scaling problem.  Yet it is a more recent development than APT or
Ivip - and I think it is the most problematic of the three.

I think the prominence of LISP-ALT is not a function of its
technical merits, but more a result of:

  1 - Four or more people working on it, rather than two or so
      (APT) or basically one (Ivip).

  2 - Some employer support for the LISP developers, including
      for travel to meetings and hardware to run a test
      network.  (The APT developers are studying and I am
      self employed.)

  3 - The greater experience of the LISP developers compared
      to that of the other developers - and likewise being
      better known to RRG participants.

  4 - Greater brand recognition for "Cisco" compared to
      the names of developers of alternatives: APT, Ivip,
      TRRP, Six/One Router and now, I think, RANGER.

A WG may overly focus attention on on LISP-ALT and so detract
from the attention and effort needed to develop the more
promising approaches.

 - Robin

_______________________________________________
routing-discussion mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/routing-discussion

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 30 Jan 2009 10:04:21 -0800

> that a WG could detract attention from more promising other

How about the world is going too slow?

Dino

---

From: Vince Fuller <[email-masked]>
Date: Fri, 30 Jan 2009 15:18:40 -0800

> Ivip - and I think it is the most problematic of the three.

As I have explained more than once already, your timeline is simply wrong.

The base LISP spec, draft-farinacci-lisp-11.txt in its current incarnation,
was a direct result of design discussions that occurred in October, 2006,
at the IAB-sponsored Routing and Addressing Workshop in Amsterdam. The first
version of draft-farinacci-lisp appeared for select reviewers on December 1st,
2006 and was published on January 22nd, 2007.

The LISP+ALT spec, draft-fuller-lisp-alt-03.txt in its current incarnation
(and originally termed "LISP 1.5"), was a direct result of discussions at a
lunch meeting during the November, 2006 IETF plenary in San Diego. Small
group discussions occurred for several months prior to taking it to the RAM
list in June, 2007 though the first version of the draft was not published 
until November, 11th, 2007.

	--Vince

---

From: Robin Whittle <[email-masked]>
Date: Sat, 31 Jan 2009 15:01:59 +1100

Short version:      LISP-ALT seems liked a rushed development
                    which won't be widely enough adopted even if
                    there are RFCs and interoperable
                    implementations by 2012.

                    The work which preceded the various I-Ds.

Dino, in response to my concern that the LISP-ALT development is

> How about the world is going too slow?

In the 2.3 years since RAWS in Amsterdam the DFZ routing table
has grown from about 200k to about 275k.  There may be a slight
glitch downwards due to the economic crisis in recent months:

  http://bgp.potaroo.net/bgprpts/bgp-active.png

That is a growth rate of about 15% PA.

We have growing problems with:

  1 - This growth rate increasing the cost of of routers
      in the DFZ, which is a burden on all ISPs and therefore
      all Internet users.

  2 - Excessive constraints in PI space which is an attempt to
      reduce the above growth rate.

  3 - With the increasing number of end-user networks which want
      multihoming, inbound traffic engineering and portability
      either not being able to get any of these, or having to
      pay much more for it (and cause burdens on the DFZ) than
      would be the case with a good scalable routing solution.

Whenever IPv6 is widely adopted, the same things will happen
there too.

However, I don't think these problems are so urgent that we
should rush into re-engineering the Net with a poorer solution
than if we spent a few more years working on it.  We will be
stuck with the solution for decades.

I imagine you could create LISP-ALT RFCs by the end of 2010 and
have interoperable implementations in software and in some
hardware routers by the end of 2011.

But I can't see how you are going to get the widespread adoption
of LISP-ALT EID space by most end-user networks (which want
multihoming, TE and portability), from the smallest (SOHO etc.)
to the largest (corporations, universities and government
departments) in 2012 or at any other time.

By then you will probably have figured out business arrangements
for EID allocation and for running PTRs.  I imagine it would be
be similar to parts of:

You will probably have also solved the PMTUD problem with a
judicious use of the OpenLISP stateful approach, requiring all
DFZ routers to return 16 bytes or more past the IP header in
their PTB messages, minimising the number of packets the ITR
stores the nonce for and allowing for occasional checking for
growth or reduction in the path MTU to each ETR.  I think the
result would resemble:

  http://www.firstpr.com.au/ip/ivip/pmtud-frag/

LISP-ALT would still have the initial packet delay problem, the
long-path problem and the difficulties in building a big ALT
network which has redundancy and meshiness - or whatever is
required for robustness.

None of these problems would present serious practical
difficulties in the test network or in the first months or year
or two of deployment - because there would be small number of
ALT routers.

However, anyone concerned about the long-term future would be
well aware of these problems.  I don't think you can solve these
problems in any elegant, efficient manner - because they are
fundamental problems with the LISP-ALT architecture.

ALT is certainly more elegant than CONS, with its new network
elements of CARs and CDRs which needed to be created from scratch.

AFAIK, an ALT router can be created by correctly configuring
various functional elements which are already present in a good,
flexible, hardware-based or software-only router.

The ITR and ETR functions need new code.

By 2011, I figure you would also have figured out standardised
approaches by which an ISP could manage its ETRs to securely
make tunnels to the upstream ALT routers it needs to get map
request messages.

You would also need a secure and reasonably automated method by
which end-user networks could identify themselves to ISP ETRs
for the purposes of telling the ETR their EID, so the ETR would
be prepared to convey mapping for that EID to the ALT routers.

Maybe you will make the ALT network handle the initial packets
(which also function as map requests) - but then the ALT network
needs much greater capacity, for a small reduction in the delay
of initial packets.

That would worsen the traffic burden on ALT routers and require
more care about the commercial relationships between all the
organisations who run these ALT routers.  No-one is going to
carry packets with a bunch of mapping requests, and especially
the greater volume of initial traffic packets, unless they get
paid by whoever sends them and/or whoever accepts them.

Circa 2012, I think it would be unlikely that any large end-user
network which already has PI space would want to convert it to
LISP EID space.  They would need to run PTRs for that space.
That is not so hard if all the EIDs are still in the one
geographic area.  However, if they wanted to use their EIDs
anywhere on the Net, they would need PTRs in or near such
locations.  They would gain the ability to split their address
space for physically separate (in terms of BGP border router)
sites into smaller pieces than the current BGP /24 limitation
allows.  This will become more important as IPv4 space becomes
more and more valuable - but only for networks which have many
sites they want to connect via multiple, physically separate, ISPs.

However, all their network would now suffer from the initial
packet delay problem.  Initially, that might be tolerable for a
university.  But any corporation which is vying for customers
and wants its web-sites to be responsive would look into the
future and see that LISP-ALT's long-path problem is going to get
worse if and when it becomes very widely adopted.

You may well be able to get some adoption of LISP EID space by
smaller networks which can't get PI space today.

ISPs would presumably be able to run ITRs and ETRs without major
expense - either with firmware updates for their existing
routers of via inexpensive or open-source software to run on
COTS servers, at least initially.

By then - 2013 or beyond - other scalable routing solutions will
be better developed and probably have their own WGs, test
networks etc.  I think the scalable routing problem would be
more widely known within and beyond the IETF - especially as
IPv4 space becomes more scarce and valuable.

I can't imagine that LISP-ALT could scale well to your stated
goal of 10^10 EIDs.  As I wrote previously, the ALT network
would have to be so vast that the initial packet delay problem
would be serious, - especially with meshiness or whatever for it
to be robust.  This will affect some DNS requests and replies
too, not just TCP SYN and ACK packets.

The smaller and more numerous the ITRs (to spread the load of
traffic) the more problems there will be with initial packet
delay due to the ITRs not having the mapping already cached.
More numerous ITRs will also drive up the query traffic on the
ALT network.

The local query server schemes (APT and Ivip) have no initial
packet delay problems.  They need to distribute and store
mapping data all over the Net, slowly or in real-time.  Today's
COTS (Commercial Off The Shelf) servers can store and transfer
the data just fine for 10^8 to 10^9 EIDs/micronets.  If and when
we get to 10^10 EIDs/micronets (the vast majority of which must
be for personal mobile devices), a COTS server will probably
have 10^1 or 10^2 times more RAM, CPU power etc.

The mapping distribution for APT and Ivip can be done with
purpose-built, streamlined, mechanisms which are not as fussy as
a LISP-ALT-style global query server network where the whole
thing depends on a lonely query packet traversing a global-scale
network, and its equally lonesome response packet making its way
back via the Internet.

I doubt that anything other people write will convince you not
to push/rush ahead with LISP-ALT.  If you can't imagine a better
architecture, then I think you should develop it ASAP - in a WG.

My point in writing is to remind onlookers that there are other
people in this field who think ALT has too many fundamental
problems to be the optimal long-term scalable routing
architectural enhancement for the Internet - and that there are
other approaches, which are being developing more slowly, which
show much greater promise.

Vince, thanks for filling in the background on the Internet

> As I have explained more than once already, your timeline is simply wrong.

My timeline was stated to be in terms of I-D publication dates,
which are correct.

> 2006 and was published on January 22nd, 2007.

I recall someone writing that APT development was taking place
in late 2006 too.

> until November, 11th, 2007.

The only June 2007 message I can find which relates to ALT is
your "Preliminary thoughts on LISP 1.5" (2007-07-21), which is
largely a private message from 2007-04-04:

  http://www.ietf.org/mail-archive/web/ram/current/msg01562.html

This includes "we're looking for alternative methods  that don't
introduce enormous amounts of  complexity".  Looking back, this
is clearly an early description of LISP-ALT.  Maybe it is also
an early description of CONS, which appeared in an I-D on
2007-07-30.

If you developed ALT so early, then I don't understand why you
published and promote CONS in July 2007  CONS is harder to
understand, design and implement than ALT, and is still a global
query server system.

I was the only person to write a substantial response to the
above mentioned message, noting that I couldn't clearly
understand it - and that point 6 appeared to describe the same
thing as my Ivip "Anycast ITRs in the core", now known as Open
ITRs in the DFZ (OITRDs).

Despite Dino and I think others either criticised or ignored my
"Anycast ITR in the Core" idea from 2007-06-15 onwards.  The the
same idea materialised as LISP Proxy Tunnel Routers on
2007-12-05 in draft-lewis-lisp-interworking-00.

My timeline concerned I-D dates.  Since you mention the work
which preceded this, here are the Ivip dates:

I thought of (what were later known as) OITRDs and the overall
TTR mobility scheme on 2007-06-14 and posted it to the RAM list
the next day.  All other Ivip things were developed after that.

Urged on by Dino and others, I wrote up an I-D ASAP.  This, in
2007-07-15, also included the fast push mapping distribution
system and local query servers - both full-database and caching,
with the ITR being notified if the mapping changed for any
micronet it recently requested mapping for.  This I-D also
included ITR functions in sending hosts.

 - Robin

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 30 Jan 2009 23:55:42 -0800

> hardware routers by the end of 2011.

Writing RFCs is probably the first 10% to completion.

You know how long it takes for multiple vendor implementations to get  
released, documentation to be complete and accurate? And then the long  
evaluation cycle for deployment and then what you learn through this  
process needs to be fed back in. And sometimes you have to restart  
some of these steps?

Of course we don't want to rush, but never assume we have time.

Did we assume we had time for Dual-Stack deployment so when IPv6-only  
sites get deployed (which might possibly happen in the next year or  
so), we could have IPv6 in the infrastructure? We missed that boat and  
we are in a heap of trouble now. Sad.

Dino