Subject: Re: [lisp] Economic issues of long-path / stretched routing

From: David Meyer <[email-masked]>
Date: Sun, 25 Jan 2009 13:26:08 -0800

> Referring to the graphic Dino sent, which can also be found at
> http://www.lisp4.net/, one can see that ALT traffic between Savvis and
> Level(3) has several organizational AS-hops to traverse. 

	This is just an artifact of the way I built it. You could
	put up a ALT tunnel betwen Savvis and L3 if that was of
	interest (and Savvis & L3 wouldn't have to further
	advertise those routes to the rest of the ALT). Also,
	its a somewhat artifcial observation because the L3 box
	is in London, so I thought it made sense to home it to
	RIPE. If L3 had ALT routers in the US, the situation
	would be likely different.

	BTW, the current configuration of the ALT is a first
	cut. There may be better ways to do it, but I did want to
	get something running so we could gain some experience.

	Or am I answering the wrong question?

	Dave

_______________________________________________
routing-discussion mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/routing-discussion

---

From: Benson Schliesser <[email-masked]>
Date: Sun, 25 Jan 2009 17:53:49 -0600

John-

I agree, in LISP the volume of ALT traffic is very small versus the volume
of encapsulated data traffic. This is one of the primary reasons I like
LISP-ALT as a near-term solution, to be honest.

The ALT peering, though, isn't 100% clear to me yet. Sorry if I'm just slow
today or have forgotten something that's already been discussed. (The recent
holiday seems to have effectively cleared my mind.)

My single-homed xTR (savvis) has one ALT peer right now (asp) and he
announces an aggregate net that covers my EID prefix. That's great from a
routing scale perspective. However if I were to multihome to another net
(dmm, for example) wouldn't I have to announce my EID into the ALT via that
path if I wanted to continue serving that EID during an outage of my primary
(asp) connectivity?

At the very least it has to be announced such that my primary upstream sees
it and can divert map-requests to my secondary path. So do all of my xTRs
need to have ALT tunnels to all of my EID allocators in order to avoid
de-aggregation in the ALT tables while maintaining fault tolerant
connectivity? What happens if an AS-wide fault takes down all of my
allocator's ALT routers? I understand that my RLOCs are still aggregated by
the underlying network's routing regardless.

It would be good to discuss what all of this looks like for network
engineering purposes. For instance from the perspective of ALT traffic
engineering, an end-site xTR with end users surfing the web will probably
have a moderate collection of EID mappings that stay refreshed (Google,
Twitter, corp intranet, etc) and a small collection of mappings that get
used and expire (my blog, which nobody reads twice). This is different from
an xTR in front of a public website, which will basically build a huge
collection of mappings and have to worry more about the database size than
anything else.

Cheers,
-Benson

PS - I hate to say this, but I will volunteer to edit a draft covering this
topic if one or more LISP researchers will help write it.

On Jan 25, 2009 4:22 PM, "John Zwiebel" <[email-masked]> wrote:

FWIW:
The sub-optimal routing will only be for the LISP map-request.
The reply and all encapsulated data traffic is between RLOCs
The ALT is merely there to provide the correct mapping.

Also, since a map-reply from any ETR for a given site will
provide you with all the RLOCs for that site, there is no
need to advertise any sub-prefix into the ALT.  Rather you
would advertise the most highly aggregated route you have
for your EID-site.  So the ALT table should be considerably smaller.

On Jan 25, 2009, at 11:49 AM, Benson Schliesser wrote:

> > Dave- > > On Sun, Jan 25, 2009 at 3:26 PM, David Meyer <[email-masked]>
> wrote: >> >>       This is ...
>
> > _______________________________________________ > lisp mailing list >
> [email-masked] > https://www....
>

---

From: Benson Schliesser <[email-masked]>
Date: Sun, 25 Jan 2009 15:06:04 -0600

(This is a good place to segue, so pardon the borrowed thread...)

I have a concern about the economic aspects of stretch, specifically
related to peering in routing systems where topology follows address
assignment more closely than today's Internet. If this has already
been discussed I missed it and would appreciate a pointer. (I've heard
discussion of the performance issues, but not economic.) My concern is
not specific to LISP but is expressed nicely by LISP-ALT so I would
like to use that as an example here.

Referring to the graphic Dino sent, which can also be found at
http://www.lisp4.net/, one can see that ALT traffic between Savvis and
Level(3) has several organizational AS-hops to traverse. This isn't
the case under today's BGP4 routing because the Savvis and Level(3)
networks have direct peering. However, connectivity via the
ASP-ARIN-RIPE path doesn't use these peering interconnects. (FYI, it
seems to traverse networks such as NTT and Sprint, in case you were
wondering.)

In the Savvis-Level(3) case I believe the interconnects mentioned
above are all settlement-free, so the economic impact is primarily the
cost of usage-driven infrastructure items like router ports and
interconnects. But networks farther from the default-free zone are
likely to have more significant costs associated with transit
connectivity "upstream". Some of these networks (i.e. "tier-2"
providers) will have an existing set of settlement-free peers that
exists in order to reduce transit costs. But when these providers and
their downstream customers implement LISP-ALT there may very well be
an increase in traffic via their transit providers.

It seems to me that the driver behind this shift in traffic is the
network location of address assignment "authorities" and their ALT
routers. For instance, if ARIN's ALT router was only on-net behind a
single service provider network then all of the ARIN-served region
would need connectivity to that network. The trade/economic issues
should be obvious. Therefore I'm struck that any potential longer-path
solution to the address/topology scaling problem that moves beyond the
scope of the RRG (i.e. to the IETF) should provide input to the
addressing authorities on how to mitigate the resulting economic
issues. This might include, for instance, suggestions on where ALT
routers need to reside and what sort of peering/interconnectivty
should be in place.

Further, potential solutions should be considered for their ability to
reduce long-path traffic. Again using LISP as an example it is
important to consider how much traffic may follow the ALT path before
it switches to the direct LISP-encapsulated path. Quantifying this may
be difficult since it's in-part dependent on the latency of the ALT
path. But a relative view could be given.*

Has this issue already been discussed or addressed? I think that the
IETF is the right place to discuss the more concrete details of any
given solution (i.e. IETF LISP WG). But I would also like to see the
RRG consider these issues in abstract.

Cheers,
-Benson

* - For each new flow, given the flow's rate and size versus the
long-path latency and knowledge of how long it takes for a flow to
switch from the long- to short-path, as well as an estimate of the
number of "unique" flows (unique within the solution's context), a
relative view of traffic could be given. Comparison could then be
based on historical data gleaned from end-sites such as users and
datacenters to determine what the real impact may be.

On Jan 25, 2009 12:02 PM, "Dino Farinacci" <[email-masked]> wrote:

> Just a simple question: Is the ALT GRE-tunnel hierarchy a tree ?

It is not required, but it should follow the address allocation delegation.

What we have deployed on the pilot is in the enclosed diagram and the
EID-prefix assignments follow a registry allocation hierarchy. This
topology supports both IPv4 and IPv6.

Once the apnic box is up, the container with ntt, dscott, and iij will
connect to it.

Dino

lisp mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lisp

---

From: Benson Schliesser <[email-masked]>
Date: Sun, 25 Jan 2009 15:49:50 -0600

Dave-

On Sun, Jan 25, 2009 at 3:26 PM, David Meyer <[email-masked]> wrote:
> ...

I think you're answering the right question from the LISP-ALT
perspective. It theoretically exists for other solutions in the same
class, too, i.e. solutions that create a stretched path in order to
follow address-assignment topology instead of underlying forwarding
topology.

In the LISP-ALT case, however, I'd like to explore your answer a bit
further. Does this suggest that the answer is for my ALT AS to peer
with all of my existing Internet peers, if I want to avoid suboptimal
forwarding? If I did so, how would my ALT routing table size compare
to my current routing table size? I'm imagining them the same size
(i.e. comprised of supernets learned by my peers as well as
less-specifics learned when my peers' customers multihome) but I might
be missing something.

Cheers,
-Benson

---

From: John Zwiebel <[email-masked]>
Date: Sun, 25 Jan 2009 12:22:35 -1000

FWIW:

---

From: John Zwiebel <[email-masked]>
Date: Sun, 25 Jan 2009 15:02:37 -1000

On Jan 25, 2009, at 1:53 PM, Benson Schliesser wrote:

> My single-homed xTR (savvis) has one ALT peer right now (asp) and  
> he announces an aggregate net that covers my EID prefix. That's  
> great from a routing scale perspective. However if I were to  
> multihome to another net (dmm, for example) wouldn't I have to  
> announce my EID into the ALT via that path if I wanted to continue  
> serving that EID during an outage of my primary (asp) connectivity?
>

Yes you should, but that doesn't add another route to the ALT routing  
table.
> At the very least it has to be announced such that my primary  
> upstream sees it and can divert map-requests to my secondary path.
>
If your primary router goes down, yes.
> So do all of my xTRs need to have ALT tunnels to all of my EID  
> allocators in order to avoid de-aggregation in the ALT tables while  
> maintaining fault tolerant connectivity?
>
All of your xTRs need to connect to the ALT to get mapping  
information. (ie to send
map requests over the ALT and receive map-replies natively).  Each  
xTR would
advertise the same EID-prefix.  So no matter how many xTRs you have,  
only the
single EID-prefix is advertised. The ALT only carries EID-prefix  
routes (and next-hop
information for the ALT).  It need not carry the RLOC routes at all.   
Those are in the DFZ.
> What happens if an AS-wide fault takes down all of my allocator's  
> ALT routers? I understand that my RLOCs are still aggregated by the  
> underlying network's routing regardless.
>
Hopefully you'd have RLOCs from two different allocators.  When one  
goes down, the
other one is still available for map-requests.

Hope this helps

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 26 Jan 2009 08:34:15 -0800

> announces an aggregate net that covers my EID prefix. That's great  
> from a routing scale perspective. However if I were to multihome to  
> another net (dmm, for example) wouldn't I have to announce my EID  
> into the ALT via that path if I wanted to continue serving that EID  
> during an outage of my primary (asp) connectivity?
>

Yes, you would. And you would peer at the level before the aggregation  
level router. So in the real world, you would peer with sub-parts of  
your region's registry.

> So do all of my xTRs need to have ALT tunnels to all of my EID

Not divert, but you want the shortest path to your site for Map- 
Requests. So each of your xTRs will have GRE tunnels where their  
"tunnel source <foo>" configuration will have <foo> be the locator  
address from each attached upstream.

> maintaining fault tolerant connectivity? What happens if an AS-wide  
> fault takes down all of my allocator's ALT routers? I understand  
> that my RLOCs are still aggregated by the underlying network's  
> routing regardless.
>

Right, no deaggregation anywhere. You don't have to TE with more  
specific injection on the ALT, you use your tunnel locators for this.

Also, we want you to have a low opex xTRs, so when *you do not run BGP  
on your xTRs*, it will be your upstreams will "redistribute static tag  
<foo>". And they won't need to deaggregate as well because you will be  
*within* their aggregation level.

> engineering purposes. For instance from the perspective of ALT  
> traffic engineering, an end-site xTR with end users surfing the web  
> will probably have a moderate collection of EID mappings that stay  
> refreshed (Google, Twitter, corp intranet, etc) and a small  
> collection of mappings that get used and expire (my blog, which  
> nobody reads twice). This is different from an xTR in front of a  
> public website, which will basically build a huge collection of  
> mappings and have to worry more about the database size than  
> anything else.
>

Mappings get refreshed locally in the xTR based on data flow. So you  
don't need to continually Map-Request the site. Plus, you can use the  
old ARP trick. That is, give yourself enough time before the entry  
times out, to request to see if the mapping is still current. And only  
do that when there is data flow against the entry.

Dino

---

From: Scott Brim <[email-masked]>
Date: Tue, 27 Jan 2009 12:58:29 -0500

I don't know which of the lists this is going to should be trimmed, or
I would gladly do so.

> during an outage of my primary (asp) connectivity?

You peer with your upstream ALT routers based on prefixes they are
responsible for, and where the ALT routers are doesn't (necessarily)
follow physical topology.  For any prefix there will probably be more
than one upstream ALT router.  Yes, you announce your prefix to all
upstream ALT routers responsible for the shorter including prefix.
Table size should be minimal, depending on how things are configured
in your neighborhood.  It could just be default from each.  At most it
would be a set of highest-level prefixes, maybe some special ones, and
some longer ones for local efficiency.  

> that my RLOCs are still aggregated by the underlying network's

Allocation is decoupled from ALT routers.  RIRs may run high-level ALT
routers but there may be intermediate ones too.  For a particular
prefix there should be multiple redundant ALT routers.  For the most
part they need to be within the region in which they are most used,
even though some prefixes may be allocated and then move to completely
different continents.

> collection of mappings and have to worry more about the database size than

Right.  The range in possible tactics is huge, and they all work as
long as they speak the same protocol.  

Scott

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 28 Jan 2009 05:14:41 -0800

> some longer ones for local efficiency.

And your neighborhood really doesn't need back-doors so we can keep  
the ALT hierarchical by shaping it as an acyclic tree and not a mesh  
of trees or a mess of links connecting up and down at *different*  
hierarchical levels like the underlying Internet is configured today.

Dino