Subject: Re: [rrg] The Internet DFZ as a giant NBMA "link"

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 1 Dec 2008 06:26:27 -0800

>-----Original Message-----
>From: Scott Brim [mailto:[email-masked]]
>Sent: Sunday, November 30, 2008 9:56 PM
>To: Templin, Fred L
>Cc: RRG
>Subject: Re: [rrg] The Internet DFZ as a giant NBMA "link"
>
>Excerpts from Templin, Fred L on Sun, Nov 30, 2008 01:49:12PM -0800:
>> I have not seen this widely discussed, but in the map/encaps
>> approaches the Internet DFZ is essentially viewed as a giant NBMA
>> "link" that connects all *TRs. In other words, all *TRs on the
>> virtual link can reach each other in a single IP hop and can use
>> neighbor discovery mechanisms such as IPv6 ND.  This also paves the
>> way for new ND mechanisms such as SEND.
>
>How would this scale?  We're considering O(10^9) prefixes and rather
>more xTRs.

Simply add more xTRs and make sure their RLOCs are
published in the mapping database.

Thanks - Fred
[email-masked]

>> This model exactly parallels that which was established in ISATAP
>> [RFC5214], and has been further developed in RANGER/VET/SEAL. I made
>> an effort to articulate this back in 2002 in a note titled: "The
>> Inside-Out ISATAP Model":
>>
>> http://www.ops.ietf.org/lists/v6ops/v6ops.2002/msg00657.html
>>
>> but we seem to now be coming back around to the same concept.  The
>> main point of this note therefore is to (re)introduce the useful
>> concept of the Internet DFZ as a giant NBMA "link" that connects
>> routers the same as for any link.
>
>I'll take a look again.
>
>Scott
_______________________________________________
rrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/rrg

---

From: Yakov Rekhter <[email-masked]>
Date: Mon, 1 Dec 2008 09:53:23 -0800

Tony,

> |Recent postings suggest drop the first packet during map 
> |lookup. Oeps!!!!
> |I think first packet delivery is a MUST.
> 
> 
> The problem is that first packet delivery requires buffering.  To quote an
> esteemed colleague:  "Buffering bad."  ;-)

This discussion focuses on the implication of dropping or buffering
the first packet of a given flow. But there are cases where other
than the first packet of a given flow would need to be either
buffered or dropped.

Consider the scenario where an ITR goes down, and as a result *all*
the flows that used to go through that ITR would need to be shifted
to other ITR(s), and other ITR(s) have no mapping info for these
flows. What would these other ITR(s) do with the packets of these
flows as these other ITR(s) are in the process of acquiring mapping
info ? Would these other ITR(s) buffer all of them until they acquire
the mapping info ?  (with link speeds of of several Gb/s that may
be a *lot* to buffer). Or would they drop all of them until they
acquire the mapping info ? (with link speeds of several Gb/s that
may be a *lot* to drop).

Yakov.

---

From: Dan Jen <[email-masked]>
Date: Mon, 01 Dec 2008 10:14:17 -0800

On Mon, 2008-12-01 at 09:53 -0800, Yakov Rekhter wrote:
> What would these other ITR(s) do with the packets of these

The other option would be for the ITR to punt the packets to its default
mapper, as in APT.  The ITR's default mapper has the full mapping table,
and delivers the packet while sending the ITR the mapping info for its
cache.

Dan

---

From: "William Herrin" <[email-masked]>
Date: Mon, 1 Dec 2008 12:52:01 -0600

On Sun, Nov 30, 2008 at 11:52 PM, Scott Brim <[email-masked]> wrote:
> The issue will rarely come up.  In order for it to arise, a site
> border router must come up cold with no knowledge of prefixes where it
> wants to find its NTP servers.  Once it maps these prefixes, which
> will very probably be few and well-used, it will have mappings for
> them until it comes up cold again.  So as yet another alternatives,
> avoid completely cold restarts, or give it what it needs via
> configuration or synchronization.  There are probably other
> possibilities.

Scott,

It need only expire the cached map entry for inactivity. Which it will
tend to do over tens of minutes with no use and plenty of competition
from other maps, something typical of the traffic pattern for NTP. Nor
can you store the cached map for long: in a pull-cache system, cache
expiration is your only guarantor of convergence. Everything else you
throw at it is a heuristic that only works *most* of the time.

That's one of the things that makes NTP good for illustrating this problem.

I remind you about:
http://www.ops.ietf.org/lists/rrg/2008/msg00465.html

and the thread at:
http://www.ops.ietf.org/lists/rrg/2008/msg00509.html

Also, look at: http://bill.herrin.us/network/statechange.html for an
explanation of why you can't cache entries for very long.

Regards,
Bill Herrin

-- 
William D. Herrin ................ [email-masked]  [email-masked]
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004
_______________________________________________
rrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/rrg

---

From: "Christopher Morrow" <[email-masked]>
Date: Tue, 2 Dec 2008 03:31:54 -0500

On Mon, Dec 1, 2008 at 12:53 PM, Yakov Rekhter <[email-masked]> wrote:
> to other ITR(s), and other ITR(s) have no mapping info for these/

if the ITR goes down won't the IGP/EGP take care of shifting
some/all/most of this traffic away? for the ETR that might be more
complex, but for the ITR there is either a backup/neighbor/vrrp-pair
or another option (another link) for this traffic to go to, right?

(or there SHOULD be, or the  customer is in the same problem space as
they are in today's internet... their PE device died and has no
routes, so no traffic gets passed)

> flows as these other ITR(s) are in the process of acquiring mapping

I suspect that the same thing that happens today happens tomorrow,
packets/flows get dropped. Losing a router isn't a recoverable
situation today, if you have no routing info you are SOL.

> acquire the mapping info ? (with link speeds of several Gb/s that

agreed, this is a bunch of traffic, but it's a problem we have today
already. This isn't changed with any proposal going forward, as with
APT even you don't necessarily have enough routing info to see the
default-mapper at boot-time (depending on deployment scenarios of
course).

-chris

---

From: Michael Meisel <[email-masked]>
Date: Tue, 02 Dec 2008 00:49:10 -0800

> default-mapper at boot-time (depending on deployment scenarios of

Though I agree that many of the other problems above already exist
today, I don't think this last one does. If I understand correctly,
Yakov's concern isn't about boot time, it's about failover from one
active ITR to another. The original ITR has the appropriate mapping for
some prefix with a huge amount of data being sent to it. This ITR fails,
and the second ITR doesn't have the right mapping info. However, as Dan
already pointed out, this should not be a problem in APT, since default
mappers prevent ITRs from ever having to buffer packets.

-Michael

---

From: "Christopher Morrow" <[email-masked]>
Date: Tue, 2 Dec 2008 04:00:43 -0500

> active ITR to another. The original ITR has the appropriate mapping for

ah, sorry! yes, that was his question in grow/rrg I believe... I think
it makes sense to have the ability to share mapping info among the
'vrrp pairs'. That's not hard to do, it's just a feature request. In
practice the 'vrrp pair' probably is servicing traffic on both members
of the 'pair' and keeping the mapping info sync'd (within a reasonable
time lag) is sensible.

This isn't a change to ITR/ETR functionality, it's just an added
feature to the implementation. In fact, it maybe that an operator
wants to share the mapping info among devices in a 'region' or 'pop'
to avoid mapping request overhead in a local region... that fits into
the 'vrrp pair' model above as well.

Note: 'vrrp pair' doesn't have to be 'vrrp' it could be paired devices
(as seen from the CE side) in a provider network... Something like
diverse PE devices of course.

> mappers prevent ITRs from ever having to buffer packets.

sure, and I agree that APT has potential in certain deployment
scenarios, but in reality isn't likely the 'best' solution for
everyone, so isn't the 'only solution', and honestly isn't very
different from a LISP deployment on it's own... save the ::0/0 ->
to-the-left with TTL 999999999 entry in the mapping database.

-chris

---

From: Yakov Rekhter <[email-masked]>
Date: Tue, 2 Dec 2008 08:03:46 -0800

Chris,

> >>>> The problem is that first packet delivery requires buffering.  To quote 
> >>>> an esteemed colleague:  "Buffering bad."  ;-)
> ah, sorry! yes, that was his question in grow/rrg I believe... 

That is correct.

> I think it makes sense to have the ability to share mapping info among 
> the 'vrrp pairs'. That's not hard to do, it's just a feature request. In

Consider the case where ITRs belong to ISPs, and customers who are
multi-homed to different ISPs. In this case ITRs in a 'vrrp pair'
may belong to different ISPs.

Moreover, a given ITR may need to participate not in one, but in
multiple 'vrrp pairs', as different customers connected to that ITR
may have their other connection(s) to different (other) ITRs.

Do you think that is still "not hard to do" ? 

Yakov.

---

From: "Christopher Morrow" <[email-masked]>
Date: Tue, 2 Dec 2008 11:16:57 -0500

> Do you think that is still "not hard to do" ?

the multi-isp situation is harder, indeed... in practical cases some
of the missing data on the alternate provider should exist (due to
other customers on the relevant edge device(s) ), but yes some traffic
will be dropped.

Do you have an alternate possible solution for that case?  (someone
off-list pointed out to me that the ITR might not actually be the PE,
which side-steps this situation I think, right?)

As to the second portion, whether it's one or more 'vrrp pair' devices
there exist already many ways to distribute this sort of info... I
don't see that as a large problem, no.

-Chris

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 2 Dec 2008 19:35:15 -0800

Even though I didn't make Michael Meisel's short-list of

>and delivers the packet while sending the ITR the mapping info for its

If we are going to do this, I'd like to see it done in a way
that preserves the model of the DFZ as a giant NBMA link on
which all xTR's are one-hop neighbors. That means that, when
the ITR punts a packet to the Default Mapper (DM), I want for
the DM to *not* decapsulate the packet but rather examine its
mapping tables based on the inner destination address and
*rewrite* the outer destination address to an RLOC of the
ETR. So, the packet coming in to the DM would have:

  inner_src=EID(A)         inner_dst=EID(B)
  outer_src=RLOC(ITR)      outer_dst=RLOC(DM)

and (after the DM does the mapping lookup) the packet coming
out of the DM would have:

  inner_src=EID(A)         inner_dst=EID(B)
  outer_src=RLOC(ITR)      outer_src=RLOC(ETR)

In this way, the DM is "off-link" from the viewpoint of the
ETR, and the ETR gets to perform link-scoped operations (e.g.,
IPv6 ND) with the ITR as if it were an on-link neighbor. Is
this what you had in mind?

The alternative is for the DM to decapsulate, perform the
mapping, then re-encapsulate while injecting its RLOC as
the outer_src in the transaction, which makes it such that
the ITR is off-link from the viewpoint of the ETR - not
so good for link-scoped operations.

I would prefer option A, if that is OK with you.

Fred

---

From: Dan Jen <[email-masked]>
Date: Tue, 02 Dec 2008 19:55:29 -0800

Hi Fred,

On Tue, 2008-12-02 at 19:35 -0800, Templin, Fred L wrote:
> So, the packet coming in to the DM would have:
> 

Yes it is, for the reasons you stated.  However our current APT design
allows TRs to communicate with the default mappers when necessary(for
failure handling and such).  Our recent focus on an incrementally
deployable design from evolutionary principles may cause us to
re-examine these choices though.

> I would prefer option A, if that is OK with you.

Very okay, unless we discover good arguments to do otherwise.  Thanks
for putting thought into this!

Dan

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 2 Dec 2008 20:18:20 -0800

Dan,

OK to what you said, but I made a couple of conceptual leaps
in my last message that might bear a bit more discussion.

As can be told by most of my messages to this list, I am
fixated on IPv6 as the inner protocol and IPv4 as the outer
protocol. That said, when I said that the ITR and ETR would
appear as on-link neighbors, what I meant was that the ETR
(on receipt of a packet with inner_src=RLOC(ITR)) can
construct an IPv6 link-local ISATAP address of the form:

  fe80::0200:5efe:RLOC(ITR)

and use it as the destination address of an IPv6 ND message
such as an RS/RA. The encapsulated ND message would have:

inner_src=fe80::0200:5efe:RLOC(ETR) inner_dst=fe80::0200:5efe:RLOC(ITR)
outer_src=RLOC(ETR)                 outer_dst=RLOC(ITR)

So, the ITR and ETR get to do an IPv6 RS/RA exchange with
each other over the ISATAP link (i.e., the Internet DFZ)
and exchange nice information like route information options
useful for populating the FIB. If they are smart about it,
they can even arrange to use SEND with CGAs so they can each
know they are talking to the correct neighbors. Finally, if
the DM acts as an IPv6 ND proxy, the ITR could even send an
RA as its first packet out to the ETR (via the DM) so the
ETR can get an early start on populating its FIB.

This just to give a bit more insight as to my thought process.

Thanks - Fred

>From: Dan Jen [mailto:[email-masked]]
>Sent: Tuesday, December 02, 2008 7:55 PM
>To: Templin, Fred L
>Cc: Michael R Meisel; RRG
>Subject: RE: [rrg] NTP and first packet delivery
>
>Hi Fred,
>

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Dec 2008 07:42:24 -0800

Dan,

>From: Templin, Fred L
>Sent: Tuesday, December 02, 2008 8:18 PM
>To: Dan Jen
>Cc: RRG
>Subject: Re: [rrg] NTP and first packet delivery
>
>Dan,
>

I meant "outer_src=RLOC(ITR)"; not "inner_src".

>ETR can get an early start on populating its FIB.

Let's forget the bit about "IPv6 ND proxy" for now; I'm
not seeing the immediate functional value to warrant the
added complexity. Everything else is still good, though.

Fred

---

From: Scott Brim <[email-masked]>
Date: Wed, 3 Dec 2008 13:18:35 -0500

Excerpts from William Herrin at 12:52:01 -0600 on Mon  1 Dec 2008:
> tend to do over tens of minutes with no use

Correct me if I'm wrong but it seems to me that whatever higher
stratum NTP servers are used will be (1) well used, and (2) in popular
prefixes.  What servers do you use?  My little MBP uses apple.com and
nist.gov.  I doubt that they would be removed from the cache.  If a
site communicates with zillions of others, so there is a lot of churn
in the cache, it probably wants to reconsider its caching strategy in
the first place, but second, steps can be taken to make sure a map for
a particular prefix doesn't get squeezed out.  (That doesn't mean it
doesn't get refreshed as TTL expiration approaches.)

Scott

---

From: "William Herrin" <[email-masked]>
Date: Wed, 3 Dec 2008 17:28:25 -0500

On Wed, Dec 3, 2008 at 1:18 PM, Scott Brim <[email-masked]> wrote:
> nist.gov.  I doubt that they would be removed from the cache.

In Red Hat Linux, the default installation is:

server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org

pool.ntp.org consists of a large, random assortment of volunteers who
have decided it would be cool to run a public NTP server. Two lookups
over time get two more or less randomly chosen sets of IPs:

nslookup 0.pool.ntp.org
Server:         10.187.99.2
Address:        10.187.99.2#53

Non-authoritative answer:
Name:   0.pool.ntp.org
Address: 208.38.65.35
Name:   0.pool.ntp.org
Address: 4.79.132.221
Name:   0.pool.ntp.org
Address: 64.247.17.255
Name:   0.pool.ntp.org
Address: 65.255.217.202
Name:   0.pool.ntp.org
Address: 72.167.54.201

nslookup 0.pool.ntp.org
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   0.pool.ntp.org
Address: 65.111.164.223
Name:   0.pool.ntp.org
Address: 65.255.217.202
Name:   0.pool.ntp.org
Address: 69.31.43.10
Name:   0.pool.ntp.org
Address: 72.167.54.201
Name:   0.pool.ntp.org
Address: 75.144.70.35

Bear in mind that an individual ntp client in normal operations backs
off to 17 minutes between packets and can be configured to go more
than a -DAY- without sending a packet.

> If a

Here's a free solution:

Don't remove entries from the cache unless you run out of memory.
After the short, perhaps 30-second TTL expires, the next packet you
send to that destination triggers a map refresh in order to statisfy
http://bill.herrin.us/network/statechange.html . The results of the
refresh will, if necessary, delete the old entry. While you wait for
the results of the refresh, keep using the expired map. 99.9% of the
time this will be the correct action. For the rest, its a normal,
everyday black hole anomaly that clears in a few hundred milliseconds
when the refresh comes in.

The hitch is that lisp sits rather closer to the core than most of the
pull-cache solutions that have been discussed. Are you sure you're
going to have enough cache to keep 20 or 30 minutes worth of *all* the
destination addresses that the site's folks have sent packets to? I
think I'd want to see some metrics on how many unique destination
addresses leave the biggest site that would be behind just one ITR.

Regards,
Bill Herrin