Subject: Re: [rrg] IRON-RANGER - Route Reflectors & DNS?

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 23 Mar 2010 19:31:58 -0700

Hi Robin,

> -----Original Message-----
> From: Robin Whittle [mailto:[email-masked]]
> Sent: Tuesday, March 23, 2010 4:28 PM
> To: RRG
> Cc: Templin, Fred L
> Subject: IRON-RANGER - Route Reflectors & DNS?
>
> Short version:    Fred proposes BGP with Route Reflectors for
>                   IRON-RANGER.  I suggest using DNS to enable
>                   IR(EID) routers to discover the IR(VP) routers
>                   which handle the VP which covers their EID prefix.
>
> (This is the same as a message I posted 11 hours ago, which did not
> appear on the list.  I tried again, but with the same result.  Now I
> am posting from a different email address rw2 which will disappear as
> soon as my ordinary address works for the RRG list.)
>
>
> Hi Fred,
>
> I am replying to your msg06326 with a new title, because you are now
> proposing an IRON-RANGER (I-R) overlay structure which is different
> from what I understood you were previously proposing.
>
> Firstly, a summary of my understanding of the previous plans - which
> is not necessarily the same as what you were planning.
>
> Then, my understanding of your new plans, based on BGP with RRs - and
> some questions about how it would work and scale well.
>
>
> I-R overlay with BGP
> --------------------
>
> In the past, as best I understood it, you were proposing that each
> IRON router would have tunnels to a number (2 to 10, typically, I
> guess) typically nearby IRON routers and that each would run their
> I-R BGP instance over these links.  The Internet role of these IRON
> routers would not necessarily be at the borders of ASes connecting to
> other ASes - they could also be entirely within their AS.
>
> The IP address each IRON router uses in the Internet is the same as
> it uses in the I-R overlay network.
>
> Each IRON router would discover, via the 2 to 10 links to
> neighbouring IRON routers, all the VPs (Virtual Prefixes) which were
> advertised in the I-R overlay.
>
> Typically each VP would be advertised by two or perhaps more IRON
> routers, so a given IRON router XX would discover, via ordinary BGP
> best-path announcements of its 2 to 10 neighbours, a best path to one
> of the two or more IRON routers which advertise each VP.  This raised
> some problems with the system by which you wanted some IRON routers
> (those delivering packets to end-user networks using I-R "edge"
> space) to register the EID prefix of each such end-user network with
> each of the two or more IRON routers advertising this VP in the I-R
> overlay network.  In that system, each IRON router needed to know the
> IP addresses of every one of the IRON routers advertising a
> particular VP.  But see below where I suggest using DNS for this
> second purpose.

There has been yet a third approach for this, which I
think really alleviates all of the complexity we have
been discussing. Since the list of VPs as well as the
individual VP->RLOC mappings changes only very rarely,
it should be sufficient for every IRON router to simply
upload a flat file containing the full mapping table
from a content server when it boots up. The IRON router
should probably also check for deltas periodically to
detect additions and deletions. That's about it.

> I-R overlay with BGP and RRs
> ----------------------------
>
> There is new terminology for the various roles an IRON router could play:
>
> > From now on, I will
> > use IRON Router (IR) as the base acronyom and extend
> > it with parenthesized functions as follows:
> >
> >   IR(VP) - an IR that advertises one or more VPs into
> >            the IRON BGP RIB
> >
> >   IR(GW) - an IR that connects the IRON to the DFZ
> >
> >   IR(EID) - an IR that connects a customer EID network
> >
> >   IR(RR) - an IR that is a BG Route Reflector in the IRON
> >
> > Note that it is possible for a single IBR to serve
> > *more than one* of these roles, but I think that it
> > is sufficient to discuss one function at a time so
> > I don't think we will need even more complex TLAs.
>
> I understand that the IR(GW) role is roughly equivalent to that of an
> Ivip DITR or LISP PTR in that it advertises the full range of I-R
> "edge" space to neighbouring ASes and so accepts packets sent from
> any of those networks which have not been upgraded with IRON routers.
>
> But see further discussion below where you mention:
>
>    > IR(GW)s forward data packets from the IRON into the DFZ.
>
>
> In addition, there are the IRON routers which perform none of the
> above roles, but which advertise the IRON "edge" space in the
> internal routing system of the AS they are located within, and so
> accept packets sent from within that AS which are addressed to I-R
> EID prefixes.   I figure any IRON router does this, no matter what
> other roles it may play.

I think what you are describing here is the same as what
I meant by IR(EID).

> The new structure is based on BGP Route Reflection: RFC 4456.
>
> > The way the IRON BGP is going to work is as follows. There
> > will be a well-known FQDN, e.g., "isatapv2.net", which
> > resolves to a list of IP addresses - possibly many. Each
> > IP address is the RLOC of an IR(RR) per RFC4456. Quoting
> > from RFC4456, Section 6:
> >
> >    "In a simple configuration, the backbone could be divided into many
> >    clusters.  Each RR would be configured with other RRs as Non-Client
> >    peers (thus all the RRs will be fully meshed).  The Clients will be
> >    configured to maintain IBGP session only with the RR in their
> >    cluster.  Due to route reflection, all the IBGP speakers will receive
> >    reflected routing information."
> >
> > and this is precisely the arrangement that will be used
> > for the IRON "backbone". In particular, upon startup each
> > IBR(RR) resolves the name "isatapv2.net", and forms a full
> > mesh of BGP sessions with all other IBR(RR)'s.
>
> I assume that the IP address used on the I-R overlay network by the
> IRON router performing an IR(RR) role will be the same as that
> router's Internet address.
>
> In the conventional use of BGP Route Reflection, which is just within
> a single AS, each RR runs an iBGP session with every other RR.
>
> This fully meshed arrangement for I-R would set some kind of limit on
> the number of RRs in the entire system.  I guess the maximum figure
> would be 50 to 100 or so.  (Below, it turns out this was a good guess.)
>
> For a single router to maintain 50 iBGP sessions would be quite
> demanding, I think - since each one concerns however many VPs are
> advertised in the I-R overlay, and so takes up RAM in the router's
> CPU and involves communication and processing effort whenever a
> best-path changes.
>
> From what I understand about Route Reflection in an ordinary BGP AS
> setting (not the IRON RANGER intended usage) when a RR router NN has
> a client router XX, RR sends XX its best path for all advertised
> prefixes.  Then, in an ordinary AS setting (not as you will be using
> Route Reflection in I-R), when the XX client router forwards a packet
> matching the prefix of a given best-path, it will do so according to
> that best-path, which will be through NN, which is the next hop in
> the best-path it received.
>
> This pattern will not be followed in the I-R use of Route Reflection,
> as far as I know.  If an IRON router QQ is a Route Reflector, with a
> client IRON router YY, QQ will send YY a best-path (with QQ as the
> next-hop) for some prefix P.  The next-hop is not important to YY
> because its method of forwarding a traffic packet is to tunnel it
> directly to the destination router, whose IP address appears in the
> best-path, rather than forward it to the next-hop router.   So if
> three IRON routers AA, BB and CC all advertise a VP "P" and of these,
> the RR router QQ has its best path for this prefix to BB, then QQ
> will send YY a best path with BB as the destination router.  QQ won't
> know about AA or CC, but it will be able to tunnel traffic packets
> whose destination address matches P to the destination router BB -
> via using SEAL over the Internet.
>
> While conceptually it might be said that YY is tunnelling a packet to
> BB over the IRON overlay network, YY is only using the overlay to
> find the address of BB.  The SEAL tunneling is done via the Internet,
> and this does not necessarily involve the packet going via QQ as the
> next hop.  YY could be in a completely different AS network from QQ.
>
> I understand the purpose of the new arrangement is as before, to
> allow all IRON routers to find, for each VP, one of the typically two
> or more IRON routers which are playing the IR(VP role for this VP.

We're not going to use BGP over the IRON anymore at all;
the whole map will be kept in a flat file.

> So there still needs to be some new mechanism for IRON routers which
> play the IR(EID) role to register their EID with the typically two or
> perhaps more IRON routers playing the IR(VP) role for the VP which
> covers the EID prefix in question.
>
> Below, I contemplate using DNS for this purpose - and you made a
> similar suggestion for IRON-RANGER when discussing Ivip in your
> recent message.
>
> In the new model, I understand that the entire set of IRON routers is
> split up into clusters, with each cluster having a single IRON router
> playing the IR(RR) role and all the other IRON routers in the cluster
> being a client of that IR(RR) role router.
>
> However, these clusters need not be within a single AS network, or
> even be restricted to any part of the Net.
>
> You suggest below how you would give each of these client routers the
> ability to use another IR(RR) in the even that its current one fails.
>  RFC 4456 (Section 7) mentions two or more RR routers in a cluster
> which work together and which prevent routing loops between them by
> using a common 4 byte CLUSTER ID.
>
> I guess this approach could be used in I-R, if we assume that any
> given IRON router which is not playing the IR(RR) role will be
> configured by some means to tunnel to the one set of IR(RR) routers
> for whichever cluster it is within.
>
> However, I think the goal is to make any non-IR(RR) IRON router be
> able to get best-paths from two or more widely separated IR(RR)
> routers, which wouldn't I guess be using the same cluster ID.  So I
> think each such client router should not pass on best-paths learned
> from one IR(RR) router to another.
>
>
> > Then, when each IBR(VP) starts up it likewise resolves the
> > name "isatapv2.net" and forms a BGP session with one or
> > a few IR(RR)s that are "nearby" (the IR(VP) can decide
> > for itself what constitutes "nearby"). The IR(VP)s then
> > advertise their VPs into the IRON, and the IR(RR)s ensure
> > that all VPs are reflected to all IR(VP)s.
>
> OK - I will assume there is some way the DNS can return dozens or
> hundreds of IP addresses of all the IRON routers playing the IR(RR)
> role.  That wouldn't fit into a single DNS reply packet - I have not
> studied how DNS returns more information than would fit into a single
> packet.  I understand that you plan for 50 to 100 IR(RR) routers.
>
> I guess a given IRON router playing the IR(VP) role which connects to
> two separate IRON routers playing IR(RR) roles will act like a client
> to each one separately, and would not pass on any best paths back to
> the other IR(RR):
>
>
>    IR(RR)  [AA]------\
>                      [GG]  IR(VP)
>    IR(RR)  [BB]------/
>
> See below where this guess about an IRON router GG using two or more
> Route Reflectors turns out to be valid:
>
>       > When each IR that needs to form a tunnel with an IR(RR)
>       > comes up, it checks through the list of all IR(RR) IP
>       > addresses and picks one or a few that are "nearby". It
>       > is up to each IR to decide what is "nearby".
>
>
> So GG would be like a client to AA and a client to BB, getting a best
> path from both AA and BB for each of the VPs.  It wouldn't advertise
> any best paths for VPs other than the ones it was playing the IR(VP)
> role for.
>
> It wouldn't matter that GG it gets two best-paths, one through AA and
> the other through BB, for each VP other than its own ones, because it
> is not going to use the paths themselves.  It is only using the best
> paths to find out the IP address of the "nearest" of the two or more
> IRON routers playing the IR(VP) role for each VP which it is not
> advertising itself.  However, the above arrangement may result in GG
> getting best-paths for two different routers playing the IR(VP) roles
> for a given VP.  Then it would need to choose one.
>
>
> > After all of the VPs have been disseminated to all of the
> > IR(VP)s, then the IRON-RANGER data packet forwarding and
> > route optimization can be coordinated in the forwarding
> > plane. That's it.
>
> But to the above description, I think you would need to add how the
> IRON routers which are not playing the IR(RR) and IR(VP) roles are
> brought into the overlay network.  These may be ordinary IRON routers
> with no other roles, or they could be playing the IR(GW) role and/or
> the IR(EID) role.
>
> I guess you could have them look up the full list of IRON routers
> playing the IR(RR) role and decide which of these was the "closest".
>
> There are potentially hundreds of thousands of IRON routers - all but
> those playing the IR(RR) role - which need to choose a single IR(RR)
> router as being "closest".  I think there may be scaling problems in
> doing this, since each one might have a hundred or so IR(RR) routers
> to choose from - but presumably this process is spread out due to not
> many of these routers being rebooted at any one time.   It is
> probably not essential that the absolute "closest" one be chosen -
> but we don't want IRON routers in Siberia choosing an IR(RR) router
> in South Africa.

We won't need this anymore at all if we go with the flat
file; all IRs will have the full map to all other IRs.
There will be no need for IR(RR)s.

> Below I will assume that not just the IR(VP) routers, but all routers
> which are not playing the IR(RR) role also need to choose at least
> one IR(RR) router to tunnel to, and so become a client of the IR(RR).
>  So each such non-IR(RR) IRON router decides becomes a member of the
> cluster of each IR(RR) router it chooses.
>
>
> In an ordinary BGP network, with may ASes and many links between many
> routers, when there are several routers advertising a given prefix,
> any one other router will typically choose a best-path (and advertise
> this to its neighbours) of the advertising router which is "closest".
>   This would typically be the advertising router with a path
> involving the fewest number of ASes.
>
> However, in the I-R overlay network with Route Reflection, as best I
> understand it, the full set of IRON routers can be divided into four
> subsets regarding a given VP:
>
>    1 - Those which advertise the VP (the IR(VP) role) and are not
>        playing the IR(RR) role.
>
>    2 - Those which advertise the VP and are playing the IR(RR) role.
>
>    3 - Those which do not advertise the VP and are not playing the
>        IR(RR) role.
>
>    4 - Those which do not advertise the VP and are playing the
>        IR(RR) role.
>
> Those in subsets 3 and 4 need to find best paths to one of the
> routers in subsets 1 and 2.  Presumably there will be two or more
> routers in the union of subsets 1 and 2.
>
> I am unsure of what ASNs all these routers will be using.  Will they
> use the ASNs of their own ASes?  Or will those of a given cluster use
>  one AS, and those of another cluster another AS?
>
> Either way, the BGP metric of distance - the number of ASes in a best
> -path - is not going to be very meaningful for any router when
> choosing between multiple best paths.
>
> All the routers playing the IR(RR) role are fully meshed.  Those in
> subset 3 simply get a best-path from their one IR(RR) role router -
> unless they are using two or more, as I suggested in the diagram above.
>
> Assuming for simplicity they are just using one IR(RR) router, then
> every router in the subset 3 above simply gets the best-path for a
> given prefix which is chosen by its single IR(RR) router.   From that
> single IR(RR) router, every router performing the IR(VP) role for a
> given VP is either one hop or two hops away.  Those IRON routers in
> subset 2 are all one hop away, since all IR(RR) routers are fully
> meshed.  If one or more IRON routers are in subset 1, then these are
> two hops away, unless they are a client of the current IR(RR) router.
>
> So I think the system will reliably provide a single best-path to
> every IRON router for every VP, but that this will not necessarily be
> for the IR(VP) router which is closest.
>
>                          [XX]
>                         /
>           [AA]------[CC]-----[YY]
>                 \/
>                 /\
>  [WW]-----[BB]------[DD]-----[ZZ]
>
>
> AA, BB, CC and DD are playing the IR(RR) role, and so are fully
> meshed with SEAL tunnels to carry BGP messages and any tunneled
> traffic packets each of these needs to send to any of the others.
>
> If YY is not playing the IR(VP) role for a given VP "P", then the
> question is which of multiple best-paths will it get for P, depending
> on which other routers are playing the IR(VP) role for P.
>
> YY's Route Reflector is CC.  If CC is playing the IR(VP) role for P,
> then YY will get a best-path to CC.
>
> Otherwise, if CC is not playing the IR(VP) role, then YY will get
> whichever best-path CC chooses.  If all the routers were using ASNs
> from their own ASes, then there could be various scenarios.  But YY
> won't necessarily get a best path to the topologically closer router
> playing the IR(VP) role.
>
> The closest one may be chosen if client routers (such as WW, XX, YY
> and ZZ) typically choose a nearby router to be their IR(RR) router
> and if, in this example, either CC or XX is advertising VP "P".
> Presumably, if CC were advertising P, then CC would give itself as
> the destination router in its best paths.  So far, so good.   Also,
> if AA, BB, CC and DD were not advertising P, but XX was, presumably
> CC would send to all its clients, such as YY, a best path to XX.
>
> But note that for CC, all the routers XX, AA, BB and DD are a single
> hop away, and that all the remaining IRON routers are only two hops
> away.  There's no way in BGP that CC, if it was in Glasgow, could
> tell that BB was in London (and so presumably WW is not too far away)
> and that DD was in New Zealand (and so presumably that ZZ was a long
> way from Scotland).

There are no IR(RR)s anymore, so no concerns for making
terribly wrong guesses.

> >>> but only some IBRs have knowledge of prefixes
> >>> advertised within the DFZ.
> >>
> >> I don't think any IRON routers need to know what prefixes are
> >> advertised in the DFZ, since they don't forward packets to DFZ routers.
> >
> > IR(GW)s forward data packets from the IRON into the DFZ.
>
> I don't understand this at all.  My understanding of the IR(GW) role
> is to advertise the I-R "edge" space in the DFZ and so attract
> packets sent from networks without IRON routers, where those packets'
> destination addresses match any of the I-R EID prefixes.

That's correct; I meant to say: "IR(GW)s forward data packets
between the IRON and the DFZ.".

> I don't see any need for an IR(GW) router, or any other IRON router,
> to accept packets from the "IRON" network and to forward them to
> non-IRON routers of other ASes.

If the packet originates from an IRON network and is
destined for a non-IRON end system, it will need to
be forwarded by an IR(GW).

> As far as I know, all IRON routers are connected to the Internet, so
> if they have a packet to send to any global unicast address which is
> not part of the I-R "edge" subset, they do so using their ordinary
> forwarding techniques.

All IRON routers are connected to the IPv4 Internet, yes.
But, not all will necessarily connect to the IPv6 Internet.
So, there will need to be a set of IRs that provide a
default route to the IPv6 Internet - right?

> As far as I know, no traffic packets actually traverse the I-R
> overlay network as such, in terms of going from one IRON router to
> another via multiple other IRON routers - since whenever an IRON
> router (either playing the IR(VP) or IR(GW) roles, or playing no such
> role and just handling a packet which arrived from its local routing
> system) needs to get a traffic packet to some other IRON router
> (playing the IR(VP) or IR(EID) roles) then it simply uses SEAL to
> tunnel the packet directly to the destination router, via the Internet.

Right. I consider the act of using SEAL to tunnel the
packet directly to the destination router as "traversing
the IRON". Consider that there is a virtual full mesh of
tunnels between all IRs, but that most of the time the
vast majority of tunnels are dormant and not represented
in soft state. It is this virtual full mesh that I
consider to be "The IRON".

> (This is leaving out discussion of an IPv6 IRON router tunneling to
> an IRON router with an IPv4 address which can forward packets to an
> enterprise network with IPv6 I-R EID space.)
>
> >>> This latter class is known as
> >>> IBGs, and they advertise "default" into the IRON.
> >>
> >> Yes, but this is for the purpose of being like an Ivip DITR or LISP
> >> PTR, as described above.  They don't forward packets to DFZ routers
> >> so as far as I know, they don't need to know what prefixes their DFZ
> >> router neighbours are advertising best paths for.
> >
> > Yes they do. Again, the IRON is used for both the control
> > and forwarding planes.
>
> In my understanding of the control plane and forwarding plane actions
> of IRON routers, as just mentioned, that IRON routers tunnel directly
> via the Internet to other IRON routers.  For BGP messages (control
> plane) these flow directly over these tunnels.  In the above diagram,
> BB has BGP sessions with WW, AA, CC and DD.  So all control plane
> packets go via a single tunnel, via the Internet.  There BGP control
> plane does propagate information, which propagates from one IRON
> route to the next, but each actual BGP session is over a direct
> tunnel, between IRON routers, using SEAL over the Internet.
>
> For the "forwarding plane" - when the packet is a traffic packet -
> the ETE is either:
>
>    1 - An IRON router playing the IR(VP) role.  If it is also
>        playing the IR(EID) role for the destination EID prefix,
>        it delivers the packet directly to the destination network.
>        If not (which would typically be the case) it tunnels it to an
>        IRON router which is playing the IR(EID) role.  (In both cases
>        it also sends a "route redirection" packet to the ITE IRON
>        router, to give it "mapping" so that ITE router will be able
>        to tunnel other traffic packets for the same destination EID
>        prefix directly to the proper IR(EID) router.)
>
>    2 - An IRON router playing the IR(EID) role for the destination
>        EID prefix.  This delivers the packet directly to the
>        destination network.
>
> So I can't see any instance where an IRON router delivers a traffic
> packet "to the DFZ" - that is, to any ordinary Internet router of
> another AS.
>
> I see three types of packet being sent in tunnels between IRON routers:
>
>   1 - BGP messages, always between the two tunnel endpoints.
>
>   2 - Traffic packets as just mentioned.
>
>   3 - Route redirection packets with "mapping" as just mentioned.
>
> There may also be some SEAL things I have forgotten.  But in all
> cases, for an IRON router to send a packet to another IRON router, it
> tunnels with SEAL through the Internet.

Yes.

> >>>> I think the reasoning for this must come from your mixed IPv4 / IPv6
> >>>> plans, which I have tried to avoid thinking about so far.
> >>>>
> >>>> Can you explain more about your vision for this?
> >>>
> >>> My reasons for thinking so strictly about mixed IPv4
> >>> and IPv6 was the nice property of stateless address
> >>> mapping when only an IPv6 address is known and not the
> >>> corresponding IPv4 address. However, with a routing
> >>> protocol now in use in the IRON we have state - so, my
> >>> rationale no longer applies.
> >>
> >> OK - but isn't "stateless address mapping" what is contemplated below
> >> for these?:
> >>
> >>     IPv6-EID/IPv4-RLOC
> >
> > Yes, because an IPv4-RLOC address can fit inside of
> > an IPv6-EID address.
> >
> >>     IPv4-EID/IPv6-RLOC
> >
> > No, because an IPv6-RLOC address cannot fit inside
> > of an IPv4-EID address.
>
> Ahh . . . OK.
>
>
> >>> With this in mind, IRON
> >>> applies equally well for IPv6-EID/IPv6-RLOC, IPv4-EID/
> >>> IPv4-RLOC and IPv6-EID/IPv4-RLOC (however, I need to
> >>> think more about IPv4/IPv6).
> >>
> >> IPv6-EID/IPv6-RLOC  This is what would happen if I-R was purely
> >>                     for IPv6.
> >>
> >> IPv4-EID/IPv4-RLOC  This is what would happen if I-R was purely
> >>                     for IPv4.
> >>
> >> IPv6-EID/IPv4-RLOC  I understand this as the ability of the mapping
> >>                     (which the VP router always has, as developed
> >
> > VP router == IR(VP) in my nomenclature.
> OK.
> >>                     the potentially multiple "DEL" router
> >
> > DEL == IR(EID) in my nomenclature.
> OK.
> >>                     registrations for a given EID prefix, and which
> >>                     it sends as "mapping" - AKA route redirection -
> >>                     to any IRON router which tunnels a traffic packet
> >>                     to this VP router) to tell IRON routers to
> >>                     somehow tunnel IPv6 traffic packets to an IRON
> >>                     router (this is where I want to use my DEL term)
> >>                     which will deliver them to an EID-using
> >>                     enterprise network which is an IPv6 network, but
> >>                     which presumably doesn't have native IPv6
> >>                     connectivity since this delivering role IRON
> >>                     router ("DEL"!) is on an IPv4 address.
> >>
> >>                     So this is to support isolated IPv6 networks
> >>                     which use I-R "edge" (EID) space and which
> >>                     receive their incoming packets via an IPv4
> >>                     service.  (Or potentially a multihomed such
> >>                     network where one or more of its "DEL" routers
> >>                     is on IPv4, rather than IPv6.)
>
> I understand that there would be some kind of IPv6 RLOC address for
> what would normally be the IPv6 IR(EID) router, but because it is in
>   a particular prefix, it really means to take 32 bits from this
> address and deliver the packet by some means to an IRON router
> playing the IR(EID) role for this IPv6 I-R EID prefix, but which is
> itself on an IPv4 address.
>
>
> >> IPv4-EID/IPv6-RLOC  This would be an I-R "edge" using (EID)
> >>                     network which lacked IPv4 connectivity (at
> >>                     least for this particular "DEL" router) and
> >>                     so which accepted incoming packets via such
> >>                     a router on an IPv6 address.
> >>
> >>                     Maybe a MN doing IPv4 applications when it is
> >>                     physically connected only to an IPv6 network.
> >>
> >>                     Or a non-mobile network which for some reason
> >>                     only has an IPv6 service, but wants to run
> >>                     IPv4 space, using I-R "edge" EID space.
> >>
> >>
>
> >>> I will have to finally take the time to understand Ivip.
> >>> I will try to do so soon so I can converse with you on
> >>> more even terms.
> >>
> >> OK - I would really appreciate this.
> >
> > OK.
>
> Thanks for doing this - I will reply soon.
>
>
> >>>> However, I think it is wildly unrealistic to assume that IPv4 will
> >>>> die or become anything but *the* Internet everyone relies upon for a
> >>>> very long time, perhaps forever.  I am not saying this is a good thing.
> >>>>
> >>>> If you can articulate your vision for mixed IPv4 and IPv6 IRON-RANGER
> >>>> operation, I can go along with it.  But I don't believe at all that
> >>>> IPv6 will take over from IPv4 for most end-users before 2020.  As I
> >>>> mentioned, there's still a lot of unused advertised space - and (I
> >>>> assume) unused unadvertised - global unicast IPv4 address space.
> >>>>
> >>>> I can't envisage a situation where it will be better to sell ordinary
> >>>> (non-mobile) users purely an IPv6 service, without even behind-NAT
> >>>> IPv4 connectivity, than to sell them a service which is either a
> >>>> single global unicast IPv4 address or behind-NAT IPv4.
> >>>>
> >>>> Mobile users could be different, since many functions and services
> >>>> suitable for hand-held cellphone-like devices could be done via IPv6
> >>>> - and since there would always be an option to tunnel through IPv6 to
> >>>> an IPv4 NAT box so people can run client-style IPv4 applications on
> >>>> their MN when they want to.
> >>>
> >>> For many of the reasons you have mentioned, I am going
> >>> to back down and say that IRON-RANGER can be agnostic to
> >>> whatever IPvX/IPvY protocol combination gets used. I
> >>> still believe that the expanded address space of IPv6
> >>> will eventually steer new growth toward IPv6, but I
> >>> won't be so brave as to guess a timeframe for this.
> >>>
> >>> Still, one of the salient features of IRON-RANGER is
> >>> support for IPv6 transition.
> >>
> >> OK.  I understand there is potential value in the two crossover cases
> >> mentioned above:
> >>
> >>    IPv6-EID/IPv4-RLOC
> >>    IPv4-EID/IPv6-RLOC
> >>
> >> but for now I am imagining Ivip development being completely separate
> >> for IPv4 and IPv6.  Before finalising any protocols, I would be keen
> >> to investigate possible interworking - so I am not saying it is
> >> impossible or a bad idea.  Just that in this stage of development I
> >> am keeping the two systems entirely independent.
> >
> > OK, but from a legacy perspective IRON-RANGER is descended
> > from ISATAP which deals only with IPv6-EID/IPv4-RLOC.
> > (ISATAP is itself descended from mechanisms such as
> > RFC2529, RFC4213, etc.)
>
> OK.
>
>
> >>>>>> Then there are ways of using space more efficiently, as Ivip, LISP
> >>>>>> and probably IRON-RANGER could do, by slicing and dicing it into much
> >>>>>> smaller chunks than is possible with the /24 limit on prefixes in the
> >>>>>> DFZ.
> >>>>>
> >>>>> OK.
> >>>>
> >>>> So to me, a successful implementation of IRON-RANGER would be as good
> >>>> as Ivip or LISP in enabling really high levels of address utilization
> >>>> in IPv4.  This will considerably extend the ability of IPv4 to handle
> >>>> new users, including new end-user networks which need real global
> >>>> unicast space (not behind-NAT) because they are running servers.
> >>>
> >>> Can do.
> >>
> >> OK.  Ivip will be able to slice and dice IPv4 space into 1, 2, 3, 4,
> >> 5, 6, 7 or any integer number of IPv4 addresses in a single micronet.
> >>  It is not restricted to "prefixes".
> >>
> >> LISP and IRON-RANGER work in prefixes, but can still do 1, 2, 4, 8
> >> IPv4 addresses, which will be pretty much as good for finely slicing
> >> the space and mapping it to wherever it is to be used.
> >>
> >> I predict the vast majority of IPv4 micronets (EID prefixes for LISP
> >> and I-R) will be less than 256 IPv4 addresses.  If you look at the
> >> huge preponderance of /24 currently advertised in the DFZ:
> >>
> >>   http://bgp.potaroo.net/as2.0/bgp-active.html
> >>
> >>   /19  18191    5.74%
> >>   /20  22214    7.00%
> >>   /21  22356    7.05%
> >>   /22  28914    9.12%
> >>   /23  28732    9.06%
> >>   /24 166028   52.35%
> >>
> >> I conclude that the great majority of end-user networks find 256 IP
> >> addresses sufficient.  Since this is the smallest number of addresses
> >> they can advertise in the DFZ (and have the best-paths propagated
> >> through the whole DFZ) I believe it is reasonable to assume that many
> >> would be happy with 128 addresses, 64, 32 or whatever.  Perhaps quite
> >> a few would be happy with 1 or 4 addresses.
> >>
> >> At present, these /24s only take up a fraction of a percent of the
> >> IPv4 advertised address space, so arguably they are not wasting much
> >> space now, by being forced to be 256 addresses when less would suit
> >> the needs of the advertising networks.  However, a good scalable
> >> routing solution will be catering to many more end-user networks than
> >> those which currently advertise their own prefixes in the DFZ.
> >> Assuming the new kind of "scalable" "edge" space of LISP, Ivip or I-R
> >> has few or no performance problems, then it could be widely used and
> >> be used in just the right quantities required, without wasting much
> >> space.
> >
> > All right. I think I understand the concept of IPv4
> > micronets now. IRON-RANGER can handle this too.
>
> OK - I understand you mean that I-R will be able to use smallish
> chunks of IPv4 space, separately mapped to wherever they are needed,
> to help with using IPv4 space efficiently - without burdening the DFZ
> control plane with a prefix for each one, and without wasting space
> in BGP advertised prefixes which are at least 256 IPv4 addresses long.

Right.

> Do you also mean that I-R would allow separately mapped portions of
> IPv4 address space to start and end on integer boundaries, as with
> Ivip - rather than retaining the conventional binary boundary prefix
> method of delineating address space?

You mean like have a "prefix" that starts with something
like 192.0.2.9 and ends with 192.0.2.35? That just seems
to weird for me to contemplate at the moment.

> >>>>>> I think that most growth in Internet usage will occur in the IPv4
> >>>>>> Internet for at least the rest of this decade.  The only time it
> >>>>>> would make sense to use IPv6 instead of direct IPv4 or IPv4 behind
> >>>>>> NAT would be for some service where it wasn't important to be able to
> >>>>>> connect to IPv4.  At present, you couldn't sell any such service. I
> >>>>>> guess that it may be possible to do this for large IP cell-phone
> >>>>>> deployments where there are enough IPv6 services available to do a
> >>>>>> reasonable subset of what people want in a hand-held device, and
> >>>>>> where tunneling to a server which provides behind-NAT IPv4
> >>>>>> connectivity would also be possible.
> >>>>>
> >>>>> I agree that the IPv4 Internet is not only not going away
> >>>>> but also continuing to grow. But, I still think that users
> >>>>> will want to have both IPv4 (behind NAT if necessary) and
> >>>>> IPv6 as we move forward from here.
> >>>>
> >>>> At present, there's only one scenario in which I can imagine there
> >>>> being a real demand among non-mobile customers for IPv6.  Let's say
> >>>> that one or more large mobile phone companies decides to make their
> >>>> new, or existing, 3G systems work with each MN having its own global
> >>>> unicast IPv6 address (or perhaps /64).   This would enable direct
> >>>> host-to-host connectivity between any of these MNs.  (Though carriers
> >>>> typically want to avoid this, to stop people running VoIP and instead
> >>>> to use their voice call services, for which they charge more than
> >>>> they can for basic IP connectivity).
> >>>>
> >>>> Now let's say there are hundreds of millions or billions of these
> >>>> MNs, each with its own global unicast IPv6 address.  That address
> >>>> could be stable as long as the MN is in the one carrier network.  If
> >>>> it roams to another network, it would probably get another address.
> >>>> However, the TTR Mobility system would fix this - and give each MN
> >>>> its own permanent /64, no matter how it connected to the Net, as long
> >>>> as it is via IPv6.  (I do not currently plan any connections between
> >>>> Ivip or TTR Mobility for IPv4 and IPv6 - best to keep them as
> >>>> separate systems.)
> >>>>
> >>>> In this situation, people on non-mobile networks would have a genuine
> >>>> reason to get native IPv6 connectivity.  Firstly, they might want to
> >>>> sell or give services to these MN users.  Secondly, from home, they
> >>>> might want to run a web-cam, file sharing, VPN or whatever which the
> >>>> MN could access directly, on a host-to-host basis, without mucking
> >>>> around with IPv4.
> >>>>
> >>>> So I can imagine this trend happening - but only once there are a
> >>>> substantial number of ordinary users with native IPv6 connectivity.
> >>>> I guess this is most likely to occur with cell-phones.
> >>>
> >>> I honestly don't know what the drivers will be, Robin,
> >>> but I still believe (and I still believe that the *IETF*
> >>> believes) that IPv6 is where we need to go in the long
> >>> run. Again, however, I agree with you that IPv4 will
> >>> still be around for a very long time.
> >>
> >> None of us know anything about the future - we have to make do with
> >> educated guesses.
> >>
> >> I agree we should plan for widespread IPv6 adoption.  I am only
> >> arguing against assumptions such as:
> >>
> >>    IPv6 widespread adoption will being real soon now.
> >>
> >>    IPv4 usage is near its peak - so there's no need to plan for
> >>    it to be more widely used, to solve its scaling problem etc.
> >>
> >> For an example of the latter position, and implicitly the first, see
> >> Tony Li's recent message:
> >>
> >>   http://www.ietf.org/mail-archive/web/rrg/current/msg06192.html
> >>
> >>      IPv4 is done.  Over.  Cooked. Fully toast.  It will either
> >>      enter a black market where we deaggregate and no proposal
> >>      will help, or we shift to v6 and v4 is irrelevant.  In
> >>      either case, we're not in time to do anything significant
> >>      for v4.  And we still need a v6 solution, that's clearly
> >>      higher priority.
> >>
> >>
> >>>>>>> 3) IPv6 addresses can embed IPv4 addresses such that there
> >>>>>>>    is stateless address mapping between an EID nexthop and
> >>>>>>>    an RLOC.
> >>>>>>
> >>>>>> Can you explain this with an example?  I can't clearly envisage what
> >>>>>> you mean.
> >>>>>
> >>>>> I mean, if the IPv6 EID FIB includes entries with a next-hop
> >>>>> address such as: 'fe80::5efe:V4ADDR' (i.e., an IPv6 address
> >>>>> with embedded IPv4 address), then V4ADDR can be statelessly
> >>>>> extracted as the RLOC address of the ETR.
> >>>>
> >>>> So the "mapping", which the LFR-role and IDR-role routers get from
> >>>> the VP router is actually telling them to tunnel subsequent traffic
> >>>> packets to an IPv4 address?   That would only work if every LFR-role
> >>>> and IDR-role router had IPv4 access - unless you were to establish
> >>>> special routers to act as gateways for delivering to IPv4 addresses,
> >>>> which is not out of the question.
> >>>
> >>> Public IPv4 RLOCs that are routable within the IPv4 DFZ
> >>> is what I am suggesting.
> >>
> >> Yes, but if you are able to specify this in the mapping sent by a VP
> >> router to an IRON router which is accepting traffic packets and
> >> tunneling them (initially to the VP router, and then to whichever
> >> "DEL" role router
>
> Now IR(EID).
>
> >>                    it decides to from the mapping sent by the VP
> >> router in response to this initially tunneled packet), then it will
> >> only work if all these IRON routers (IBR and IBG in your terminology)
> >> can tunnel packets to any IPv4 "RLOC" address.  Yet these are all
> >> IRON routers which are on IPv6 addresses.
> >>
> >> They would need either direct IPv4 connectivity to do this, or a
> >> means of forwarding the packet, in tunneled form, to some other IPv6
> >> router which could send them to the IPv4 address of the "DEL" role
> >> IRON router.
> >
> > Direct connectivity to the IPv4 Internet is required and assumed.
>
> OK - I understand from this that all IRON routers have IPv4 addresses
> and that some, perhaps many, of them have IPv6 addresses.
>
> If so, I understand that there are two separate I-R overlay networks.
>  All IRON routers will participate in the IPv4 I-R overlay, and all
> those with IPv6 addresses will also participate in the IPv6 I-R overlay.
>
>
> >> Neither of these things are in the current design, as far as I know.
> >
> > I'll try to make the assumption explicit.
>
> OK.
>
>
>
> >> I imagine you would want load balancing with generally the "nearest"
> >> VP router being used, when an IBR or IBG router tunnels the initial
> >> one or more traffic packets (before it gets "mapping" from the VP
> >> router.
> >>
> >> The use of both nearest and the load sharing would generally make the
> >> system work better.  Also, when one VP router dies, if you had three,
> >> it would only affect on average 1/3 of the IBR and IBG routers.
> >>
> >> Load sharing would be vital for scaling purposes - even if VRRP
> >> somehow handled the robustness problem, there's no way you want all
> >> the initial packets for any set of EID prefixes having to go to just
> >> one physical router in the Net.
> >
> > As I mentioned from the beginning, I am going to relax the
> > requirement for VRRP as an underpinning of the architecture.
> > Instead, VRRP can be used as an optimization if desired, but
> > the architecture will allow for a single VP to be advertised
> > by multiple IR(VPs). That would satisfy the scenario you
> > are describing above.
>
> OK - but I don't yet know how an IRON router which is one of the
> potentially multiple IRON routers playing the IR(EID) role regularly
> and securely registers itself as such for a given EID prefix, with
> all the two or perhaps more IRON routers which play the IR(VP) role
> for the VP which covers this EID.

Using the flat map file described above, the IR(EID) can
discover all IR(VP) routers with a VP that covers the EID
prefix.

> If each such IR(EID) role router is to do so directly and
> independently for each router playing the IR(VP) role for this VP,
> then at present, there's no way for the IR(EID) role router to find
> out the addresses of all the IR(VP) role routers.  With BGP, they get
> a single best path, which will be for one of the two or more IR(VP)
> routers.
>
> If you decide on a mechanism by which a given IR(EID) role router
> registers with all the two or more IR(VP) role routers, when it only
> knows initially the address of one of them, then I think there needs
> to be some new kind of protocol.  I can't yet see how VRRP could
> handle this, since as far as I know it requires multicast in some
> form and it has the multiple redundant routers all on the one IP
> address, which would definitely not be the case with the multiple,
> widely spaced (for robustness and load sharing) routers playing the
> IR(VP) role for a given VP.

All of this is overcome by events.

> But see below for how you intend to use VRRP.
>
>
> >>>> I don't want to try to read this RFC in order to imagine how it might
> >>>> work with I-R, so if you can describe how it would work, that would
> >>>> be good.
> >>>
> >>> I touched on this above, which is just about as deep
> >>> as my understanding goes. In an nutshell, with VRRP
> >>> each router shares the same IP address, and each
> >>> router maintains synchronized state. One of the
> >>> routers is chosen as the primary, and the others
> >>> are designated as backups. If the primary fails,
> >>> one of the backups takes over sort of like an
> >>> uninterruptible power supply.
> >>
> >> I can't see how you can have this shared IP address arrangement for
> >> IRON routers which are going to be in different places, and therefore
> >>  in different parts of the topology.  The diverse placement is
> >> essential for robustness - and for the goals of load-sharing with
> >> generally shorter paths.
> >>
> >> In the I-R overlay system, each IRON router which advertised a VP
> >> does so giving its IP address as being the same as the IP address it
> >> uses in the Internet.  This is because IRON routers playing your IBR
> >> or IBG roles tunnel traffic packets directly (via the Internet, not
> >> the I-R overlay network) to the VP router.
> >>
> >> So I don't see how you could have multiple such routers, which must
> >> be on different IP addresses on the Internet, behaving on the I-R
> >> overlay as if they all had the one IP address.
> >
> > From now on, for the base case there will be multiple IR(VP)s
> > for each VP; each with its own IP address. However, each IR(VP)
> > could also be cloned using VRRP, so there may also be more than
> > one IR(VP) per IP address for fault tolerance.
>
> OK - so VRRP is purely a local system to each site which appears to
> runs an IRON router performing the IR(VP) role for one or more VPs.
> It is not going to be used as a method by which the routers playing
> this role at multiple sites communicate so that an IR(EID) role
> router could register with them all, when it only knows the address
> or one of them.

VRRP - if used - would be a local system to each site, yes.

> >> But that's fine - I regard this as a brainstorming phase of design,
> >> so its OK to consider things just because they look like they might
> >> do the job, without assuming that they really can do it.  Even if
> >> they can't, it might lead to a line of thought which turns out to be
> >> of lasting value.
> >
> > OK.
>
> >> Yes - IBGs are the equivalents of Ivip DITRs and LISP PTRs.  I think
> >> they will need to monitor traffic on these and charge the destination
> >> networks accordingly.  Ivip anticipates this, but so far LISP and
> >> IRON-RANGER don't.
> >
> > Again with the new terminology, IBG == IR(GW). But,
> > I'm not sure about any business model that would charge
> > the end-users specifically for the use of the IR(GW).
> > I think the business model is for charging the end-users
> > for the use of the VP, and then amortize the cost of
> > operating an IR(GW) into the VP costs passed on to
> > the end-user.
>
> Yes, but some end-users might have very little traffic going to their
> networks and others might have huge volumes - and the volumes would
> generally have little or no correlation with the amount of "edge"
> (EID) address space each such end-user network had.
>
> I can't imagine there being a viable business model for IR(GW)
> routers without some kind of usage-based cost recovery.
>
>
> > The question is how does an IR(EID) forward EID Prefix
> > Advertisements (PAs) to multiple IR(VP)s? The answer I
> > am planning to use is "one at a time". In other words,
> > I am going to ask the IR(EID) to forward each PA to
> > each and every IR(VP) that advertises a matching VP.
> > So, no purpose-built protocol between the IR(VPs).
> > Does that work for you?
>
>
> Yes - except that at present, with BGP, each IRON router playing the
> IR(EID) role will get a best-path to (and therefore the IP address
> of) only one of the potentially multiple IRON routers playing the
> IR(VP) role for the prefix in question.  So some additional mechanism
> will be required for them to discover this information.

OBE.

> The overlay as it stands, as far as I know, only exists to handle BGP
> messages by which IRON routers can discover the IP addresses of the
> other IRON routers which are performing the IR(VP) role for each of
> the potentially million or so VPs.

OBE.

> As I wrote earlier, for each VP, there are two reasons why an IRON
> router would need to know the addresses of any of these IR(VP) routers:
>
>   1 - The IRON router is playing an ordinary role of being able
>       to forward traffic packets from its local routing system
>       to either IR(VP) routers (initially) or after receiving
>       mapping from the IR(VP) router, directly to one of the
>       potentially multiple IRON routers playing the IR(EID) role
>       for the EID prefix which the destination address matches.
>
>       This is also for IRON routers playing the IR(GW) role.
>
>       In this case, its fine if only one of the IR(VP) routers
>       is known - ideally a nearby one, but this is not essential.
>
>       This needs to be already known, since we don't want there
>       to be any delays in looking this information up when a
>       packet arrives for a VP which the router doesn't yet know
>       the address of an IR(VP) router.
>
>       Each IRON router needs to already known this for all VPs.

Right - a flat file that contains all mappings.

>   2 - The IRON router is one of the potentially multiple IRON
>       routers playing the IR(EID) role for a given I-R EID.
>
>       Then, according to what you wrote about "one at a time"
>       each of these IRON routers needs to know the addresses of
>       all the IRON routers currently playing the IR(VP) role for
>       the VP which covers this EID.
>
>       In this case, the IR(EID) IRON router only needs to know
>       this about a small subset of the VPs.
>
>       The router needs to know about this on a continual or
>       repeated basis, since it will regularly be re-registering
>       itself with these IR(VP) routers.
>
>
> I think BGP, with or without Route Reflectors, is perfectly good for
> the first purpose.  Without Route Reflectors, and using the
> "previous" approach I described at the start of this message, I think
> the best-path any one IRON router gets for a VP will frequently be
> that of a close or closest of multiple IR(VP) routers for this VP.
>
> With Route Reflection, I think it would be more difficult to ensure
> that the single best-path would be for a close or closest IR(VP) router.
>
>
> For the second purpose, I think BGP won't work.  However, since any
> one IRON router playing the IR(EID) role only needs to know the IP
> addresses of all the IR(VP) routers for a small subset of VPs, I
> suggest that DNS would be a perfectly good way they could discover this.
>
> It wouldn't matter much if a cached DNS entry pointed to an IRON
> router which previously performed the IR(VP) role for the VP in
> question, but had recently died or become unreachable - since
> registration would proceed with the others.
>
> The trick would be to allow a router to come up in an IR(VP) role and
> make sure it had registrations from all the thousands or hundreds of
> thousands or whatever IR(EID) role routers before it advertised the
> VP on the I-R overlay network.   To do that via DNS in, say, 5
> minutes, would require all the DNS caching times for this lookup
> process be 5 minutes or less.

No DNS.

> Alternatively, a newly started IR(VP) router which hadn't yet been
> found by all the IR(EID) routers could advertise itself, but would
> then need to have a way of handling traffic packets tunneled to it,
> but for which it did not know any IR(EID) router to forward them to.
>  The most obvious approach would be to forward it to one of the other
> IRON routers which are playing the IR(VP) role for the same VP, with
> that one doing the "route redirection" secured "mapping" reply to the
> IRON router which tunneled the packet, so in the future it would
> tunnel such packets directly to the correct IR(EID) router.

No, all IR(VP)s that own the same VP will check
their FIBs to find a more-specific route to an
EID prefix taken from the VP. If there is no entry
in the FIB, the packet is dropped.

This means that if an IR loses its memory there will
be a lag time until the EID prefix mappings are
re-established. So, the IR should probably keep
EID mappings in non-volatile storage across reboots.

> One might think of using DNS instead of BGP for the first purpose,
> but that would involve some problems:
>
>   1 - Each IRON router would need to repeatedly query the DNS for
>       each of the million or so VPs - which is a serious and
>       probably unacceptable scaling problem.

Right - let's not do that.

>   2 - If they did this, then it would sometimes try to tunnel
>       traffic packets to and IR(VP) role router which was alive
>       when it last queried the DNS, but has since died.  This would
>       cause lost traffic packets, or require some two-way
>       acknowledgement so the IRON router could quickly detect the fact
>       that this supposed IR(VP) router was no longer reachable, alive
>       and still performing this role.   While the I-R system involves
>       that IR(VP) router normally sending a "route redirect" AKA
>       "mapping" packet to this router, the only way of making this
>       work as an acknowledgement for the purpose of avoiding lost
>       traffic packets due to a dead IR(VP) router would be the
>       initial IRON router buffering these traffic packets for a few
>       seconds so it could resend them to some other IR(VP) router
>       instead.  This would be costly and is not anticipated in the
>       current design.

No buffering of packets. Route failures are detected through
neighbor unreachability detection, and a different neighbor
is tried.

> > OSPF looks like a dead end due to its requirements for
> > using areas for very large networks. I want everything
> > to be one big, flat network with a backbone of IR(RR)s.
> > I think BGP with Route Reflectors fits the bill well.
>
> OK.
>
>
> >>>> But it is important for IRON routers to run their new BGP instance
> >>>> with neighbouring IRON routers which are generally physically or
> >>>> topologically close.  Otherwise, the "distance" metrics in the
> >>>> overlay network won't resemble the real "distance" to the other
> >>>> routers, and your routers playing the LFR or IDM role won't
> >>>> automatically discover the address of the "closest" VPR for a given VP.
> >>>
> >>> Do you mean distance as in hopcount? Because, every IRON
> >>> router is a neighbor on the link - i.e., hopcount is 1
> >>> always.
> >>
> >> I meant the "distance" metrics of BGP - each best-path offered by a
> >> neighbouring router is assessed according to the number of ASes it
> >> contains, and then, subject to local policy, the one with the lowest
> >> number of ASes is usually chosen - with this being offered to all the
> >> neighbours, with an additional AS added (or a few, to make it less
> >> attractive, if this is desired).
> >>
> >> I assume that in the BGP overlay each IRON router uses as its AS the
> >> AS it is within on the Internet.
> >
> > The IRs actually use IBGP within the IRON, so everything
> > is within a *single* AS. (Yes, that's a very big AS, but
> > that's the way it is.) IR(GW)s use EBGP within the DFZ.
>
>
> OK - so how would any RR choose the a best-path to the closest IR(VP)
> router?  As far as I know, the distance metrics in BGP only work in
> the DFZ (not inside and AS with iBGP), and only work according to the
> number of ASNs a path traverses.

OBE.

> >>>> These tunnels surely need to be manually configured - and that
> >>>> defines the membership in the I-R overlay network and its structure
> >>>> for the purposes of its BGP (or OSPF?) control plane.
> >>>
> >>> Automatic tunneling is the goal I am working toward.
> >>
> >> But if you have 100k IRON routers, how does any one IRON router
> >> decide which small subset of these to create tunnels to?
> >
> > By resolving the name "isatapv2.net".
>
> OK - with the Route Reflector approach, some as-yet unspecified
> number of IRON routers playing the IR(RR) role discover the complete
> set of all such routers and set up SEAL tunnels to each other in a
> complete mesh, for the purposes of running their BGP instance to
> create the overlay.  This happens for both the IPv4 and IPv6 overlays.

Domain name resolution will no longer be needed to
discover IR(RR)s since there are no longer any IR(RR)s.
Domain name resolution may still be used to discover
IR(GW)s.

> >> You can't have all 100,000 IRON routers tunneling to the 99,999 other
> >> IRON routers.
> >
> > No; the 100k IRs tunnel to one or a few IR(RR)s among
> > perhaps 100 or so IR(RR)s around the world. Each IR(RR)
> > is fully meshed to all other IR(RR)s.
>
> OK - so my guess above of 50 to 100 was good.

OBE.

> >> If you are going to use BGP to provide each IRON router with a
> >> best-path to the "nearest"  VP router of several VP routers
> >> advertising a given VP, then you need these tunnels to be with
> >> topologically nearby routers.   I can't imagine how you could do this
> >> automatically.
> >
> > When each IR that needs to form a tunnel with an IR(RR)
> > comes up, it checks through the list of all IR(RR) IP
> > addresses and picks one or a few that are "nearby". It
> > is up to each IR to decide what is "nearby".
>
> OK - each IRON router could use traceroute or ping on all hundred or
> so IR(RR) routers.  This is probably scalable since it only occurs
> when each IRON router boots up.

OBE.

> >> Also, if I ran an AS with one or more IRON routers, I would want to
> >> manually configure which IRON routers in other ASes each one tunneled
> >> to, rather than trusting some automagic system to do this.  There
> >> will be real flows of BGP messages over those tunnels, and I would
> >> want most or all of them to be with ASes I had a zero-cost peering
> >> arrangement with.
> >
> > How about all IRs within the same AS using IBGP? That
> > would keep the AS peering costs at zero always.
>
> I guess you could do this - but this would need to be part of how you
> structure the Route Reflection process and how individual IRON
> routers within an AS coordinate themselves in choosing which one or
> more IR(RR) routers to tunnel to.
>
>
> >>>> I regard the whole question of registering EIDs with VPRs as being
> >>>> undecided until you propose an exact mechanism.
> >>>
> >>> The mechanism is periodic transmission of signed router
> >>> advertisements with credentials that prove ownership of
> >>> the advertised prefixes. These are what I have formerly
> >>> called "bubbles", but as discussed above we should
> >>> probably try for a new name.
> >>
> >> OK - but depending on your choice of BGP or OSPF I understand you
> >> will devise a mechanism, potentially using VRRP.
> >
> > Right. The IR(EID) will forward a separate PA to each
> > IR(VP) that advertised a VP that covers the EID prefix.
>
> Yes - but BGP only provides the address of one of the IR(VP) routers.
> Above I suggest using DNS for this purpose instead.

OBE

> >> Thanks for the continuing discussion.
> >
> > Thanks - Fred
>
> OK - to be continued, no doubt!

I think its time to start wrapping this up. Let's try to
be brief for any additional follow up.

Thanks - Fred
[email-masked]

>   - Robin

---

From: Robin Whittle <[email-masked]>
Date: Thu, 25 Mar 2010 22:27:35 +1100

Short version:  The new plan for I-R is to not use BGP or Route
                Reflection, but to have all IRON routers download a
                multi-megabyte (in a full deployment) file which
                lists every Virtual Prefix (VP) and for each VP the
                addresses of the 2, 3, 4 etc. IRON routers which
                perform the IR(VP) role for this VP.

                Each IRON router would keep this information up-to-
                date via delta checks with some central servers.

                This avoids the need for BGP etc.

                I think it is highly scalable, since the number of
                VPs will be finite, each VP will cover many EID
                prefixes, and since the file only lists the subset
                of IRON routers which perform the IR(VP) role.

                This is unique Core-Edge Separation architecture
                which avoids excessively long paths for initial
                packets (as may occur with LISP-ALT) and which
                uses tunneled traffic packets as map requests.

                It avoids the NERD approach of ITRs knowing mapping
                for every EID prefix.

                I think it is much superior to LISP-ALT - but that
                Ivip would be better still.

                There needs to be a way of tunneling a traffic
                packet to an IR(VP) router which has never been
                used by the tunneling router before, and finding
                out quickly (such as in the next 0.5 to 1.0 seconds
                whether the IR(VP) router is alive and ready to
                accept such traffic packets.  The obvious way to do
                this is to await the map reply message.  Until
                then, I think it cannot be assumed the tunneled
                packets are going to be delivered, so perhaps they
                should be buffered and sent to an alternative IR(VP)
                router if they are not too stale after some time-out
                if no mapping arrives from the IR(VP) router.

                Otherwise, every IRON router would need to
                continually test reachability to all the IR(VP)
                routers - which would be unscalable.

Hi Fred,

I am responding to (msg06342) "Re: [rrg] IRON-RANGER - Route
Reflectors & DNS?" with a new title to reflect your new plan of a
single file to tell all IRON routers which IRON routers are
performing the IR(VP) role for each VP.

> detect additions and deletions. That's about it.

OK, I understand this means something like:

  For every VP there are at least 2 and probably 3, 4 or 5
  IRON routers performing the IR(VP) role.

  There is a centrally maintained list of all the VPs and
  for each VP, the IP addresses (or FQDNs?) of the IRON
  routers which are currently playing the IR(VP) role for
  each such VP.

  With 100k VPs, this is, for instance, 400k addresses.
  IPv4: 1.6 Mbyte;   IPv6: 6.4 Mbyte
  In reality, the files would be somewhat bigger than this,
  and bigger still if FQDNs were used.

Each IRON router (say 300k to 600k of them) downloads a copy of
this file and then uses some protocol to check for changes
periodically ("delta check"), somehow keeping its own copy up-to-
date.   The frequency of these delta checks and the ability to load
share them amongst multiple generally "nearby" servers will affect
the scaling of the whole system, including how frequently these
delta checks can be performed - once every minute, 10 minutes, or
hour.

I think this is a promising approach, but . . .

  When a given IRON router playing the IR(VP) role for a prefix P
  ceases doing this, or becomes unreachable, each IRON router needs
  to be able to detect this really quickly - otherwise data will be
  lost by tunneling a traffic packet to that IR(VP) router.

  If the IR(VP) router is unreachable, will SEAL tunneling tell the
  IRON router which is tunneling traffic packets to it?  (If not,
  then there would be ongoing data loss.) If so, would that ITE IRON
  router be expected to have buffered the traffic packet so it could
  be tunneled to another IR(VP) router instead?  If not then there
  would be ongoing data-loss.

More on this below.

When an IRON router adopts the IR(VP) role for a given prefix,
depending on the registration mechanism, it will take it a while
to be sure it has received all the registrations for the EID prefixes
in its VP.  Only then can it declare itself open for business - so
only then could its address or FQDN be added to the master list.
(But it the master list is the one which is used for registration -
see below for a solution to this problem.)

Then it would take time, depending on the speed of delta checks, for
all the IRON routers to know of this IR(VP) role.

> I think what you are describing here is the same as what

Your mention of IR(EID) was in msg06329.  I quoted that in my
message of this new thread (IRON-RANGER - Route Reflectors & DNS?)
which did not make it to the RRG list, but to which you replied in
msg06342.

Your description was:

I understood this role to be an IRON router accepting tunneled
traffic packets whose destination address (once detunneled) matches
that of an I-R EID prefix which this router is accepting packets
for.  This router is then able to forward the traffic packets to
the end-user (AKA "enterprise") network whose EID prefix this is.

You didn't have a name for the role played by an IRON router where
it advertises the totality of the I-R "edge" space in the routing
system of its network (where this IRON router is inside an ISP or
end-user network and not advertising these prefixes to other ASes).
The role of this router is to:

  1 - If it has no mapping for an EID which covers the destination
      address of the packet, to tunnel it to one of the routers
      which is playing the IR(VP) role for the VP which covers the
      destination address - and then to await "mapping" being sent
      back from that IR(VP) router, which it will cache and use to
      handle any subsequent traffic packets whose destination
      address matches that of the EID in the mapping, as described
      in the next point.

  2 - If it has the mapping for an EID matching the destination
      address to use this to choose which IR(EID) role IRON router
      (as per my understanding) to tunnel the packet to.

When these things are done by an IRON router which advertises all
the I-R "edge" space in the DFZ, you call this the IR(GW) role:

but you have no name for the role of an IRON router doing the same
things but only accepting traffic packets from its local routing
system.

I understood that every IRON router did this anyway, so you didn't
have a name for this role.  Still, I think there should be a name
for this role, even if all IRON routers are capable of doing it.
In the following discussion, I will give this role a name:

  IR(LF) - forwarding traffic packets which arrive from its local
           routing system.

So here is my rewrite of the roles, including this one:

  IR(LF) - Forwarding traffic packets which arrive from its local
           routing system.  Advertises all I-R "edge" space in
           its local routing system.

           (All IRON routers are capable of doing this, but it is
           still a distinct role.)

  IR(GW) - Like an IR(LF) role except that the IRON router
           advertises the I-R "edge" space in the DFZ (that is: in
           the DFZ, to DFZ routers of other ASes but its own).
           (This is similar to LISP's Proxy Tunnel Routers and
           Ivip's DITRs, though a DITR normally only advertises a
           subset of the "edge" space.)

  IR(VP) - One of typically 2, 3, 4 etc. IRON routers which handles
           a given Virtual Prefix of I-R "edge" space.  This means
           it:

             1 - Accepts registrations of EID prefixes which fall
                 within this VP, from the IRON routers which
                 perform the IR(EID) role for these EID prefixes.

             2 - Accepts tunneled traffic packets from IRON routers
                 performing the IR(LF) and IR(GW) functions and
                 then tunnels the packet to the correct IR(EID)
                 role router for the EID prefix which covers the
                 packet's destination address.  (This may be the
                 same router, so no tunneling is required - but
                 usually it will be another IRON router.)

             3 - After 2, sends mapping for the matching EID to the
                 IR(LF) or IR(GW) role router.

            What does it do with a packet which matches its VP, but
            for which it has no EID registration?  Should it tunnel
            it to another IRON router which is also performing the
            IR(VP) role for this EID prefix?  According to your
            reply below, it should not buffer the packet - so it
            can't resend any traffic packets to another IR(VP)
            router if the first one appears to be dead.

   IR(EID) - Accepts tunneled packets from IR(VP), IR(LF) and IR(GW)
            role routers and then delivers the packet to the
            destination network.  A multihomed end-user network
            will have two or more ISPs and there will be an IR(EID)
            role router for each such ISP.  (Whether this router is
            in the ISP network or in the end-user network is not
            fixed, but it is always on a conventional "RLOC" - AKA
            "core" address.)

> the whole map will be kept in a flat file.

OK - that saves a lot of trouble!

That is not my understanding of what you wrote earlier:

    > Since the list of VPs as well as the
    > from a content server when it boots up.

That file has an entry for each VP, and each entry contains the IP
addresses or FQDNs of the 2, 3, 4 or whatever IRON routers which are
performing the IR(VP) role for that VP.

This is not a complete list of all IRON routers - it only mentions
IRON routers which perform the IR(VP) role for at least one VP.

So it is not what you just mentioned:

   >  all IRs will have the full map to all other IRs

That was something BGP achieved - but I don't think you need it.

If you want to use a single file to identify all IRON routers, this
would be a lot bigger.  Ideally, you would have millions of IRON
routers, with most of them performing just the IR(LF) role.  The
more there are of these, and of IR(GW) routers, the better the
traffic load is shared among them, so the smaller they can be.

You could also put the IR(LF) function in a sending host, as I do
with Ivip's optional ITR function in the sending host.

I don't think there is a need for any IRON router, no matter what
role it is playing, to know the address of every other IRON router.
So I think the list of VPs, with the IR(VP) routers for each VP, is
all you need.

This is fine for the two purposes:

  1 - IR(EID) role routers registering their EIDs with the 2, 3, 4
      etc. IR(VP) role routers.

  2 - IR(LF) and IR(GW) routers tunneling traffic packets to one of
      the IR(VP) routers for the matching VP.  However, the list
      doesn't provide any information on which of these is closest,
      which ones of them are currently reachable and which of them
      have had time since they were added to the list to get all
      the EIDs in that VP registered by all the various IR(EID)
      role routers which handle those EIDs.

OK.

> between the IRON and the DFZ.".

This still doesn't imply a direction, yet they only take packets
from the DFZ and tunnel them to IRON routers, which are playing the
IR(VP) or IR(EID) roles.   My previous description makes this clear.

> be forwarded by an IR(GW).

I don't see how this would occur.  If some ISP or end-user network
is using I-R "edge" space, it must have one or more IRON routers in
its own network or in the networks of its ISPs, which are
performing the IR(EID) role.

When a host in this network sends out a packet addressed to an EID
address, it will be forwarded to one of these IRON routers, which
in its IR(LF) role (which I understand every IRON router performs)
is advertising prefixes covering all I-R "edge" space.  That IRON
router will tunnel the packet either to the correct router
performing the IR(EID) role for the matching EID prefix, or it will
tunnel it to one of the routers playing the IR(VP) role for the
matching VP.

However, when a host sends a packet to an ordinary address outside
the network - not within the I-R "edge" subset of the global
unicast address space, this may go through the same IRON router as
just discussed which is performing an IR(LF) role, but that role
does not affect packets addressed to ordinary global unicast
addresses.  So the IRON system is only for carrying packets
addressed to I-R "edge" (EID) addresses.  That packet, with an
ordinary destination address, would be forwarded as it is today,
out to the ISP's BR and then to other DFZ routers and eventually to
the ISP or end-user network where the destination host is located.

> default route to the IPv6 Internet - right?

IPv4 hosts can't send packets to IPv6 addresses, so I guess you
must be referring to IPv6 hosts sending packets to IPv6 addresses
which are either IR "edge" addresses or not - and that the network
these hosts are in is an island IPv6 network, without native IPv6
connectivity, but have IRON routers which are somehow linking this
IPv6 network to the real IPv6 Internet.   I won't speculate on how
this could be done.

OK.

However, there is no need for every IRON router to know the address
of every other IRON router.  They need to know, for each VP, the
addresses of the IRON routers which play the IR(VP) role, and once
they have mapping for particular VPs, this tells them the addresses
of the one or more IRON routers which are playing the IR(EID) role
for this EID prefix.  Then they use SEAL to tunnel to these other
IRON routers.

You could refer to this as a virtual full-mesh, but in fact, the
ITEs (ingress tunnel endpoints), but there are never tunnels
between two IRON routers which only perform the IR(LF) or IR(GW)
roles.

>>   1 - BGP messages, always between the two tunnel endpoints.

Not needed now.

OK.

OK.

> to weird for me to contemplate at the moment.

OK - its not a prefix, of course - so I call it a "micronet".

In Ivip micronets are integer numbers of IPv4 addresses or IPv6
/64s, with arbitrary start and end points - but a micronet is
always within a MAB (Mapped Address Block - a prefix advertised in
the DFZ by DITRs).

OK.

OK.

But here's a problem with the currently proposed scheme, in which
the one file tells all IRON routers about each VP and for each VP
the 2, 3, 4 etc. IRON routers which are performing the IR(VP)
function for the VP.

This is fine for IR(EID) role routers finding out the addresses of
the 2, 3, 4 etc IR(VP) routers to register their EID with.

What is not OK is using this file alone for IR(LF) and IR(GW)
routers to decide which IR(VP) routers to tunnel traffic packets to.

This is because when you make some IRON router NN take on the
IR(VP) role for some VP "P", it will take time (at least the delta
check repeat time, and then some) for all IR(EID) routers to be
able to register with these IR(VP) routers.

Yet, at the same time, IR(LF) and IF(GW) routers are using the very
same list and may tunnel traffic packets to NN the moment they see
it appear on the list.

The solution is probably to fix a time for delta checks, such as 10
minutes, add a fudge factor and then arrive at a time such as 15
minutes after NN is mentioned as having an IR(VP) role for "P", that
all P's IR(EID) routers will have registered with NN.  Then add an
algorithm in the IR(LF) and IR(GW) routers to make them wait for 15
minutes after they see NN has this IR(VP) role before actually
tunneling any traffic packets to it.

> EID mappings in non-volatile storage across reboots.

This would make sense for direct reboots, but if the router was
turned off for a long time, or was not able to keep itself updated
due to being unreachable for a long period of time, the delta check
process could lead to so many updates that it would be better to
start afresh and download the full copy of the latest file.

OK.

> neighbor unreachability detection, and a different neighbor

But if some router AA performing the IR(LF) or IR(GW) role has a
traffic packet to tunnel to router XX which is performing the IR(VP)
role, and it has never had any relationship with XX before, there
is no SEAL tunnel in place.

SEAL would tunnel the packet without any preliminaries, as far as I
know - if there are preliminaries, then this will delay the
delivery of initial packets in a new communications flow.

The SEAL layer on AA has no idea whether XX is alive.

If AA tunnels the traffic packet without any buffering, and XX is
dead, unreachable, not playing the IR(VP) role, or if it is playing
the role but for some reason has not received a registration from
any of the IR(EID) role routers, then the packet will be lost
forever.

There isn't any "neighbour unreachability" since to AA, XX is just
some IRON IR(VP) router it has an address for.  It has no
established tunnel and no way of knowing whether it is alive or
dead, unless it either:

  1 - Performs preliminary handshakes, which would be expensive and
      would unacceptably delay traffic packets.

or

  2 - Buffers the traffic packet and any other packets it sends to
      XX until it gets a mapping packet from XX, which confirms that
      at least one of these packets got to XX and that XX is able
      to handle them.  If, after a second or so, it didn't get
      any mapping from XX, then it would assume that XX was dead and
      could try sending them to another IR(VP) router - but by then
      the traffic packets are getting a bit long in the tooth.

At present I think this is an unsolved problem: how the IR(LF) or
IR(GW) router detecting the fact that a listed IR(VP) router is in
fact dead or unreachable, especially when it is just about to
tunnel a traffic packet to this IR(VP) router for the first time.

OK.

  - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 25 Mar 2010 08:25:05 -0700

Hi Robin,

Main point is that you have identified an IR(VP) liveness
detection issue, but I think I have an idea that might help.
See below for more detailed responses:

> From: Robin Whittle [mailto:[email-masked]]
> Sent: Thursday, March 25, 2010 4:28 AM
> To: RRG
> Cc: Templin, Fred L
> Subject: IRON-RANGER - Single VP->routers file with delta checks
>

Yes.

Yes.

> delta checks can be performed - once every minute, 10 minutes, or

I was thinking the list of all IRs would be capped off
somewhere around 100k or so. As far as I can tell, IRs
should be the border routers of large corporations, ISP
networks, academic institutions, government agencies,
etc. They should not be placed on home user residential
gateways, individual airplanes, etc. In other words, the
IR(EID)s would be placed in a manner that would be somewhat
analogous to the way in which EBGP routers are placed in
the Internet DFZ today - and I don't think we have
millions and millions of those, right?

> I think this is a promising approach,

OK.

> but . . .

There's always a "but" when we talk about these things,
isn't there?

>   lost by tunneling a traffic packet to that IR(VP) router.

But, how long does it take for BGP peers to detect
that a neighbor has gone unreachable? I should think
that with SEAL we should be able to get within the
same ballpark for unreachability detection.

>   would be ongoing data-loss.

No buffering, but isn't the concern the same as for
when a BGP router goes unreachable? There will be a
period of data loss in that case too, right?

> all the IRON routers to know of this IR(VP) role.

Right, but adding a new IR(VP) (or adding a new VP to an
existing IR(VP)) would be a major event that will take
some time to percolate throughout the IRON. This is not
something that should happen every few minutes or even
hours.

>             router if the first one appears to be dead.

If the IR(VP) does not have an EID registration that
matches the packet, it should drop the packet. But,

>             fixed, but it is always on a conventional "RLOC" - AKA

I think your definitions are fine for use for the purpose
of this discussion, but I'm not convinced yet that there
is a need to have IR(LF) in addition to IR(EID). In my way
of thinking, all IRs are IR(EID)s even if the EID space
aggregated by the IR is NULL.

> OK - that saves a lot of trouble!

Right, but the tradeoff is that we will not get
liveness detection in the control plane so we will
need to test liveness in the data plane.

>    >  all IRs will have the full map to all other IRs

I was thinking that it may be valuable to require all IRs
(not just IR(VP)s) to register themselves with the IRON
map administrative authority. That way, there is a list
of all nodes that are authorized to act as IRs.

> routers, with most of them performing just the IR(LF) role.

Why millions? Why can't we make do with just ~100k
or so that are border routers of major enterprise
networks as I discussed above?

> The

This is an area where our viewpoints may be diverging
just a bit. I am thinking that IRs would be major pieces
of equipment at the borders of medium to large-sized
enterprise networks, and that those networks would use
EID-based routing (possibly using RANGER recursion)
internally. Whereas, you seem to believe that pushing
the IRs out to the extreme network edges would be
desirable. Remember that with RANGER recursion, however,
we can have recursive re-encapsulations so that the
IRON principles can apply at any level of the recursion.

>       etc. IR(VP) role routers.

OK, but I still would like to think of *all* IRs as IR(EID)s,
even if some IRs configure a NULL set of EIDs.

>       role routers which handle those EIDs.

All right; I'll accept the IR(LF) and IR(GW) roles as
useful for this discussion.

> IPv6 network to the real IPv6 Internet.   I won't speculate on how

All right. So, this means that if we want to stand up
IRs that use IPv6 EIDs then those IRs must be able to
connect to the global IPv6 Internet and not just the
global IPv4 Internet, right? I don't know enough about
this, but I think that the IPv6 Internet is not all
that big yet, and that not all enterprises, sites, etc
connect to it yet. So, I don't know what sorts of special
arrangements would be necessary in order for an IR to
get connected.

> for this EID prefix.  Then they use SEAL to tunnel to these other

I was thinking of having each IR discover the list of
all IRs for access control purposes. Again, in my
view there should never be more than about ~100k
total IRs (both VP and non-VP).

> between two IRON routers which only perform the IR(LF) or IR(GW)

There are no tunnels btw IR(LF)s initially, but they will
be created on demand following a mapping update.

> always within a MAB (Mapped Address Block - a prefix advertised in

It is an interesting idea, but for now I'd prefer to
stick with powers-of-two prefix lengths.

> tunneling any traffic packets to it.

Using an idea I will propose below, this may not be
necessary.

> start afresh and download the full copy of the latest file.

Good point.

> tunnel a traffic packet to this IR(VP) router for the first time.

The idea is this. When an IR(LF/GW) has a packet to send,
but the only thing it has in its FIB is a VP, it sends a
replicated copy of the data packet to *each IR(VP) that
is listed for the VP in the map*. Each IR(VP) that gets
the data packet will forward it onwards to the final
destination and send back a mapping reply.

So, the IR(LF/GW) will intentionally duplicate initial
packets until it gets a map reply back from an IR(VP).
The IR(LF/GW) then begins tunneling packets through the
route optimized path instead of through the IR(VP).

Note that when the IR(LF/GW) receives a map reply, the
map reply may list multiple RLOCs - perhaps 2, 3, 4, etc.
For robustness purposes, the IR(LF/GW) may then also
send duplicate packets through all of the RLOCs until
the closest one send back a SEAL acknowledgement (at
which point it can cease the packet duplication).

> > be brief for any additional follow up.

We have gotten away from the discussion of whether
there is a way to tell whether an IR is authorized
to forward packets over the IRON using a given
EID prefix. Using SEAL, we can tell whether a packet
has come from an on-path ITR as opposed to an off
path attacker that uses spoofed RLOC addresses, but
that still does not tell whether the ITR is authorized
to send from a given EID prefix. But, this is why I
advocate placing all IRs in the IRON map - so, if
one of the registered IRs starts sending packets
with spoofed EID addresses, it can be reported to
the authorities and punished for its uncivil
behavior. What do you think?

Thanks - Fred

---

From: Robin Whittle <[email-masked]>
Date: Sat, 27 Mar 2010 01:20:45 +1100

1 hour and 40 minutes before the RRG meeting at IETF 77:

Short version:     Fred envisages IRON routers being relatively big
                   and few in number - say 100k or so in total.

                   I suggest that there should be numerous IR(LF)
                   role routers (the equivalent of an ITR) to
                   spread the load of this work, and do this
                   closer to sending hosts.

                   Fred proposes a solution to the problem I raised
                   of tunneling a packet to an IR(VP) role router
                   which may be dead or unreachable, or unable to
                   handle the packet.  Fred suggests tunneling the
                   packet (these are "initial" packets only) to
                   all the IR(VP) role routers, which would be 2, 3,
                   4 or so.   This seems like a good solid solution
                   to me, as long as the potential duplication of
                   packets at the destination network can be handled
                   in some way.  Also, this multiplies the workload
                   of all the IR(VP) role routers in proportion to
                   how many there are of them - compared with
                   load-sharing between then and sending just one
                   copy of the traffic packet to one IR(VP) role
                   router.

Hi Fred,

OK.

OK.

> millions and millions of those, right?

This may be OK for some or many of the IRON routers performing
the IR(EID) role, which to my understanding means delivering
packets to EID-using end-user networks.  This is separate from
the IR(LF) role I invented, which is accepting packets from the
local routing system which are addressed to I-R EID prefixes and
then tunneling those packets.

I think there should be lots of these IR(LF) role routers - the
more there are the less load there is on each one, and so the
cheaper they can be.  This includes putting the function in
sending hosts and all manner of "routers".

> There's always a "but" when we talk about these things,

This field is full of "gotchas"!  Better to find out about them
sooner rather than later.

> same ballpark for unreachability detection.

The IRON router playing an IR(VP) role will be located on some
piece of address space which is advertised in the DFZ.  However
there are other reasons why this router could be unavailable
beyond what would be detected by BGP.  BGP would only detect the
whole route to that prefix being unreachable.  This could be
fine, but the router itself might be dead, being rebooted etc.
Or the router might not be performing this role - or it might be
performing it, but hasn't yet got all its EIDs registered, in
which case it can't handle all the traffic packets which might
be sent to it.

> period of data loss in that case too, right?

As noted above, I think that BGP in the DFZ detecting a prefix
advertised from a BR is now unreachable is not sufficient to tell
any IRON router whether it should or should not try to tunnel to
some IRON router in that prefix which is listed (in the centrally
controlled file) as playing the IR(VP) role for some VP.

> something that should happen every few minutes or even

I agree, but there needs to be a way of dealing with it happening
without traffic packets being sent to it when it suddenly dies,
or when it is listed as performing this role, but is not yet
ready to do so.

OK.

> aggregated by the IR is NULL.

OK - all IRON routers are capable of performing the roles I
describe as IR(LF) and IR(EID).  Still, I think, for discussion,
it is important to be able to identify these roles with a simple
term.

OK.

> of all nodes that are authorized to act as IRs.

OK - but the complete set of IRON routers is much larger than the
subset which perform IR(VP) roles, so the file would be larger.

I am not convinced you need each IRON router to have the address
of every other IRON router.

I prefer the idea of anyone being able to perform IR(LF)
functions on their own servers or routers, without having to
register the device in some central database.

Generally, the more there are of these IR(LF) functions, the
closer they are to sending hosts, the less work each one has to
do and the less subject each one will be to congestion, needing
large caches etc.

> networks as I discussed above?

As I just argued - the more devices which perform the "ITR"
function the better - with people being able to run them without
needing permission, on their own hardware.

Doing this ITR = IR(LF) role is work.  A major goal with Ivip and
I think LISP is to make it easy for people to run their own ITR
functions, closer to the sending hosts and for Ivip, inside the
sending hosts if this is locally desired.  In a sending host, the
cost can be zero in terms of capital, software, extra power,
extra space etc.

> IRON principles can apply at any level of the recursion.

There is a definite divergence here.   My only interest in IRON-
RANGER is as a scalable routing system.  I know it descends from
a broad-scope RANGER system, which did far more things than I
thought were interesting, including interlinking IPv4 and IPv6
in some way, and various "recursive" things I couldn't see the
need for.

> even if some IRs configure a NULL set of EIDs.

Sure - all IRON routers are capable of performing the IR(EID)
role, but only a subset do.  So it makes sense to have a term for
this role.

> useful for this discussion.

OK!

OK.

> arrangements would be necessary in order for an IR to

I am not sure what you are planning regarding this notion of IPv6
sites using IPv4 for all their connectivity, and I don't yet

OK.

OK.

OK.

OK.

OK.

> which point it can cease the packet duplication).

This sounds like a good approach.

It might be subject to critiques about the fan-out of packets to
up to 5 or so VPs being a DoS amplifier of some kind.

However, as long as there are not too many IRON routers playing
the IR(VP) role for each VP, then this will be a reasonably good
arrangement.

We want multiple IR(VP) routers for robustness.  However whatever
the number is, say 4, this quadruples the incoming load of each
such IR(VP) role router.

Also, for these initial packets, it will cause 4 times as many
packets to arrive at the one or more IR(EID) role routers.

Some of the 4 IR(VP) role routers might send to IR(EID) role
router EE and some to another such router FF.

You would need to have some kind of system in EE and in FF to
discard second and subsequent packets, since they should not be
sent to the end-user network, but I can't see a way that EE and
FF could coordinate this.  So there would be cases where EE
forwards its first of 1, 2 or 3 copies of the packet to the
destination network and when FF forwards the first of its 3, 2 or
1 copies of the same packet.

Duplicate packets packets could confuse applications.  They may
not be disastrous, but it would depend on the application.

Would it be relatively easy for a single IR(EID) role router to
discard duplicates of the same packet?  They are tunnelled from
separate IR(VP) role routers, and the decapsulated packets would
look identical.  However, what if the sending host sent multiple
such packets?  That would appear the same, if by chance they
came first via different IR(VP) routers.  Then, the IR(EID) role
router should not be regarding them as duplicates.
router

> behavior. What do you think?

I can't imagine how you could securely and scalably ensure that
each IR(LF) or IR(GW) role router was handling packets whose
source address matched some allowed subset of the address space,
which would be different for each such IR(LF) or IR(GW) role
router.

Nor do I see how this is important or required to make IRON-
RANGER work.

You have however suggested a solution to the problem I mentioned
- of an IR(LF) or IR(GW) router sending a packet to an IRON
router which was listed as performing the IR(VP) role, but was
either not performing it, or was not reachable at that time.  I
suggested buffering, awaiting a mapping reply and then, if no
mapping reply arrives, timing out and sending the buffered
packet to another IR(VP) role router instead.  But as I
acknowledged, by then, the traffic packet would be about a
second old - which is probably too old to be useful.

The "scattergun" approach of sending it to all IR(VP) routers
raises some moderate difficulties with increasing the load
on IR(VP) role routers, and with duplicate packets - but I
think it is in principle a good, solid, solution.

  - Robin