Subject: Re: [rrg] IRON-RANGER, an interesting Core-Edge Separation (CES) architecture

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 6 Apr 2010 14:11:44 -0700

Hi Robin,

> -----Original Message-----
> From: Robin Whittle [mailto:[email-masked]]
> Sent: Tuesday, March 30, 2010 10:19 PM
> To: RRG; Templin, Fred L
> Subject: IRON-RANGER, an interesting Core-Edge Separation (CES) architecture
>
> *** 104 kbyte message ahead. ***
>
> Short version:    Fred's IRON-RANGER CES architecture has been
>                   significantly redesigned in the last few weeks.
>
>                   Here I present an explanation of a subset of this
>                   architecture, as I understand it, since I think
>                   this subset is novel and interesting.
>
>                   I call this subset IRON-RANGER-lite (I-R-lite) -
>                   and I suggest some improvements to this which
>                   may be different from what Fred has in mind.
>
>                   I compare the scaling properties of this modified
>                   subset of I-R to Ivip, which is a more complex
>                   architecture.  I think Ivip scales better - because
>                   I-R's overly simple architecture does not allow
>                   any "cached concentration", or "aggregation" of
>                   the querying demands of ITR-like functions - so
>                   all these queries need to be handled directly by
>                   the authoritative query servers: IR(VP) routers.
>
>
>                   These improvements simplify the I-R system - by
>                   having End User Networks (EUNs) tell VP companies
>                   directly which IR(EID) routers their EID should
>                   be mapped to.  This removes the need for IR(EID)
>                   routers to register the EIDs with IR(VP) routers.

They already do tell the VP companies about their mappings,
but see more below:

>                   With a relatively minor addition to this - Map
>                   Update  messages, which are similar or identical
>                   to Map Reply messages - I-R would have a real-time
>                   mapping system which would be as fast as Ivip.

IRON already has a Map Update message which is a network
layer, unacknowledged, Route Advertisement message with
a signed certificate proving authorization to use a PI
prefix. Since it is an unacknowledged network layer
message (actually just an ICMP) it needs to be sent
repetitively - hence, the IRON-RANGER "bubble-blowing"
behavior. But, see more below:

>                   Real-time mapping distribution to the ITR-like
>                   devices - IR(LF/GW) routers - enables the mapping
>                   to be a single IR(EID) (AKA "ETR") address.  If the
>                   system is then designed to allow only a single such
>                   address, then IR(LF/GW) devices can be much
>                   simpler, since they don't need to test reachability
>                   or choose between multiple IR(EID) routers.  EUNs
>                   would then be responsible (as they are in Ivip) for
>                   changing the mapping in real-time to achieve their
>                   portability, multihoming, inbound TE or other
>                   goals.  (This would enable I-R to support TTR
>                   Mobility as efficiently as Ivip - and better than
>                   LISP - since the MN could drop the tunnel to the
>                   old TTR earlier than if non-real-time mapping was
>                   used.  Note: TTR Mobility only requires mapping
>                   changes when the MN moves more than about 1000km.)

A single mapping would be attractive but then I don't get
the part about "inbound TE" since outside networks would
see only a single way to get to the EUN. Can you define
what you mean by "inbound TE"?

>                   This "I-R-lite" subset of the current IRON-RANGER
>                   design, and its modified version with real-time
>                   mapping could be used for dynamic TE on a
>                   minute-by-minute basis - irrespective of whether
>                   the mapping is simplified to always be a single
>                   IR(EID) address.
>
>                   Still, this souped up real-time version of I-R
>                   won't scale as well as Ivip.  The reason is I-R's
>                   simplicity: the ITR-like devices - IR(LF/GW) - work
>                   directly with the authoritative query servers -
>                   IR(VP).  In Ivip, the ITRs do not work directly
>                   with the authoritative QSAs - their query burden
>                   is "aggregated" by the use of QSR intermediaries.
>                   Optional QSCs between the ITRs and QSRs further
>                   reduce the burden on QSRs.
>
>                   Another scaling problem of I-R is the "scattergun"
>                   approach of tunneling initial traffic packets to
>
>                   all (I assume 3 or so) IR(VP) routers.  This is
>                   somewhat inefficient, but more importantly sets
>                   strict limits on the number of IR(VP) routers
>                   there can be for each VP.  Ivip has no such
>                   inefficiencies or restrictions in the number of
>                   authoritative QSA query servers.

I will discuss a modification to the scattergun approach
below that relaxes the inefficiencies yet still improves
convergence.

> I guess few people have been closely following the discussions
> between Fred and I on his proposal, originally "RANGER", but now
> known as "IRON-RANGER" (I-R).  I wouldn't want to be examined on all
> we have discussed.
>
> Fred responded to my critiques with clarifications and design changes
> and the result is a Core-Edge Separation architecture which I think
> interesting.
>
> I still think Ivip is the best CES architecture (msg06219), but I
> think the current I-R design, or at least my understanding of the
> following subset of it, is superior to LISP-ALT or to any other LISP
> approach I know of.
>
> Here is my description of my understanding of what to me are the most
> interesting aspects of Fred's design.  I will refer to as "I-R-lite".
>
> I haven't checked this with Fred.  Please await Fred's response
> before drawing any inferences about what he is planning.
>
>   - Robin          http://www.firstpr.com.au/ip/ivip/
>
>
>
> I-R terminology
> ===============
>
> Here is a version of the roles which an IRON router can perform,
> based the discussions which lead to (msg06351).  There is a minor
> update to the IR(VP) description.
>
>
>   IR(LF) - Forwarding traffic packets which arrive from its local
>            routing system.  Advertises all I-R "edge" space in
>            its local routing system.
>
>            (All IRON routers are capable of doing this, but it is
>            still a distinct role.)
>
>
>   IR(GW) - Like an IR(LF) role except that the IRON router
>            advertises the I-R "edge" space in the DFZ (that is: in
>            the DFZ, to DFZ routers of other ASes but its own).
>            (This is similar to LISP's Proxy Tunnel Routers and
>            Ivip's DITRs, though a DITR normally only advertises a
>            subset of the "edge" space.)
>
>
>   IR(VP) - One of typically 3 or so. IRON routers which handles
>            a given Virtual Prefix of I-R "edge" space.  This means
>            it:
>
>              1 - Accepts registrations of EID prefixes which fall
>                  within this VP, from the IRON routers which
>                  perform the IR(EID) role for these EID prefixes.
>
>              2 - Accepts tunneled traffic packets from IRON routers
>                  performing the IR(LF) and IR(GW) functions and
>                  then tunnels the packet to the correct IR(EID)
>                  role router for the EID prefix which covers the
>                  packet's destination address.  (This may be the
>                  same router, so no tunneling is required - but
>                  usually it will be another IRON router.)
>
>              3 - After 2, sends mapping for the matching EID to the
>                  IR(LF) or IR(GW) role router.

The exact mechanism used for mapping in I-R is the ICMP
Redirect message (actually, the SEAL SCMP redirect message).

>             If it has no registered EID prefix for a traffic packet
>             it receives, it drops the packet.
>
>    IR(EID) - Accepts tunneled packets from IR(VP), IR(LF) and IR(GW)
>             role routers and then delivers the packet to the
>             destination network.  A multihomed end-user network
>             will have two or more ISPs and there will be an IR(EID)
>             role router for each such ISP.  (Whether this router is
>             in the ISP network or in the end-user network is not
>             fixed, but it is always on a conventional "RLOC" - AKA
>             "core" address.)
>
>
>
> How I-R-lite differs from the full I-R proposal
> ===============================================
>
> "I-R-lite" is described here as a CES system for either IPv4 or IPv6,
> where each of these protocols is handled independently by a separate
> I-R-lite system.  While I-R as Fred plans it is intended to be a
> single system linking IPv4 and IPv6, for simplicity and clarity,
> I-R-lite involves one complete independent system for IPv4 and
> another for IPv6.
>
> (Ivip is the same - two independent systems.  However, I don't rule
> out there being some kind of linkage between the two if the extra
> complexity is justified by some important inter-working or transition
> benefits.)
>
> In this description I won't go into details of encapsulation or of
> how to solve the resulting Path MTU problems.  I figure they can be
> solved one way or another.  I am assuming something like Fred's SEAL
> tunneling would be used, where the outer header's source address is
> that of the Ingress Tunnel Endpoint.  This is different from what I
> plan for Ivip.
>
> The I-R-lite subset I describe below does not support (at least in
> any simple, inexpensive, manner) any ISP BR filtering of packets by
> their source address.
>
> Fred's proposal also apparently involves "recursion".  I don't really
> understand this and I can't see what purpose it might serve in
> scalable routing, so it is not part of I-R-lite.

The recursion part is from RANGER, which talks about
recursively-nested RLOC-addressed local routing regions
joined together by EID-addressed border routers. Without
this recursive approach, there would need to be NAT
traversal even if there were multiple layers of NAT.
I don't want to get involved with designing yet another
tunneling mechanism that supports NAT traversal at this
point, but am rather expecting that border routers can
be established at local routing region boundaries. That
is essentially the same difference as the ISATAP approach
compared to the Teredo approach.

> Fred intends I-R to support Mobility in some way - but as far as I
> know, this is not along the lines of TTR Mobility (Translating Tunnel
> Router), which is the only way I know of doing global mobility, for
> IPv4 and IPv6.  Ivip uses TTR Mobility this, and so could LISP.
> Maybe I-R-lite could use TTR Mobility, but below I ignore Mobility
> and focus only on providing scalable support for non-mobile end-user
> networks (EUNs) gaining portability, multihoming and inbound TE.

The current vision for mobility is to use I-R mapping
updates for PI prefix portability and to use an adjunct
mechanism (e.g., HIP, MIP, etc.) for end system
mobility. If your TTR system could be used instead,
perhaps we should give it a closer look.

> Fred has discussed his intention that each ETE (Egress Tunnel
> Endpoint) router, such as the second in these pairs:
>
>    IR(LF/GW) -> IR(VP)
>    IR(VP)    -> IR(EID)
>    IR(LF/GW) -> IR(EID)
>
> will be able to establish whether the ITE (Ingress Tunnel Endpoint),
> the first of these pairs, is authorised to handle packets from a
> given prefix.  I understand this as meaning:
>
>   ITE-A tunnels a traffic packet which has a source address of XXX.
>
>   ETE-B receives this, and assumes it came from ITE-A.  (However, it
>   may not have, since perhaps it has never received a packet from
>   ITE-A, and so this is the first SEAL encapsulation with a new
>   SEAL-ID which can only later be used to verify packets as having
>   come from the same ITE).
>
>   ETE-B is somehow able to ascertain which prefixes of address space
>   ITE-A is authorised to handle.
>
>   ETE-B drops the traffic packet if its source address does not
>   match one of these prefixes.
>
> I can't imagine how ETE-B could do this - much less how it could do
> so in a tiny fraction of a second, because any longer would
> unreasonably delay the traffic packet.  Also, if the ITE is an IR(GW)
> router, it could be handling packets sent from any network in the
> world - so how could the ETE decide which source addresses were and
> were not acceptable for traffic packets tunneled by an IRON router
> performing the IR(GW) role?

I think this latter point is one which I had not
considered. Clearly, there is no way for an IR(GW) to
tell whether an EID source address has been spoofed
and so there is no way for an ETE to validate source
addresses coming from an IR(GW). Looking at it this
way, it seems reasonable to say that the ETE should
not try to validate EID source addresses coming from
any form of ITE (be it an IR(GW), IR(VP) or IR(LF)).
I will keep this thought in mind as I respond to
further points below.

> I-R currently has no business plan for how the operators of IR(GW)
> routers pay for their operation.  If an IR(GW) router advertised all
> VPs into the DFZ, then it is doing work for all the companies which
> control VPs.  An IR(GW) could also advertise just some VPs in the
> DFZ.  In both cases, there are no current plans for the operators to
> charge the VP companies.  I think this won't work.  In Ivip,
> companies which run DITRs will charge the MABOC (Mapped Address Block
> Operating Companies) whose MABs these are, and furthermore will
> provide the MABOCs with itemised traffic figures for individual
> micronets, so the MABOCs can charge their end-user networks according
> to the DITR traffic which was addressed to the micronets of each
> end-user network.

Why not just have each VP company set up their own
IR(GW)s as part of their cost for doing business?

> I am not promoting my "I-R-lite" idea as a solution to the routing
> scaling problem.  I am suggesting that this subset of Fred's proposal
> is interesting and worthy of the attention of RRG folks.
>
> I also discuss some things which I would do to improve I-R-lite which
> I thin differ from Fred's plans.
>
> Then I discuss how well I think this modified version would scale -
> comparing it to Ivip
>
>
> I-R-lite
> ========
>
> I describe I-R-lite using the current terminology Fred and I
> developed recently for the various roles an IRON router ("IR") could
> perform - though perhaps Fred doesn't see the need for every such
> term.  Any statement below about "I-R" (Fred's IRON-RANGER
> architecture) also applies to my "I-R-lite" unless there is a note to
> the contrary.
>
>
> Traffic packets are tunneled from one IRON router to another.  I-R
> uses Fred's SEAL approach to tunneling and Path MTU discovery, but in
> this description I will simply assume encapsulated tunneling can be
> done, without going into details.
>
> The goals of I-R-lite include:
>
>    Scalable support for end-user networks using an "edge" subset
>    of the global unicast address space to achieve portability,
>    multihoming and inbound traffic engineering (TE).
>
>    Works the same in principle for IPv4 and IPv6.  However,
>    IPv6 address allocation would probably be simpler, since
>    all the "edge" space could come from fresh prefixes, while
>    for IPv4, it must come from potentially very numerous
>    prefixes scattered throughout the global unicast range.
>
>    Complete decentralisation of all crucial aspects of the system.
>    (I-R, as currently designed, does not achieve this because it
>    relies on a single file to be read by all IRs, and for a
>    centralised source of changes to this.  So I-R-lite doesn't
>    achieve this either - but I suggest improvements which would
>    do so.)

I still think we can use a centrally-managed flat
file for this. As an example, the linux kernel is
maintained in a central repository, then pushed out
to "ftp mirror" servers where it can be fetched in
a manner that distributes the load and provides fault
tolerance. So, the master VP mapping file could be
pushed out to mirror sites in realtime, then IRs could
fetch the info from their closest mirror site.

> The non-goals include:
>
>    Mobility - just for this discussion.  (In fact, IRON-RANGER
>    and I-R-lite would support TTR Mobility.  My modified
>    version of I-R-lite with real-time mapping would support it
>    marginally better, as does Ivip.)
>
>    Support for ISP BR source address filtering.
>
>    Real-time control of tunneling.  Therefore, I-R-lite, like
>    LISP (and unlike Ivip) involves mapping for a multihomed
>    EUN consisting of multiple IR(EID) router addresses, with the
>    ITE device having to choose which of these to use.   This
>    raises many problems, and I don't suggest it is the best way
>    to run a CES system, but the LISP team and Fred seem to be
>    happy with it.  (However my modified version of I-R-lite
>    supports real-time control of mapping, like Ivip.)
>
>
>
> Both LISP and Ivip use the term "ITR" and "ETR" in the same way:
>
>   ITR - Ingress Tunnel Router.  Accepts traffic packets and tunnels
>         them to an ETR.
>
>   ETR - Egress Tunnel Router.  Receives these tunneled packets and
>         delivers them to the destination network.
>
> I-R does not use these terms, but the terms ITE and ETE (Ingress /
> Egress Tunnel Endpoint) means much the same thing.
>
> LISP, Ivip and I-R are all capable of handling packets sent by hosts
> in networks which lack ITRs (ITEs).  In order to do this, each
> architecture has a special subset of these devices which, instead of
> advertising "edge" space in an ISP or End User Network (EUN),
> advertise it to neighbouring ASes - they advertise "edge" space in
> the DFZ.
>
>   Ivip:    DITR (Default ITR in the DFZ).  Previously known as
>            OITRDs and before that, erroneously, as "Anycast ITRs
>            in the Core/DFZ".   DITRs generally only advertise a
>            subset of the "edge" space.
>
>            Ivip "edge" space is advertised in the DFZ by these
>            DITRs as separate "Mapped Address Blocks" (MABs),
>            each of which typically covers the space used by many
>            EUNs.  An IPv4 /16 MAB would typically provide "edge"
>            space for thousands or tens of thousands of EUNs.
>
>   LISP:    PTR - Proxy Tunnel Router.  As far as I know, each PTR
>            advertises all the "edge" space.  There is no specific
>            LISP term equivalent to Ivip's "MAB", but such a term
>            is needed.  Dino recently used "coarse prefix" to refer
>            to the same thing.
>
>   I-R:     IRON routers performing the IR(GW) role.  As with LISP
>            I understand each such router advertises all the
>            "edge" space.  The closest thing to a MAB in I-R is
>            a "Virtual Prefix" - "VP".
>
> In LISP and I-R, for IPv6, if the "edge" space is always in a
> currently unused part of the address space, then each PTR or IR(GW)
> router could probably advertise the whole "edge" space to its
> neighbouring DFZ routers with a single prefix, or just a few prefixes.
>
> In the same circumstance, Ivip's DITRs would not do this, since each
> one typically only advertises a subset of the total set of MABs.
> This is because each DITR site is run by a company which typically
> works for a subset of the MABOCs.
>
> This means that for IPv6, in the above scenario, both LISP and I-R
> could have very large numbers of "coarse" prefixes or VPs and still
> only add one or a very small number of prefixes to "the DFZ routing
> table" (shorthand for the set of globally advertised prefixes which
> every DFZ router needs to handle).

My original thinking stemmed from IPv6 as EID space and
IPv4 as RLOC space. The expectation was that every IR(VP)
would attach to the IPv4 DFZ, but not every IR(VP) would
attach to the IPv6 DFZ. The further expectation was that
IPv6 space would be separated into EID space and non-EID
space. This bifurcation of the IPv6 address space would
make it such that non-EID IPv6 prefixes would be routed
wholly within the IPv6 DFZ while EID IPv6 prefixes would
be routed within the IRON.

Since there will be O(10^5) or even more IR(VPs), however,
we cannot have each of them participate in the IPv6 DFZ
BGP since the scaling would become untenable - right? But,
if we stipulated that each VP company has to have at least
one IR(GW) then we could have that company's IR(GW)
advertise all of its VPs into the IPv6 DFZ but no other
VPs. So, each VP company is responsible for running its
own IR(GW) if it wants its VP space to be reachable via
the IPv6 DFZ.

> Both Ivip and LISP-ALT involve a single tunneling arrangement: from
> the ITR to the ETR.  LISP-ALT optionally allows the ITR to send (in a
> different form of tunneling) the initial(*) traffic packets to the
> ETR, via the ALT network.  But otherwise, LISP-ALT involves ITRs
> tunneling traffic packets directly to ETRs, just like Ivip.
>
>   * "Initial traffic packets" is shorthand for those packets
>     an ITR or ITE device receives before it has mapping for how
>     to properly tunnel such packets to the final destination device -
>     which for Ivip is a single ETR, for LISP is one of several ETRs
>     and for I-R is one of several IRON routers performing the IR(EID)
>     role for the matching prefix.
>
> I-R is fundamentally different from Ivip and LISP in this regard.
> The initial traffic packets are tunneled by an IR(LF/GW) router to an
> IR(VP) router which tunnels them to the correct (of potentially
> several) IR(EID) routers.
>
> This is the general process, but it is possible that the IRON router
> playing the IR(VP) role (and there are typically 2, 3 or 4 such
> routers for any one VP) is also the correct router of the potentially
> multiple IR(EID) routers for the EID prefix which the packet is
> addressed to.  In this case, there is no need for a second stage of
> tunneling.  Also, if the IR(VP) role router is also acting as an
> IR(LF) or IR(GW) router, it may accept the traffic packet from its
> local network or the DFZ with this role, and so have no need to
> tunnel it to a separate IR(VP) role routers, since this router also
> happens to be one of the IR(VP) role routers for the VP which covers
> the traffic packet's destination address.

Yes, when the EID prefix is "at home" (i.e., the IR(VP) is
also the IR(EID) that owns the EID prefix) there is no need
for the indirection and even initial data packets flow along
the correct path.

> As soon as the IR(VP) router receives such a traffic packet, it
> regards it as an implicit Map Request, and sends back some "mapping"
> information to whichever IRON router (performing an IR(LF) or IR(GW)
> role) tunneled the packet to the IR(VP) router.   This was initially
> described as a "route redirection" message, but I will refer to it as
> simply a Map Reply message.

The mechanism is a network layer ICMP redirect. That will
remain as the mechanism for map-reply, since it is the
means by which routers on NBMA links inform neighbors of
a better nexthop.

> In the following example, I refer to an IR(LF) router - which
> advertises all the "edge" space in the routing system of whichever
> ISP or end-user AS network it resides in.  The same example would
> apply to an IR(GW) router which does the same thing - except that it
> advertises all "edge" space in the DFZ.
>
> In this example, an IR(LF) router advertises all the "edge" space in
> its local routing system and so receives a traffic packet addressed
> to 44.44.01.02.  The IR(LF) router recognises this as being within
> the "edge" subset of the global unicast address range, but does not
> have any mapping for an EID prefix which covers this address.   It
> does this by knowing already every VP in the system, and recognising
> that this address falls within one of those VPs.
>
> In this example, there is a VP 44.44.0.0 /16 - though perhaps this is
> rather large for VP - maybe Fred intends they be somewhat smaller,
> such as a /18 or /20, to better share the load on a greater number of
> IR(VP) routers.  The load depends very much on the traffic volumes,
> not so much on how much address space is within each VP.
>
> The EID prefix which matches this destination address is 44.44.01.00
> /28 - but the IR(LF) router doesn't have any knowledge of this.  If
> it did, it would have mapping for this EID prefix, and wouldn't need to
> tunnel the packet to an IR(VP) router.
>
> The IR(LF) router tunnels the packet to *all* of the IRON routers
> which are playing the IR(VP) role for this VP: 44.44.01.0 /28.  (How
> they know the addresses of these IR(VP) routers is described below.)
>  The reasons for tunneling to all these IR(VP) routers is to avoid
> the need for buffering the packet, while ensuring that at least one
> copy of the packet will arrive at a working IR(VP) router.  (See the
> most recent discussions with Fred about this.  I referred to this as
> the "scattergun" approach.  It raises a few inefficiency - and
> therefore scaling difficulties - but I think it is a good way of
> solving the problem of IR(VP) routers being suddenly unreachable or
> dead.)

I have a new proposal - call it the "modified scattergun".
The whole reason for the scattergun approach was to avoid
packet and mapping loss due to an unreachable ETE. But,
the main problem was that the final destination would get
multiple copies of the same packet and potentially become
confused. In the "modified scattergun" approach, the ITE
selects 2-4 IR(VPs) that are close by (see more below).
It selects one of these as the "primary" and names the
others as "backups". The ITE sends initial packets to
each IR(VP), but marks the packets sent to the primary
as "live" and the packets sent to the backups as "blanks".
When an IR(VP) receives a packet for which a mapping
reply must be sent, it returns an ICMP redirect. If
the packet is marked "live" the IR(VP) also forwards
the packet toward the final destination. If the packet
is marked "blank", however, the IR(VP) simply drops the
packet. In this way, initial packets will *probably*
get through, but a mapping reply will *almost certainly*
be received. For that matter, the "blanks" need not even
contain identical data to that contained in the original
data packet; they could simply be NULL packets that are
used solely for the purpose of triggering a map reply.

> Each IR(VP) has (ideally) complete knowledge of the mapping for all
> the EIDs in its VP.  (How it gains this is discussed below.)  For
> each traffic packet it receives from an IR(LF) or IR(GW) router, it
> sends back a Map Reply packet with mapping information for the
> matching EID.
>
> (Whether it should do this for every such packet, or just for the
> first few, could be debated.  Since there could be a flurry of such
> traffic packets, it is probably good enough to send mapping for just
> the first 2 or 3.)

The solution to this is to use ICMP rate limiting the same
as specified in RFC4443. It is fine if no mapping reply is
received for each and every packet that needs to be redirected
as long as at least one mapping reply eventually gets through.
The modified scattergun approach described above will ensure
faster convergence, however.

> Below, I assume there are always 3 IR(VP) routers for each VP.
>
> Assuming, for simplicity, that there is a single IR(EID) router, the
> paths taken by an initial traffic packet are:
>
>                 /--->--IR(VP)1-->--\
>                /                    \
>   SH-->--IR(LF)---->--IR(VP)2-->----IR(EID)--->--Destination-EUN.
>                \                    /
>                 \->--IR(VP)3-->----/
>
> Fig 1.
>
> Presumably, there will be a method by which the IR(EID) router can
> ignore the second and third copies of these packets, and only send
> the first one to the destination EUN.  I don't know how Fred plans to
> do this, but I will assume something can be added to the SEAL
> tunneling protocol to make it easy for the IR(EID) to recognise these
>  three packets as all resulting from the one initial traffic packets.

No need for this using "modified scattergun". There will
only be one "live" packet, and the rest are "blanks".

> This can't be done by simply looking at the traffic packet, since
> perhaps the sending host sent several identical packets.
>
> Probably the way to do this is something like this:
>
>   The IR(LF) creates a nonce for this particular traffic packet
>   and includes it in the SEAL headers when it tunnels it to each
>   IR(VP) router.
>
>   This can't be the SEAL ID, since it maintains separate (random
>   start, monotonically incrementing) SEAL ID sequences for each
>   IR(VP) router.
>
>   Each IR(VP) router includes this nonce in the packet it sends to
>   the IR(EID) router.
>
>   The IR(EID) router uses the nonce to identify unique traffic
>   packets and so to discard the second and subsequent copies
>   it receives, sending only the first to the destination EUN.
>
> This looks OK, but it adds to the length of the SEAL header.  See
> below for a gotcha when there are two or more IR(EIDs).  Such nonces
> are only needed for these initial packets which get tunneled to all 3
> or so IR(VP) routers - the nonce is not needed for traffic packets
> tunneled directly from an IR(LF) router (which in this section of the
> discussion includes all IR(GW) routers) to an IR(EID) router.
>
>
> Before, while or after each IR(VP) router tunnels the traffic packet
> (perhaps with the nonce I suggested above) to the IR(EID) router, it
> also sends a Map Reply packet to the IR(LF) router:
>
>
>                 /---<--IR(VP)1
>                /
>          IR(LF)----<--IR(VP)2
>                \
>                 \-<--IR(VP)3
>
> Fig 2.
>
> The map reply packets are secured with the SEAL-ID the IR(LF) router
> used when tunneling to each IR(VP) router.
>
>     (I recall that SEAL-IDS are 32 bit integers, starting at some
>     random value by any particular ITE router at the time it first
>     tunnels a packet to any particular ETE router.  Then the number
>     is incremented for every packet tunnelled from this ITE to ETE.
>     Off-path attackers are assumed not to be able to guess the
>     starting or the current value.  There may be some gotchas in how
>     the routers at each end easily recognise valid and invalid
>     SEAL-IDs without having to cache each one.  Fred and I discussed
>     this in the past month or two, and I recall there was probably a
>     way of doing it.  However I think it would be tricky or perhaps
>     impossible to implement by using simple counters to frame a range
>     of acceptable values, rather than by caching the use of each
>     value along with a caching time, or time of use - to give the
>     effect of maintaining a timer for each individual SEAL-ID.)
>
> All traffic packets this IR(LF) receives before it gets a "map reply"
> are "initial" packets.  As soon as the IR(LF) has mapping in its
> cache which covers the EID prefix - in this case 44.44.01.0 /28 -
> then all further traffic packets it receives which match this EID
> prefix are no longer "initial" and so are not tunneled to all the
> IR(VP) routers for the matching VP.  All these subsequent traffic
> packets are tunneled to the IR(EID) router:
>
>
>   SH-->--IR(LF)---->------------->--IR(EID)--->--Destination-EUN.
>
> Fig 3.
>
> For a multihomed EUN, there will be two or more IR(EID) routers, so
> the process could be more complex.  Depending on how the priorities
> and weightings or whatever are defined, and the individual
> perspective of each of the three IR(VP) routers, it is possible that
> one or more IR(VP) routers will choose to tunnel the packet to a
> different router IR(EID)1 than one or more of the other IR(VP)
> routers, which tunnel their copies of the traffic packet to IR(EID)2.
>
> Then instead of the initial packet flow of Fig 1, we could get:
>
>
>                 /--->--IR(VP)1-->--\
>                /                    \
>   SH-->--IR(LF)---->--IR(VP)2-->----IR(EID)1--->--Destination-EUN.
>                \                                 /
>                 \->--IR(VP)3-->----IR(EID)2-->--/
>
> Fig 4.
>
> This would result in a duplicate copy of the traffic packet arriving
> at the destination EUN.  A assume this cannot be tolerated, so I
> guess Fred will devise a way of preventing this.
>
>    Note: below I suggest I-R adopting Ivip's approach of the mapping
>          containing just a single address, that of one IR(EID)
>          router.  If this is adopted, then this duplicate packet
>          problem can't occur.
>
> If both IR(EID)1 and IR(EID)2) were at the destination EUN, then
> there would be a simple solution, since they could compare notes
> about nonces and so determine which was the first arrived copy of the
> traffic packets and which were duplicates which could be dropped.
> For instance, the two functions of IR(EID)1 and IR(EID)2 could be
> performed by the one router.  Each would have a different "core"
> address of course, IR(EID)1 from PA space from one ISP and IR(EID)2
> from PA space from another ISP.
>
> However, if IR(EID)1 and IR(EID)2 are not in the same place or device
> - such as if they were in separate ISP networks - then I think the
> problem of duplicate packet delivery to the destination EUN is a
> serious "gotcha".  I think it could only be solved by having a
> special router function at the EUN to distinguish between duplicate
> packets, and by the IR(EID) routers tunneling packets to it, with the
> nonce I suggest above which would be generated for each initial
> traffic packet by the IR(LF) router.

The whole business of duplicate packet detection goes
away when we use "modified scattergun". "Blanks" are
discarded before any IR(EID) ever sees them, and only
the "live" packet gets through to the final destination.

> The IR(VP) routers somewhat resemble a "Default Mapper" (DM) in APT.
>  However, in APT, the DM was within the same (ISP, typically) network
> as the ITR, and I recall the ITR simply forwarded the traffic packet
> to one of the potentially multiple DMs in its network.  I also recall
> that the DM interpreted the mapping information, chose a particular
> ETR to tunnel the packet to, and then in the Map Reply message to the
> ITR, told it simply to tunnel packets to this ETR if they matched the
> the EID prefix which was also in the Map Reply message.

Yes, each IR(VP) is essentially a "default mapper" for the
portion of VP space that it owns.

> I-R's forwarding of initial packets also resembles the use of
> LISP-ALT where traffic packets are sent on the ALT network.  I think
> the ALT network is a bad idea, since the paths can easily cris-cross
> the globe and because there are fundamental difficulties making it
> scale well, while maintaining generally short paths while also
> avoiding single points of failure.  In LISP-ALT, there isn't an exact
> equivalent to an IR(VP) router or an APT DM - since the ALT network
> would deliver the packet to one of the correct ETRs.
>
> I-R involves tunneling initial packets directly from the IR(LF)
> router to the IR(VP) router (actually to all the 3 or so IR(VP)
> routers) and then directly from the IR(VP) router(s) to the IR(EID)
> router(s).  This tunneling is via the Internet - there are two such
> tunnels, or really three parallel parallel paths, each with two
> tunnels in series, as shown in Fig 1 and Fig 4.
>
> LISP-ALT's delivery of initial packets is by forwarding over an
> overlay network composed of Internet tunnels between ALT routers.
> The traffic packet has to be forwarded typically by multiple ALT
> routers, and this involves multiple Internet tunnels, since there is
> a tunnel between each router.  There could be half a dozen or a dozen
> such tunnels - maybe more, it depends on how the ALT system is
> structured, which has never been described in detail.
>
>
> There needs to be a careful choice about how many IRON routers
> perform the IR(VP) role for each VP.
>
>   1:  Too few.  Single point of failure.
>
>   2:  Better, but still pretty highly strung with dependence on
>       only two VP routers being alive and reachable - and lots
>       of EIDs prefixes depend on this.
>
>   3:  Better still - maybe a good choice.
>
>   4:  Maybe a good choice.
>
>   5: Probably an excessive number.
>
>
> I suggest 3 or 4 as good choices.  More than this and there are
> efficiency concerns, since each initial packet needs to go to every
> IR(VP) router.

Actually, I am beginning to think that the number of
IR(VPs) that service the same VP need not be limited
based on the number of IR(VPs) that an IR(LF) will
choose to forward initial packets to. There could be
perhaps as many as 10-20 IR(VPs), and the IR(LF) need
only select the 2-4 IR(VPs) from the list that are
"nearby".

> Also, for a choice of 4, assuming all the IR(VP)
> routers are up and reachable, this quadruples the workload of the
> IR(EID) routers in handling initial packets and the nonces I suggest
> above.   Furthermore, since there are likely to be two or perhaps
> more IR(EID) routers, and since different IR(VP) routers might choose
> a different IR(EID) router, there would be more trouble with
> duplicate initial packets going over the final links to the
> destination network.

This will not be a problem with "modified scattergun".

> Choosing a higher number of IR(VP) routers has one advantage in
> addition to increasing robustness.  It will also tend to mean that
> the shortest path from the IR(LF) router to the IR(EID) router(s) via
> any of the IR(VP) routers will tend to be shorter - since the more
> IR(VP) routers there are (presumably scattered widely around the
> Net), the greater the chance that one will result in a short total
> path from the IR(LF) to it, and then to the IR(EID).

I was actually thinking that each IR(VP) could be marked
with its LAT/LON coordinates in the VP master map file.
That way, the IR(LF) can always choose the IR(VPs) that
are physically closest to it. That way, initial packets
would never go Sacramento->Sydney->Seattle when they
could have gone Sacramento->Portland->Seattle.

> Below, I will assume there are exactly 3 IR(VP) routers for each VP.
>  I also assume that these are geographically and topologically
> located in a diverse manner, so an outage which makes one unreachable
> is unlikely to affect the other two.
>
> I will also assume that all the EUNs which use I-R "edge" space are
> multihomed - typically with two ISPs and so with two IR(EID) routers,
> but occasionally with 3 or 4.
>
>
> This "I-R-lite" subset of IRON-RANGER is a novel and interesting
> arrangement.
>
> Every IRON router can perform the IR(LF) role, and some of them are
> configured to do the same to the DFZ, and so be perform the IR(GW)
> role.   Neither of these roles involves the router initially knowing
> any mapping at all.  All they need to know is:
>
>    1 - A complete list of VPs.
>
>    2 - For each VP, 3 (in my example) IRON routers which are
>        performing the IR(VP) role for this VP.
>
> Then, after they tunnel traffic packets to all the IF(VP) routers for
> a given VP, they get back a Map Reply which covers a specific EID
> prefix - and don't need to bug the IR(VP) routers again about traffic
> packets whose destination addresses match this EID, for a period
> specified by the caching time in the Map Reply message.
>
> Fred suggested that all IRON routers discover the above by
> downloading a single file which contains all this information - when
> they boot - and then doing regular checks, which I call "delta
> checks" with some network of centrally coordinated servers to find
> out how the file changes over time.
>
> For a full-scale deployment with tens of thousands of VPs, perhaps
> 100k of them - the size of this file is going to be a few megabytes,
> and the rate of change is going to be quite slow.  The file would
> contain either IP addresses or FQDNs of the IRON routers playing the
> IR(VP) role.

I like the idea of FQDNs, since it would allow each VP
company to have finer-grained control over its RLOC
mappings. However, this would not allow for labeling
each RLOC with "location" information like LAT/LON
such as I mentioned above unless there were some sort
of DNS resource records that could convey location
information. I will look into this.

> I think this is practical, but there objections to the centralised
> nature of this one file, which I discuss below.  For now, I assume a
> single file and that each IRON router would do a delta check, on its
> own schedule unsynchronised with other IRON routers, every 10 minutes.
>
> This is my choice for an example.  Exactly what the time would be, or
> should be, hasn't yet been discussed.
>
>
> Let's say that at 3:00 UTC a particular IRON router 22.22.88.88 is
> configured to be one of the 3 IR(VP) role routers the VP previously
> discussed: 44.44.0.0 /16.
>
> By 3:10 UTC, in theory, we can assume that every IRON router knows
> this.
>
> This information is used for two purposes:
>
>   1 - As described above, for an IR(LF) or IR(GW) router to
>       decide which 3 (in my example) IRON routers to tunnel
>       an initial packet to, based on the traffic packet's
>       destination address matching a given VP.
>
>   2 - As not so far mentioned, how the 2 (or perhaps 3 or 4)
>       IR(EID) routers for a given EID in this VP register themselves
>       with each of the three IR(VP) routers for this VP.
>
> Registration will take some time.  Fred roughly described a way of
> doing it, involving a message with some crypto stuff (such as using
> signatures an PKI I think) so the IR(VP) router could tell from the
> message sent by the IR(EID) router that it really was authorised as
> performing this role for a given EID.  Below I suggest some
> improvements to this, but for now I an discussing the I-R-lite subset
> of what I understand to be Fred's I-R design.
>
> Let's say it takes each IR(LF) router up to two minutes to register
> itself with the newly established IR(VP) router.  Now, for Justin
> (Just In Case) we add a 3 minute fudge factor and decree that within
> 15 minutes of an IR(VP) role for a router being made available on the
> master list (and via delta checks) that any self-respecting IR(EID)
> router for an EID within this VP can be assumed to have completed its
> registration with the new IR(VP) role router.
>
> This enables a simple solution to the problem of IR(LF) and IR(GW)
> routers tunneling packets to newborn IR(VP) routers before they have
> had time to get registrations for all their EIDs:
>
>    IR(LF) and IR(GW) routers will wait 15 minutes after the
>    appearance of a new IR(VP) router before tunneling any
>    traffic packets to it.
>
> However, since it could take 10 minutes for any one IR(LF) or IR(GW)
> router to discover that the new IR(VP) router has been established,
> then this blows out the maximum time before the IR(VP) router is
> doing its full workload to 25 minutes.  This can be fixed by all IRON
> routers knowing the current UTC and by the master file and its delta
> updates including the UTC time each IR(VP) role for a given VP was
> established.  Then, we would have a 15 minute time before each IR(VP)
> role was fully operational.
>
>   (Fred did not want an IR(VP) role router to tunnel a traffic packet
>    to any other IR(VP) router just because it lacked a registration
>    for an EID which matches its destination address.  I think this
>    dropping of such packets is a good idea.)

OK.

> One thing I haven't explored with Fred is the creation of the
> "mapping" information, and how the 3 or so IR(VP) routers might work
> together on this.
>
> I understand that each IR(VP) role router would receive an EID
> registration from each of the typically 2 or more IR(EID) routers for
> a given EID.  Somehow, the IR(VP) router (perhaps after comparing
> notes with the other 2 or so IR(VP) role routers) must develop
> "mapping" which can be sent to the IR(LF) and IR(GW) routers.
>
> I understand from previous discussions that the mapping will resemble
> that of LISP:
>
>   The EID: start address and the length of the prefix in bits.
>
>   The addresses of the typically 2 or perhaps more IR(EID) role
>   routers for this EID.
>
>   Information on preferences and weightings to control load sharing
>   between them, including no load sharing: sending all packets to
>   IR(EID)1 and not IR(EID)2, unless IR(EID)1 is dead, unreachable
>   or unable to get packets to the destination network.
>
>      (IRON-RANGER seems to have the same problems as LISP in the
>      ITR / IR(LF) / IR(GW) routers having no direct way of testing
>      the reachability of the destination network through each
>      ETR / IR(EID) router.  Ivip has no such problem, since there
>      is a single ETR address in the mapping, and the end-user network
>      is responsible for changing the mapping in real-time.  The end-
>      user network, or some company they appoint to do so, can easily
>      test reachability of the network through each ETR, since it
>      knows one or more hosts or routers on the network it can try to
>      get a response from via each ETR.  LISP ITRs, APT ITRs/DMs and
>      I-R IR(LF/GW) routers are given no information about a host or
>      router in the destination network by which they could test
>      actual reachability of the network through each ETR / IR(EID)
>      router.)

Let's come back to this. Mapping to a single IR(EID)
would certainly be a big improvement in terms of reducing
complexity for the IR(LF). It also places the reachability
burden on the end user network, since it will need to
test that its mappings are reachable in real time.

However, if the IR(LF) is told of a single mapping to
an IR(EID), and the network is somehow partitioned,
the IR(LF) will have no alternate path and traffic
will black-hole. Like I said, let's come back to
this...

> It is not clear how the IR(EID) role routers give this preference or
> weightings or whatever to each of the IR(VP) routers they register
> with - or how the IR(VP) routers individually or collectively decide
> on the mapping for this EID.  Somehow, each IR(VP) router must have
> already computed mapping for each registered EID in its VP.  It needs
> this when it tunnels a traffic packet addressed to this EID - and it
> needs it to create the Map Reply it sends to IR(LF/GW) routers.
>
> In recent discussions, there was a difficult problem when the IR(LF)
> router tunneled a packet to just one IR(GW) role router.  What if the
> IR(GW) role router was dead, being rebooted, or unreachable?  The
> traffic packet would be lost, and there is no reliable way of finding
> out in a fraction of a second that this had occurred.
>
> Fred's "scattergun" approach solves this nicely - the IR(LF) router
> tunnels the traffic packet to all IR(VP) role routers (according to
> the 15 minute start-up time arrangements just described).  So as long
> as at least one of these IR(VP) routers is reachable and alive, the
> traffic packet won't be lost.  The first Map Reply from any of these
> tells the IR(LF) router not to tunnel packets to the IR(VP) routers
> any more - but to use the mapping to tunnel them directly to the
> correct IR(EID) router out of 2 (or perhaps 3 or 4) specified in the
> mapping.

"Modified scattergun" will only send one live packet and
2-3 blanks. Chances are still very good that the live
packet will get through, but the chances that a mapping
reply will eventually be received are almost certain.

>   (In the above discussion, for brevity, I used "IR(LF)", but
>   all the above also applies to routers playing the IR(GW) role
>   too.)
>
>
> Fred also suggested that the central file should list every IRON
> router.  I think this is unnecessary since in the functionality I
> described above for I-R, all of which is part of my I-R-lite subset,
> there is no need for IRON routers to know about all other IRON
> routers.  They only need to know about the subset which is performing
> the IR(VP) role for one or more VPs.

The reason for listing all of the IRs was so that
there could be an ingress filter of RLOCs that an
ETE could legitimately accept traffic packets from.
RLOCs not listed in the file would be considered
as not authorized to act as ITEs. But, let's save
that thought for now and discuss further below:

> He mentioned there would only be 100k or so IRON routers, and that
> these would be larger routers in ISPs, or similar.
>
> Fred's ~100k estimate is subject to the critique that it forces all
> this ITE work to be done by a relatively small number of routers,
> when it would be desirable to have the option of spreading it out
> over more numerous, lower capacity, less expensive devices which are
> generally closer to sending hosts.

OK, but I really prefer to not get into the business of
NAT traversal. NAT traversal comes with a host of sticky
issues even if UDP encapsulation is used. NAT traversal
mechanisms such as STUN, ICE and Teredo are complicated,
and have to deal with the ways in which the myriad different
types of NATs maintain state. I want to keep the ITE/ETE
implementations simple, which to me means no NAT traversal.

> Below, in the improvements section I envisage many more IRON routers
> - primarily to perform the IR(LF) role.
>
>
> This single file arrangement, with a single source of information for
> the delta checks (even though the delivery of the file and the delta
> checks themselves might be farmed out in a distributed manner) is
> subject to the critique of being overly centralised, and so having a
> single point of failure.  Below, in the improvements section I
> suggest some alternative arrangements which would not be subject to
> these critiques.
>
> I can't remember what Fred's design involves in terms of "Map
> Updates".   An IR(VP) router may at some stage become aware that a
> given IR(EID) registration has lapsed or been cancelled.  (Fred
> hasn't described a cancellation arrangement, but I assume this could
> be added securely as an extension of the registration arrangement, so
> an IR(EID) role router could de-register itself.)

The current mechanism of sending periodic network layer
router advertisements has two possibilities for EID
registration cancellations: 1) prefix expirations, and
2) explicit prefix cancellations. Prefix expirations
occur when a prefix refreshment has not been received
within a timeout window, and prefix cancellations
occur when a prefix registration with zero lifetime
is received. This is all network layer, however, so
has no means for reliable delivery. I think we will
probably need a method for reliable prefix registrations
and cancellations (see more below).

> Does the IR(VP) role router maintain in its cache a record of every
> map reply it sent, in a form by which it can find those sent in the
> current caching time which concerned an EID prefix whose mapping has
> just changed?   If so, can it then send a Map Update message, with
> the new mapping for this EID, in a similar fashion to the Map Reply
> message it sent to each such IR(LF/GW) router which tunnels an
> initial packet to the IR(VP) router.

NO; I do not want the IR(VP) routers to send updates to
every IR(LF) that it previously sent a map reply to. The
map reply process is stateless. Instead, when an IR(EID)
no longer services an EID prefix it will begin dropping
packets addressed to this EID prefix and send back ICMP
(or SEAL SCMP) unreachable messages. The unreachables
tell the IR(LF) to delete the current mappings and allow
subsequent packets to flow through the IR(VP)s again.
More on this later.

> With these caveats, the subset of I-R I have presented here - my
> "I-R-lite" subset - has some interesting properties:
>
>   1 - All IRON routers can perform - and generally do perform -
>       the IR(LF) or perhaps IR(GW) roles.  To do this, they need
>       only to cache the mapping information sent by IR(VP)
>       routers.

Right, but the caching information has a non-infinite
lifetime and is subject to invalidation based on
receipt of "unreachable" messages (see above).

>   2 - There is no central repository of mapping.  (Only a central
>       repository of which IRON routers are performing IR(VP) roles
>       for which VPs.)

Right. Mappings are maintained between the IR(VP)s and
IR(EID)s only and are kept out of the global IR(VP)
mapping table. That is the way in which the system is
kept scalable.

>   3 - Initial packets get delivered reliably and pretty quickly.

Delivery of initial packets will be best-effort, which is
pretty much the existing condition in the Internet today.
Delivery of mapping replies will be better than best effort,
however. These features are due to the characteristics of
"modified scattergun".

>       With three or so IR(VP) routers, this means at worst-case the
>       packet will traverse the Earth and come back again.  For
>       instance, if the SH and IR(LF/GW) router is in South Africa
>       and the closest operational IR(VP) router is in Vancouver,
>       and the destination network and its IR(EID) router is in
>       Italy, then the initial packets need to go from South Africa,
>       to North America, and then back to Italy.

No; the IR(LF) will choose only the closes IR(VP)
to forward initial packets to, and also the 2-3
next-closest IR(VP)s to forward blanks to. Initial
paths will therefore be kept short.

>       The typical initial packet path for I-R will probably be
>       longer than the typical outcome for Ivip, but I think it
>       will typically be better than using LISP-ALT to deliver
>       initial packets.  It will certainly be superior to the current
>       LISP-ALT arrangement of ITRs dropping all initial packets until
>       a new packet arrives after the map reply arrives.
>
>
>
>
>
> Improving I-R-lite
> ==================
>
> Splitting up the VP file
> ------------------------
>
> The first obvious improvement is to decentralise the master VP file.
>  Having a single file or a single organisation controlling files in
> multiple locations, represents a single-point of failure for the
> entire system.  It would be better if a failure by one organisation
> only affected a subset of the IR "edge" space - the subset which that
> organization is paid to be responsible for.

I really like your idea of having the centralized file
contain only an FQDN for each VP, and having the VP
maintain the resource records in the DNS for each FQDN.
If there would only be a way for the RR's to include
location information we would then be in business.

> I-R "edge" space is all within VPs.  Presumably each VP has a single
> organization which is responsible for it.  That VP organization is
> the only one which would want to run this VP's 3 or so IR(VP) routers
> - and the only organization which should be allowed to control them.

There could be many more IR(VP)s than "3 or so" for
each VP as long as our "modified scattergun" is not
required to send to each and every IR(VP). Or, is
there some other constraint that I am not seeing?

> It seems likely that one organization will be responsible for more
> than one VP, at least in IPv4 where various small (long prefix) VPs
> might be carved out the address range wherever possible.  In IPv6,
> there's almost endless space, so all the VPs would be on fresh unused
> space, and have vast address space within them - so each organization
> could have a single VP with huge capacity.
>
> However, if there is a lot of traffic to the VP, as there would be,
> this would overload a single set of IR(VP) servers - so even with
> IPv6, VPs will need to be kept small enough that the traffic for each
> VP is within the scaling limits of the set o3 3 or so IR(VP) servers.
>
> One way or another, the VP organizations could decide which of some
> smaller number of "VP file" companies they wanted their VP to be
> handled by.  Then, for load sharing and splitting up the whole system
> into smaller chunks, each IRON router would download a file from each
> such "VP file" company, and likewise do the 10 minute or whatever
> interval delta checks with each such company.
>
> However, if there were more than a few dozen such "VP file"
> companies, this means each IRON router has to do quite a lot more
> delta checks to the servers of each such company.
>
> The servers for the VP file companies could be discovered by DNS in
> some way.
>
> There's no free lunch here - but the current design of a single file
> for all VPs is not the only way the same basic architecture could be
> implemented.  Having some kind of flexibility and multiple smaller
> files seems like a good idea.
>
>
> Alternatives to the VP file
> ---------------------------
>
> It would also be possible for IRON routers to discover all the VPs
> and which 3 or so IRON routers perform the IR(VP) role for each VP by
> a method similar to what I propose in Distributed Real Time Mapping.
>
> Later, this will be fully described in:
>
>   http://tools.ietf.org/html/draft-whittle-ivip-drtm
>
> At present (version 01) this ID doesn't yet include this, so please
> refer to the section "Stage 2 needs a DNS-based system so TRs (QSRs)
> can find DITR-Site-QSDs (QSAs)" of:
>
>   http://www.ietf.org/mail-archive/web/rrg/current/msg06128.html
>
> This would involve each IRON router walking a special part of the DNS
> which describes VPs and non-VP areas, where the DNS replies for the
> VPs also contain by some means the IP addresses of the IR(VP)
> routers.  Instead of "delta checks", there would be DNS cache
> time-outs for the information which was discovered in this way, so
> there would be a continual series of DNS requests for these items
> from all IRON routers.   This has some scaling problems, because
> there are larger numbers of IRON routers than there would be QSRs in
> Ivip, but it is probably a valid way of replacing the file-download
> with delta check approach with something more decentralised.  If
> multiple IRON routers shared a single caching DNS resolvers, then the
> DNS resolver would cache a lot of this, so the authoritative
> nameservers for these items wouldn't have to handle all the queries.

I think the approach that is beginning to make the most
sense to me is flat-file of FQDNs for VP-to-RLOC mappings,
with the actual VP-to-RLOC mappings maintained as DNS
resource records. But, I would still like to get some
sort of per-RLOC location information, and I'm not sure
if there is a DNS resource record type that supports
that. I will check...

> The delta-check to one or a few dozen servers (or more local servers
> with the same information) would probably be more efficient than the
> DNS approach.  It depends a lot on how many VPs there are, and how
> rapidly all the IRON routers need to know about IRON routers taking
> on new IR(VP) roles.
>
>
> "Aggregating" the VP discovery process
> --------------------------------------
>
> I-R has no intermediate buffer layer between the IRON routers and the
> source of this VP information.
>
> For I-R to have as many IR(LF) routers as Ivip has ITRs, this would
> place a much greater load on I-R's VP discovery system (whether with
> a file and delta checks or via DNS as suggested above).  This doesn't
> happen in Ivip, because the ITRs don't need to know very much at all.
> All they need to know is how to query one or more local (such as in
> the same ISP network) QSR Resolving query servers, perhaps via local
> caching QSC query servers.   The QSRs need to know about all the
> MABs, which is the rough Ivip equivalent of I-R's VP.
>
> I-R could be enhanced with some local or nearby server which enables
> multiple IRON routers to more efficiently discover the VP file(s) and
> changes to it.  This is a form of aggregation, to make it unnecessary
> for a bunch of IRs in one area to all, individually, get files from
> and send delta checks to some far distant server.   With the DNS
> model, this can be easily achieved by multiple IRON routers sharing a
> nearby caching resolving DNS server.
>
> Another approach might be some IRON routers passing on the changes
> they discovered to neighbours, but this needs to be done securely.
>
> Both these approaches would reduce long-distance communications, but
> would do little or nothing to reduce the workload of each IRON router
> in keeping itself up-to-date with the VPs.
>
> Probably the most efficient arrangement would be for multiple IRON
> routers to connect to a nearby "resolving and notifying" server of
> some kind, which took responsibility for directly sending all changes
> to the VP information to each IRON router.  Then, each IRON router
> doesn't need to ask, or poll (delta check) for changes.  It just sits
> there and is securely told whenever there is a change.  The resolving
> server would take responsibility for discovering all the VPs - by the
> multiple files approach, or the DNS approach - and would be devoted
> to either polling these (delta check) or in some other way getting
> all the changes on a timely basis.  Then it would send out the
> changes to all its client IRON routers.   Each IRON router would
> probably use two or perhaps three of these for robustness, which
> would multiply the total workload and the workload of each IRON router.
>
> Note that this arrangement would be somewhat analogous to how
> multiple (dozens to hundreds) of Ivip ITRs rely on two or three local
> QSR servers.  However, with I-R, each IRON router is required to know
> all the VPs and for each VP the 3 or so IR(VP) routers - which is a
> more onerous task than that of the Ivip ITRs.
>
> Ivip ITRs only need to know the full subset of the global unicast
> address space which is "edge" space.  They don't need to know
> anything about individual MABs.  In IPv6, a whole short prefix, such
> as a /8 or /16, could contain all the edge space, so this becomes
> trivially simple.  For IPv4, there will be MABs of various sizes
> scattered all about.  Even then, the ITRs don't need to know specific
> MABs, so if four are adjacent and can be covered by a single shorter
> prefix, then the shorter prefix of "edge" space is all each ITR needs
> to know.
>
> Ivip ITRs don't need to know about where one MAB ends and another
> starts, since they always send their mapping queries to an upstream
> QSC or QSR query server which is in the same network, or in an
> upstream ISP's network.  The QSRs need to know about every MAB, and
> the addresses of 2 or so typically nearby authoritative QSA servers
> for each MAB.
>
> I think this last approach - multiple IRON routers sharing a single
> "resolving and notifying" server of some kind - is the one which
> holds most promise for both reducing the workload of each IRON router
> and for reducing the burden a larger number of IRON routers would
> place on whatever centralised or somewhat centralised systems specify
> the VPs and which routers are performing the IR(VP) role for each VP.

I really would rather not have yet another functional
building block beyond the current IR(LF/EID), IR(GW)
and IR(VP). I want to keep the number of "moving parts"
down to a bare minimum. You seem to think there should
be many, many IR(LF/EID)-role routers. I think there
may be lots of them, but I do not want any of them
tucked behinds NATs which is a constraint you did
not mention but perhaps should think about.

In any case, even if all holders of public IPv4 addresses
were to become IR(LF)'s, I still think the flat file
list of FQDNs and the DNS distribution of FQDN-to-RLOC
mappings would work in a scalable fashion.

> More, cheaper, devices for the IR(LF) role?
> -------------------------------------------
>
> Fred envisages about 100k total IRON routers, most of which would be
> performing the IR(LF) role and many of which would also be performing
> the IR(EID) roles and I guess quite a few of which will be performing
> IR(VP) roles for one or more VPs.
>
> In order to reduce costs and spread the load better, it would be
> desirable to have many more devices playing the IR(VP) role.  For
> instance, if it could be done in software by cheap servers, as well
> as in large, centrally located, hardware-based routers, then this
> could be a lot more cost-effective and easier to introduce.  However,
> to allow for ten times more IR(VP) role devices, there needs to be a
> pretty lightweight method by which they discover all the VPs and
> their IR(VP) routers.
>
> As noted above, some kind of nearby "resolving and notifying" server
> looks most promising.

As above, I am thinking:

  1) No NAT traversal
  2) Use DNS with FQDN-to-RLOC mappings
  3) Stateless route replies from IR(VPs) to IR(LFs)
  4) really prefer not to have an additional functional
     element beyond IR(LF/EID), IR(GW) and IR(VP).

> If the global impact of each IRON router performing only the IR(LF)
> role could be really minimised, and if the workload for each such
> device could also be minimised, then the IR(LF) role could be
> performed in an sending host which is not in itself on an I-R "edge"
> address.

"Performed in a sending host" is exactly why I call it
{E,I}TE instead of {E,I}TR, since the "endpoint" could
be either a host or a router. However, I repeat that
I really would prefer not to bring NAT traversal into
scope. Maybe you think NAT traversal is not so much
of a problem?

> IPv6 allows the provision of vast amounts of address space in each
> VP, so few are needed to get sufficient address space.  Still, it
> seems that in order to split up the IR(VP) workload effectively,
> there needs to be a large number of VPs for both IPv4 and IPv6.
>
> Ivip ITRs only need to know what parts of the global unicast address
> range are "edge".  This is done by them learning from their local QSR
> the details of every MAB (Mapped Address Block), each of which is a
> DFZ-advertised prefix.  They don't need to know anything more about
> the MAB.  Any traffic packet addressed to "edge" space will be
> handled by the ITR function.
>
> I-R IR(LF) functions need not only to know the VPs (the closest I-R
> equivalent to Ivip's MABs) but to know 3 or so IR(VP) routers for
> each VP.  Unlike Ivip ITRs, they don't need to buffer traffic packets
> when awaiting mapping - they simply tunnel the traffic packet to the
> 3 or so IR(VP) routers, which is a non-trivial amplification of the
> basic workload of tunneling each traffic packet once.  IR(LF) routers
> don't need to re-request mapping, as an Ivip ITR might if it gets no
> response after a while, because each of the traffic packets tunneled
> to IR(VP) routers is a map request.
>
> I think the biggest scaling bottleneck in I-R is the reliance on a
> small number of IR(VP) routers to handle initial packets.  The
> "scattergun" approach means that it is a bad idea to have more than
> about 3 or 4 such IR(VP) routers per VP.

I no longer think that is true. The IR(LF) using "modified
scattergun" can simply select 2-4 IR(VP)s out of a list of
perhaps a small number of 10's of IRs that service the same
VP. Then, the IR simply sends a live packet to one of the
IR(VP)s and sends blanks to the others.

> Due to each such IR(VP)
> router for a given VP getting every initial packet handled by any
> IR(LF/GW) router, having more IR(VP) routers per VP doesn't achieve
> load sharing - it just creates more work for the IR(LF/GW) routers,
> and for the IR(EID) routers which have to handle 3 or more traffic
> packets, one from each IR(VP) router.

Blanks are discarded at the IR(VP), so the IR(EID) never
sees them.

> This means that VPs need to be made small enough not to have so much
> traffic as to overload the IR(VP) routers.

Right.

> A MAB is advertised by a set of DITRs around the world, and it is up
> to the MABOC (MAB Operating Company) how many DITRs it runs, and
> where, to handle that part of the traffic sent to the MAB's micronet
> addresses which comes from hosts in networks without ITRs.  The more
> DITR sites which handle this MAB, the less work the DITR at each site
> needs to do.
>
> I think this means that I-R's VPs need to be more numerous than
> Ivip's MABs.
>
> The next bottleneck is that each IR(LF/GW) router needs to know about
> all this larger number of VPs, and for each VP, needs to know 3 or so
> IR(VP) routers.
>
> Ivip ITRs need to know only what is "edge" or not ("core"). They
> don't need to know exactly where MABs begin and end.  When expressing
> this "edge" space as prefixes, the number of prefixes each Ivip ITR
> needs to know could be reduced in some circumstances when
> neighbouring MABs could be covered by a single shorter prefix.  For
> instance, if there are two MABs:
>
>    5.5.0.0   /17
>    5.5.128.0 /17
>
> the ITR only needs to know that 5.5.0.0 /16 is "edge" space.  ITRs
> will discover this from their upstream QSCs or QSRs by a simple
> mechanism I haven't yet designed.
>
>    (For IPv4, it might be easier for each ITR to learn about what
>    is "edge" space from its local QSR by downloading a single
>    bit-map, with 2^24 bits (actually, a little less: 224 x 256 x 256
>    = 14.6Mbits ~= 1.83 Mbytes), one for each /24, to flag whether
>    each /24 in the global unicast address range is "edge" or not.)
>
> A further bottleneck with I-R is that the ITRs have a greater need of
> changed information about VPs than Ivip ITRs have about which parts
> of the address range are "edge".  This is because with I-R, even if
> VPs remain reasonably stable, as would Ivip MABs, there would be a
> degree of instability in which 3 or so IRON routers are performing
> the IR(VP) role for each VP.
>
> An Ivip DITR doesn't need to know about all the MABs in the Ivip
> system or what all the "edge" space is.  It only needs to know which
> MABs it is handling.
>
> Compared to Ivip's MABs (or a simpler expression of what is "edge"
> space or not), IR is likely to involve a greater number of VPs which
> every IR(LF/GW) router needs to know about.  The IR(LF/GW) router
> needs to know more about each VP, and it needs to get more updates
> about this information.
>
> This makes it more difficult to introduce larger numbers of IR(LF/GW)
> routers into I-R than it is to introduce larger numbers of ITRs into
> Ivip.   This has the effect, in I-R, of concentrating work into a
> smaller number of IR(LF/GW) routers than the number of ITRs (and
> DITRs) which would handle the work in Ivip.  To avoid congestion,
> each of this fewer number of IR(LF/GW) routers needs to be more
> powerful, with greater bandwidth.
>
> Also, the "scattergun" approach of I-R increases to some extent the
> upstream bandwidth required compared to an Ivip ITR handling the same
> traffic.
>
> Due to Ivip's real-time mapping system, Ivip ITRs are simple compared
> to the ITRs of LISP or the IR(LF/GW) routers of I-R.  The lack of
> real-time mapping in LISP and I-R requires more complex ITRs, to
> cache more complex mapping, and to do some kind of reachability
> testing in order to choose for themselves which ITR (LISP) or IR(EID)
> router to tunnel the packets to.
>
> Also, Ivip ITRs can be on "edge" space, whereas LISP ITRs and IR(LF)
> routers need to be on "core" addresses.

Think about it this way. With the I-R vision, IR(LF/EID)s
are enterprise edge routers that service large enterprise
networks, ISP networks, etc. Within each enterprise, the
RANGER recursion is used to break the enterprise down into
(possibly) multiple levels of hierarchically arranged
networks. Traversal of each layer of the hierarchy
entails a tunneling arrangement between enterprise network
border routers. At the highest level of the hierarchy, we
have IR(LF/EID) routers connected to core RLOC addresses.
So, IRON is simply the highest layer of the IRON-RANGER
recursive hierarchy. So yes; the number of IR(LF/EID)
routers will generally be constrained to a few 10^5's
or so. But, there will be arbitrarily-complex enterprise
networks stacked up behind each one.

> All these factors make it a lot easier with Ivip to have large
> numbers of ITRs, closer to sending hosts than with LISP or I-R.  Ivip
> ITRs can be in in sending hosts (not behind NAT), which cannot be
> attempted with LISP and I-R as currently described.

As long as it is "not behind NAT", I-R should support
ITRs in sending hosts just fine.

> The use of large numbers of server-based ITRs in Ivip spreads the
> load and enables this work to be done with less expense than by
> concentrating it into a smaller number of expensive, dedicated,
> routers - as is the current plan with I-R and LISP.

I don't have so much of a concern for scaling of IR(LF/EID)
routers as you seem to have. The scaling would seemingly
be no different than for large enterprise network border
routers in the current Internet today. So, if there are
only a few IR(LF/EID) routers for a large enterprise
network I think the scaling considerations would be roughly
the same as for the border routers that connect large
enterprise networks to the Internet today.

> But even if this goal is achieved - more IR(LF) routers closer to
> hosts, each handling less traffic and so being potentially cheaper
> and/or less congested - this means that there will be more work for
> the IR(VP) routers than with a fewer number of IR(LF) routers.
>
> For instance, if an ISP only had 3 IR(LF) routers, on average more
> traffic packets would be covered by currently cached mapping than
> would be the case for any one of a more numerous arrangement of
> IR(LF) routers.   If there were 20 of these routers, in each one, it
> is less likely that any one IR(LF) router would already have the
> mapping.
>
> Due to the "unaggregated" exposure of the IR(VP) routers to every
> single IR(LF/GW) router, I-R involves more map query and reply
> activity, all other things being equal, than Ivip or perhaps LISP.
> (Also, the Map Request in I-R consists of tunneling a potentially
> bulky traffic packet to each of the 3 or so IR(VP) routers, and
> continuing to do so until a map reply packet comes back from one of
> them.)
>
> LISP-ALT on its original form doesn't provide any such "aggregation",
> since each ITR gets its mapping from the authoritative query server
> (usually an ETR) via the ALT network.  However, LISP-ALT with Map
> Resolvers would have such aggregation IF the Map Resolvers cached map
> reply information they got from the authoritative query servers.  I
> am not sure whether LISP-ALT's Map Resolvers are caching or not.
>
> With Ivip (Distributed Real Time Mapping) there is a great deal of
> this "aggregation", since dozens or hundreds of ITRs may use one or a
> few QSRs, which are caching query servers.  (QSRs in turn will handle
> less queries if there are caching QSCs between the ITRs and these
> QSRs, but this doesn't alter the degree to which the QSRs reduce the
> mapping query load on the authoritative QSA query servers.)
>
>
> Map Updates, resulting in real-time mapping distribution
> --------------------------------------------------------
>
> IR(VP) routers - individually or perhaps working with the other
> IR(VP) routers for the one VP - as part of making up their mapping
> information to be ready to send to IR(LF/GW) routers, will at various
> times change the mapping for a given EID prefix.  This would happen
> when, for instance:
>
>    1 - The EID prefix is defined by one or more LF(EID) routers
>        registering it.
>
>    2 - The EID prefix becomes undefined if no LF(EID) routers
>        register it - including by not re-registering it at the
>        rate required by the IR(VP) router.
>
>    3 - One or more new IR(EID) routers register an existing EID
>        prefix - so adding their addresses to the mapping for
>        that EID prefix.
>
>    4 - The registration of one or more IR(EID) routers expires
>        due to it not reregistering in time.
>
>    5 - The registration is cancelled.  (I am not sure if Fred
>        intends an IR(EID) router to be able to cancel it, but
>        I guess it could be part of the registration protocol.)
>
>    6 - Some other reasons, such as changes to the nature of
>        EID registrations which affect the currently undescribed
>        methods by which the "preference and weighting" part of
>        the mapping is generated.
>
>
> When the mapping changes, it really needs to change in an identical
> fashion for all the 3 or so IR(VP) routers - by some inter-router
> protocol which is yet to be described and would be closely related to
> the yet-to-be described EID registration protocol.

Yes. Currently, I-R calls for network-layer "bubbles" to
maintain and update the mappings. It could be that what
we need instead is a connection from the IR(EID) router
to the VP company by which the IR(EID) router securely
updates the mapping, which the VP company pushes out
to all of the IR(VP) routers servicing the VP.

> When the mapping changes, it would be possible for each IR(VP) router
> to send out a "Map Update" message, similar in form to the Map Reply
> message, and likewise secured by the SEAL ID of the tunneled traffic
> packet which gave rise to the original Map Reply.  I don't recall if
> this if part of Fred's design, but I think it would be possible and
> desirable.

No; I want the IR(VP) routers to be stateless and simply
return mapping replies. The IR(LF) routers will discover
that a mapping is changed either when their current
mapping times out or when they receive unreachable
messages from the previous IR(EID) router.

> Then, if the IR(VP) router kept a record of which IR(LF/GW) routers
> it had sent Map Replies to, concerning the prefix whose mapping has
> just changed, then it could send out Map Update messages to those
> routers.
>
> There is some inefficiency in this, due to 3 or so IR(LF) routers
> sending out their own, identical content (different SEAL ID), Map
> Update to each IR(LF/GW) router.  But perhaps that means that this
> "reverse scattergun" effect can be used to advantage - to assume that
> there is no need for the IR(VP) router to expect a handshake,
> acknowledgement or whatever from the IR(LF/GW) router, because
> generally at least one of the Map Updates will arrive.
>
>    (There's no need for IR(LF/GW) routers to acknowledge the
>    receipt of a Map Reply message.  If all 3 or so Map Reply
>    messages are lost, then the IR(LF/GW) will continue to tunnel
>    traffic packets to the 3 IR(VP) routers, and presumably these
>    will be sending back more Map Replies for these, so the process
>    will be self-limiting without need for ACKs.)

Correct. This is exactly the way network-layer
redirects on NBMA links work. This mechanism is
at the heart of the I-R design.

> These proposed "Map Update" messages will cause IR(LF/GW) tunneling
> to be changed more rapidly to a better arrangement than the only
> alternative: not sending the Map Update and waiting for the cached
> mapping in the IR(LF/GW) routers to expire - at which time the
> routers will request fresh mapping, by sending traffic packets to all
> 3 or so IR(VP) routers again until at the first Map Reply response
> arrives.

No "map update" messages, since IR(VP)s are stateless.
Instead, the IR(LF) will receive unreachables if the
EID is shifted from an old IR(EID) to a different one.

> These proposed Map Update messages involve extra state in, and effort
> by, IR(VP) routers.  However, it would enable several benefits,
> including:
>
>   1 - IR(LF/GW) routers could cache their mapping for a much longer
>       time, since the cache time no longer controls the ability of
>       the system to get fresh mapping to the IR(LF/GW) routers.
>
>   2 - This extended cache time reduces the frequency with which
>       IR(LF/GW) routers have their cached mapping time out, and so
>       reduces their effort in tunneling traffic packets to all 3
>       or so IR(LF/GW) routers.
>
>   3 - This in turn will significantly reduce the workload of the
>       IR(VP) routers.
>
>   4 - Point 4 means that VPs can be bigger and less numerous.
>
>   5 - Point 5 means that there will be less VPs in the total
>       list, enabling there to be less work done by each
>       IR(LF/GW) router, and therefore makes these cheaper and
>       more able to be installed in greater numbers, closer to
>       hosts.
>
> IF I-R was upgraded to do Map Updates from IR(VP) routers to
> IR(LF/GW) routers, then this would make its mapping distribution a
> genuinely real-time arrangement, like Ivip's.

With I-R, receipt of unreachables will flush stale state
in IR(LF) routers, and new state will be established when
the IR(LF) allows new packets to flow through an IR(VP).
Yes, this means that there will be a "brown hole" with
a short duration of packet loss until the new state is
established. But, this will typically affect only one
or a few sessions and reliable transport protocols will
retransmit.

> If so, then I-R could adopt Ivip's very simple mapping information -
> a single ITR address for Ivip, or a single IR(EID) address for I-R.
> This would simplify IR(LF/GW) routers considerably.  This would also
> externalise the detection of reachability and the decisions about how
> to restore connectivity after a multihoming service failure, making
> it the responsibility of end-user networks or of whoever they appoint
> to do this.  I regard this as one of the major benefits of Ivip over
> LISP and the current I-R design.

I think you meant "a single ETR address"? But, what
happens if that ETR is unreachable from the vantage
point of the IR(LF) that needs to forward a packet to
it? For example, if there is some sort of temporary
route outage or network partition? Can you see some
sort of way around requiring the IR(LF) to see
multiple RLOCs per each EID prefix?

> A different approach to "registering EID prefixes"
> --------------------------------------------------
>
> The current I-R design appears to be based on the same assumptions as
> LISP - that the ETR (LISP) or IR(EID) (I-R) function is responsible
> for the mapping of an EID it handles.
>
> Therefore, in I-R, IR(EID) routers are assumed to be configured
> securely, and given some authentication items - and then it is the
> job of the IR(EID) router to register itself with the 3 or so IR(VP)
> routers for the VP its EID prefix is within.
>
> I have never thought of Ivip this way - and I think there's no reason
> why I-R needs to work this way either.
>
> The 3 or so IR(VP) routers which handle a given VP will definitely be
> run by, or run for and controlled by, whatever single organization is
> responsible for this VP - the "VP Company".  The rough equivalent in
> Ivip is the MABOC (MAB Operating Company).
>
> These IR(VP) routers are not going to accept registrations without
> each registration passing stringent security arrangements - hence the
> need to give the 2 or so IR(EID) routers the correct authentication
> items which the IR(VP) routers will accept.  This could be a signed
> message to the effect that this particular IRON router (specified by
> its IP address) is authorised to perform the IR(EID) role for a
> specified EID prefix.  Then the IR(VP) routers will need to use some
> arrangements (PKI?) to verify these signatures, and to cache this
> verification in some way so they don't need to do this for every
> re-registration - and to occasionally check to see the public key
> used for the signature has not been revoked by the relevant PKI CA
> (Certification Authority).

I think this would work best if each VP company ran its
own PKI CA. That way, the ownership of a VP taken from
the VP company can best be verified by that VP company
itself without the need to involve a 3rd party entity.

> So the current design is something like this, for each EID prefix,
> assuming there are two ISPs and so two IR(EID) routers:
>
>   1 - The multihomed EUN (end-user network) chooses which 2 IRON
>       routers will perform the IR(EID) role for its EID prefix.
>       This is basically a matter of receiving an IP address for
>       an IR(VP) router from each of from its ISPs.
>
>   2 - The EUN creates two messages, one for each of the IP
>       addresses, attesting that its EID prefix is to be registered
>       by an IRON router with the given IP address.
>
>       One approach to this would be the EUN to sign these messages
>       with its own key pair.  To do this, it would have to use a
>       key pair which is covered by a PKI system which is recognised
>       by the IR(VP) routers for the VP its EID prefix is within.
>       This clearly involves the company which controls the VP, which
>       is probably the company it leases this EID space from.
>
>       Another approach is to obtain these signed messages by having
>       them signed by the VP company's key-pair.  But to do so, the
>       EUN will need to authenticate itself with that VP company.
>
>       So either way, the EUN needs to authenticate itself to the
>       VP company in some way, and provide the IP addresses of the
>       IR(EID) role routers.

The business of discovering the IP addresses of the IR(EID)
role routers becomes complicated if there is a NAT between
the EUN and the IR(EID) router.

>   3 - These signed messages are now in the possession of the EUN,
>       who passes one to one ISP and the other to the other ISP.
>       This may involve the EUN authenticating itself to the ISP, but
>       it already has a business relationship with the ISP - so this
>       is probably trivial.

Yes, but the EUN still has to convince the ISP to set
up routing for its EID prefixes within the ISP network.

I think this might work best if the EUN contacts a
border router in the ISP network acting as an IR(EID)
router, which in turn contacts the VP company. Then,
tunneling would go first to the IR(EID) router (i.e.,
the router in the ISP network) and then to the EUN
router.

>   4 - Each ISP loads its message into the IRON router which is
>       performing an IR(EID) role for this VP.  Now this IRON router
>       will be able to register itself with the 3 or so IR(VP)
>       routers, and keep doing so for as long as is required.
>
> This raises a tricky question if the EUN stops using one or both
> ISPs.  How can it prevent these messages still being sent by these
> IRON routers?   It should be able to prevail on the ISPs to take
> these signed messages out of their routers, but let's say the ISP has
> gone feral and won't respond to reasonable requests.   This would be
> an intolerable situation, black-holing traffic packets.  So there
> needs to be some additional mechanism by which the EUN can contact
> the VP company and have the 3 IR(VP) routers ignore the registration
> message from the errant router.
>
> I think this is all a lot of work for no good purpose.
>
> A much better idea is for the EUN to get IP addresses of the routers
> from its ISPs, and securely communicate these to the VP company.
> Then the VP company programs that into the 3 IR(VP) routers it
> controls.  That's it - there's no need for the two IR(EID) routers to
> register anything with the 3 IR(VP) routers.

OK, but if the EUN is within a NATed ISP network (i.e.,
the ISP internally uses private IP address space) then
the EUN still has to make arrangements for the ISP
network to route its EID prefixes. Otherwise, there
would need to be a NAT traversal scheme for the EUN
to associate its IR(EID) routers with the VP company.
Lacking heavy-duty NAT traversal (when there may
actually be multiple NATs in the path) I don't see a
way for the EUN to coordinate with the VP company
without also coordinating with the ISP network.

> But now let us consider two scenarios - the original, non-real-time
> mapping arrangement and my proposed real-time approach.
>
> In the original non-real-time approach, the absence of a registration
> from a previously registered IR(EID) router would signify to each
> IR(VP) router that that IRON router should no longer be included in
> the mapping for the given prefix.  Depending on the caching time
> specified in the Map Reply messages sent from IR(VP) routers to
> IR(LF/GW) routers, it would take some time for all IR(LF/GW) routers
> to gain updated mapping which did not mention this IRON router as one
> of the IR(EID) routers for this EID prefix.
>
> But if, as just suggested, the IR(VP) routers don't figure out the
> mapping of an EID prefix based on repeated registrations from IR(EID)
> routers, because they are simply configured with the IP addresses of
> supposed IR(EID) routers by the VP company, then how do the IR(VP)
> routers figure out the mapping and how would they change this mapping
> to remove a particular IR(EID) router if it became "unreachable"?  I
> think the IR(VP) router has no business deciding what the mapping
> should be once the above suggestion is implemented.  In the
> non-real-time arrangement (the original I-R design) the ITR-like
> IR(LF/GW) routers are supposed to figure out which IR(EID) router are
> reachable or not, and which ones can be used to reach the destination
> network as part of their built-in independent per IR(LF/GW) router,
> approach to multihoming service restoration.
>
> So with the original non-real-time mapping, and the new, direct (no
> registration) arrangements where the VP company simply configures its
> 3 VP routers with the IP addresses of the IR(LF/GW) routers, then I
> think there is nothing more to do.  Just let the IR(LF/GW) routers do
> their reachability testing however Fred intends this to be done.
>
> With the real-time-mapping arrangements (Map Update), if the EUN
> chooses a new ISP, or the ISP tells them they have a different
> IR(EID) router, then the EUN must simply communicate the new IP
> address securely to the VP company, who will pop it straight into
> their 3 IR(VP) routers.  This removes the need for the just suggested
> extra mechanism by which the VP company could prevent its 3 IR(VP)
> routers from accepting a registration from the errant IRON router in
> the feral ISP.

I don't see a way around also involving the ISP if the
ISP network uses private address space instead of public.
The only other way would be through the use of heavy
duty NAT traversal, which I can't see happening.

> In the real-time approach (Map Updates), combined with the
> just-suggested (no registration) direct "VP company controls the
> mapping directly via its IR(VP) routers" approach, then the moment
> the VP company changes the mapping in its 3 IR(VP) routers, each one
> will send a Map Update message to whichever IR(LF/GW) routers to
> which it has, within the caching time, sent the mapping for this EID.

In the I-R approach, I want the IR(VP)s to be stateless
from the standpoint of mapping. So, IR(LF)s will discover
that a mapping has changed through the receipt of
unreachables from a former IR(EID) router. (Either that,
or through expiration of a mapping over which packets
have not recently flowed.)

> So this gives the VP company direct real-time control of the mapping
> in all the ITRs which need it.  Since the EUN directly tells the VP
> Company which IP addresses to use, and since these real-time updates
> directly control the cached mapping in all ITRs currently tunneling
> packets to this EID prefix, this means the EUN has direct, real-time,
> control of the mapping of its EID prefix and so direct real-time
> control of which IR(EID) role router the IR(LF/GW) routers will
> tunnel packets to.  This is just like Ivip.  There's no longer a need
> for ITRs to do any reachability testing, since the EUN can easily
> hire a company to do better reachability testing of its actual
> network via the two or more IR(EID) routers - and give that company
> the authentication details it needs to be able to tell the VP Company
> to change the mapping.

There is still the problem that, if IR(LF) routers only
get a single RLOC mapping for a given EID, that that
RLOC may be unreachable from the standpoint of the IR(LF)
even if it appears to be reachable from the standpoint
of the IR(EID) router. I don't see a way around having
to give the IR(LF) something like 2-4 alternative RLOC
mappings for the same EID so that it can test reachability
itself (i.e., the "locator liveness" problem).

> So, with these to elaborations - which look feasible to me -
> IRON-RANGER could have real-time mapping just like Ivip.   This is at
> odds with the LISP tradition and with the expectation of many RRG
> folks that such things are impossible.
>
> With this proposed pair of modifications:
>
>   1 - "EUN -> VP Company -> IR(VP) router" instead of "IR(EID)
>       routers registering their EID prefixes with the IR(VP)
>       routers".

OK, but it still needs to involve the ISP networks.

>   2 - Map Updates for real-time mapping.

No map updates; destination unreachables as indication
of mapping change instead.

> I-R would still not be as scalable as Ivip if there were a very large
> number of ITRs tunneling packets to this EID prefix at the time the
> mapping is changed.  With this modified version of I-R, each such
> change would require each of the 3 or so IR(VP) routers to send a Map
> Update to each of these IR(LF/GW) routers which is (or might be)
> tunneling packets - all those IRON routers which, within the caching
> time, were recently sent Map Replies for this EID prefix.
>
> This is a triplication of total workload for the IR(VP) routers,
> compared to a single device sending the Map Update to the IR(LF/GW)
> router.  (However, as noted previously, this "reverse scattergun"
> approach might be robust enough to send the Map Update without any
> requirement for acknowledgement, retries etc.)
>
> It is also an escalation of workload for each affected IR(LF/GW)
> router, because it will receive not just one Map Update, but 3.
>
> The real problem is for the IR(VP) routers which must each send a Map
> Update packet to every IR(LF/GW) router which, within the caching
> time, was previously sent a Map Reply for this EID.   This will not
> scale well.  If there are 100 such IR(LF/GW) routers, then it will be
> bad.  If there are 10,000, it will be really bad.
>
> Ivip avoids these scaling problems, because the mapping change goes
> out to a dozen or more DITR sites, and is received at each site by
> the authoritative QSA query server there.  That sends out any needed
> Map Updates to its queriers, who likewise were sent a Map Reply for
> this EID prefix within the current caching time.
>
> But these will be far fewer in number per QSA than the total number
> of ITRs which need to get such a Map Update.  Firstly, the work is
> spread over a (typically) dozen or more QSAs at this many widely
> distributed DITR sites.  Secondly, each querier - each QSR - which
> needs to get a Map Update will sometimes, or frequently, pass the
> information on to multiple ITRs.
>
> So I think the Ivip approach will scale better, and not result in an
> overly large workload for any server (QSA, QSR or QSC).  Also, in
> Ivip, the ITR gets a single Map Update which it will acknowledge.
> This is probably less expensive than the I-R approach where the
> IR(LF/GW) router gets 3 Map Updates, one from each IR(VP) router.
>
>
> Scaling vs. simplicity - and more on real-time mapping distribution
> ===================================================================
>
> Here I compare Ivip (with Distributed Real Time Mapping) and version
> of the I-L-light subset with two significant modifications, as
> described above:
>
>   1 - No registrations of EIDs from IR(EID) routers.  Instead, EUNs
>       tell the VP company the IP addresses of the IR(EID) routers
>       which will be handling each of their one or more EID prefixes.
>
>       This is inherently a real-time process, since the VP company
>       receives the new information and can easily configure the
>       3 or so VP routers to have new mapping, in less than a second.

This can and should be initiated by the EUN, yes, but
will still need to involve the ISP network if the ISP
network is internally using private IP address space.

>   2 - By adding Map Updates (if they were not already in Fred's
>       design), the mapping changes (which always result from the
>       above real-time process) can be sent to each IR(LF/GW)
>       router which is caching the mapping for this EID prefix.

No; passive destination unreachables instead as negative
indication of mapping change.

> These two changes would make the I-R mapping distribution system just
> as real-time as Ivip.  Then, the mapping could still consist of
> multiple IR(EID) routers with priorities, weightings etc. to tell
> IR(LF/GW) routers how to do load sharing and choose between IR(EID)
> routers them more than one of them appears to be working - as with
> LISP and the current I-R design.
>
> However, it would also be possible to simplify the system so that,
> like Ivip, the mapping can only consist of a single address of a
> single IR(EID) router.  This would mean that IR(LF/GW) routers would
> never be required to choose between IR(EID) routers - and all EUNs
> would be required to achieve their multihoming service restoration
> and/or inbound TE goals by changing the mapping in real-time.

Locator liveness testing would suggest that we need
more than one RLOC->EID mapping.

> Even if this decision is not taken, the above two points make the
> system real-time - so any EUN which wants to control the mapping of
> its EIDs in real-time can do so.  The only question is whether the
> system is simplified, as in Ivip - which requires all EUNs to do this.
>
> Either way, the real-time control of mapping and tunneling is a major
> advance on LISP and the original I-R design.
>
> Some EUNs might want to use the system for dynamic (responsive to
> traffic flows, minute to minute, or even faster) inbound TE.  Some
> such EUNs will find it highly advantageous to steer inbound traffic
> over their two or more ISP links, being able to maximise the
> utilization of each link within some acceptable level which avoids
> congestion.  I guess content distribution networks could also steer
> traffic sent to various EID prefixes to their IR(EID) routers at
> various separate sites all around the Net, to dynamically load
> balance the workload of these sites.
>
> For instance, a single site EUN has 5 hosts, or groups of hosts, each
> with a differing pattern of incoming traffic - where the levels of
> traffic for each changed hour-to-hour or minute-to-minute.  By
> splitting each group into two smaller groups, and defining an EID
> prefix for each small group, the EUN will be able to dynamically
> steer these 10 streams of traffic between two or more ISP links,
> perhaps of different capacities, and so maximise the utilization of
> these expensive links while avoiding congestion which would occur
> without this dynamic inbound TE capability.
>
> The next thing which will happen is the VP Companies will want to
> charge for mapping changes, or at least for frequent mapping changes.
>  That's fine - they should do so.  The EUNs which have a high enough
> need for inbound TE to pay the fee per mapping change will continue
> to make those changes.  The fee might be between a few cents and a
> few tens of cents - and still be highly worth the expense for some
> EUNs, since it is cheaper to do this at peak times than to pay for
> higher capacity links to their ISPs.
>
>
> This need to charge for frequent mapping changes results directly
> from the ability of the system to let EUNs (or companies they
> appoint) to directly control the tunneling behavior of IR(LF/GW)
> routers all over the world.  This applies whether the system uses the
> current complex mapping, with complex IR(LF/GW) functions - or adopts
> the simpler Ivip-style single address mapping, and so enables all
> IF(LF/GW) routers to be significantly simpler.
>
> Sill, this modified I-R architecture seems to have more scaling
> problems than Ivip.  It is simpler than Ivip - and I argue below that
> the extra complexity in Ivip is justified by the way this complexity
> enables Ivip to scale better.
>
> Anything which increases the number of ITR / IR(LF/GW) routers is
> assumed to be good, since the more there are of these, the less work
> each one has to do, and the cheaper each one can be.  This makes it
> more attractive to implement the ITR or IR(LF/GW) functions in cheap
> COTS (Commercial Off The Shelf) servers - and this enables the system
> to be introduced rapidly with fewer costs and risks than relying on
> major upgrades to the functionality of routers from the major router
> manufacturers.
>
>
> The I-R architecture (or at least the above modified real-time
> version of I-R-lite) is simpler than the architecture of Ivip with
> DRTM.
>
> The change "l" listed above - EUNs telling VP companies directly what
> the mapping should be - is a considerable simplification of the
> original I-R design.  As far as I know, it is superior in every
> respect to the the plan for each IR(EID) router to have a signed
> message which it uses to repeatedly register an EID with each of 3 or
> so IR(VP) routers.

Sounds great, but if the ISPs (and enterprise networks)
use private addressing internally then they would also
need to be involved in the EID mapping coordination.

> The change "2" above adds some complexity to the I-R design, but the
> Map Update message is similar or identical to the Map Reply, so there
> are no more complex protocols required.

Negative acknowledgements as an indication of mapping
change is the mechanisms assumed by I-R. The penalty
is some packet loss "brown-holing" until the new
mapping is discovered.

> The additional complexity is partly in the IR(VP) router needing to
> retain state about, within the current caching time, which IR(LF/GW)
> router it has sent mappings to for each EID in the VP.
>
> The other extra complexity is in the IR(LF/GW) functionality - these
> need to be able to recognise Map Update messages and act on them
> (which is very similar to how they respond to a Map Reply message).
> They may need a fancier SEAL-ID recognition system, since the SEAL-ID
> the IR(LF/GW) sent with the original traffic packet to the IR(VP)
> router is used to secure both Map Reply and Map Update messages from
> the IR(VP) router.  These Map Update messages may arrive quite a time
> (whatever the caching time is) after the initial traffic packet was
> tunneled, so the "window of acceptable SEAL-IDs" needs to be
> correspondingly wider, in time.
>
> Depending on the caching time, this may involve recognising
> significantly older SEAL-IDs than in the current design.  I think
> Fred's current plan is to use some kind of sliding window arrangement
> on SEAL-IDs to recognise them as valid, rather than to cache each one
> and maintain some kind of timer for each one.  Still, I think adding
> Map Updates will not involve excessive complexity.
>
> I will dub this real-time souped up version of my I-R-lite subset
> "I-R-RT".
>
> I-R-RT has virtues of simplicity.  It has only three kinds of network
> element which handle traffic packets, if we consider the IR(GW) role
> of IRON routers - like LISP PTRs or Ivip DITRs, receiving traffic
> packets from the DFZ - to be not significantly different from the
> IF(LF) role, which does the same thing, but only advertised the
> "edge" space in a local routing system, and so doesn't attract
> traffic packets from the DFZ (from other ASes).
>
> All network elements except the newly proposed VP List servers are
> roles for IRON routers.  There are three types of role, each with
> different responsibilities.  All IRON routers are assumed to be
> capable of performing the IR(LF/GW) role, and a subset of them will
> also perform one or both of the other two roles.
>
>   IR(LF/GW)   Must know all the VPs and for each VP must know
>               the IP addresses (or FQDNs?) of the 3 or so IRON
>               routers which are performing the IR(VP) role for
>               that VP.  Do this by either:
>
>                  1 - Download file(s) and then do delta checks,
>                      using the VP File servers mentioned below.
>
>                  2 - A DNS-based approach as suggested above.
>
>               Initially, in the absence of its cache containing
>               mapping for a matching EID, the IR(LF/GW) router
>               tunnels any traffic packet which is addressed to an
>               "edge" address to all 3 IR(VP) routers for the
>               matching VP.
>
>               Accepts Map Reply messages from these IR(VP) routers
>               with a caching time.  (This time can now be quite long,
>               since we are no longer relying on cache time-outs for
>               IR(LF/GW) routers to discover mapping changes.)
>
>               Tunnels each subsequent traffic packet matching the
>               EID specified in the Map Reply message to one IR(EID)
>               router.
>
>               Also accepts (within the caching time) Map Update
>               messages, which are similar or identical to Map Reply
>               messages, and alters its tunneling instantly according
>               to the new mapping.
>
>               Work with IR(VP) and IR(EID) role routers to solve
>               PMTUD problems.
>
>
>   IR(VP)      Typically 3 IRON routers will perform the IR(VP) role
>               for a given VP.  Some IRON routers will do this for
>               many VPs.
>
>               Receives mapping for each EID directly from the VP
>               company which is responsible for the VP.  Presumably
>               this role router is run by - or at least controlled and
>               paid for by - the VP Company.
>
>               When it receives a traffic packet in a tunnel from
>               an IR(LF/GW) router, or from its own internal IR(LF/GW)
>               function, does four things:
>
>                 1 - Tunnels the packet to a single IR(EID) router
>                     according to the mapping of the matching EID.
>
>                 2 - Sends back (perhaps within itself) to the
>                     IR(LF/GW) function the mapping for this EID
>                     prefix, with a caching time.   (This is
>                     secured by the SEAL-ID in the SEAL tunnel
>                     headers of the traffic packet just received.)
>
>                 3 - Stores the EID, the SEAL-ID and maintains some
>                     kind of timer, so it can perform the following
>                     if necessary:
>
>                 4 - If the mapping for an EID prefix is changed,
>                     send the new mapping as a Map Update message to
>                     all the IF(LF/GW) routers which, in the recent
>                     N minutes (whatever the caching time) were sent
>                     a Map Reply for this EID.
>
>               Work with IR(LF/GW) and IR(EID) role routers to solve
>               PMTUD problems.
>
>
>    IR(EID)    These accept tunneled packets sent by IR(LF/GW) routers
>               - all but the "initial" packets - and from IR(VP)
>               routers - the "initial" traffic packets the IR(LF/GW)
>               router received before it got mapping for a matching
>               EID.
>
>               Forwards the decapsulated traffic packets to the
>               destination EUN (End User Network).
>
>               Assuming there are 3 IR(VP) routers and that all are
>               working, ignore the 2nd and 3rd packet from these -
>               only forward the first to the EUN.
>
>               Work with the IR(LF/GW) and IR(VP) routers to solve
>               PMTUD problems.
>
>               (If existing complex mapping is used, then the IR(EID)
>               role routers would need to work together in some way
>               to prevent the destination EUN receiving duplicate
>               initial traffic packets if some IR(VP) role routers
>               tunneled packets to IR(EID)1 and others to IR(EID)2.
>               However, if the mapping is only allowed to be a single
>               IR(EID) address, like Ivip, then there will be no such
>               problem with duplicated traffic packets.)
>
>
>     VP-list servers
>
>               These currently have no formal name or function - and
>               they are probably not routers.  There needs to be
>               some kind of global, redundant, load-shared,
>               system by which all IRON routers get to know what the
>               current VPs are, and which 3 or so IRON routers are
>               playing the IF(VP) roles for each VP.
>
>               This involves providing files for download and
>               responses to delta checks.
>
>               This is only if a DNS-approach is not used.
>
>               Ideally, I think, there should be no single file, or
>               single server - but some more distributed and fault-
>               tolerant system than the one originally proposed by
>               Fred, which involves a single file, from potentially
>               multiple servers.

These lists need to be modified modulo the points that
were discussed earlier in the message. I think we touched
on the points a number of times, so I won't reiterate them
here.

All of the rest that follows seems to be focused on Ivip
and then on a comparison of Ivip and I-R. I think the
above discussion has already touched on most of this,
and I think a lot has been learned that will improve
I-R. I am going to have to stop here for now, but would
be interested in any further ideas that might help
improve things beyond the insights discussed above.

Thanks for all of your attention to this, Robin!

Fred
[email-masked]

> As with Ivip, EUNs must make their own arrangements to control the
> mapping themselves, or appoint some other organization to do so.
>
> If single address mapping is used, the above is a reasonably complete
> description of the system.  If the existing LISP-like multi address
> (two or more IF(EID) address) mapping is retained, then the IR(LF/GW)
> role, and the IR(VP) role must also retain whatever mechanisms Fred
> proposes for testing reachability and deciding which of the mapping's
> multiple IR(LF/GW) role routers to tunnel the traffic packet to.
>
> Ivip's network elements are as follows.  Please see these for more
> information:
>
>   http://tools.ietf.org/html/draft-whittle-ivip-arch
>   http://www.firstpr.com.au/ip/ivip/drtm/
>
> These could be roles so a single server or router performed multiple
> such roles - but generally  I consider these to be separate classes
> of device.  All these could be implemented as software on a server.
>
>
>    ITR     Learn about all the "edge" space, such as by getting
>            a list of MABs, from their local QSR (perhaps via
>            one or more intermediate QSCs) - or a simpler list
>            which doesn't mention individual MABs when two or
>            more are adjacent.
>
>            Advertise this "edge" space in the local routing system
>            and so tunnel each received traffic packet which is
>            addressed to an "edge" address to an ETR.
>
>            If the ITR has no cached mapping matching the destination
>            address, buffer the packet and send a Map Request (which
>            includes a nonce and the packet's destination address) to
>            a local QSR or QSC.  (Each ITR will auto-discover, or be
>            configured with, the address of 3 or so QSRs or QSCs
>            which it uses for all its Map Queries.  The ITR needs
>            to resend the Map Request if no Map Reply arrives within,
>            say, 80ms.)
>
>            The Map Reply specifies a micronet of SPI (Scalable PI =
>            "edge" space) and a single ETR address, with a caching
>            time.  When this mapping arrives, tunnel the buffered
>            packet to the single ETR specified in the mapping.
>
>            If the mapping is already cached, when the traffic
>            packet arrives, tunnel the packet to the single ETR
>            specified in the mapping.
>
>            Cache this mapping for the caching time, together with
>            the nonce of the original request, and accept Map Update
>            messages from the QSR or QSC, which will be secured by
>            the same nonce.  These updates will either change the
>            ETR address or tell the ITR to flush this micronet from
>            the cache.  (The latter would be for when the existing
>            micronet is split or joined to some other micronet - so
>            if the ITR is still handling packets addressed to the
>            old micronet, it will buffer them and make a new Map
>            Request, to receive mapping for a different micronet which
>            covers the destination address.)
>
>            ITRs can be in ISP and EUNs, including EUNs using
>            conventional edge space and those using SPI "edge"
>            space.  So ITRs can be on a micronet address - SPI "edge"
>            space.  ITRs cannot be behind NAT in the current design.
>
>            ITRs work with ETRs to handle PMTUD problems caused by
>            encapsulation.
>
>
>    DITR    Like an ITR, but:
>
>              1 - Advertises only a subset of the "edge" space -
>                  specific MABs (Mapped Address Blocks) into the
>                  DFZ, and so receives packets addressed to these
>                  MABs.  An ordinary ITR receives packets addressed
>                  to all MABs.
>
>                  A DITR doesn't need to know all the MABs or all
>                  "edge" space - it just gets the list of MABs it
>                  needs to advertise from its QSA.
>
>              2 - Looks up the mapping, if it is not already cached.
>                  Sends a Map Request to a QSA which is in the same
>                  rack at the same DITR site, so there is a fast
>                  100% reliable, connection, and only a few ms
>                  delay in being able to tunnel the packet.
>
>              3 - Is located at a DITR site, where the site and its
>                  one or more DITRs and QSAs typically only handle
>                  a subset of the MABs.  (According to which MABOCs
>                  this DITR site operator is working for.)
>
>              4 - Analyses traffic so the company which operates the
>                  DITR site can bill the MABOCs (MAB Operating
>                  Companies) for the traffic handled for each MAB.
>                  This analysis will include time and micronet
>                  details so the MABOC can bill its SPI-leasing
>                  EUN customers for each such customer's DITR traffic.
>
>              5 - May connect to the QSA via a QSC, but most likely
>                  just sends Map Requests, and receives Map Replies
>                  and Map Updates, directly to the QSA which is at
>                  the same site, and presumably in the same rack.
>
>                  Conceptually, there is a single QSA, but in
>                  fact there may be two or more for redundancy,
>                  and perhaps the DITR will be configured to use
>                  a QSA at another DITR site run by the same
>                  company as a backup if its own site's QSAs
>                  fail.  (This last option is not mentioned in
>                  the DRTM ID or in Ivip-arch.)
>
>              6 - Stops advertising its MABs in the DFZ if its
>                  QSA is dead, or can't get up-to-date mapping.
>
>
>    ITFH    Like an ITR, but is built into the sending host.
>
>            Handles traffic packets sent to all MABs, but does
>            not "advertise" routes to these - it simply intercepts
>            outgoing packets generated by the host's otherwise
>            conventional stack.
>
>            The sending host can be on conventional space or on
>            "edge" space (SPI, micronet space).  In the current
>            Ivip design, it can't be behind NAT.
>
>
>    QSA     Authoritative Query Server.  These are only located at
>            DITR sites.  In theory a QSA could be authoritative for
>            the mapping of all MABs, but in practice, each DITR site
>            will only support a subset of MABs.
>
>            Gets, by some means, a real-time feed of mapping changes
>            for all its MABs and so maintains a complete mapping
>            database for each MAB.  (How this is done is not
>            currently specified, but since this is only for the QSAs
>            in a single DITR network, and since each such network only
>            handles a subset of MABs and will probably have no more
>            than a few dozen DITR sites, this is assumed to be
>            possible in secure, scalable, fashion.  Private network
>            links could be used between these sites.)
>
>            Responds to Map Requests from DITRs at this site -
>            sending them Map Reply messages within a few milliseconds
>            and sending them Map Update messages if and when
>            required.
>
>            Conceptually, there is a single QSA at each DITR site.
>            In reality, there may be one for the use of the DITRs
>            there and one or more for accepting Map Replies from
>            typically nearby QSRs.
>
>            QSAs are in DITR sites.  A single DITR network might have
>            one or two dozen such sites, each handling the same
>            subset of MABs.  These would be scattered around the Net
>            to share the load and generally minimise total path
>            lengths.
>
>
>    QSR     Caching Resolving Query Server.  ISPs run 1 or more
>            likely 2 or 3 of these for their own ITRs and for the
>            ITRs in their customer networks.
>
>            Auto-discovers, via a DNS mechanism, all the MABs and
>            provides a form of this information - the complete set of
>            "edge" space - to all the ITRs it serves.  Also, for each
>            MAB, discovers the address of 2 or 3 typically nearby QSAs
>            which handle that MAB.
>
>            Accepts Map Requests from queriers (ITRs and QSCs) and
>            sends them Map Replies and Map Updates.
>
>            Answers the queries from its own cached mapping or by
>            sending a query to one of the nearby QSAs, depending
>            on which MAB the queried address lies within.
>
>            Sends its own Map Requests to QSAs, depending on which
>            VP the queried address fits within.  Accepts Map Replies
>            and later potentially Map Updates from these QSAs.
>
>
>    QSC     This is an optional device - a Caching Query Server.
>
>            It accepts Map Requests from ITRs and/or other QSCs -
>            and sends them Map Replies and Map Updates.
>
>            It sends its own Map Requests (when it receives a
>            Map Request it can't answer from its cache) to
>            one of the handful of QSRs and/or QSCs which are
>            "upstream".   QSCs, when they serve multiple ITRs,
>            can frequently answer Map Requests from their cache
>            - since a previous request by another ITR filled the
>            cache.  So QSCs can reduce the workload of QSRs.
>
>            (The code for the ITR/DITR, QSA, QSR and QSC functions
>            will have many common elements.)
>
>
>    ETR     Egress Tunnel Router.  Accepts the tunneled traffic
>            packets from ITRs and forwards them to the destination
>            network.  May be in the ISP network and so shared by
>            multiple destination networks, or may be located in
>            the destination network, such as on a PA address from
>            the ISP.
>
>            Works with ITRs to handle PMTUD problems caused by
>            encapsulation.
>
> Please see the text and diagrams at:
>
>   http://www.firstpr.com.au/ip/ivip/drtm/
>
> for how all these fit together, and for how initial services and
> substantial scalable routing benefits will result without any ISP
> investment, just by using DITRs, QSAs and ETRs.
>
> The network elements are, in their logical groups:
>
>    ITR        QSC          QSR        QSA      ETR
>    DITR       (Optional)
>    ITFH
>
> All but the ETR would share some common code elements.
>
>
> Ivip scales better than the above modified version of I-R for a
> number of reasons:
>
>   1 - Ivip ITRs are simpler, since they don't need to know all
>       the MABs - just the subset of global unicast space which is
>       "edge" (SPI) space.
>
>       They don't need to know each MAB, or know anything about each
>       MAB - while the I-R equivalent - IR(LF/GW) - needs to know all
>       VPs and know 3 or so IR(VP) routers for each VP.
>
>       If I-R retains its current multi-address mapping arrangements,
>       Ivip ITRs are much simpler because they use single ETR address
>       mapping and so do not do any reachability testing or make any
>       choices between multiple ETR addresses.
>
>       This means ITRs can be more numerous, closer to hosts and even
>       in sending hosts.  This reduces the load per ITR, enabling them
>       to be cheaper, including being implemented with software on an
>       inexpensive server.
>
>   2 - An Ivip ITR only tunnels all traffic packets to a single ETR,
>       while an I-R IR(LF/GW) router tunnels initial packets to all 3
>       or so IR(VP) routers.
>
>   3 - An Ivip ETR only receives a single initial packet, while an I-R
>       IR(EID) router typically receives 3 or so, and must use only
>       the first.  This is only true if the current multi-address
>       complex mapping is retained.  If I-R adopts the single address
>       (single IR(EID) router) mapping like Ivip, then this problem
>       of duplicate traffic packets arriving from two or more IR(EID)
>       routers won't occur.
>
>   4 - In I-R, the authoritative query servers are the IR(VP) routers.
>       Due to the "scattergun" approach to handling their potential
>       unreachability - tunneling initial traffic packets to all
>       IR(VP) routers, the number of IR(VP) routers needs to be
>       strictly limited.  I assume a figure of 3 or so.
>
>       This means that the VPs must be made small enough that each
>       IR(VP) router can handle the load of all initial traffic
>       packets handled by all the IR(LF/GW) routers in the world.
>
>       Consequently, there will be more VPs in I-R than MABs in Ivip.
>
>   5 - The current I-R design involves each IR(GW) advertising all the
>       VPs in the DFZ.  (With IPv6, this might be achieved with a
>       single short prefix, if all the "edge" space is within that
>       prefix.)
>
>       This means there must be a lot of these IR(GW) routers, since
>       at any one location in the DFZ, it must be assured that no such
>       IR(GW) router becomes congested.
>
>       While an Ivip DITR could advertise all MABs, in practice each
>       DITR (each DITR site) will only handle a subset of MABs.  To
>       the extent that a single DITR can't handle all the MABs of a
>       site, there can be multiple DITRs there.  (A MAB can even be
>       split and handled by 2 or more DITRs at a given site, but
>       still advertised by a router there as the single MAB prefix.)
>
>   6 - Ivip includes plans (charged-for mapping changes and DITR
>       traffic) which allow attractive business cases to be made for
>       DITRs and the DITR sites, which will be run by, or for,
>       MABOCs.
>
>   7 - In I-R, each IR(LF/GW) directly queries the authoritative
>       query server(s).  It actually queries all 3 or so IR(VP)
>       routers.  Furthermore, the query is not a short packet, but
>       a potentially long traffic packet.
>
>       These implicit queries continue, in their triplicated form,
>       until the IR(LF/GW) router receives at least one Map Reply from
>       one of the IR(VP) routers.  Also, the authoritative query
>       servers - IR(VP) routers - need to tunnel these traffic packets
>       to IR(EID) routers.
>
>       In short, there is no "aggregation", "cached concentration" or
>       whatever it might be called, between the querying ITR-like
>       devices and the authoritative query servers.
>
>       Likewise, in this modified version of I-R with real-time
>       mapping distribution due to Map Update messages, the
>       authoritative query servers need to do all the work of
>       sending Map Updates directly to each ITR-like IR(LF)
>       router which is caching a Map Reply mapping for the EID
>       prefix whose mapping just changed.  (Also, all 3 or so
>       IR(VP) routers have to do this, so each IR(LF) router gets
>       typically 3 Map Updates.  This is simple and more robust than
>       getting one, but it is less efficient.)
>
>       In Ivip, the authoritative QSAs are not queried directly by the
>       ITRs, except by the DITRs at that site.
>
>       The ITRs may query via one or more levels of caching query
>       server (QSC), each level of which tends to reduce the workload
>       of the upstream query server, which may be another QSC or one
>       of the QSRs.
>
>       Even without any QSCs, ITRs always query local QSRs,
>       and when each QSR serves a large number of ITRs, each QSR will
>       frequently be able to answer from its cached mapping, thereby
>       reducing the number of Map Requests the authoritative QSAs must
>       handle.
>
>   8 - This reduces the workload of QSAs.  Firstly, they receive fewer
>       Map Queries and send less Map Replies.  Secondly, when a
>       micronet's mapping changes, they send typically far less Map
>       Updates than there are ITRs which need Map Updates, because
>       they send the Map Update to a QSR, which will typically send an
>       equivalent Map Update to multiple ITRs, either directly or via
>       one or more levels of QSCs.
>
>   9 - In Ivip, there is no very low, such as 3 or so, limit on the
>       number of authoritative query servers.  As noted above, for
>       each I-R VP, there can probably be no more than 3 or 4 IR(VP)
>       routers.
>
>       With Ivip, there can be as many QSAs as there are DITR sites -
>       and even within a DITR site, there can be multiple QSAs to
>       spread the load of the queries for concerning multiple MABs.
>
> There may be other scaling benefits to Ivip as well.
>
> To summarize:
>
>     I-R is an interesting and in principle relatively simple CES
>     architecture.  However, this simplicity involves a direct
>     communication path between the ITR-like devices and the
>     authoritative query servers - with a further "scattergun"
>     inefficiency in these interactions.
>
>     Ivip has more types of network elements and is more complex,
>     but this enables the workload to be split up in a manner which
>     reduces total effort - and in particular reduces the total
>     effort by the authoritative QSA query servers or any other
>     single network element.
>
>     With no low (3 or so) limit on the number of authoritative
>     QSA query servers, Ivip can have dozens, or in principle
>     hundreds of QSAs, to share the total load for the MABs they
>     handle - though I expect most DITR networks to work fine with
>     10 to 20 sites.
>

---

From: Robin Whittle <[email-masked]>
Date: Tue, 13 Apr 2010 20:37:43 +1000

Short version:  The Chronicles of IRON-RANGER continue.  In this
                instalment:

                   Fred responds to my "I-R-Lite" subset, including
                   with a "modified scattergun" approach to solving
                   a problem I think existed in I-R and therefore in
                   I-R-Lite.

                   He suggests a latitude and longitude approach for
                   choosing a (typically) "close" IR(VP) router from
                   a list of several of these.  I intend to apply
                   this to a similar problem in Ivip.

                   I tried to talk Fred into converting I-R into a
                   real-time mapping system, like Ivip.  This
                   involves the IR(VP) routers holding more state
                   so they can send map-update messages to their
                   queriers (IR(LF/GW) routers).  He doesn't want
                   to do this, so I-R retains its LISP-like non-real-
                   time mapping, and so needs LISP-like complex
                   mapping and the ability for IR(LF/VP) routers
                   to determine which of several IR(EID) routers
                   can deliver packets to the destination network.

                   I also tried to talk him into having some caching
                   intermediaries between the ITR-like functions -
                   IR(LF/GW) routers - and the few authoritative
                   query servers - IR(VP) routers.  These functions
                   in Ivip - QSRs and optional QSCs - are a crucial
                   part of making Ivip, in my opinion, more scalable
                   than IRON-RANGER.   Fred is keen to keep his
                   system simple - and not to add anything in
                   between.

                   These are two forks in the road where Fred takes
                   his proposal one way and I take mine another.
                   From these choices, many differences result.

                   I am also trying to convince him to have the end-
                   user network register the IP addresses of the two
                   or more ETR-like IR(EID) routers directly with the
                   VP company, along with any other mapping details
                   to do with which one to use in preference, or how
                   share the load between them.  Then, the VP company
                   creates the mapping from that and sends it to its
                   multiple IR(VP) routers which handle this VP.

                   Fred believes there is no need to monitor the
                   usage of the DITR-like (PTR-like) IR(GW) routers.
                   I argue that since individual EUNs will place such
                   varying and sometimes high loads on these IR(GW)
                   routers that it will be absolutely necessary to
                   monitor and bill them for this usage.  The same
                   arguments apply to LISP.

                   I give an example of how Ivip can do fine-grained
                   real-time responsive inbound TE.  This is with the
                   mapping of each micronet being to a single ETR
                   address.

Hi Fred,

OK.

> behavior. 

Here I think you are referring to the repetitive messages by which
IR(EID) role router registers an EID prefix with (typically) two or
more IR(VP) routers which handle the VPs which cover this EID prefix.
 My use of the term "map-updates" is for something different, which
you don't do - and as you explain below you don't want to do:  having
the IR(VP) router maintain state about map-replies it sent out so if
the mapping changes, it can send a map-update to the queriers
(IR(LF/GW) routers) which it sent the map-replies to.

> But, see more below:

OK.

> what you mean by "inbound TE"?

OK, here is a mini-tutorial on Ivip's real-time TE:

The idea is to split up the destination address space for a given
traffic flow into multiple micronets (Ivip) or EID prefixes (I-R and
LISP).  This means it needs to be split over two or more IP
addresses.  If all the traffic in a body of incoming traffic which
needs to be split over two or more ISPs is to a single host, then the
host needs to be given two or more IP addresses, each within a
different micronet.   If the traffic to be split results from a
single FQDN, then this will be made to resolve to the two or more IP
addresses so the traffic will generally be split evenly between those
IP addresses, and therefore between the two or more micronets or EID
prefixes.

Let's say, for example, the network has two ISPs ISP-A and ISP-B.  If
there was only one major stream of incoming traffic and the only
desire was to split the traffic evenly between the ISPs, then it
would suffice to make this traffic be to IP addresses in just two EID
prefixes.  If the traffic was to a single host, then it would need to
have two IP addresses, so if the traffic resulted from a single FQDN,
that would need to resolve to these two IP addresses.

On average, assuming the traffic results from multiple sending hosts
which choose one IP address or the other at random, then the traffic
will be spread generally evenly between the two EID prefixes: EIDP-A
and EIDP-B.

To simply split the traffic between the two ISP links, it would
suffice to have EIDP-A mapped to the ETR (Ivip, LISP) or IR(EID)
router which uses ISP-A and likewise for the EIDP-B prefix to be
mapped to the ETR or IR(EID) which uses the link from ISP-B.

This is the simplest form of load sharing, and doesn't require
real-time control of mapping.  In LISP and I-R as currently defined,
you have some method by which the ITR (LISP) or routers playing the
IR(LF/GW) roles will automatically detect the unreachability of the
ETR or IR(EID) router they are currently using, and so tunnel the
packets to the other one.   So normally, as long a the ITR or
IR(LF/GW) router thinks both ETRs are working, each ITR or IR(LF/GW)
router will split the split over both links evenly.  But if one link
goes down, the traffic goes over the one remaining operating link.

With LISP and I-R as currently defined, there's no need to do the
above, since there is a way of specifying in the mapping that the ITR
or IR(LF/GW) devices automatically do the same kind of 50/50 split,
when both links are working, and to switch to the one remaining
operational link as soon as the other one dies.  (Yet, what are the
decision criteria for how long one ETR or IR(EID) router must be
unreachable before this occurs?  How long before the ITR or IR(LF/GW)
retries?  How long must the ETR or IT(EID) be up for, and able to
deliver packets to the destination network, before each ITR or
IR(LF/GW) router decides to start using it again?)

In Ivip, with real-time control of the mapping of each micronet, the
ITRs only have a single ETR address to use, so it is the
responsibility of the end-user network (EUN) to make sure the mapping
is changed in real-time so that when ISP-B, its link, or the ETR on
that link becomes unusable, that the mapping for the micronet which
is currently to that ETR-B will be changed instead to ETR-A, which is
still working.   As I describe in the Ivip documentation, this
detection of failure and change of mapping would typically be done by
a specialised "Multihoming Mapping Company" company hired by the EUN.

So far, there's no real advantage in the Ivip approach - assuming the
LISP or I-R system could detect the failure as rapidly and reliably
as the Multihoming Mapping Company.  I tend to think they can't, so I
think that Ivip would typically respond better to failures than LISP
or I-R.

Now I will consider Ivip being used for a more complex inbound TE
scenario.

Let's say there are four streams of incoming traffic to be managed -
TA, TB, TC and TD.  TA and TB are streams of packets coming from
multiple hosts to two sets of web servers.  TC is SMTP, DNS queries
and various other things.  TD is VPN traffic from multiple staff
members at home and travelling around the world,  including VoIP over
the VPN links, plus other VoIP sessions with servers all around the
world.

I will assume that the SMTP traffic can be split over two IP
addresses for the one server.  If it can't be, then this is a class
of incoming traffic which this example and Ivip's inbound TE
arrangements don't apply to.

I will assume that the rates of traffic of these 4 streams vary
significantly during the day, hour-to-hour, perhaps minute-to-minute.
 If the only aim was to split each stream 50/50 between the two ISP
links, then each stream could be directed to two IP addresses, in two
micronets, and one micronet of each stream could be mapped to the ETR
used by one ISP and the other micronet of each stream mapped to the
ETR used by the other ISP.

Here I will describe an example where real-time control becomes
useful.  This is a pretty contrived example, but it demonstrates the
flexibility which can be achieved with real-time mapping and single
ETR-address mapping.  In principle, this arrangement could be used to
solve arbitrarily complex inbound TE requirements.

Let's say these four streams vary greatly in their volume over the
week, due to populations of customers in various continents doing
e-commerce sessions, web-browsing sessions or whatever at different
times of the day and week.

In this example, ISP-A (and so ETR-A) is a 10Mbps low-cost service,
which for some reason has high latency.  ISP-B (and so ETR-B)
provides a 5Mbps link, and their service is more expensive (per
megabyte of incoming traffic) - but has a low latency and low
packet-loss-rate.

Let's say the 4 incoming streams of traffic are to IP addresses
covered by 8 micronets (TA0 to TD7) - two for each stream:

  TA:   TA0 and TA1
  TB:   TB2 and TB3
  TC:   TC4 and TC5
  TD:   TD6 and TD7

Each of these micronets can be mapped to one ETR or another.  A
typical arrangement would be:

  TA0       ETR-A        } Web sessions from Europe and Nth America.
  TA1       ETR-A        }

  TB2       ETR-A        } Web sessions from China, Asia and Oceania.
  TB3       ETR-A        }

  TC4       ETR-A        } SMTP.
  TC5       ETR-A        }

  TD6              ETR-B } VPN and VoIP.
  TD7              ETR-B }

If the TD stream grew to a point where it was too much for the 5Mbps
ISP-B link - such as occasional spikes leading to congestion and
dropped packets above what is tolerable - and if the ISP-A link was
not congested, then it would be a simple matter to change the mapping
of one micronet:

  TD7       ETR-A

to spread half of this load onto the ISP-A link instead.

If there was a flurry of TB activity from Asia, which, in addition to
reasonably high levels of TA and TC traffic was causing congestion in
the ISP-A link, then (assuming that TD7 was mapped as before to ETR-B
and that the TD traffic is reasonably low) it would be desirable to
send half of the TB traffic over the ISP-B link.  This can be done
with a single mapping change:

  TB3               ETR-B

There is tremendous flexibility in such an arrangement to achieve
real-time, minute-to-minute, inbound TE for a variety of traffic streams.

I won't elaborate further examples, but I hope you can see that this
enables traffic to be split up pretty much as desired and steered as
desired via whichever of multiple ISP links the EUN uses.

The EUN itself is in charge of this process, which can be done
manually or automatically.  The MABOC (Mapped Address Block Operating
Company, equivalent to I-R's VP Company) would charge a small fee (a
few cents I guess) per mapping change.  This would reduce the
temptation to be sending mapping changes every few seconds, but it
would be an attractive arrangement for sending changes every few
minutes if this enabled better utilization of the two or more ISP
services.

This system could handle all sorts of rapid and unanticipated changes
in traffic patterns.  Perhaps one or more teleworking staff members
are sending very large files over the VPN.  Perhaps, due to a TV
advertisement or a mention in a talk-show there is a sudden burst in
web-sessions and e-commerce transactions in the TA stream.

The more traffic streams there are, the more ISPs are used, and the
more variation there is in the traffic volumes per stream, the
greater the opportunities for generating benefits via this kind of
real-time, potentially very fine-grained, control of incoming traffic.

In general, this approach would enable a larger total volume of
traffic to be handled and/or with higher peaks per single traffic
stream (assuming all streams don't peak at the same time) over links
of a given capacity.  So, in principle, the costs of real-time
mapping changes could be much smaller than the only obvious
alternative - paying for faster links all the time.

A manual or automatic system for generating mapping changes for the
purposes of inbound TE could be established at the EUN.  These
mapping changes would be sent to the Multihoming Monitoring Company,
which has the credentials to actually control the mapping.  This way,
the MABOC only gets mapping change commands from the Multihoming
Monitoring Company, rather than from this company and from the EUN at
the same time.

Normally, as long as all the ETRs are operating properly, the
Multihoming Monitoring Company simply passes on the map change
commands it receives from the EUN to the MABOC.  (By this I mean that
the Multihoming Monitoring Company securely receives these changes
and then uses the information (the micronet and the new ETR address)
to generate its own map change commands via whatever system the MABOC
provides.)

If the Multihoming Monitoring Company detects that one ETR - say
ETR-B - cannot be used to reach the destination network, then it will
ignore the TE related mapping change commands it receives from the
EUN and send mapping changes so that all the micronets are mapped to
ETR-A, until ETR-B is found to be working again.  (ETR-B might have
failed, or ISP-B might have failed, or the link between the ISP and
the EUN - with ETR-B being either at the ISP-B or at the EUN site.)

OK.

OK.

> compared to the Teredo approach.

This explanation doesn't help me - even the first sentence doesn't
make any concrete sense to me.  In the future, if you can document
"recursively-nested RLOC-addressed local routing regions joined
together by EID-addressed border routers" with some examples, with
diagrams, I should be able to understand it.

Perhaps what you are alluding to relates to the mixed IPv4-IPv6 basis
of RANGER.  Below, you wrote:

   > IPv4 as RLOC space.

> perhaps we should give it a closer look.

OK:  http://www.firstpr.com.au/ip/ivip/#mobile

OK.

> IR(GW)s as part of their cost for doing business?

They would need to do this, I agree.  The question is how they would
apportion the costs of running these IR(GW) routers.  They might have
10,000 customers who are leasing space within the VP.  (This is just
like a MABOC in Ivip leasing SPI space within a MAB to 10,000 EUNs.)

But let's say 70% of the cost of running these IR(GW) routers around
the world was a direct function of the bandwidth they needed - and
that 80% of the traffic through these IT(GW) routers was going to
just 100 of the 10,000 EUNs, with a single EUN getting 10% of the
total traffic through all the IR(GW) routers?

It would be unreasonable to simply charge every one of the 10,000
EUNs the same amount, directly or indirectly, for this totally
EUN-dependent 70% of the cost of running the IR(GW) routers.

Let's say VPC-A did the above - flat-rate, charging (perhaps by
amount of address space used) but with no traffic-related charges for
any of its 10,000 EUN customers.  Meanwhile, a VPC-B company monitors
the usage of its IR(GW) routers and charges its 10,000 EUN customers
a fee which depends, in part, on the the load each EUN places on the
IR(GW) routers.

VPC-A's base charge is going to be higher than VPC-B's charge for a
small volume EUN.  So all the lower-traffic EUNs choose VP-B.  VP-B's
total charge for high-traffic EUNs - including usage charges which
would be very high for these EUNs - would be much higher than VPC-A's
flat-rate fees.  So all the moderate and high-volume EUNs would flock
to VPC-A.

Then, VPC-A would have a vastly busier and more expensive set of
IR(GW) routers to run than VPC-B's set of IR(GW) routers.

VPC-A would either go out of business, or be forced to increase their
flat rate to roughly match the costs imposed by their high-traffic
EUN customers.  When this happens, the moderate traffic EUNs will
find some other VPC-C/D/E with lower flat-rate costs, or choose
VPC-B, which even with its usage costs, is cheaper than VPC-A's now
much higher flat rate.  That leaves VPC-A with a handful of really
high traffic EUNs, and then, with this small number of flat-rate fees
coming in, which is unable to pay for the traffic costs of their
IR(GW) routers, they VPC-A would go out of business.

> fetch the info from their closest mirror site.

It is technically possible and could be efficient.  However, this
single file, single point of control, is subject to critiques
regarding single points of failure and administrative and competition
policy arguments that their should be multiple files, each with a
different organisation, so VP companies can choose which one to
depend upon and pay for.

> be routed within the IRON.

I might have a rough understanding of what you are proposing here,
but I am not going to try to fully understand the mixed IPv4-IPv6
arrangements you have for I-R.  I am not saying they could have no
value - just that I don't have time to fully discuss and understand
them, and that I am happy to plan Ivip as two separate systems.
Maybe, later, there will be a reason to link the two Ivip systems.  I
think people would get really bamboozled if I only described the two
systems as being interlinked.

I think your inbuilt IPv4-IPv6 linkage adds a whole extra level of
complexity to I-R.  Ideally, I think, you could explain I-R as a set
of principles and mechanisms or whatever which would be attractive
just for IPv4, in case IPv6 never becomes widely adopted - or even if
it does become widely adopted without the need to make IPv6 in any
way dependent on IPv4.

Likewise, I think you should be able to explain I-R for IPv6 alone,
as if IPv4 never existed, or has ceased to be important.

Then, ideally, you could explain what costs and benefits there are in
a form of I-R which supports both IPv4 and IPv6 with the recursion,
interlinking or whatever which you propose.

As it stands, with I-R only being described in terms of an
interlinked system, I think many people would reject it as being
over-complex, insufficiently modular or whatever - or they may simply
ignore it because they are convinced of one of the following:

   IPv4 will be, or should become, irrelevant in the timeframe of
   interest (say to 2030 or so).

   IPv6 may never be widely adopted - so the benefits of I-R would
   need to be for IPv4 alone.

   Even if it is assumed that IPv4 and IPv6 are both going to be
   widely used in the time-frame of interest, the two systems should
   not depend on each other.

I believe that the Core-Edge Separation architectures should be shown
to provide benefits for IPv4 and IPv6 separately.  Maybe there will
be good reasons to link the two systems, but I think it is too early
to assume that this must be done.

> BGP since the scaling would become untenable - right? 

I disagree.  I think it would be fine to add 100k prefixes to the
DFZ, provided it supports a very much higher number of prefixes for EUNs.

It would be technically possible and perhaps desirable for Ivip, in
the long-term, in IPv4, to have 100k MABs.  As long as each MAB
supported dozens of EUN micronets, this would still be a major
scalable routing improvement compared to each such EUN advertising
these millions of prefixes as PI prefixes in the DFZ.

I don't suggest 200k or 1 million MABs as a good idea.  Its hard to
imagine 100k MABs, but I suppose it could happen if most of them were
small prefixes (long prefix length) as more and more of the IPv4
global unicast space was turned into MABs to provide SPI space.
Then, there would be a honeycomb of address usage - one prefix being
used conventionally and the next as a MAB - all through the global
unicast address range.

In IPv6, all the MABs could be within a currently unused short
prefix, so there is no need to create them between the conventional
prefixes.  Therefore, there only needs to be as many MABs as there
are MABOCs, subject to some scaling constraints on how many micronets
can be in a MAB.  For instance, if there are too many micronets in a
MAB, then the time required for a QSA to download the complete
mapping data for this MAB might become a problem.  It needs to do
this at boot time, and it may need to do this if it ever looses synch
with the MABOC's feed of real-time updates.

I can't imagine there needs to be more than a few thousand MABOCs,
maybe 10k at most.  Probably a few hundred would be more than enough
for competition and diversity.  Each MABOC, in general, would have
one or a few MABs.  Even with 10^10 micronets (almost all of them for
mobile devices), with some MABs having 10^6 or 10^7 micronets, I
can't imagine that the total number of MABs in IPv6 would get much
beyond 10k.

I understand you don't want each VP advertised in the DFZ, as Ivip
has DITRs advertising MABs in the DFZ.

But still, I think my arguments remain for the necessity of VP
companies charging their end-user networks in proportion to the
amount of traffic each such end-user networks imposes on the VP
company's one or more IR(GW) routers.

There really needs to be multiple such IR(GW) routers for each VP, in
order to spread the load and to result in generally shorter total
paths between:

 Sending host > IR(GW) router > IR(EID) router > destination network.

> But,

Sure - each VP company needs to provide at least one IR(GW) covering
its VPs, otherwise their customer's EID address space would not be
reachable from any network which did not have its own IR(LF) router.

For various reasons, one VP company would probably have more than one
VP - just as with Ivip, one MABOC would probably have more than one MAB.

OK.

> means by which routers on NBMA links inform neighbors of

OK.  The "NBMA link" really means that some router X performing an
IR(LF/GW) role uses SEAL to tunnel a traffic packet to another router
Y which is performing the IR(VP) role for the Virtual Prefix which
covers the destination address of the traffic packet.  X and Y may
never have communicated before.  X knows about Y's address and its
role from some list or other mechanism, but Y has no prior knowledge
of X.

There are no preliminary packets required for this tunneling.  Y
simply gets a SEAL-encapsulated packet, and assumes that it came from
X, because X's address is in the outer source address and perhaps the
SEAL headers.

When the map-reply goes from Y to X, SEAL tunneling is used,
including a SEAL-ID from the header which came with the traffic
packet, which is sent back in the map-reply to authenticate this
map-reply.

While you refer to this one-way and then two-way communication taking
place over an "NMBA link", I don't regard it as a "link" because
there need for prior communication, no previously established tunnel
state etc.  No initial packets required to set anything up before
sending either SEAL-encapsulated packet.

I think the term "NMBA link" may imply, to some people, that X and Y
already had established a two-way tunnel, or that they both already
knew of the other and had communicated in the past.  This is not the
case, but if it had to be the case, then it would not scale.

> I have a new proposal - call it the "modified scattergun".

OK - let's go!  I was thinking that Elmer Fudd used a blunderbuss,
but I see he used a straight-bore shotgun, without the trumpet-like
flare which I think is characteristic of the blunderbuss type of
"scattergun".

> used solely for the purpose of triggering a map reply.

OK - I think this is a big improvement, with one additional problem.
 It completely solves the problem of duplication of packets at the
destination network.  Since the "blanks" can be short, it also
greatly reduces the burden which would result with long traffic packets.

Furthermore, this enables you to have large numbers of IR(VP) routers
for each VP, if you want do do so - overcoming the practical limit I
suggested of 3 or maybe 4 or 5.  With a larger number of IR(VP)
routers, the IR(LF/GW) router sends a live packet to a "closest"
IR(VP) router and shorter, lighter, "blanks" to some subset of the
remaining IR(VP) routers.  There will be less recoil on these
IR(LF/GW) routers too . . .

You could use the apparent geographic closeness just on its own, or
use it to select two or three apparently close-by routers, and then
ping them to check they are alive and to figure out which one is
topologically closest by the crude but good-enough metric of having a
short ping time.

But for large numbers of VPs and IR(VP) routers, and for each of the
large number (100k or more) IR(LF/GW) routers there is a potential
scaling problem with how each IR(LF/GW) router decides, for each VP,
which of the multiple IR(VP) routers is the "closest".   (Below, you
suggest a latitude-longitude solution to this.)

With the new system, there is an additional problem - there may need
to be good assurance that the "closest" IR(VP) router is actually
alive, since this is the only one which will forward the traffic
packet to the IR(EID) router.

This too raises scaling problems, since the IR(LF/GW) router needs to
know already which "closest" IR(VP) router to use for all the VPs.
It can't muck around at the time the traffic packet arrives, probing
these IR(VP) routers to see which ones are closest and/or alive.  It
needs to tunnel the full traffic packet as a "live" packet to one
IR(VP) router which must have a very good probability of being alive,
and then send "blanks" to a few more - all without time for sending
and receiving any other packets beforehand.

I think there would be scaling problems with each IR(LF/GW) router
having to repeatedly test the liveness of the IR(VP) routers it chose
as the "closest" for each VP.  Even if there were 100k VPs and only
10k IR(VP) routers which handled them all - most IR(VP) routers
handling multiple VPs.

There would be a significant scaling problem at the point of each
IR(LF/GW) router having to probe the liveness of all 10k IR(VP)
routers on a continual basis.

There would be a big scaling problem with each such router having to
respond securely to probes from the 100k+ IR(LF/GW) routers - even if
in practice each IR(LF/GW) router only chose a subset of these IR(VP)
routers as being "closest".

Perhaps it would suffice to do the following.  I will assume that
there may be 4 or more IR(VP) routers for a given VP, and that each
IR(LF/GW) router LLL has already chosen, but some means a single
"closest" IR(VP) AAA router and 3 others: BBB, CCC and DDD.

  1 - At boot-up, by some means - and perhaps occasionally in the
      meantime, if there is no other communication with AAA -
      LLL probes AAA to determine if it is alive.

      This may be integrated with whatever method is used to
      choose the "closest" IR(VP) router for a given VP.

  2 - As already proposed, LLL tunnels the full traffic packet as a
      "live" map request to AAA, and sends shorter "blanks" to BBB,
      CCC and DDD.  LLL then uses the first valid map-reply from
      any of these, and then tunnels subsequent matching traffic
      packets to whichever IR(EID) router it chooses from the
      mapping.

  3 - If, after half a second or so, LLL does not get a valid map-
      reply from AAA, it subsequently sends a blank, or some other
      packet which functions as a reachability probe.  If it gets
      nothing in return, after another half or one second or so,
      it regards AAA as being unreachable and therefore promotes
      another IR(VP) router for this VP to the role of "closest".
      This process may take some time, with probing various
      other IR(VP) routers it got from the list, from DNS or
      whatever.

With this arrangement, there's no absolute certainty that LLL can
reach AAA at the time it sends the "live" packet, but there is a
reasonable chance it will be able to do so, due to the initial step
in 1 above, and regular checks of this as also noted in 1.

If LLL discovers AAA is unreachable, which it could after sending a
single "live" packet, LLL will find another IR(VP) router within a
few seconds to use instead.   Unfortunately, during that time, it
could be receiving multiple traffic packets which it also needs to
send to AAA.  So perhaps, until it has found a new "closer" IR(VP)
router, it should choose another IR(VP) router at random and send
these packets to that router instead.  This would, of course, become
part of the search for a new, "closest" IR(VP) router.

In many protocols today, a second packet is only sent after receiving
a reply from the other end.  However, some protocols might send
multiple initial packets initially.  (This behaviour of applications
might be exacerbated if programmers perceive that the CES system is
less reliable at delivering initial packets.)

OK.

OK.

OK.

OK.

However, it isn't quite right to say that an IR(VP) router "owns" a
VP, unless there are no other IR(VP) routers for this VP.  Typically
there would be multiple IR(VP) routers for each VP, so I refer to
such a router "handling" the VP.

> only select the 2-4 IR(VPs) from the list that are

With the modified scattergun approach, I entirely agree.

OK.

> could have gone Sacramento->Portland->Seattle.

That's an interesting idea . . .   I think it is a good approach.

Geographical coordinates as a good-enough method to choosing
"closest".  "Closest" doesn't absolutely, always, need to be
topologically closest.  It is more a case of generally choosing a
closer IR(VP) router from various alternatives, where some of the
alternatives are on the other side of the Earth.  If a handful of
routers were geographically close, they could be pinged to determine
which was topologically the closest.

I suggest specifying the latitude and longitude in degrees with, say,
3 decimal places, rather than arc minutes and arc seconds, to make
storage and numerical comparisons easier.  For most populated parts
of the globe, it will suffice to treat latitude and longitude as
Cartesian coordinates.  IR(LF/GW) routers near the poles will need to
use a more elaborate algorithm to choose the "closest" IR(VP) router.

Thanks for this.  I will probably use it in Ivip's DRTM, where there
is also a need to choose a typically "closest" QSA authoritative
query server from a list of many.

At boot time, and occasionally thereafter, as I suggested above, the
IR(LF/GW) router could probe the "closest" IR(VP) router to see if it
is reachable and happy to accept tunneled traffic packets.

> information. I will look into this.

I think it would be good to add such records.

OK.

> test that its mappings are reachable in real time.

Yes - the reduction in ITR complexity (IR(LF/GW) for your proposal)
is a big reason for using a single ETR (IR(EID) router) mapping.
Likewise the modularisation of the system, totally externalising
reachability testing.  But these things are only feasible and
desirable with real-time control of mapping to all the ITRs
(IR(LF/GW) routers) which need the mapping.

> will black-hole. Like I said, let's come back to

It is possible to imagine circumstances where (using Ivip and LISP
terms):

   ITR-1 ------------ISP-A ------ ETR-A
                                       \
                                       EUN
                                       /
   ITR-2 ------------ISP-B ------ ETR-B

but where ITR-1 can't reach ISP-B and therefore ETR-B and ITR-2 can't
reach ISP-A and therefore ETR-A.  Normally, both ITRs can reach both
ISPs and therefore both ETRs.

Then, *if* each LISP ITR could correctly figure this out in time,
LISP would be able to achieve full connectivity when Ivip couldn't.

However, I think such conditions would be transitory since the DFZ
routers will be re-organising themselves to fix this - within seconds
I think.

The current BGP-based multihoming service restoration arrangements
involve the same problems and delays.

More common reasons for an ITR being unable to reach a given ETR
would be :

  1 - The ITR itself is isolated - unable to send packets to either
      ETR.

  2 - The ETR is dead.

  3 - The ETR is at the ISP and is alive but the link from it to the
      EUN is dead.

  4 - The ETR is at the EUN and is alive but the link from the ISP to
      the ETR is dead.

Nothing can fix 1, but Ivip can cope fine with 2, 3 and 4.

2, 3 and 4 are the sort of long-lasting multihoming connectivity
fault which I think the CES architecture needs to deal with.

> reply will eventually be received are almost certain.

I agree.

OK.

> implementations simple, which to me means no NAT traversal.

I entirely agree about having ETRs not behind NAT.  Ivip ETRs are
always on ordinary ("RLOC" in LISP terms) addresses - never on SPI
("EID") addresses.

At present, Ivip ITRs are not allowed behind NAT either.  Perhaps in
the future I will have a tunneling system, with keepalives, so they
can send map-queries to one or more QSCs or QSRs, and receive
map-replies and map-updates from them.   ITRs (including ITR
functions in hosts) can be on conventional ("RLOC) addresses or SPI
(micronet - "EID") addresses.

I wasn't suggesting that you allow IR(LF) routers to be behind NAT.
Just that if you restrict their numbers, by the expensive mechanism
of having them all listed in some file, or for other reasons, then
you prevent ISPs and EUNs from putting the IR(LF) functionality in
lots of their internal routers.  Putting this functionality in many
more routers means that the traffic packets are intercepted earlier,
by more routers, and this reduces the traffic load on each such
IR(LF) router.  This enables them to be less expensive devices.

However . . . with the current structure of I-R, where you have no
intermediary, caching, devices in the map-request - map-reply system
between the IR(LF/GW) routers and the authoritative query server
systems (one of the IR(VP) routers) there is a downside to having
larger numbers of IR(LF) routers.

If, in a given ISP, a single router did all the IR(LF) work, then it
would be very busy with traffic packets.  However, it would have a
sizable mapping cache, and would frequently be able to satisfy its
mapping requirements for new streams of packets from separate sending
hosts from its own cache.

If, instead, you had this work farmed out over 10 or 100 smaller
IR(LF) routers, then each one would be doing less work in terms of
traffic packets and in terms of map-requests and map-replies, caching
etc.  But each one would have less chance of already having the
mapping cached, so collectively they would generate more map-requests
and require more map-replies than if there was only a single such
IR(LF) router in the whole ISP.  So this would worsen the load on all
IR(VP) routers.

Also, a larger number of these IR(LF) routers means more scaling
problems for IR(VP) routers responding to any probes they make when
choosing which IR(VP) router is both "closest" (or close enough) and
alive.

OK.

> map reply process is stateless. 

OK - then I think this rules out changing I-R to do real-time
mapping.  Therefore, AFAIK, you can't do single ETR mapping either.
I think this means that like LISP, you have serious problems in the
ITR function - IR(LF/GW) routers - in scalably, reliably and quickly
determining which ETR functions (IR(EID) routers) are currently
reachable and capable of delivering packets to the destination network.

In Ivip, the QSA authoritative query servers do cache the information
about which querier got a map reply for which micronet, within the
current caching time.  Then, if the mapping changes, the QSA will
send map-updates to their queriers: QSRs in the ISP and in some
larger end-user networks.  The QSRs in turn do the same to their
queriers - ITRs and QSCs.  Likewise QSCs do it for their queriers:
ITRs and other QSCs.  I think this extra effort, which enables
real-time mapping being sent to every ITR which needs it, is well
worth the trouble.

> Instead, when an IR(EID)

OK - assuming the IR(EID) router is reachable and alive, it sends
back an "unreachable" message which triggers the IR(LF/GW) to start
afresh by tunneling the traffic packets ("live") to one "close"
IR(VP) router and by sending "blank" map-requests to some other
IR(VP) routers.

This might be OK - but what if the IR(EID) router is dead, and the
network didn't detect this or for some other reason did not send
unreachable messages to the IR(LF/GW) router?  I am unsure how much
we can rely on DFZ routers or routers in ISPs sending back such ICMP
unreachable messages.

There would need to be a way for the IR(LF/GW) router to securely
accept or reject such messages.  This gets back to the format of the
SEAL header, the placement of the SEAL-ID and whether or not the
SEAL-ID is returned in the packet fragment which is included in the
ICMP destination or network unreachable message.  I recall that IPv6
implementations are required to send back the largest fragment they
can fit in a certain ~500 byte packet length, but that IPv4
implementations are not required to do so, or are required to send
only the IPv4 header and the next 8 bytes.   I think there are
difficulties securely authenticating IPv4 destination unreachable
messages unless more of the packet is returned.  Ideally, IPv4
routers should send back more bytes - perhaps many of them already do.

But this doesn't solve the problem of the ITR-like device - the
IR(LF/GW) router - being able to know whether the ETR-like device -
the IR(EID) router - is able to deliver packets to the destination
network _at_the_time_ it tunnels the first packet to this ETR-like
device.

Ivip solves this by making the end-user network responsible for
changing the mapping to point to an ETR which can deliver packets
right now.

With LISP and I-R as currently defined, the ITR-like function may be
able to find out after it sends the packet whether or not this
ETR-like function can deliver it.  But by then, the packet has been
lost.  It could take the ITR-like function quite a lot of effort to
prove to itself that this ETR-like function really was dead, or
unreachable, or incapable of delivering packets, and then to find
another which could deliver the packets.

The ETR-like devices themselves could be peppered by such activities
from tens of thousands of such ITR-like devices at the same time,
which raises serious scaling problems.

None of these problems arise in Ivip, since the end-user network pays
another company (the Multihoming Monitoring Company) to probe real
reachability of their network via various ETRs.  This probing is
perfectly scalable and robust.  Then, the Multihoming Monitoring
Company will make decisions and mapping changes according to whatever
algorithms the end-user network desires regarding multihoming service
restoration.

> receipt of "unreachable" messages (see above).

OK - In my original conception of "I-R-Lite" I hadn't considered
these invalidations, and thought that perhaps the IR(VP) router would
send map-updates - but this is not part of your proposal.

OK.

OK.

> No; the IR(LF) will choose only the closest IR(VP)

OK.

> location information we would then be in business.

That could be done with new resource records.

But the single file is still subject to critiques about centralised
control etc.   DNS-based discovery of these VPs and FQDNs would not
be subject to this, as long as we accepted multiple organisations
being responsible for managing the DNS servers of the upper level
domain names at the root of the tree, which we do already with
ordinary TLDs.

> there some other constraint that I am not seeing?

I agree, the "modified scattergun" approach means you can have quite
a number of IR(VP) routers for a given VP, provided you have a
scalable way each of 100k or so IR(LF/GW) routers can choose the
"closest" and a subset of the others to use.

> if there is a DNS resource record type that supports

One of the luxuries of designing a solution to the routing scaling
problem is that you can and should think BIG.  The whole project is
so important that it perfectly acceptable to invent things out of a
hat, like new DNS resource records.  As far as I know, there is
little cost or complication creating such things.  They only need to
be understood by certain DNS servers and the IR(LF/GW) routers, and
whatever caching resolvers they might be configured to use.

> not mention but perhaps should think about.

I don't suggest IR(LF) routers be behind NAT.

However, as I wrote in the latter part of my message, I think I-R has
more scaling problems than Ivip because its overall structure is too
simple - you have no kind of "aggregation", "caching" or whatever
between your queriers (IR(LF/GW) routers) and the authoritative query
servers (IR(VP) routers).  So the IR(VP) routers get a hammering and
compared to their closest equivalent in Ivip: QSAs.

Ivip has at least one level of "aggregation" or "caching" between the
front line of the ITR and the authoritative query servers: the QSRs
(caching Resolving query servers, such as one, two or more at each
ISP).

All this can easily be seen from the PDF text file and the 3 diagram
files at:

Furthermore, one or more extra layers of caching can be introduced
between the ITRs and the QSRs via the optional QSC caching query servers.

With this greater complexity - very roughly the addition of QSRs and
optional QSCs, but also the load sharing amongst QSAs - I think Ivip
is able to scale better than I-R.  This enables more ITRs to be used,
which makes them cheaper and enables the ITR function to be performed
in sending hosts (at present, not behind NAT).

Since the Ivip mapping system has a map-update (requiring state in
QSAs, QSRs, QSCs and ITRs), Ivip can do real-time mapping.  This
greatly reduces the complexity of the ITR function, and enables
single ETR address mapping, so reducing the size of the mapping
information in total.

> mappings would work in a scalable fashion.

The flat file is technically feasible and would be quite efficient.
I think that for reasons of competition policy, avoidance of single
point of control / failure etc. that some less centralised approach
would be better.

>      element beyond IR(LF/EID), IR(GW) and IR(VP).

OK - but my critique above still applies.  Ivip has greater
complexity in terms of having QSRs and optional QSCs.  These and the
QSAs (most akin to IR(VP) routers) all carry state about previous
map-reply messages they sent, so they can send map-updates.  This
scales better and enables real-time mapping, which reduces complexity
in other ways and removes the problem of ITRs having to figure out
which ETRs can deliver packets to the destination network.

> scope. Maybe you think NAT traversal is not so much

With Ivip, for an ITR to be behind NAT - as a router or in a sending
host - I think there would be little trouble with NAT traversal.  I
would need to check this carefully regarding the IPTM protocol by
which the ITR sometimes sends a long and a short packet to the ETR
and expects a reply, as part of testing Path MTU to that ETR.  In
Ivip, the ITR doesn't otherwise communicate with the ETR, or listen
for ICMP unreachable messages.  It is up to the EUN - or a company
hired by the EUN - to get the mapping right.

The only elaboration I think is needed for an ITR to be behind NAT is
 a somewhat different protocol than the current UDP packets, perhaps
just TCP with keepalives, by which the ITR as a querier can
communicate with one or a few QSCs or QSRs to get mapping
(map-replies and map-updates), as well as to download and update the
list of "edge" space (all the MABs).   This could be done, I am sure.
 As part of defining the protocols in detail in the future I would
probably add this as an option.

With I-R as currently defined, your ITR-like functions (IR(LF)
routers) need to be able to receive ICMP destination unreachable
messages from ETR-like functions (IR(EID) routers) as well as getting
the map-replies from IR(VP) routers.  I am not sure how well this
would work through NAT, but perhaps it would work fine.

As a general principle I think the more ITRs the better - the more
IR(LF) routers the better.  However, in the absence of anything
between these and the authoritative query servers (IR(VP) routers),
the more you have, the more work the authoritative query servers will
need to do, as I mentioned above - since each IR(LF) router will have
less chance of already having the required mapping in its cache.

While Ivip's scalability doesn't depend on putting ITR functions
behind NAT, or in sending hosts, it is an option to put them in a
sending host on ordinary address space or on SPI space.  In the
future, I may well provide for ITR functions behind NAT too.

As long as you have no caching intermediaries between your IR(LF)
routers and the IR(VP) routers, I don't think you can have both:

      IR(LF) routers behind NAT

and   My previously suggested notion of real-time mapping with
      map-update messages being sent from IR(VP) routers.

While I argue it is a worthwhile expense having the extra state in
the authoritative query servers - IR(VP) routers, or QSAs in Ivip -
doing this without any caching intermediaries (QSRs and optional QSAs
in Ivip) would lead to some prohibitive problems:

  1 - Without the intermediaries, each authoritative query server
      would have to send out a very large number of map-updates.

  2 - Without any intermediaries (such as for Ivip, QSRs and QSCs
      with keepalive sessions established from their ITR queriers)*
      there is no scalable way the authoritative query server can
      keep open the NAT channel with sufficiently frequent keep
      alives - nor would it be desirable for the queriers (IR(LF)
      routers) to send keepalives to all these distant authoritative
      query servers.

  * With or without NAT between the querier and the query server,
    the querier needs to periodically check that the query server:

       querier:  query server:

       ITR  ----> QSR

       ITR  ----> QSC

       QSC  ----> QSR

       QSC  ----> QSC

       QSR  ----> QSA

    is still reachable, alive and hasn't been rebooted.  If any of
    these conditions are true, then the querier should flush any
    cached mapping it got from this query server.  This is because
    in ordinary operation, we expect the query server to send a
    map-update to queriers which it recently sent a map-reply to for
    the micronet whose mapping just changed.  This update couldn't
    be sent if the query server is unreachable, dead or has been
    rebooted - so the cached mapping might be stale.

I think this is an interesting discussion, because it illuminates how
you make certain choices which prevent your design from more closely
resembling mine.  Then, within those constraints, we are exploring
the best way of doing things.

Within those constraints, I think my system is more complex in some
ways, simpler in others, scales better and modularly externalises
reachability testing and multihoming and TE decision-making.  Yours
is simpler in some ways, more complex in others, and I think doesn't
scale as well - or enable end-user networks to directly control the
mapping of their space in real-time.

In summary, you are keen to have few elements and for those elements
to be simpler - in particular no state in the IR(VP) function.

However, I think this means you get poorer scaling - the need for
fewer, larger, more expensive, IT(LF) devices - and these need to be
more complex, with more complex mapping and be able to solve the
still unresolved problem of scalably determining reachability to
ETR-like devices, and their ability to deliver packets to the
destination network.  Also, you you can't give end-user networks
real-time control of mapping.

> IR(VP)s and sends blanks to the others.

I agree.  Now I think the biggest scaling problems in I-R are:

  1 - The lack of "aggregation" or "caching" between the ITR-like
      queriers and the authoritative query servers - causing a
      greater load on the latter: IR(VP) routers.

  2 - Difficulties and scaling problems with large numbers of
      IR(LF/GW) routers trying to establish which of multiple
      IR(EID) routers can delivery packets to the destination
      network, ideally *before* the IR(LF/GW) router sends
      any packets to them.

  3 - The authoritative query servers (IR(VP) routers) also having
      to receive and tunnel full initial traffic packets, which
      will frequently be short and few, but which could be long and
      numerous.

OK.

>> traffic as to overload the IR(VP) routers.

OK.  But my statement was rather simplistic.  You could have a small
VP which has a lot of traffic, and so need to make it smaller still
to split this traffic over more IR(VP) routers.  This was on the
assumption that you could only have a few IR(VP) routers for each VP.
 With the "modified scattergun" approach, this no longer applies.
You can have dozens of IR(VP) routers for a given VP.  Each of these
IR(VP) routers will only get full "live" traffic packets from the
subset of IR(LF/GW) routers which choose it as being "closest" from
the complete set.

Even with multiple IR(VP) routers and the geographic "closest"
algorithm, you might, in rough (oversimplified) theory, get very high
traffic loads of initial packets on a single IR(VP) router.

If, due to a football game, earthquake, political/celebrity scandal
or some other equally ground-shaking event, everyone and their dog
(Aussie term for pretty much every person) in a given city hops on
their computer and accesses some website, then most of the work is
going to be done by the IR(LF/GW) routers in that city, and they will
all probably chose the same single IR(VP) router, based on the
geographic algorithm.

That IR(VP) router will need to handle all the initial packets.  A
saving grace is that, assuming this IR(VP) router really is close to
the many IR(LF/GW) routers, that its map-reply messages should get to
the IR(LF/GW) routers quickly and reliably.  A further saving grace
with this scenario is that all the IR(LF/GW) routers in the city will
quickly get the mapping they need - and so not need to send traffic
packets or "blank" map-queries to any IR(VP) routers.

These saving graces are pretty powerful, so while in crude theory it
might look like the one IR(VP) router gets a lot of initial packets,
in practice, it would probably work out much better than this.

> networks, ISP networks, etc. 

OK.

> Within each enterprise, the
> networks. 

I don't yet understand how this would work or what benefits it would
have.

> Traversal of each layer of the hierarchy

If you can illustrate this with a detailed example and diagrams I
should be able to understand what benefits you think will arise from
this recursion, and why you believe people will want to rearrange
their Internet connectivity in this manner.

> ITRs in sending hosts just fine.

OK - but subject the concerns mentioned above that the lack of
"aggregation" or "caching" between these and the authoritative query
servers leads to greater workload for those authoritative query
servers the more the ITR-like work is split up, as would be the case
with ITR-like functions in sending hosts.

With Ivip, an ITR or ITR-in-sending host could be on SPI ("edge" =
"EID") address space.  Map-replies, map-updates (from QSRs and QSCs)
and IPTM messages from ETRs need to go via an ITR and an ETR to reach
these ITR functions on SPI addresses, but that is fine.

I suspect that with I-R, you couldn't have an IR(LF) like function on
an EID address.

> enterprise networks to the Internet today.

I agree - in principle you could make these border routers do all the
extra work of the IR(LF) functions.

I am keen to make Ivip easily deployable without any need to add
functionality to existing border routers.  The ITR functions could be
done in a suitably programmed server, which participates in the
internal routing system by advertising all MABs.  Servers are cheap
and easily reprogrammed.   Likewise DITRs can be servers too.  All
the query servers: QSAs, QSRs and optional QSCs would be servers, of
course.  The ETR function could be done in a server too.

Quad core 3GHz, 12Gbyte RAM servers with dual 1GBps Ethernet ports on
the motherboard can probably handle quite a few packets.

So no routers need be altered to deploy Ivip.  Of course it would
make sense to add ITR and probably ETR functions to new and existing
routers - but this would not be required to establish substantial
initial services: MABOCs hiring out SPI space.  The whole DRTM
(Distributed Real Time Mapping) system is a phased approach in which
ISPs do almost nothing initially - and make no investment in software
or hardware at all.  All the initial deployment work is done by
MABOCs and could be done with suitably programmed COTS (Commercial
Off The Shelf) servers.

I think the I-R architecture requires the new functionality be
available in the existing routers before it can be initially
deployed.  This might be easy - a low or zero cost software update
from the router vendor (or am I being hopelessly over-optimistic?).
But if the current router in a particular position doesn't have such
functions available, and if you feel the functions really need to be
done at that router, or nearby with a single device which needs more
packet handling capacity than a server, then it makes I-R harder to
deploy, I think.

OK.

> messages from the previous IR(EID) router.

OK - as discussed above.

OK.

OK.

OK.

In the absence of any prior system of the IR(LF/GW) router knowing
that the IR(EID) router can currently deliver packets to the
destination network, there may be a significant fraction of initial
packets getting lost in this way.

The initial packet flow is not from the IR(LF/GW) router but:

  SH --> IR(LF/GW) -------> IR(VP) ------> IR(EID)

so the destination unreachable will go from the IR(EID) router to the
IR(VP) router, not to the IR(LF/GW) router.

I guess you will have a mechanism in the IR(VP) router to securely
recognise this (if it can be securely recognised in IPv4, as
mentioned above) and for the IR(VP) router to choose an alternative
IR(EID) router from the multiple such routers in the mapping.

But by then, the IR(LF/GW) router will probably have received the
mapping from the IR(VP) router and will be sending one or more
subsequent traffic packets to the same IR(EID) router.  So it would
not be surprising for two or so traffic packets to be lost.  Then, I
am not sure how you intend the IR(LF/GW) router to choose another
IR(EID) router, if there were three or more in the mapping.
Frequently, there would be just 2 IR(EID) routers, so the decision
involves simply using the other one.

In the above system, every IR(LF/GW) router needs to find out "the
hard way" (two or more packets being lost) that the IR(EID) router
which the mapping currently suggests be chosen first is not able to
deliver packets to the destination network.

This is assuming there is an unreachable message of some kind sent
back.  I assume that if the IR(EID) router is alive and reachable,
and if it can't deliver packets to the destination network at that
instant in time, that it will send an unreachable message.  But
perhaps, depending on how it connects to the destination network,
that might only be generated half a second or more after the packet
is lost.

> I think you meant "a single ETR address"? 

Yes.

> But, what

As discussed above, I expect such "splits" within the DFZ to be
transitory and fixed within seconds.  I could elaborate Ivip's
mapping and its ITRs to have multiple ETR addresses for intrinsic
load sharing, but I think it is not worth the trouble.  I certainly
do not want to have multiple ETR addresses in the mapping so ITRs can
somehow choose different ETRs if they somehow figure out an ETR is
unreachable, since I don't want such complexity in ITRs and because
the ITR on its own can't know whether an ETR can reach a destination
network.  (This could be fixed with greater protocol complexity, or
by the mapping containing a test IP address to ping through the ETR,
but that would still lead to scalability problems with 10k or more
ITRs trying to do this all at the same time via the one ITR to the
one test address.)

> itself without the need to involve a 3rd party entity.

OK - but then you need a PKI each IR(LF/GW) router can use to
authenticate the signature of each VP company's CA!

> the EUN and the IR(EID) router.

I can't see how an EUN could be behind NAT.

I don't think there needs to be an autodiscovery by the EUN of the IP
addresses of the IR(EID) routers it is using.  Firstly, these could
be its own routers, on PA space it gets from its ISP.  Alternatively,
if they are routers at the ISP, then the ISP will tell the EUN
administrators what their IP addresses are.  I assume that these will
be stable in both cases - that is the PA prefix the EUN gets from
each ISP is stable, not reliant on DHCP etc.  Also, I assume that the
ISP-run IR(EID) router will be on a stable address, with some locally
arranged hot-swap to replace it if the physical hardware fails.
(Also, an EUN could have ISP link handled by an IR(EID) router at the
ISP site, and another link from another ISP handled by its own
IR(EID) router.)

So I think the multihomed EUN administrators would know the IP
addresses of their two or more IR(EID) routers - and would securely
communicate this to the VP Company from whom they lease their EID
prefix, by any of various means.  As noted below, these means need
not include communications through any of these IR(EID) routers.

> the router in the ISP network) and then to the EUN

I see the act of the EUN telling the VP company the IP addresses of
its IR(EID) routers as a static action to be taken once and usually
not repeated, unless it changes ISPs.  It could be done securely by
any Internet connection using SSH or HTTPS, or via phone or fax.

In I-R it is not the job of the EUN, or the VP company, or the IR(VP)
router to respond instantly to some ISP going down, or some IR(EID)
router being unreachable or unable to deliver packets.  That is all
handled in the ITR-like IR(LF/GW) function - by mechanisms which are
yet to be fully defined.

I also think there needs to be a way of planning changes to the
addresses of these IR(EID) routers, rather than just changing them in
real-time.  Let's say an ISP informs the EUN that at 3 AM on Sunday,
they will be turning off one LF(EID) router and replacing it with
another on a different IP address.

The EUN doesn't want to have someone up at 3AM to tell the VP
company.  They should be able to tell them well beforehand and the VP
company should change the mapping in their IR(VP) routers at the
appointed time.  (In Ivip, I assume that the Multihoming Mapping
Company could be told of such changes, and have its system
automatically change the mapping at the appointed time.)

> network to route its EID prefixes. 

The ISP and its EUN customer certainly needs to work together to:

  1 - Have the ISP choose an IR(EID) router and tell the EUN of
      its IP address.

  2 - Make whatever physical connection, routing address arrangements
      are required, and whatever else is required to get packets in
      and out of the EUN, where those packets are from or to the
      EUN's EID address range.  This includes the ISP forwarding
      outgoing packets whose source address are in this EID prefix,
      so these packets go to other hosts in the ISP, and to hosts in
      the rest of the Internet.

Whether or not the ISP uses NATed space - address space which is not
global unicast - internally is a private matter between it and the
EUN.  Perhaps the EUN doesn't need to know about it anyway.

> Otherwise, there

OK - so the EUN's router or routers are performing the IR(EID)
function.  The ISP needs to provide a stable global unicast address
(RLOC, not EID) for that IR(EID) router to function from.  This would
be in the form of a PA prefix - PA as far as the EUN is concerned.
How the ISP does this via its routing and addressing scheme doesn't
really matter.

> without also coordinating with the ISP network.

I am not assuming that the method by which the EUN registers the
IR(EID) addresses with the VP company actually needs to go via these
IR(EID) routers.  It could be done via a phone call, or an HTTPS
session over a 3G network via a laptop or cellphone or whatever.

> duty NAT traversal, which I can't see happening.

As noted above, I don't think the ISP needs to be involved in the
path of communications by which the EUN registers the IR(EID)
addresses with the VP company.

OK.

> itself (i.e., the "locator liveness" problem).

I discussed above how this works for Ivip.  There will be some
scenarios where it won't work, but I expect them to be transitory -
the sort of thing the DFZ responds to rapidly.  This speed of
response is apparently good enough for today's BGP-based, unscalable,
approach to multihoming service restoration, so it should be OK for a
CES architecture such as Ivip or I-R.

> OK, but it still needs to involve the ISP networks.

As discussed above.

> of mapping change instead.

OK - you go the high way and I'll go the low way . . . and Ivip will
provide real-time control of mapping, with more types of network
elements, much less complex ITR-like functions and better scaling.

The code for the ITR, QSC, QSR and QSA will contain significant
commonalities.  Once we have code for an ITR and an a QSR, it would
be quite simple to cook up code for a QSC.

> network is internally using private IP address space.

As discussed above.

> indication of mapping change.

As discussed - the question of map-updates and the state required for
them in the authoritative query servers is a fork in the road.  I go
one way, and you go another.

> more than one RLOC->EID mapping.

No - the EUN either does it themselves, which is tricky without a
bunch of servers in the DFZ, or more likely hires a specialised
Multihoming Mapping Company to probe reachability of their network
via the various ETRs, and to change the mapping accordingly.

> need to be involved in the EID mapping coordination.

As previously discussed.

> is some packet loss "brown-holing" until the new

Yes - this is your choice, which involves more complex mapping,
greater complexity in the ITR, and reliance on the ETR either being
alive and sending an unreachable message, or being dead and the
network sending an unreachable message to the IR(LF/GW/VP) router.  I
am not sure how well the latter can be relied upon.

Even with the former, you have to hard-code decision algorithms into
your I-R system.  You need to specify how the ITR-like function of
the IR(LF/GW) routers will respond to such an unreachable.  How can
they authenticate the destination unreachable message, especially
with IPv4?  How long do they accept its meaning before trying again?
  How is the required probing - or potential lost packets in the
future if you don't use special probe packets - going to scale well
when there are 10k IR(LF/GW) routers all trying to figure out whether
one IR(EID) router is reachable and able to deliver packets to a
destination network?

> [email-masked]

Thank you Fred for discussing your proposal in such detail and with
such patience.

I will probably pinch your geographic "closeness" idea for Ivip's
DRTM algorithm for choosing a "nearby" QSA - and will attribute the
idea to you.

  - Robin

  http://www.firstpr.com.au/ip/ivip/

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 16 Apr 2010 17:12:59 -0700

Hi Robin,

Please see below for some follow-up:

> Sent: Tuesday, April 13, 2010 3:38 AM
> To: Templin, Fred L; RRG
> Subject: Re: [rrg] IRON-RANGER, an interesting Core-Edge Separation (CES)architecture
>

Good - it was helpful for me to see the Ivip details at
the beginning of the message instead of the end so that
I could consider them without bias from first bashing
through pages of IRON-RANGER specifics.

> that would need to resolve to these two IP addresses.

I understand from this that you intend multi-addressing
as an important aspect of the multihoming and traffic
engineering solution? But, I assume also that this would
be multi-addressing based on configuring addresses from
multiple of what you are calling "micronets"? Is that
right? So then, the mapping of each micronet determines
the means for manipulating traffic engineering for
multi-addressed networks (or something like that).

> unreachable before this occurs?

Unreachable messages provide immediate feedback when they
are available. If the ETR is alive but there is no route
to the destination, it can return SCMP unreachables
immediately. If the ETR is dead, the network may or may
not return ICMP unreachables, and any ICMP unreachables
may or may not contain enough information to determine
whether they are authentic. So, the SEAL explicit
acknowledgement mechanism provides a means for getting
periodic heartbeat keepalives from the ETR to the ITR.
This is the "hints of forward progress" indication to
the ITR's Neighbor Unreachability Detection mechanisms.

> How long before the ITR or IR(LF/GW)

This last one is actually a very good question. Once
an ITR detects an ETR as dead, in I-R it will not try
that ETR again until the mapping state expires and the
ITR goes back to the IR(VP) to get a new mapping update.
So, the ETR could potentially be back to life for quite
some time before the ITR decides to use it again.

> a specialised "Multihoming Mapping Company" company hired by the EUN.

Why can't the EUN test reachability itself, e.g., by pinging
to some well-known site w/o requiring the MMC? To me, the way
you describe Ivip makes it sound like it has way too many
moving parts, and I would be interested to see if some of
them could be eliminated.

> arrangements don't apply to.

This seems to bear out the assumption I made above, being that
you are using a multi-addressing approach where the addresses
are taken from PI (not PA) prefixes. Otherwise, there would
be no opportunity to dynamically map the prefixes between
multiple ISPs.

> desired via whichever of multiple ISP links the EUN uses.

In IPv4, I understand your micronets to be very short IPv4
prefixes - perhaps even consisting only of a small number
of consecutive addresses rather than a traditional prefix.
What would be the granularity for something to be considered
a "micronet" in IPv6? A /56?

> the EUN - with ETR-B being either at the ISP-B or at the EUN site.)

Why can't the EUN test its reachability itself? Can't the
EUN simply ping or traceroute out through each of its ISPs
to a known reference point on the Internet on its own accord
and see what comes back? I'd rather not have to require so
many 3rd party piece-parts such as an MMC.

> >>             fixed, but it is always on a conventional "RLOC" - AKA

This I think is a key point. In order for this to work,
the IR(EID) has to either be on a public address (in which
case it can tunnel to ETRs on its own accord) or on a
private address (in which case it will need for there
to be some device in the operator's network with a public
address that can act as its go-between). The former case is
what you are calling a "core" address, and the latter case
is what I call RANGER.

> OK:  http://www.firstpr.com.au/ip/ivip/#mobile

OK. I will get to the mobility part after we sort out the
core architecture.

> > I will keep this thought in mind as I respond to

OK, but one issue I think we are going to run up against
is ingress filtering for EID-addressed packets emanating
from end user networks. Doesn't the ISP need some way to
do ingress filtering so that rogue customers can't produce
packets with spoofed source addresses?

> IR(GW) routers, they VPC-A would go out of business.

I appreciate this discussion of IR(GW) economics, but what
happens if someone overloads an IR(GW) with spoofed EID-
addressed packets? Won't the EUN that owns the prefix
that covers the EID addresses be charged for the bogus
traffic?

> different organisation, so VP companies can choose which one to

All we would ask of the flat file is that it contain a
list of VPs and a FQDN for each VP. So, all that is needed
is a "VP listing organization" (maybe using a model similar
to IANA or ICANN), and each VP company gets to manage their
own VP-to-RLOC mappings. I don't think we would need to
solve world peace in order for something like that to
work...

> to assume that this must be done.

A very big reason (perhaps the *only* reason) for the linkage
of IPv6 and IPv4 was that originally I was planning to leverage
IPv6 ND and ICMPv6 as the means for propagating mapping and
reachability information. But, now I realize that that was
setting the 3rd "tone" in the Internet protocol stack one
half step too high resulting in an unresolved "sus4" sound
instead of the desired major chord.

Now, I am asking that SEAL be the mid-layer that performs
all mapping and reachability functions vis-à-vis the new
"SCMP" protocol. So, SEAL "glues" the inner network layer
protocol to the outer network layer protocol regardless
of whether the inner/outer protocols are the same. That
means that the inner/outer layers could be IPv4, IPv6 or
perhaps even something else like OSI/CLNP.

So, let's say that the inner layer is IPv4, and the mapping
returns a mixture of both IPv4 and IPv6 RLOC addresses. That
means that the ETR can be reached via IPv4-in-IPv4 tunneling
when IPv4 RLOCs are used, while the ETR can be reached via
IPv4-in-IPv6 tunneling when IPv6 RLOCs are used. In another
example, if the inner layer is OSI/CLNP, CLNP-in-IP tunneling
could be used. So, what we have is an any-in-any tunneling
capability, and it doesn't matter whether we are operating
with mixed or homogeneous network layer protocol combinations.

> DFZ, provided it supports a very much higher number of prefixes for EUNs.

I wasn't so much concerned with the number of prefixes added,
but rather the number of routers participating in the routing
protocol. I assume BGP routers peer with only a few other
BGP routers (i.e., and not a full mesh), but if we add 100k
additional BGP routers how does that affect the scaling for
example in terms of path length stretch, etc. Maybe a BGP
expert like Tony could answer?

> has DITRs advertising MABs in the DFZ.

I think you have me convinced. I think I will relax this, and
allow each IR(VP) to advertise all of its VPs into the DFZ.

> company's one or more IR(GW) routers.

OK, but I'm just concerned about DOS attacks that would
run up an innocent user's bill (as mentioned earlier).

OK.

> VP - just as with Ivip, one MABOC would probably have more than one MAB.

I agree. Multiple VPs, since the number of sub-prefixes
taken from each VP needs to be kept manageable.

Here, I am borrowing the term "sub-prefix" from the Virtual
Aggregation (VA) proposal ('draft-ietf-grow-va') and I think
we will see that I-R has a number of similarities with VA.
Plus, the "VA" (http://www.va.gov/) would seem like a natural
organization for managing my entourage of warriors.

> role from some list or other mechanism, but Y has no prior knowledge

Correct.

Yes.

> case, but if it had to be the case, then it would not scale.

Actually, I think there is plenty of literature on NBMA
links that shows that there need not be any a priori
arrangements between neighbors on the link. Indeed,
RFC4861 talks quite a bit about the operation of IPv6
ND over NBMA links.

The way one discovers neighbors on NBMA links is via
redirects from routers on the link. That is exactly
what is going on here.

> topologically closest by the crude but good-enough metric of having a

I'd rather not do any pinging and just go based on the
LAT/LON (below). But, this raises the question of whether
the IR(VP) routers can always be expected to be on the
face of the earth (e.g., and not on some other planet)
and whether the link delays in the core can be considered
to be on the same order (e.g., and without SATCOM or other
LFN links in the path).

> routers on a continual basis.

For liveness probing, I have been up to now thinking to use
the SEAL explicit acknowledgement mechanism whereby the ITR
periodically sets an "Acknowledgement Requested" bit on a
packet and uses any acks that come back as "hints of forward
progress" (see: RFC4861) as keepalive indications. But, if
we go with your real-time mapping, maybe we can do away with
the probing. More on this later.

> less reliable at delivering initial packets.)

I really don't want anything as complicated as all this.
All I really want is to just get back at least one map
reply; I don't even care too much if the live packet is
lost, because that should be due to a transient outage
anyway and the original source will retransmit. So, I
don't want to do any additional work beyond just firing
off the live packet and a few blanks. With this approach
the live packet *should* get through, and there should
be an extremely high probability of getting back a map
reply back.

> > faster convergence, however.

Check.

> > only be one "live" packet, and the rest are "blanks".

Check.

> > the "live" packet gets through to the final destination.

Check.

> such a router "handling" the VP.

OK - "handling" it is.

> With the modified scattergun approach, I entirely agree.

Good.

> > This will not be a problem with "modified scattergun".

Check.

OK.

> I think it would be good to add such records.

Actually, it could be that we can leverage the existing
"Well Known Service (WKS)" resource records that are
defined in RFC1035. As long as we mark each WKS RR with
the IP Protocol number for SEAL (to be assigned by IANA)
we should be able to define the variable length bitfield
format any way we like, including encoding LAT/LON
information and either IPv6 or IPv4 addresses.

> >>    dropping of such packets is a good idea.)

'draft-ietf-grow-va' also stipulates that an IR(VP) that
does not have a matching prefix must drop the packet. This
is in complete agreement with what we are proposing in I-R.

> (IR(LF/GW) routers) which need the mapping.

I'm going to suspend disbelief and presume that we can find
some way to do real-time mapping in which the mapping always
returns just a single address as you suggest. I will have
some questions and comments on this as we proceed below.

> involve the same problems and delays.

I will have to take your word for the above. I really
and truly do want to believe it, and as someone who
I regard as very wise said to me many years ago: "the
Internet is connected". Which would tend to support
your assertion that any transitory outages should
fix themselves within a few seconds. Much is riding
on this assumption, so we have to be absolutely sure
that it is valid.

> > reply will eventually be received are almost certain.

Check.

> > that thought for now and discuss further below:

Check.

> IR(LF) router.  This enables them to be less expensive devices.

OK. This means that if we want to avoid NAT traversal then
our IR(LF) devices will need to be assigned public addresses.
In IPv4 we are running out of these, but not in IPv6. So, if
we allow a mix of RLOCs (both IPv4 and IPv6) then we can
run I-R tunneling either over IPv4 or IPv6, and there should
be no shortage of IPv6 space.

So, to me, this means that ISPs can deploy provider-aggregated (PA)
IPv6 addressing and routing, and EUNs can use the PA addresses as
a springboard into obtaining provider-independent (PI) prefixes
from IR(VP) companies. The EUN can then arrange to have its PI
prefixes associated with any of its ISPs while using the PA
addresses as a means for transiting the ISP network.

> choosing which IR(VP) router is both "closest" (or close enough) and

OK, this brings up an interesting point. It would be good
to have an intermediate caching device so that many IR(LF)s
can share the same mapping information without overloading
the IR(VP)s. But at the same time, ISPs need to be involved
at some point so that they can do ingress filtering and not
allow rogue EUNs to spew packets with spoofed source addresses.
So, why not have a caching router in the ISP network that
all EUNs connected to the ISP are required to forward their
tunneled packets through? Maybe we can call it an IR(DR) for
"Default Router"? The EUNs could set the IR(DR) as their
default router in the I-R overlay, and the IR(DR) could
receive redirects from IR(VP)s and keep a cache from which
it serves all of the IR(LF) routers in the ISP.

This also feeds into later points about the realtime mapping
updates. As long as the IR(VP)s only need to update IR(DR)s
(i.e., and not all IR(LF)s) then perhaps the realtime mapping
can be scalable.

> > and cancellations (see more below).

Check.

> real-time mapping being sent to every ITR which needs it, is well

I think what you are referring to as QS*'s here is one and
the same with what I was calling IR(DR) above. But, I think
I only need one variety of QS* device, where you seem to
think there needs to be three different kinds?

> which raises serious scaling problems.

I had rebuttals to all of these points prepared when I
was considering sticking with the current I-R IPv6 NUD
model of liveness testing, but for now I am going to
suspend disbelief and pretend that realtime mapping per
your Ivip proposal is feasible. If it turns out to not
seem feasible following the ensuing discussion, then I
will go back to the IPv6 NUD model.

> algorithms the end-user network desires regarding multihoming service

Here is where you keep losing me. Why do we need an MMC if
the EUN can probe its own ETR reachability?

> send map-updates - but this is not part of your proposal.

Not the current proposal, but considering going to realtime
mapping.

> > mapping table. That is the way in which the system is

Hmm. I lost context for this, and it may not matter if
we determine that realtime mapping is feasible for I-R.

> > however. These features are due to the characteristics of

Check.

> > paths will therefore be kept short.

Check.

> That could be done with new resource records.

WKS should work.

> control etc.

Should be no different than the way the current IANA,
ICANN, etc. models work.

> DNS-based discovery of these VPs and FQDNs would not

DNS resolution of FQDNs into IR(VP) addresses would
be controlled by the VP companies themselves. All that
is needed in the flat file is VP->FQDN bindings. Unless
you can think of some way to query the DNS to ask it for
the list of all VPs in the universe? I don't know of a
way to do that...

> "closest" and a subset of the others to use.

We are thinking of LAT/LON as indication of closeness.

> whatever caching resolvers they might be configured to use.

WKS should fill the bill w/o requiring a new RR that
might require changes to stable DNS infrastructure.

> single ETR address mapping, so reducing the size of the mapping

I think the compromise position I am gravitating toward is to
have one level of caching server (i.e., the IR(DR)) between
the IR(LF)s and the IR(VP)s. I think that takes care of the
scaling and also gives an ISP control point from which it
can do ingress filter checking.

> point of control / failure etc. that some less centralised approach

OK, this was touched on several times above and nothing
different to add here.

> which ETRs can deliver packets to the destination network.

Can you accept the addition of just an IR(DR) as a single
level of caching between the IR(LF)s and IR(VP)s?

> real-time control of mapping.

As above, I am now considering adding an IR(DR) construct as
a means for having a mapping cache that improves scaling.

>       will frequently be short and few, but which could be long and

As above, I am going to consider the use of realtime mapping.

> in practice, it would probably work out much better than this.

I guess we will have to think about this, but maybe not
so much of an issue if we bring IR(DR) devices into the
picture? Then, its not everyone and their dog, but a
regional registry that acts on behalf of all dogs in
the region.

> I suspect that with I-R, you couldn't have an IR(LF) like function on

We have been over these points earlier, and won't try to
rehash here. New thinking is that IR(LF) devices only need
to be on public addresses as RLOCs (either IPv4 or IPv6)
and that there will be IR(DR)s between the IR(LF)s and
IR(VP)s.

> > messages from the previous IR(EID) router.

Well, let's think about this. If each IR(VP) remembers
each redirect it send to each IR(DR), then it can "replay"
the redirects if a mapping change occurs that need to be
pushed out. Mapping state will have a lifetime, so the
updates only need to get pushed out to IR(DR)s that are
expected to have valid lifetimes remaining. This, coupled
with the fact that there are not nearly as many IR(DR)s
as IR(LF)s, makes the whole thing potentially scalable.

> > EID is shifted from an old IR(EID) to a different one.

Considering bending toward your realtime mapping, as
discussed above.

> that might only be generated half a second or more after the packet

Again, the thought now is toward realtime mapping. But,
I want to do it without an MMC and by having the EUNs
test the reachability of their IR(EID)s themselves. Do
you see a problem with getting rid of the MMC role?

OK.

> transitory and fixed within seconds.

I sure hope you're right about this. It does seem also
to agree with what the wise person told me long ago,
but was cast into doubt by what some of the other
CES proposal team people seem to be saying. In fact,
I think I'll go have another look at Dave Meyer's
expired paper on this subject:

http://tools.ietf.org/id/draft-meyer-loc-id-implications-01.txt

> I could elaborate Ivip's

I need to take some time to understand this better.

> not include communications through any of these IR(EID) routers.

I think the EUNs still need to coordinate with both the
VP company (to get mappings into the IR(VP)s) and the ISP
(to get ingress filtering entries into the IR(DR)s). I'm
willing to walk away from the "bubble" proposal, but there
needs to be something else to take its place...

> automatically change the mapping at the appointed time.)

I'm still very much in favor of having the EUN test
its own ETR reachability and not have to have an MMC
company do it for them. That would make for one less
moving part, and one less business model that is
dependent on the other business models required for
all of this to work.

> How the ISP does this via its routing and addressing scheme doesn't

OK, I agree the IR(LF/EID) needs to use a PA address
given by the ISP as a springboard into PI. That would
avoid my concerns about NAT, etc.

OK.

> addresses with the VP company.

But, the ISP does need to know which PI prefixes the
EUN is authorized to source packets from, right?
Otherwise, it could be an unwitting accomplice to a
rogue EUN's source address spoofing attacks.

> > have not recently flowed.)

Turning away from this and toward RT mapping still.

> CES architecture such as Ivip or I-R.

I will examine the meyer draft closely and see where he
bases his assumptions regarding reachability of singleton
IP addresses. Maybe I read his paper wrong in the first
place? In any event, if I still have question after that
I will run the question past the wise person who long ago
told me pretty much the same thing you are telling me here.

> > OK, but it still needs to involve the ISP networks.

But, what about ingress filtering?

> elements, much less complex ITR-like functions and better scaling.

I'm obviously thinking about bending in your direction.

> be quite simple to cook up code for a QSC.

But, can we make do with just an IR(DR) and not need
three different varieties of query servers? I sure
hope so.

> > network is internally using private IP address space.

See above.

> one way, and you go another.

See above.

> via the various ETRs, and to change the mapping accordingly.

I want that the EUN does it itself without the need
for an MMC.

> > need to be involved in the EID mapping coordination.

PA as a springboard to PI - I think that alleviates any
concerns about private addressing, NATting, etc.

> one IR(EID) router is reachable and able to deliver packets to a

Bending toward RT mapping.

> DRTM algorithm for choosing a "nearby" QSA - and will attribute the

OK, and thank you.

Fred
[email-masked]