Subject: Re: [rrg] RANGER and SEAL critique

From: Robin Whittle <[email-masked]>
Date: Tue, 02 Feb 2010 23:32:56 +1100

Short version:   Fred provided some further information and I did
                 my best to understand it.

                 If I understood it approximately correctly, the
                 RANGER approach is a CES system without ordinary
                 ITRs or ETRs, and with some interesting design
                 features.  These include a general lack of
                 mapping, and lack of any problem with "initial
                 packet delays".  However, the proposal as I
                 understand it has serious problems with frequently
                 very long paths - unless Fred is really
                 proposing something like my "8 x VP router"
                 suggestion below, and I just didn't
                 understand it.

                 I think the RANGER IDs were of no help in
                 understanding what Fred has in mind.  This
                 text he wrote below seems unrelated to
                 the general description of RANGER - which is
                 applicable to all sorts of things apart from
                 being a CES solution to the routing scaling
                 problem.

                 Please see the separate thread: "SEAL critique,
                 PMTUD, RFC4821 = vapourware" regarding the
                 SEAL tunneling and PMTUD system, which would
                 be used in the many tunnels of the system
                 Fred describes.

                 I will write a critique for the RRG Report once
                 Fred responds to what follows.

Hi Fred,

I am replying to your response to my discussion of RANGER:

  http://www.ietf.org/mail-archive/web/rrg/current/msg05796.html

I am referring to:

  The RFC-to-be which is linked to from:
  http://tools.ietf.org/html/draft-templin-ranger-09

  http://tools.ietf.org/html/draft-russert-rangers-01
  http://tools.ietf.org/html/draft-templin-intarea-seal-08

> Here is what I think about RANGER and scalable routing.
> RANGER expects that the existing state of affairs in the
> current Internet BGP routing system will persist, but the
> goal of RANGER is to arrest the growth of the BGP RIB so
> that it will level off and not continue to expand along
> super-linear rates. In particular, RANGER expects that
> the current BGP will continue to maintain the RLOC-based
> RIB for the Internet, but that future growth due to
> mobility, multihoming and PI addressing will be handled
> out of EID space instead of RLOC space.

OK - this is the same as with the other Core-Edge Separation (CES)
architecture - including LISP, APT, Ivip, TRRP and ILNP.

> RANGER asks that a new BGP instance that carries EID
> prefixes be established within the DFZ, where each
> participating EID-based BGP router is an ITR/ETR that
> treats the DFZ as a virtual NBMA link through tunneling.

OK - I hadn't recognised from reading the RANGER IDS that there was
such a thing.

I will refer to each such router as an "ITR/ETR" and to the network
of these as the RANGER Overlay Network - RON.  This is a convenience
for discussion - I think these routers do not really play these roles.

My understanding of the above is that the RON is made of a subset of
DFZ routers perform both ITR and ETR functions.  (ITR and ETR are the
the terms used in LISP, Ivip etc. RANGER and SEAL uses "ITE" and
"ETE" or "iEBR" and "eEBR".)

I understand that while these ITR-ETR routers may also participate in
the DFZ via the conventional BGP instance, that they also have a
second BGP instance by which the RON is created.

I understand the RON exists to convey "mapping" information between
these ITR-ETR routers - and my guess is that it carries traffic
packets too.  "Mapping" is the term used in other CES architectures
for the information ITRs need to decide which ETR, or a set of
multiple ETRs, the packet could be tunneled to.  If all the ITRs and
ETRs participate in this RON, then I can roughly imagine this second
BGP system functioning normally to provide routes to ETRs, which is
comparable to "mapping" in other CES architectures.

However, my interpretation of what you write below makes me think
that the RON BGP messages don't attempt to carry a route for every
end-user prefix of "edge" EID space.  There could be millions of
these - say 10 million for portability, multihoming and TE for
non-mobile networks.  (Brian Carpenter and I came up with the same
figure independently.)  I think there is nothing in RANGER to state
your goals regarding how many of these prefixes you want the system
to support - to this 10 million figure is my assumption.

I can imagine two ways by which these ITR-ETR routers may be linked
for the purpose of transferring BGP messages over TCP links, between
the 2nd BGP instances of these routers:

  1 - The ITR-ETR routers use direct physical links between
      themselves for the RON sessions where such links exist - and
      tunnels to one or more other ITR-ETR routers if such a router
      does not have such direct links.

      See below - it is purely by tunnels, for BGP and I think
      for traffic packets.

  2 - All DFZ routers have the second instance, so the RON is a
      second BGP control plane for all DFZ routers.

      See below, this is not what you later describe.

I assumed this RON set of DFZ routers are the BRs of ISPs - but you
later say this need not be the case: these ITR-ETR routers may be BRs
of ISPs, but need not be.

To understand "NBMA" I am referring to:

  http://tools.ietf.org/html/draft-templin-ranger-09#section-3.3

    3.3. Virtual Enterprise Traversal (VET)

      Within the enterprise-within-enterprise framework outlined
      in Section 3.2, the RANGER architecture is based on overlay
      networks manifested through Virtual Enterprise Traversal
      (VET) [I-D.templin-intarea-vet] [RFC5214].  The VET approach
      uses automatic IP-in-IP tunneling in which ITEs encapsulate
      EID-based inner IP packets within RLOC-based outer IP
      headers for transmission across the commons to ETEs.

      For each enterprise they connect to, EBRs that use VET
      configure a Non-Broadcast, Multiple Access (NBMA) interface
      known as a "VET interface" that sees all other EBRs within
      the enterprise as potential single-hop neighbors from the
      perspective of the inner IP protocol.  This means that for
      many enterprise scenarios standard neighbor discovery
      mechanisms (e.g., router advertisements, redirects, etc.)
      can be used between EBR pairs.  This gives rise to a
      data-driven model in which neighbor relationships are
      formed based on traffic demand in the data plane, which in
      many cases can relax the requirement for dynamic routing
      exchanges across the overlay in the control plane.

IPv6 over NBMA (Non-Broadcast Multiple Access) networks is described
in RFC2491, which I had quick look at.  There is new text in this VET
ID of 26 January which I think is relevant to the RON you are describing:

  http://tools.ietf.org/html/draft-templin-intarea-vet-08#section-6.1

      Routing protocol participation on non-multicast VET
      interfaces uses the NBMA interface model, e.g., in the
      same manner as for OSPF over NBMA interfaces [RFC5340],
      while routing protocol participation on multicast-
      capable VET interfaces uses the standard multicast
      interface model.  EBRs on VET interfaces use the list
      of EBGs in the PRL (see: Section 5.2.2) as an initial list of
      neighbors for inter-enterprise routing protocol participation.

This Potential Router List (PRL) lists, for the current network (I
assume an ISP network) all the Enterprise Border Routers.  I guess
the ITR-ETR routers are all of this type - so I understand this is
how the BRs in an ISP find out about each other, at least for the
purposes of linking to each other to establish BGP links, with the
2nd BGP instance, to form this ISP's section of the RON.

However, you later state that these ITR-ETR routers need not be DFZ
routers.  I assume the EBGs are all DFZ routers.  If so, then how
would the PRL, which only contains the DFZ BRs, include those ITR-ETR
routers which are not BRs?

      EBRs that connect enterprises to the global Internet DFZ
      configure EID-based inter-enterprise routing using the BGP

This is the RON system - the routing system by which RANGER's ITR
functions in the ITR-ETR routers tunnel packets to ETR functions of
such routers, and by which the ITR-ETR routers share their routing
information.

"Enterprise" in this context means an ISP.  "Inter-enterprise" means
between all ISPs.

      [RFC4271] over a VET interface that spans the entire DFZ.

I don't have a clear understanding of the above.  In my understanding
of the term, an "interface" can't span the entire DFZ.

      Each such EBR peers with a set of neighboring routers on the
      VET interface, where the set is determined through peering
      arrangements the same as for the current global BGP.

This makes me think that all an ISPs' DFZ routers must be ITR-ETRs in
the RON - but not, perhaps the DFZ routers of transit providers,
since these are not ISP BRs.

But rather than use the physical links between the routers, as is
used for the DFZ traffic packets and BGP conversations, I understand
that these ITR-ETR routers somehow configure tunnels between each
other (where the packets go over the physical links anyway).

I conceive of these ITR-ETR routers as being physically implemented
in the same hardware, same route processor etc. as the Cisco, Juniper
or whatever DFZ routers - but that the "ITR-ETR router" behaves as a
separate entity.  Its connections to other ITR-ETR routers are all
via tunnels.  However, since they act as ITRs, they must be able to
advertise prefixes to the real routing system in the same physical
router.  Also, their ETR function must somehow connect to edge
networks.

                                                             Note
      however that this EID-based overlay BGP instance is seperate
      and distinct from the current RLOC-based BGP instance; \-- typo
      therefore, the set of peers used for the EID-based and
      RLOC-based instances need not be the same.

OK - so the previously quoted paragraph indicates they are the same
set of routers, "as determined through peering arrangements" and this
sentence indicates that the connections between them need not follow
the pattern of the peers in the DFZ system.

                                                            /-- typo
      Each EBR connected to the VET interface spanning the gobal
      Internet DFZ maintains a full routing information base (RIB)
      of EID-based prefixes.  In order to limit scaling, only

Limit "scaling difficulties"?

      highly-aggregated EID prefixes allocated according to the
      Virtual Prefix (VP) principles of Virtual Aggregation (VA)
      [I-D.ietf-grow-va] are included in the RIB.

I understand that each ITR-ETR's RIB and FIB has prefixes which cover
all the "edge" space.  However, there is not a separate prefix for
each of the individual end-user "edge" EID prefixes, of which there
are up to 10 million or so.

ietf-grow-va-01 is intended to be used within an AS and only affects
the FIB of the VA routers.  Although I haven't read the whole thing,
I don't see how you can apply this ID to guide people on doing
something totally different - reducing the contents of the RIB.

I had to read ahead and return, rewriting my interpretation to figure
out, as best I can, what you are describing here.  I found this part
particularly hard to follow:

      Specifically, only VP prefixes (e.g., PA prefixes delegated to
      the top-level of an ISP or enterprise network) are maintained
      in the RIB while more-specific prefixes (e.g., PI prefixes
      delegated to small sites) are not.  More-specific prefixes will
      instead be inserted into selective forwarding information bases
      (FIBs) on-demand of traffic flow such that only those routers
      that require the prefixes will insert them into their FIBs.

My best guess is that you mean that the ITR-ETR routers' RIBs contain
routes for ISP's prefixes and those of large PI using end-user
networks which are not using the scalable "edge" space of the RANGER
Core-Edge Separation architecture.  I think this is so that each
ITR-ETR's FIB will forward packets addressed to any of these
prefixes.  But it must do this via the DFZ - so perhaps I am wrong to
think of these ITR-ETR routers being separate from the underlying DFZ
router.

I think if you wrote it up in more detail, with an example, it would
be helpful.

I don't understand how it can be practical to have a router with a
partially populated FIB, awaiting packets, and when a packet arrives
with a destination address which does not math a prefix in the FIB,
the packet is held and by some magic the FIB causes the RIB to emit
the precise information needed to alter the FIB so it then has the
correct packet classification information for packets with this
destination address, and any other address which match whatever
prefix the RIB has which covers this address.  Then the FIB would be
able to forward the packet.

Having RIBs write stuff into FIBs may be costly.  These packets could
be arriving rapidly, so the FIB would need to buffer many of them
while the RIB is responding.  My biggest concern, apart from the
obvious problem of interupting the RIB according to traffic coming
into the FIB (and many routers have an FIB for each interface) is
that the RIB can't necessarily respond quickly or efficiently to a
request to find the most specific matching prefix for a given IP
address.  This is what the FIB is supposed to do.

> Each participating EID-BGP router will set up peering
> arrangements with a limited set of neighbors using
> tunnels according to the NBMA link model. 

These tunnels are presumably to function like physical links - to
carry both traffic packets being forwarded from one ITR-ETR router to
the next, but also to carry the TCP session for bidirectional BGP
communications.

So this RON system of ITR-ETR routers is linked entirely by tunnels -
with a BR of one ISP having tunnels to one or more BRs of other ISPs
- presumably, usually, not too far away.

> There is no
> requirement that these EID-BGP routers also participate
> in the current RLOC-based BGP routing instance, so the
> EID-BGP routers can be deployed incrementally and
> without disturbing the existing RLOC-BGP routing system.

So how can these ITR-ETR routers (which you refer to as EID-BGP
routers) are implemented on DFZ routers but as separate entities?

> This new EID-BGP instance will be used for carrying a
> relatively small number of highly-aggregated EID
> prefixes in keeping with the principles of Virtual
> Aggregation (VA). 

You cite:

  http://tools.ietf.org/html/draft-ietf-grow-va-01

which explicitly does no alter BGP or RIB operations, but only puts a
subset of routes from the RIB into the FIB, but what you are doing is
very different.  In the above paragraph you are reducing the number
of routes in the RIB and the BGP communications.   It is also a
contradiction with your previous statement that the FIB only has a
subset of the prefixes, but can get them quickly installed from the
RIB if a packet arrives which needs a prefix not currently in the FIB.

> So, this new EID-BGP instance (which
> again is completely separate from the existing RLOC-BGP
> instance) 

Here you wrote "is completely separate" but above you wrote "There is
no requirement that these EID-BGP routers also participate in the
current RLOC-based BGP routing instance."  It would be better to
state first that they are always separate

            will carry only highly-aggregated Virtual
> Prefixes (VPs) such as 4000::/8, 4100::/8, etc. So, at
> most there will be perhaps a few thousand of these VPs
> in the EID-BGP RIB (or perhaps even a few 10's or 100's
> of thousands) but the RIB size will be kept manageable
> through VA.

You describe a VA approach to the RIB, which I do not understand - at
least in terms of the reference you provide, which does involve
changes to the RIB.

> Now, RANGER has the EID-based VPs populated throughout
> the EID-BGP RIB, with all of the EID-BGP routers
> connecting service provider (SP) networks via the virtual
> NBMA link configured over the core. 

I don't clearly understand what the "virtual NBMA link configured
over the core" means.

RANGER is full of broad, high-level, statements about how things are
built, but each of these things typically has multiple options and I
find it very hard to construct a mental model of an physical
arrangement which would do what I think you are trying to do.

On one hand, you have millions of separate "EID prefixes" - the space
advertised by end-user networks using the "edge" subset of the
address space. These advertisements come from the ETR part of one or
more ITR-ETR routers.

These ITR-ETR routers don't use the DFZ routers directly, but each
ITR-ETR router has a tunnel (which typically passes over multiple DFZ
routers) to multiple other ITR-ETR routers in other ISPs.  Maybe
these ISPs are over the other side of the world, but I guess most of
them are not too far away.  There's certainly not a full mesh between
all these ITR-ETR routers.

> Customer Edge (CE)
> routers within the SP networks will want to use EID-based
> PI prefixes. Each such CE router "registers" its EID PI
> prefixes both within the SP network and with the EID-BGP
> routers 

I am calling these "EID-BGP" routers "ITR-ETR routers".

           that own the VP from which the PI prefix is
> aggregated. 

OK.  So there is an ITR-ETR router in Seattle which aggregates
42.0.0.0/16.  (I am using IPv4 addresses for brevity, though I
understand RANGER is mainly intended for IPv6.)  This is 65,536 IPv4
addresses of "edge" space.  Lets say this covers ~10,000 EID
prefixes, one of which you refer to as "the CE's PI prefix".

This means that ~ 10,000 end-user networks which have space
within this prefix need to register the "core" (RLOC) address of
their CE routers (and the EID prefix they are responsible for) with
this Seattle router.  (I will pass over questions of redundancy in
the case of the Seattle router failing.)

> Once "registered", the CE's PI prefix will
> be kept only in selected router FIBs, and will not be
> injected into the EID-BGP RIB. 

Which are these "selected" routers?

I understand that the Seattle router now knows the "core" (RLOC)
addresses of the CE routers of 10,000 or so end-user networks all
over the world.  These are to be the tunnel end-points for the
delivery of traffic packets - the equivalent of ETR addresses.

I understand that the Seattle router does not contain entries in its
RIB for any of these 10,000 or so EID prefixes.  There is presumably
only its own 42.0.0.0 /16 prefix, which it advertises through the RON
to all other ITR-ETR routers.

>                                Moreover, only the FIBs
> of those routers on the paths over which the CE's EID
> addressed packets will travel need to contain the PI
> prefix - no other routers need discover the prefix. 

I don't understand.  Which paths are you discussing.  There really
needs to be more complete descriptions, and probably some examples.

It is taking me many hours to try to understand what you wrote.

I assume the "PI prefix" you are referring to is one of the 10,000 or
so such "EID prefixes" which the Seattle router is responsible for.

> The
> location of the CE router's EID prefix is tracked through
> the FIB entries in the EID-BGP router that holds the VP
> from which the EID prefix is derived.

My attempt at translation:

   "Location" means the "core" (RLOC) address of the CE router which
   handles a given end-user network with a given "edge" (EID) prefix.

   These CE router "core" addresses for all the ~10,000 end-user
   prefixes contained within the 42.0.0.0 /16 prefix which the
   Seattle router advertises in the RON are recorded in the FIB
   of the Seattle router.

So if there are 10,000 separate EIDs of end-user networks within this
/16 prefix, then:

    1 - The Seattle router has all these ~10,000 "edge" (EID)
        prefixes in its  FIB, and for each one there is is an address
        of the CE router for this prefix.

    2 - The CE routers of these end-user networks could be in Taiwan,
        South Africa, Malaysia, Australia, Siberia and the South
        Island of New Zealand.  It would be convenient for the
        end-user network if it was located somewhere near the
        Seattle router, but in general, these CE routers are nowhere
        near the Seattle router.

        This is because EID space is portable all over the world -
        and also, because at any point in time, the role of being
        the VA router for this 42.0.0.0 prefix could be given to
        a router somewhere far from Seattle.

    3 - The hosts which are sending packets to these end-user
        networks could be all over the world, quite often in the
        same area as the end-user networks whose hosts the
        packets are addressed to.

    4 - The Seattle router advertises this /16 prefix to the RON
        system of ITR-ETR routers.  I guess the ITR function in
        each of these ITR-ETR routers will now be able to tunnel
        packets addressed to any one of these ~10,000 end-user
        network "edge" EID prefixes to the Seattle router.

        This tunneling, as far as I know, is through the tunnels
        of the RON system, since this is how the ITR-ETR routers
        use BGP to manage their best paths for each such prefix.

        These ITR-ETR routers tunnel all packets addressed to any
        one of the ~10,000 "edge" (EID) prefixes because of the
        42.0.0.0 /16 in their RIB and FIB.  They have nothing in
        their RIB or FIB about the 10,000 individual prefixes.
        Only the Seattle router's FIB has this.

        This means that none of these ITR-ETR routers need the
        full set of millions, or tens of millions, of these "edge"
        EID prefixes either in their RIB or FIB.

        It also means there is no caching, no mapping lookup and
        no delays waiting for mapping.  Interesting . . .

    5 - So a packet sent from an ISP in any location - including
        those listed above, and from others such as London,
        the North Island of New Zealand, Chile etc. will be handled
        like this:

         a - A host in the North Island of New Zealand sends a packet
             addressed to 42.0.56.78 which is in an "edge" EID of
             an end-user network of a company located in the
             town of Fox Glacier, not far from the said Glacier,
             in the South Island.  (A magnificent part of the world!)

         b - This packet is from a customer of an ISP in Auckland
             (North Island).  In that ISP, the packet is forwarded
             to an ITR-ETR router.  (I assume all these ITR-ETR
             routers advertise the "edge" (EID) prefixes such as
             42.0.0.0 /16 to their local routing system, though
             I don't see where you specified this.

         c - This ITR-ETR router has in its FIB a route for
             42.0.0.0 /16 which forwards the packet towards the
             Seattle router.

             As best I can tell, this is forwarding from the ITR-ETR
             router means tunneling it to a neighbouring ITR-ETR
             router.  Each such router has a single (multiple?)
             "core" (RLOC) address - and the tunnel packets are
             sent and received from these.  After encapsulation,
             the packet goes to the FIB of the DFZ router which
             is in the same device, or to a nearby DFZ router,
             which forwards it towards the tunnel destination
             address, via 0 or more DFZ routers.

             When it gets to that second ITR-ETR router, the
             packet is taken out of the tunnel and its destination
             address 42.0.56.78 examined by the second router's
             FIB.  This causes the packet to be tunneled again
             to another ITR-ETR.  Eventually, it makes its way
             across the RON to the Seattle ITR-ETR.

         d - The Seattle router detunnels the packet and presents
             it to its FIB.  This FIB is the only one in the
             world with an entry for the particular "edge" (EID)
             prefix of the network which contains the destination
             host: 43.0.56.76 /30  This is associated with a
             "core" (RLOC) address of the CE router which serves
             this end-user network.

             The CE router is in the office of a tour company in
             the Fox Glacier township, on a single IPv4 PA address
             33.22.22.33 which is a stable IP address of a DSL
             service.

         e - The ETR function of the Seattle router encapsulates
             the packet with an outer header destination address
             of 33.22.22.33 and presents the resulting packet to
             its FIB.

         f - The FIB has a prefix matching 33.22.22.33 - since
             this is part of a big (short) prefix 33.22.180.0 /17
             of an ISP on the South Island.

         g - Does the Seattle router's FIB forward this packet
             to a neighbour on the RON, and therefore via another
             tunnel?  Or does it forward the packet according
             to the FIB of the DFZ router - and so out into the
             DFZ, without any further tunneling?

             Either way, the Seattle router has a way of getting
             the traffic packet to the CE router in the office
             of the tour company, and the CE router decapsulates
             it and puts it on the LAN, which takes it to the
             destination host.

    6 - There are arguments for increasing the number of these VP
        ITR-ETR routers (such as the one in Seattle) - in order to
        reduce the number of packets each one has to handle.

    7 - There are arguments for decreasing the number of these VP
        routers, to reduce the load on the RON control plane -
        since each one advertises a prefix in the RON BGP
        system.

        You could however achieve the same goal by putting three
        other ITR-ETR VP routers in the same data centre, for
        adjacent prefixes,  and aggregating them in some way into
        a single shorter prefix there.  Then, from Seattle, you could
        advertise a single single 42.0.0.0 /14.

It is possible that I have partially or completely misunderstood you.
 I believe you need to document these things much more extensively,
ideally with diagrams and definitely with examples.

Based on the above understanding, here are some observations.

Let's say there are 10 million end-user networks and most of
them have a single edge (EID) prefix.  So let's say there are 12
million of these prefixes.

Let's say there are a billion IPv4 addresses in the "edge" subset
of the global unicast address space.  So the average size of these
prefixes is around 80 bytes.  However, most of them are 1, 2, 4
or 8 IPv4 addresses.

Let's say these are contained in 2^14 separately advertised prefixes
in BGP.  These may be of various sizes.  So this is a 16k prefix
burden on the DFZ - not much of a problem.  (In Ivip, each such
prefix would be a MAB - Mapped Address Block.)

On average, these prefixes are /16 - and so contain 2^16 IP
addresses.  With an average of 80 IPv4 addresses per end-user network
 "edge" (EID) prefix, on average, each such MAB prefix provides 820
"edge" (EID) end-user prefixes.  This is pretty good routing
scalability - 820 for 1 DFZ advertised prefix.

On average, each VP router such as the Seattle one mentioned above
handles a /18.  A /18 has 16,384 IPv4 addresses, and so on average
each one is used by about 205 end-user network prefixes.  If we
assume that each end-user prefix is to be tunneled to a different CE
router, then the average router such as the one in Seattle has 205 of
these special CE tunneling entries in its FIB.

There are several problems with the arrangement as I described above.

Firstly, there's a lot of tunneling as the packet makes its way
across the RON from what I called the ITR function of the ITR-ETR
routers to the ETR function of the Seattle ITR-ETR router.

In fact, ITR and ETR are not accurate terms.  I used them to give
something familiar which relates to other CES architectures.

These things are just routers.  It so happens that the multiple hops
between the RON router in Auckland and the one in Seattle each
involved a tunnel.  But there was no tunnel between the Auckland
router and the Seattle router.

Arguably the Seattle router is really playing the ITR role - and it
tunnels to the CE router which arguably plays the ETR role.  In this
model, this is very roughly like Ivip where each MAB has only a
single DITR in the world, and no ISPs or any other networks have ITRs.

The most obvious problems are:

  1 - The dependence of 205 or so end-user networks on a single
      router (such as in Seattle) creates a potential bottleneck.

  2 - Also, it creates a single point of failure.

  3 - Considering the random distribution of sending hosts and
      destination hosts, this arrangement frequently leads to
      excessively long paths, back and forth across the world.

Still, it is an interesting arrangement.  The router which forwards
the packets to the Seattle router does so without any delay, mapping,
caching or the like.

The Seattle router has only 205 or so entries in its FIB, so it has
no scaling problems in terms of FIB size.

You haven't specified how CE routers can securely register their
particular "edge" prefix with routers such as the one in Seattle.

However, assuming they can do this, then multihoming could be done by
the end-user network having two ISPs, and therefore having CE routers
(or really the one CE router) appearing on two separate "core" (RLOC)
addresses.  Then, to do multihoming failure detection and service
restoration you could take one of several approaches:

  1 - The end-user network itself senses the failure of its use of
      the "core" address at ISP1, and somehow uses its other ISP2
      link to securely re-register with the Seattle router the
      ISP2 "core" address instead.

  2 - The Seattle router is told by the end-user network about both
      its addresses, the one from ISP1 and the one from ISP2.  The
      Seattle router is then instructed to do reachability testing,
      of these addresses, or perhaps through these addresses to
      something in the end-user network itself.  Then the Seattle
      router would choose which link to use - it would do the
      multihoming service restoration.

  3 - An Ivip-like approach where the end-user network could tell
      the Seattle router whether to tunnel packets to the ISP1 or
      ISP2 address - but instead hires a Multihoming Monitoring
      company to do reachability testing and to control the
      Seattle router's tunneling accordingly.

All three approaches could be very fast.

To me, the most obvious enhancement of this arrangement would be to
create multiple routers such as the Seattle one.  Currently there is
only one of these VP routers for these 205 or whatever end-user networks.

If we had 8 VP routers, each just like the Seattle one, then this
would spread the load.  If you scattered these around the Net, the
RON's natural BGP behaviour would be to spread the load and to
forward packets to the nearest one.  This would generally lead to a
big reduction in total path length.

But then, each VP router would be sending a tunnel to the CE router -
so it would need to handle 8 tunnels.  Also, when the CE router moves
to a different "core" address, there are 8 VP routers to securely
register the new address with.

This starts to introduce the concept of "mapping" information into
the system, whereas before, there was no such thing.

If I have understood this correctly, then what you are suggesting is
a novel CES architecture without a mapping system, and without
delays.  If I misunderstood you, then I just partially invented such
a thing myself!

However, I think it still has problems with excessively long paths,
with concentration of the workload on too few routers - and I think
forwarding traffic packets across the RON network, with each hop
involving an encapsulation and decapsulation, with PMTUD management
for each tunnel . . . I think it is not a good way to solve the
routing scaling problem.

By doing most or all of the work with DFZ routers, there is a
potential critique that you are not really taking much load from
them.  However, while you have a second BGP instance, the total
number of routes the original and new RIB handles in each DFZ router
is far smaller than the 10 million or similar end-user networks you
are serving.

As end-user networks change where the "core" address of their CE
router, this does not at all affect the DFZ BGP control plane or the
new RON BGP control plane.

Mobility could be done by the MN re-registering each new CoA with the
VP router.  If the VP router makes the tunnel, then this won't work
with the MN behind NAT.  If you borrow a little from the TTR Mobility
architecture, you would have the MN tunnel to the VP and authenticate
itself.  Then the VP can take the MN's egress packets.  Most
importantly, this enables the MN to operate behind NAT.  In this
case, the VP is rather like a Home Agent.

If you took up my suggestion of 8 VPs, instead of one, you could have
a rather interesting mobility system, with typically much shorter
paths due to the 8 VPs.  However, then the MN would need to establish
8 tunnels to the 8 VPs.

While what you described is ostensibly RANGER - I think there is not
much in the RANGER ID which is relevant to the CES architecture you
described in your email.  RANGER can be used for many more things
than a CES architecture.

I think that to progress your proposal, you should write a completely
fresh documentation of it, specifically for a CES architecture for
scalable routing.  I suggest you list goals and non-goals, including
for how many non-mobile end-user networks you expect the system to
scale to.   There needs to be diagrams and examples.

It is not obvious how you do load-sharing inbound TE with this
system, except with the Ivip approach of splitting the traffic over
two micronets and mapping one micronet to one ETR and the other
micronet to the other ETR.

> In summary, the RANGER approach to scalable routing is
> to create a new BGP instance between tunnel routers for
> the purpose of keeping a limited set of highly-aggregated
> EID VPs in the RIB. 

OK.

> PI EID prefixes owned by customer
> routers are added to selected SP router FIB tables on
> demand, and are never injected into the RIB. 

Hmm - this makes no sense to me based on the (mis)understanding
I just developed.  What do you mean by "selected SP routers" and what
sort of "demand" is this?

> The way
> this works is that CE routers that are holders of PI EID
> prefixes 

OK . . .

          "blow bubbles" that percolate up through a reverse
> tree ascending through their SP networks until the bubbles
> reach an EID-BGP router that owns a VP from which the PI
> prefix is derived. 

I have absolutely no idea what this means or how it relates to the
rest of your email or to anything I read in the RANGER IDs.  If this
is the case, you really need to explain things much better - because
I tried very hard to understand what you are proposing and it looks
like I missed out on at last part of it.

> In that way, the locations of all PI
> EID prefix holders are available in EID-BGP router FIBs
> while only VPs appear in the EID-BGP RIB. This system
> of knowing where all PI prefix holders are at all times
> also has clear beneficial properties for supporting
> mobility and multihoming.
> 
> Finally, in terms of routing scaling, the end state
> benefit is that both the EID-BGP and RLOC-BGP RIBs
> remain manageable in size and only those routers that
> need to know about certain PI EID prefixes have to
> carry those prefixes in their FIBs.
> 
> Any thoughts or comments on this?

Please write up a complete, standalone, documentation of this to save
people from having to read the RANGER or SEAL IDs - and have
diagrams, examples and much more detailed descriptions of all the
network elements.

Please let me know how close I got to understanding what you have in
mind.

  - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 2 Feb 2010 14:09:17 -0800

Hi Robin,

> -----Original Message-----
> From: Robin Whittle [mailto:[email-masked]]
> Sent: Saturday, January 30, 2010 10:01 AM
> To: RRG
> Cc: Templin, Fred L
> Subject: Re: [rrg] SEAL critique, PMTUD, RFC4821 = vapourware
>
> Short version:     Mainly explaining my understanding of how
>                    SEAL does PMTUD when used as part of RANGER
>                    - when RANGER is acting as a Core-Edge
>                    Separation solution to the routing scaling
>                    problem.
>
>                    Both RANGER and SEAL can to other things
>                    besides this, and I find it hard to envisage
>                    the subset of their operations which would
>                    be used in a Core-Edge Separation setting.
>
>                    I still thing it is wrong to develop a complex
>                    and difficult protocol such as RFC4821,
>                    because of some badly designed tunnels which
>                    don't generate PTBs or due to a few networks
>                    which filter out PTBs.
>
>                    There's very little sign of anyone wanting to
>                    develop or use RFC4821 - so I guess the
>                    need or desire for it isn't be very strong.
>
>
>
> Thanks for your reply:
> >> So I disagree with your statement except to the extent of
> >> applications which do no packetization and just rely on the stack's
> >> TCP (or SCTP or whatever) packetization layers which may or may not
> >> implement RFC4821.
> >
> > Yes; thanks for the correction. RFC4821 concerns packetization
> > layer path MTU discovery, where the application itself is a
> > packetization layer when, e.g., UDP is used as the transport.
>
> OK.  My understanding is that in the stack there is, for each
> destination address, an MTU number which can be read and written by
> any packetization layer.  The TCP packetization layer in the stack is
> the most obvious one, but I guess if there was an SCTP layer there
> too, this would also read and write this variable (or is it multiple
> variables for each destination address?).  Then, for as many
> applications have packetization layers, they would be able to read
> and write these variables too.
>
>
> >> When IPTM sends a big packet (containing most of a traffic packet) as
> >> part of its PMTU probing, if this hits an MTU limit, then it is
> >> dropped, with a short PTB going back to the ITR.  With SEAL, in IPv4
> >> mode, the limiting router has to split the packet up and forward it,
> >> so other routers and the ETR have to carry all the fragments.  So my
> >> IPTM approach is arguably lighter weight than your SEAL approach.
> >
> > With SEAL-FS, the ETE does not carry all the fragments.
> > Instead, it uses first-fragments for reporting purposes
> > only and otherwise discards all fragments.
>
> OK - thanks.
>
>
> >> IPTM doesn't rely on the PTB. (See below for how it will be able to
> >> work with minimal length IPv4 PTBs.)  As long as a PTB does get to
> >> the ITR - which it would in most cases - then the ITR knows about the
> >> MTU problem without having to wait for the ETR to time out and send a
> >> message to the ITR saying the big packet did not arrive.  Also, the
> >> ITR gets an exact MTU value from this PTB, rather than having to do
> >> what SEAL does - hunt back and forth to find a packet size which is
> >> reliably delivered without MTU problems.
> >
> > SEAL doesn't hunt back and forth.
>
> 4.3.9.1.2 mentions an "iterative searching strategy" - which sounds
> like a fancy term for "hunt"!  This occurs only in IPv4 when the ETE
> gets a first fragment shorter than 576 bytes, then this is
> interpreted as a "runt fragment" and so is not regarded as a true
> measurement of the limiting MTU.

Yes, this follows directly from the RFC1191 "plateau
table" approach to guessing MTUs when old routers fail
to fill in the MTU field in their PTB messages. What I
meant was that yes, SEAL will hunt "back" when there is
a router in the path that does weird fragmentation. But,
it does not hunt "forth".

> > In SEAL, every data
> > packet is an implicit probe, and the ETE uses IPv4
> > fragmentation as an indication that it needs to tell
> > the ITE to reduce the size of the packets it is sending.
>
> To function correctly in IPv4, SEAL also relies on the first fragment
> arriving at the ETE, and that fragment being the length of the
> shorted MTU between the ITE and the ETE.  There's not much chance of
> the first fragment being lost, so this is fine.
>
> As far as I know, IPv4 routers are not required to make their
> fragments (all but the last usually) the length of their MTU limit,
> but I guess most of them do.
>
> Assuming they all do, your IPv4 fragmentation approach has an
> advantage where there are two or more MTU limits in succession.
>
> If the first limit is 1400 and the second 1300, then by relying on
> PTBs, as you do with IPv6, then first of all, the ITE discovers the
> 1400 limit, and sends a PTB to the sending host (SH) with the correct
> value for that: 1400 - 24 (IPv4 header + SEAL header) = 1376.  The SH
> tries again, and the encapsulated packet generates a PTB at the
> router with the 1300 limit.  This is propagated back to the SH in the
> same way, with a PTB with a 1276 byte MTU - and then all is well.
>
> With your fragmentation approach, the ETE should get a first fragment
> of length 1300 bytes.  It reports this back to the ITE, and it sends
> a single PTB to the SH, with a value of 1276 bytes.
>
> Which would be faster depends on the circumstances - since the PTB
> back to the ITE will often be faster than the longer path all the way
> to the ETE.
>
> If the limiting router sends a fragment of length less than the
> limiting MTU, then the SEAL ITE would adopt an unrealistically low
> MTU value.  I guess this is unlikely.
>
>
> IPTM doesn't hunt back and forth.  When a long enough packet arrives
> at the ITR for the encapsulated length to fall between its two
> current markers for the Zone of Uncertainty (low and high water
> markers, I think, are the terms used in other protocols) the dual
> packet probe technique will usually result in reliable new
> information in most instances, due to one of:
>
>   1 - The traffic packet is delivered to the ETR and the ITR is
>       informed about this - so the ITR raises the low marker.
>
>   2 - The traffic packet was not delivered and the ITR gets a PTB.
>       This enables the ITR to lower its high marker and send a
>       PTB to the SH with the correct MTU value to cause the SH so
>       subsequent packets it sends, once encapsulated, will fit the
>       MTU limit exactly.
>
>   3 - If the long packet does not arrive at the ETR, but the short
>       one does, and the ITR receives no PTB, then perhaps the long
>       packet was merely lost.  However, if this happens repeatedly
>       then the ITR should discern that there is an MTU limit below
>       this length, and so adjust its upper marker downwards.  This
>       will result in a PTB going to the SH, which will send
>       shorter packets.  Over multiple iterations, the ITR will
>       discover the MTU.  If the steps downwards taken by the ITR
>       in reducing its upper marker are small, this will take a long
>       time.  If they are larger, the true MTU will be found faster,
>       but there may be more overshoot, resulting in somewhat
>       smaller MTU for all hosts whose packets are tunneled to this
>       ETR than is really needed.
>
>       Once a pair of probe packets does arrive at the ETR, the ITR
>       is informed of this by the ETR and it raises its upper marker
>       accordingly
>
>  Pretty quickly, the two markers would reach the same value and the
> ITR would have a reliable measure of the MTU to this ETR.
>
> The most likely pattern would be this:  Initially (before the ITR has
> ever sent packets to this ETR address) the markers are wide apart.
> Then the SH sends a packet which longer, once encapsulated, than the
> low marker and also longer than the actual tunnel MTU, minus
> encapsulation overheard.  So the the ITR does the two packet probing
> protocol and gets a PTB.  Assuming this is from the router which sets
> the lowest PMTU limit in the tunnel, then this PTB enables the ITR to
> send a PTB straight back to the SH.  This also enables the ITR to
> lower the upper marker to match the MTU reported in the PTB it just
> received.
>
> The SH will create another packet of the correct length to suit the
> tunnel (after encapsulation it will be the length specified in the
> PTB from the limiting router).  The ITR gets this and because it
> would be longer, after encapsulation, than the low marker, the ITR
> again sends it to the ETR with the two-packet tunneling protocol.
> This time, it gets through, and the ETR reports this.  As soon as
> this report gets to the ITR, the ITR adjusts its lower marker to the
> same value as the upper marker - so there is no more Zone of
> Uncertainty.
>
> The SH will generally continue to send packets of this length or
> shorter.  If some other SH (or another application in the same SH)
> sends a packet which needs to be tunneled to the same ETR and which
> is longer than this now reliably known PMTU value, then the ITR will
> drop it and send back a PTB, without trying to send it into the
> tunnel.  That application or SH will then send packets of the right
> length.
>
> I haven't figured out every detail of this - at some stage I intend
> to work on it more and write it up as an ID.  For now, it is at:
>
>    http://www.firstpr.com.au/ip/ivip/pmtud-frag/
>
>
>
> >> In Ivip, most traffic packets are encapsulated by the ITR with the
> >> sending host's address as the outer header's source address.  Any PTB
> >> which results from those goes to the sending host, which will not
> >> recognise it.
> >
> > If the source address of the original packet also
> > goes as the source address of the outer packet, then
> > wouldn't that constitute mixing both EID and RLOC
> > addressing within the same routing region? I thought
> > the whole purpose of the CES approach was to keep the
> > EID and RLOC routing and addressing spaces separate.
>
> The "Separation" means that a subset of the global unicast address
> space is used as "SPI" space (Ivip) or "EID" space (LISP).  This is
> not a separate namespace, just a subset of the global unicast address
> space, in the form of multiple DFZ-advertised prefixes which have no
> name in LISP, but which are called Mapped Address Blocks (MABs) in Ivip.
>
> The ITR tunneling uses the sending host's source address as the outer
> source address for all ordinarily tunneled packets.  This enables any
> ISP BR filtering - which drops incoming packets due to them having a
> source address from any one of the ISP's prefixes - to be enforced on
> the inner packet by all ETRs, by the simple method of dropping any
> inner packet whose source address is different from the outer
> header's source address.
>
> This functionality is also enforced with the 2 packet probing
> arrangement - the short A packet is also sent with its outer source
> address being that of the sending host.
>
> When a packet would be, after encapsulation, a length within the
> "Zone of Uncertainty" then the ITR uses the special long (B) and
> short (A) protocol.  The B packet's outer source address is that of
> the ITR - so the ITR would normally get any PTB which arises from the
> B packet.
>
> The "sending host" address could be a conventional (non-SPI) address
>   such as from a host on PA or PI space - or it could be on an SPI
> address.  This is the source address of the packet the ITR is
> processing.  Whether that is the actual address of the sending host,
> or just that of a NAT box which the sending host is behind, is not
> known to the ITR.
>
> All hosts and most routers make no distinction between addresses in
> the SPI subset and the rest of the addresses in the global unicast
> address range.  Only ITRs treat them differently - and then only when
> they appear in the destination address of a packet which is forwarded
> to them.  Instead of forwarding the packet according to its
> destination address, the ITR's FIB processes the packet differently.
>
> If the ITR's FIB (which may be all in software, since the ITR may be
> in the sending host or be implemented on a COTS server) already has
> mapping for a micronet which matches this SPI address, it uses that
> mapping (a single ETR address) to tunnel the packet.  If not, it
> buffers the packet, requests mapping from a nearby QSD (full database
> query server), installs the mapping (a micronet start and end
> address, with a single ETR address) in its FIB and then tunnels the
> packet accordingly.
>
> There's no concept of "routing region" in Ivip.  ITRs in any place
> would do exactly the same thing.  All other devices - hosts and
> ordinary routers - make no distinction between SPI addresses and
> remainder of the addresses in the global unicast address range.
>
>
>
> >> In this scenario, the ITR gets back just the IPv4 header and the UDP
> >> header.  The attacker has to guess the 16 bit ID field in the IPv4
> >> header, which is tricky - but it could eventually succeed in doing
> >> so.  Here are the components of the UDP header:
> >>
> >>   Source port     The ITR could use a randomized source port.  This,
> >>                   combined with the 16 bit ID field, could extend
> >>                   the number of bits to be guessed to 32 - which
> >>                   I think is sufficiently secure, considering a
> >>                   successful attack only degrades efficiency, rather
> >>                   than causes actual loss of connectivity.
> >>
> >>   Destination port   Currently, I assume there is a single UDP port
> >>                   on all ETRs to send the long (B) packet to.  If
> >>                   I could easily randomize this too - such as making
> >>                   the most significant 8 bits fixed, and the others
> >>                   up to the ITR to choose.
> >>
> >>                   This would be 40 random bits - perfectly secure
> >>                   considering the moderate level of DoS the attack
> >>                   could result in.
> >>
> >>   Length          If the attacker created the traffic packet, they
> >>                   would know the length of what follows the UDP
> >>                   header.
> >>
> >>   Checksum        Ahhh - this is not a header checksum.  This covers
> >>                   the data behind the UDP header.  This data is
> >>                   mainly from the traffic packet, but it contains
> >>                   a nonce.  So the 16 bit checksum is affected by
> >>                   the nonce.
> >>
> >> I hadn't realised this before - the UDP checksum contains another
> >> 16 bits the attacker has to guess.  Combined with the IPv4 header's
> >> 16 bit ID field, I think this makes it highly secure.  If this is
> >> not enough, the 16 bit random ITR source UDP port should be sufficient.
> >>
> >> So the ITR doesn't need any more bits than are necessarily supplied
> >> by a minimally compliant RFC1191 implementation in the router which
> >> sends the PTB.
> >>
> >> How would this work for SEAL?
> >
> > Using the UDP/TCP checksum as a nonce requires that the
> > ITE cache copies of its recently-sent packets.
>
> The above procedure is only for the long (B) packet when the ITR is
> still uncertain of the PMTU to a given ETR and the packet, if
> normally encapsulated, would be of a length within this Zone of
> Uncertainty.  The B packet is the same length, but is sent with the
> ITR's address in the outer header.  Like the short A packet, it is a
> UDP packet with special headers.  All the normally encapsulated
> packets are IP-in-IP, with no other headers, and with the sending
> host's address in the outer source address.  So normally encapsulated
> packets can't generate PTBs to the ITR - only to the sending host,
> which would not recognise them, since they arise from an encapsulated
> packet.
>
> So only for these traffic packets for which the ITR is using to
> generate the B and A probe pair does it need to cache enough of the
> initial packet to be able to generate a valid PTB to the sending
> host.  The ITR would also cache the nonce, which it uses to secure
> the ETR's response to these packets.
>
> It would also cache the UDP header of the B packet, which includes
> the checksum which is almost impossible for an attacker to guess due
> to its dependence on the nonce.  I think the combination of an
> unpredictable 16 bit ID in the outer header of the B packet, and the
> influence of the nonce on the 16 bit checksum, would be sufficient to
> prevent attacks succeeding at a significant rate.  If that wasn't
> enough, the ITR could use 16 bit randomization of its UDP source
> port, and randomize 8 or more bits of the destination UDP port too.
>
>
> > But then,
> > it would need to do this for every tunnel it belongs to
> > and it has no way of knowing for how long it will have
> > to retain the cached copies.
>
> As I noted above, the ITR doesn't cache its ordinarily encapsulated
> packets.  The PTB would not go to the ITR.  It only needs to cache
> the start of those packets it is sending with the two-packet probing
> protocol.
>
> If the ITR estimates the PMTU to a given ETR, and gets it right - and
> then later the MTU falls, then there is a difficulty.  The normally
> encapsulated packets which are too long will be dropped, the ITR will
> not get any PTBs and the sending host will not recognise the PTBs
> which are sent to it.
>
> I can think of two approaches to minimising the impact of this.
>
> One is to have the ITR periodically, such as every 30 seconds, send a
> packet which (once encapsulated) is at, or close to, the MTU limit,
> as a B and A probe pair.  This is assuming the ITR is continually
> sending long packets to this ETR.
>
> This will normally deliver the packet fine, and the ITR will be able
> to confirm that the PMTU has not become any less than what it
> assumes.  If the packet doesn't arrive, or if the ITR gets a PTB,
> then perhaps the MTU has dropped and the ITR can find out what it has
> dropped to.
>
> It will usually find out the new PMTU from a PTB, but if there is no
> PTB, then it will need to send more probes of various lengths until
> one size does get through to the ETR.
>
> It would do this as described above, without having to generate
> special probe packets, by lowering its upper marker for the MTU
> estimate by some value, such as 8 or 16 bytes, sending a PTB to the
> sending host (or multiple sending hosts as they send packets which
> would exceed this length, once encapsulated) and allowing the sending
> hosts to create traffic packets which the ITR will send using the 2
> packet probing technique.  This will rapidly reduce the value of the
> upper marker to being equal to, or somewhat less than, the real PMTU
> limit.  This will drag down the lower marker too.
>
>
> The following second approach would only work if the returned part of
> the packet was long enough to show the SH that it really did result
> from a packet the SH sent.  In practice, this should always be the
> case with IPv6, and I have been gained the impression that it is
> common for IPv4 routers to send back more than the bare RFC1191
> minimum anyway.  This would require the sending host to have a
> modified stack which was ready to analyse PTBs which resulted from
> packets in the ITR to ETR tunnel.  Assuming this modified stack code
> could verify that the PTB was genuine, it would compute a new MTU for
> this destination address, by subtracting the encapsulation overhead
> (20 for IPv4, 40 for IPv6) from the MTU value in the PTB.
>
> This one PTB would not help the SH learn about a reduced PMTU to
> other SPI destination addresses it was sending to which also were
> being tunneled to the same ETR.  But the same code would receive PTBs
> from those packets too.  The ITR would be none-the-wiser, so the
> first technique and the one below would still be important - but this
> optional host upgrade would enable the SH to respond immediately and
> correctly to a reduction in PMTU to an ETR.  Other applications in
> other SHs would need to repeat this exercise, since the ITR doesn't
> know these PTBs are occurring.
>
> After 10 minutes, a sending host is allowed (RFC1191 / 1981) to try
> sending a longer packet than was allowed by a previous PTB.  Then,
> the ITR needs to recognise the time which has elapsed and use this
> with the B and A probe technique.
>
> This may be a little complex, considering multiple applications in
> one sending host, multiple hosts and multiple destination SPI
> addresses may all result in packets being sent to the one ETR address.
>
> I can see ways of coping with this stuff - but some of it requires
> carefully designed algorithms and considerable logic and state in the
> ITR.
>
> If we can upgrade the DFZ and other routers with firmware, then all
> this encapsulation and PMTUD stuff can be ignored - by using Modified
> Header Forwarding instead.  That is the way to do it in the long-term
> future, even if we start with encapsulation.
>
>
> > With SEAL, the ITE never
> > has to cache packets in order to match them up with
> > any PTB feedback.
>
> Yes - here's my understanding of how your SEAL ID specifies how the
> ITE and therefore the SH perform PMTUD in the tunnel path, between
> the ITE and the ETE.
>
> I assume in all cases that the ITE initially sets S_MRU for this ETE
> to "infinity" as described in 4.3.3, and then uses one of the
> following methods to reduce it.  This is me trying to imagine how the
> SEAL would be used for ITE to ETE tunneling when RANGER is used as a
> Core-Edge Elimination architecture.
>
> RANGER can be used for many more purposes than this, and so can SEAL,
> so it is quite a challenge to decide which parts of the IDs to
> ignore.  I understand that in this application, the ITE will be
> reducing the S_MRU value for each ETE it tunnels to.  I think that in
> other SEAL applications, this may not occur, and so you have
> arrangements for using SEAL segmentation to send long traffic packets
> as multiple SEAL-segmented packets to the other end of the tunnel.
> But this would never be invoked in a Core-Edge Separation
> application, since the ITE always sends PTBs to the SHs to have them
> reduce their packet length.
>
> In this application of SEAL, I understand there is no need for any
> mid-layer protocol between the IPv4 or IPv6 header and the SEAL
> header, or between the SEAL header and the traffic packet.  This is
> not clearly specified anywhere, since the SEAL and RANGER documents
> are general purpose, and their use for a scalable routing solution as
> a Core-Edge Separation architecture is only one thing they could be
> used for.

In some environments, it may be necessary to insert a
mid-layer UDP header in order to give ECMP/LAG routers
a handle to support multipath traffic flow separation.

> Firstly I describe my understanding of what your ID specifies for IPv4.
>
> Secondly I describe two other ways you might do PMTUD with IPv4,
> without using DF=0 packets.  These would avoid whatever risk there
> might be of setting the ITE's PMTU estimate too low due to a limiting
> router sending out fragments which are shorter than the limiting next
> hop MTU.
>
> Finally, I describe my understanding of what your ID specifies for IPv6.
>
> This is partly for my own reference, since it took me many hours to
> discern this by reading the SEAL ID and corresponding with you.
>
>
> IPv4:
>
>   The ITE sends a DF=0 packet into the tunnel.  This starts with
>   an IPv4 header, then has a SEAL header (there's no mid-level
>   protocol in this Core-Edge Separation usage of RANGER and SEAL)
>   and then the inner packet, the original IPv4 traffic packet.
>
>   The source address in the outer header is that of the ITE and
>   the destination address is that of the ETE.  The 32 bit
>   SEAL ID is split in two.  16 bits go into the IPv4 header's
>   ID field and 16 into the SEAL header's ID Extension field.
>
>   The limiting router in the tunnel (the one where the next-hop
>   MTU is less than the the length of this whole packet) fragments
>   it into at least two fragments.
>
>   Now the second para in 4.4.2 comes into play:
>
>         When the ETE processes the IP first-fragment (i.e.,
>         one with MF=1 and Offset=0 in the IP header) of a
>         fragmented SEAL packet, ...
>
>   The first para was for reassembling packets which had been
>   fragmented by the SEAL protocol.  But the second para is for
>   SEAL packets, as was just sent, being fragmented by a
>   router between the ITE and the ETE.  This only occurs for
>   IPv4 and I think it would be helpful to mention IPv4 in this
>   paragraph.  Maybe it needs its own section.
>
>        ...  it sends a "Reassembly Report - Fragmentation
>        Experienced" message back to the ITE with the S_MSS field
>        set to the length of the first-fragment and with the
>        S_MRU field set to no more than the size of the reassembly
>        buffer (see Section 4.4.5).
>
>   I think this last part about the value of S_MRU is not clear
>   enough.  What value should it be set to?
>
>   I will assume it is set to some non-zero value.
>
>   Assuming the limiting router sent out the first fragment with
>   a length equal to the limiting next-hop MTU, then this MTU
>   value is now in the S_MSS field of the message sent to the
>   ITE.
>
>
>   This message arrives at the ITE.  This message, according to
>   Figure 4, contains:
>
>        As much of invoking packet as possible without the
>        message exceeding 576 bytes.
>
>   Maybe your ID specifies this, but I am having trouble
>   following it - there has to be a way the ITE securely
>   accepts this "Fragmentation Experienced" message.
>
>   As far as I know, the ITE looks into the message, finding
>   the initial part of the packet which the ETE received as
>   a first fragment.  That will contain the outer IPv4 header
>   and the SEAL header, and from this these the 32 bit SEAL ID
>   in the SEAL encapsulated packet can be found.
>
>   I think you either cache the recently sent 32 bit SEAL IDs
>   or maintain a sliding window function over their range so
>   you can easily identify a value which was used in the last
>   second or two.  In a given ITE, each ETE has its own SEAL
>   ID counter.  Its value is intitialized randomly when the
>   state for this ETE is created.  After that, its value
>   increments with each each packet sent to the ETE.
>
>   (I may adopt this incrementing value per ETR arrangement,
>   with its sliding window, rather than using a nonce.)
>
>   The wider the window in time, the longer you can accept these
>   messages.  Since the ETE and the ITE could be on opposite
>   sides of the Net, I guess you need to have a window which
>   accepts SEAL IDs sent at least a second ago.
>
>   The longer the window in time, and the more packets the
>   ITE sends to this ITE, the wider the window is numerically
>   and the easier it is for an attacker to guess a valid value
>   and have the ETE accept a PTB with a low enough value
>   to cause lost efficiency - for the next 10 minutes or so.

The above is all correct wrt the window management. The
ITE can ensure that the window size remains bounded by
sending periodic explicit probes (e.g., once explicit
probe per every N data packets).

>   Now to 4.3.9.1.2:
>
>        4.3.9.1.2. Fragmentation Experienced (Code=1)
>
>        If the value in the S_MRU field is non-zero, the
>        ITE records the value in its soft state for this ETE.
>
>   This means this value is stored in the S_MRU variable for
>   this ETE, as defined in 4.3.3.  As noted above, I am not
>   clear on what value was written into this field of the
>   report by the ETE.
>
>        The ITE then adjusts the S_MSS value in its soft state.
>
>   This means this value is stored in the S_MSS variable for
>   this ETE, as defined in 4.3.3, subject to the instructions
>   in the next few sentences.
>
>   I am a bit confused about the differing roles of these two
>   variables.
>
>        If the S_MSS value in the Reassembly Report is greater
>        than 576 (i.e., the nominal minimum MTU for IPv4 links),
>        the ITE records this new value in its soft state.
>
>   OK - this is based on the assumption that the length of the
>   first fragment received by the ETE reflects the limiting
>   MTU of the ITE to ETE path.
>
>        If the S_MSS value in the report is less than the current
>        soft state value and also less than 576,
>
>   How could the ITE's S_MSS value for this ETE be less than
>   576?  I can't see how.  If it can't be, then the first part
>   of the above sentence may be redundant.

No, the sentence is correct. It is possible for the ITE
to need to reduce its cached S_MSS value to a size less
than 576 if there is truly a link with a small MTU (e.g.,
256) on the path. Although 576 is often considered to
be the "nominal" minimum MTU for IPv4 links, the actual
minimum MTU is only 68 bytes per RFC791.

>        the ITE can discern that IP fragmentation is occurring
>        but it cannot determine the true MTU of the restricting
>        link due to a router on the path generating runt
>        first-fragments.
>
>   Then the next paragraph describes the "iterative searching
>   strategy" to find the correct (or near enough, but perhaps
>   lower) value for the S_MSS variable for this ETE.
>
>   I think this paragraph is unclear.  I think it should state
>   that the probes are occurring only due to traffic packets
>   arriving at the ITE and being tunneled to this ETE, and
>   these being long enough.  Since SEAL treats all packets
>   as probes, this use of the term "probe" may be confusing -
>   since in fact all packets may be probes.
>
>   I think this paragraph should describe the process in detail
>   - I guess it is only occurs with real traffic packets.
>
>   The reference to section 5 of RFC1191 is not very helpful
>   because it describes several algorithms.
>
>
>
> IPv4 - my suggestion for doing it with DF=1 packets
>
>   A - If you can be sure the routers send back more than
>       the bare minimum IPv4 header + 32 bits.  (Its just
>       firmware updates to have routers do this and maybe
>       most or all of them already do.)
>
>       Send the SEAL packets as noted above, but with DF=1.
>
>       If there is an MTU problem, the ITE will get a PTB
>       with the MTU value it needs, plus enough of the
>       SEAL packet to extract sufficient of the traffic
>       packet to make a valid PTB for the SH.
>
>       The MTU value from the received PTB is written into
>       the S_MTU for this ETE.  This gives an exact value
>       without the problem of potential "runt packets" which
>       arises with DF=0 in your current process.
>
>       The ITR subtracts 24 from the MTU value it received
>       in the PTB from the limiting router (20 bytes of IPv4
>       header + 4 bytes of SEAL header) and uses this value
>       in the PTB to the SH.
>
>       This will work fine - the SH will then send packets
>       of the correct size, so when they are SEAL encapsulated
>       they will not have a problem with this MTU limit.
>
>       If any other SH, or another application in the same
>       SH, sends a packet whose length exceeds the new MTU
>       value minus 24, then the ITE will send back a PTB
>       accordingly.
>
>       If there is a further, lower, MTU limit en-route to
>       the ETE, then the above process will be repeated.
>       This is similar in principle to your IPv6 approach.
>
>
>   B - If you have to assume that some or all routers between
>       the ITE and the ETE only send back the bare minimum
>       amount of packet in their PTB, then you can still
>       accept these packets securely, and calculate a proper
>       MTU value to send to the SH, as described above.
>
>       In order to be able to generate a valid PTB, you need
>       the ITE to have cached the IPv4 header and the next
>       32 bits which follows (24 bytes) for each packet sent
>       which you think might give rise to a PTB.  You don't
>       need to do this with packets shorter than some constant
>       - depending on whatever is the lowest PMTU you ever
>       expect to find between an ITE and an ETE.
>
>       I guess you only need to cache these 24 byte items for a
>       second or so.  You need to be able to index into the
>       cache by using the 32 bit SEAL ID retrieved from the
>       16 bit IPv4 ID and the 16 bit SEAL ID Extension in the
>       initial part of the encapsulated packet, which is in
>       the first fragment, as returned in the PTB.
>
>
> IPv6:
>
>   The ITE creates an IPv6 header and a Fragment Header.  As far
>   as I know, there is no SEAL header.  I think this should be
>   made more clear in the final part of 3.4.3.
>
>   The 32 bit SEAL ID is written into the Identification field
>   of the Fragment Header.  The ITE then appends the traffic
>   packet.
>
>   The result is forwarded towards the ETE.  If it is too big for
>   a next-hop MTU in any router en-route to the ETE, that router
>   sends back a PTB to the ITE with an MTU value, and enough of
>   the original packet for the ITE to construct a valid PTB to the
>   SH.
>
>      (I can't find where your ID describes the reception of
>       the PTB.  Section 4.3.8 should cover this, but makes no
>       specific mention of PTB messages.)
>
>   The ITE secures the acceptance of the PTB by using comparing
>   the 32 bit SEAL ID, as noted for IPv4 above, via a cached set
>   of recently used values or some kind of window function
>
>   The MTU value is written to the S_MTU for this ETE.
>
>   The ITE subtracts 48 from the MTU value (40 bytes for the IPv6
>   header and 8 bytes for the Fragment Header) and uses this to set
>   the MTU value in the PTB which is sent to the SH.
>
>   As with the IPv4 approach, any packets arriving at the ITE
>   which will be tunneled to this ETE, if longer than the MTU
>   value minus 48, will be dropped and used to send a PTB to
>   the SH.
>
>   This is identical in principle to my IPv4 suggestion A above.
>
>
> Returning to Ivip's IPTM protocol:
>
> With IPv6, I could avoid caching any part of the packet if I could
> rely on the ITR getting a PTB, since the PTB is guaranteed to contain
> plenty of the inner packet - enough for the sending host to
> recognise.  (Actually, the minimum amount of original packet returned
> could be less than what should be returned to the SH, due to the
> encapsulation IP, UDP and IPTM header which precedes it.  Still,
> enough should be there that any SH should be able to recognise it.)
>
> I still need to send two packets, since the long B packet does not
> contain the full traffic packet.  It has extra things - a UDP header
> and an IPTM header, with a 32 bit nonce.   The last part of the
> traffic packet is not in the B packet
>
> The part which doesn't fit is contained in the A packet.
>
> If both A and B arrive at the ETR, the whole traffic packet is
> delivered.  The A packet is matched to its B packet with the nonce
> they both contain in their IPTM headers.
>
> The ETR will send a message to the ITR, also secured by the nonce,
> to tell it that both parts arrived.
>
> If the B part doesn't arrive, after a ~0.5 second time-out, the ETR
> will send a message to the ITR telling it that only the A part arrived.
>
> If only the B part arrives, the ETR sends a message to the ITR to
> that effect too.
>
> With IPv4, if it could be assumed that all routers would return
> sufficient of the packet to include the first 24 bytes of the inner
> packet, then I could use the same approach as for IPv6 - and so avoid
> caching any part of the traffic packet in the ITR.
>
> If I couldn't assume this, then there are two approaches:
>
>   1 - Do as for my suggestion B above - have the ITR cache the
>       first 24 bytes of traffic packets used for this 2 packet
>       probing technique.  The cache would be indexable via the
>       nonce sent with each packet - and the caching time would
>       be about a second.
>
>   2 - Avoid caching in the ITR, by including the first 24 bytes
>       of the traffic packet in the A packet.  If the A packet
>       arrives at the ETR, and the B packet doesn't, the ETR
>       can report this, and return the 24 bytes from the A
>       packet.
>
> The ITR is only doing this 2 packet probing technique infrequently,
> so the caching approach is not particularly expensive.  Caching the
> first 24 bytes of the packet has an advantage that when the ITR gets
> a PTB - as would normally be the case if the B packet was too long -
> then the ITR can send the PTB immediately, rather than waiting for a
> message from the ETR, which would necessarily arrive at least a
> second after the ITR tunneled the packet.
>
>
> The detection of a PMTU limit doesn't have to be absolutely
> bullet-proof.  It should not result in the ITR deciding that the PMTU
> is lower than it actually is, but if, for some reason, the probe
> process produces an indeterminate result - such as the ITR not
> getting anything back from the ETR as a result of the A or B packet,
> and no PTB either, then the ITR takes no further action.  This is
> indistinguishable from ordinary packet loss.  The most likely outcome
> is that the SH will try again, with a similar sized packet (unless it
> is doing RFC 4821 ... which no hosts appear to be doing at present)
> and the ITR will again generate the B and A packets.  Then, the most
> likely outcome will be that the ITR learns something definite about
> the PMTU, and so reduces its Zone of Uncertainty.
>
> Doing this PMTUD stuff in the FIB of a big router handling gigabit
> and 10 gigabit links could be quite challenging.  It doesn't have to
> be done this way, since Ivip ITRs and ETRs can be implemented in
> software on ordinary servers, which are inexpensive and can still
> handle (I guess) gigabit traffic rates.  Also, having the ITR in the
> sending host is a zero cost way of ensuring each ITR doesn't have to
> juggle too many of these PMTUD probing sessions at once.
>
>
> >> At present, for IPv4 and IPv4, your ITE (ITR) functions emit packets
> >> with an outer header of IPv4 or IPv6, followed by a 32 bit SEAL header.
> >>
> >> Immediately following the SEAL header you may have some "mid-layer
> >> headers" which I don't properly understand.  Then you have the IPv4
> >> or IPv6 traffic packet, or perhaps a segment of it.
> >>
> >> You could make the SEAL ITE work fine with minimal length IPv4 PTBs
> >> if the SEAL header was extended to 64 bits, with the additional 32
> >> bits being a nonce.  That would always be returned in any PTB.
> >
> > SEAL uses the 32bit ID (gotten from the 16bit IPv4 ID
> > concatenated with SEAL's 16bit ID extension) as a nonce.
> > There is no need that I can see for including an
> > additional nonce.
>
> OK.
>
>
> >> So I think your objection to using RFC1191 PTBs should only be based
> >> on your concern about the PTBs being systematically dropped due to
> >> filtering.
> >>
> >> I assert that such filtering is a symptom of a badly administered
> >> network - and that it should be fixed in the network, not worked
> >> around with a protocol such as SEAL or IPTM.
> >
> > In my understanding, in the interdomain routing region of
> > the Internet there is no close coordination regarding the
> > way "the network" is administered. There is also a wide
> > variety of network vendor equipment deployed in the
> > Internet which may have widely varying default behaviors.
> > So, in general it seems overly optimistic to assume that
> > all of the diverse policies, implementations and operational
> > practices out there could be brought into strict uniformity.
>
> OK - but at some point we need to stop adopting band-aid measures
> like artificially limiting MSS or MTU values.  That just lets the PTB
> filtering and lousy tunnels be less noticed.   We should not be
> trying to upgrade the stacks of all hosts in the world because a few
> end-user networks filter PTBs or ISPs and perhaps end-user networks
> run tunnels which don't support the otherwise perfectly good RFC 1191
> / 1981 PMTUD techniques.
>
> We would just be heaping limitations and complications on ourselves
> in an overly-defensive, expensive and inefficient attempt to cope
> with failure of a few ISPs and end-user networks to run the Internet
> as it needs to be run.  We are paying the ISPs.  The end-user
> networks which are filtering PTBs are disrupting a subset of their
> own communications.

I still think there are problems out there. I will post
another message on this soon.

> I just think it wrong in principle to develop messy new protocols
> such as RFC 4821 to cope with these failings.

In my opinion, packetization layers are operating "at risk"
if they use packet sizes larger than 1500 but are not in
some way checking with the final destination to ensure that
the big packets are actually getting through. RFC4821 is
a method for the source to do just that without requiring
any changes on the destination. But to be sure, SEAL does
not *depend* on RFC4821 but rather *sets the stage* for
RFC4821 and/or any functional equivalents.

Thanks - Fred
[email-masked]

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 2 Feb 2010 15:52:17 -0800

Hi Wesley,

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Eddy, Wesley M. (GRC-MS00)[ASRC
> AEROSPACE CORP]
> Sent: Friday, January 29, 2010 1:23 PM
> To: Dan Wing; 'Robin Whittle'; 'RRG'
> >From: [email-masked] [mailto:[email-masked]] On Behalf Of
> >Dan Wing
> >Sent: Thursday, January 28, 2010 10:01 AM
> >To: 'Robin Whittle'; 'RRG'
> > ...
> >> Where is the evidence that PTB filtering is ever more than a
> >> transitory, mistaken, condition?
> >
> >Sounds like you want a research paper?
> 
> 
> Here are three for Robin's sake; this issue is well known to the
> transport layer folks:
> 
> http://www.icir.org/tbit/tbit-Aug2004.pdf
> 
> http://www.caida.org/workshops/wide/0603/slides/mluckie2.pdf
> 
> http://listserver.internetnz.net.nz/pipermail/ipv6-techsig/2009-October/000708.html

Thanks for pointers to this work. Robin pointed out some
of this on the RRG list earlier, but its good to have the
additional material.

>From Ben Stasiewicz' post on the ipv6-techsig mailing list
(above), I retrieved the Alexa list of the top 1M websites
and from this took the top 1000 as my sample set. I used
the "tbit" tool (http://icir.net/tbit) to run IPv4 Path MTU
discovery (PMTUD) tests on each of the top 1000 using the
command syntax:

  # tbit -m 1460 -M 576 -n www.example.com -t PMTUD example.com  

This command has tbit advertise an MSS of 1460 during the
TCP connection, but whenever tbit receives a packet larger
than 576 from a website it drops the packet and sends back
an ICMPv4 "Fragmentation Needed" (aka "PMTUD") message.

The test was to determine how each website in the top 1000
responded to PMTUD messages. Of the websites sampled, I
observed the following results:

  Transfer Fail:   249 (24.9%)
  PMTUD Disabled:  109 (10.9%)
  PMTUD Success:   394 (39.4%)
  PMTUD Fail:      196 (19.6%)
  Connection Fail:  52 ( 5.2%)  

The "Transfer Fail" class was mostly due to HTTP 301/302
responses (tbit did not know how to handle these), and in
those cases tbit quit before any PMTUD tests were run. I
otherwise did not make any effort to diagnose the failures
more closely.

The "PMTUD Disabled" class included websites that set DF=0
in the IPv4 header, and thereby disabled PMTUD making data
packets eligible for fragmentation in the network.

The "PMTUD Success" cases correctly implemented PMTUD, but
in many cases tbit needed to send multiple PMTUD messages
before the website reduced the size of its data packets.
In some cases, websites responded to the PMTUD messages in
strange ways, including making curious guesses at packet
sizes instead of honoring the MTU size advertised in the
PMTUD message, turning PMTUD off altogether after receiving
a few PMTUD messages, etc.

The "PMTUD Failed" cases represented true failures of the
website to correctly honor the PMTUD messages, as verified
by wireshark captures. This could only happen if the PMTUD
messages were filtered in the network and not delivered to
the website, or if the website received the PMTUD messages
and ignored them.

The "Connection Fail" cases represented websites that
were unreachable, but may have also included sites that
sent large packets that were silently dropped due to a
true MTU bottleneck before reaching the test machine.

All of this closely agrees with the results documented
in "Measuring the Evolution of Transport Protocols in
the Internet" (Medina, Allman, and Floyd, ACM Computer
Communication Review, April 2005):

 http://icir.net/tbit/TCPevolution-Mar2005.pdf

The end result is that the IPv4 Internet still seems
to be vulnerable to PMTUD failures along paths that
lead to the top websites in the Internet today. This
suggests that PMTUD is likely not being invoked very
often in the first place since most links in the Internet
configure an MTU of 1500 or larger and few applications
send packets larger than that. However, as more and more
tunnels over the IPv4 Internet are used the frequency
of PMTUD failures is likely to increase as well.

Fred

> --
> Wes Eddy
> MTI Systems
> _______________________________________________
> rrg mailing list
> [email-masked]
> http://www.irtf.org/mailman/listinfo/rrg

---

From: Robin Whittle <[email-masked]>
Date: Wed, 03 Feb 2010 12:54:04 +1100

This is a response to Fred's message:

  Re: [rrg] SEAL critique, PMTUD, RFC4821 = vapourware
  http://www.ietf.org/mail-archive/web/rrg/current/msg05907.html

  in which he shows that the failure of RFC1191 PMTUD is far
  more prevalent than I had imagined:

     Some servers sending DF=0 packets (eg. 1470 bytes).

     Presumed dropping of PTBs in some networks.

     Some servers failing to respond properly or at all to PTBs.

     Perhaps tunnels which do not generate PTBs to the sending
     host (which may be the entry router of an outer tunnel).

  It seems the Internet is getting by with dodgy assumptions about
  packet sizes which enable everything to work (mostly) despite
  these failings.

  I can't see how we could transition to jumboframes in the DFZ
  with the current messy situation.

  Does this justify retreat into the development of RFC4821
  packetization layer PMTUD?  That would seem to be a capitulation
  to the bad practices which have undermined RFC1191 PMTUD,
  allowing these practices to continue and become more prevalent.
  Yet RFC4821 would would only solve the problem once all stacks
  and all applications were modified accordingly.

  How can an encapsulation-based CES architecture cope with
  these problems, assuming the encapsulation overhead leads
  to PMTU problems between the ITR and ETR which are not
  properly handled with a PTB message?  It can be done, but
  it is complex - relying on ITR-to-ETR checking that
  packets actually arrive at the ETR, rather than relying on
  PTBs.

  IPv6 would probably have similar problems - and perhaps
  more tunnels at present - than IPv4.

Hi Fred,

> packets eligible for fragmentation in the network.

Google did this this last time I looked, with 1470 byte packets:

http://www.firstpr.com.au/ip/ivip/ipv4-bits/actual-packets.html#google-no-pmtud

Trying again, using a Google logo image file, from a US server:

  tcpdump -n -x  -i eth0 > tcpdump-01.txt
  wget http://74.125.159.105/images/google_sm.gif

    IP 74.125.159.105.http > myhost.48611: ....

    0x0000:  4500 05be 7a19 0000 3806 c132 4a7d 9f69
    0x0010:  ae8f a978 0050 bde3 22f2 47ec d1c4 152f
    0x0020:  8010 006a 7ad2 0000 0101 080a 2570 5989

    0x0000:  4500 05be 7a19 0000
                  ----      ^
           1470 bytes        \ DF=0

the same thing is occurring.

> of PMTUD failures is likely to increase as well.

Thanks for doing this research.  It indicates that PMTUD is a mess in
the IPv4 Internet - and yet in general things seem to work OK.

As long as almost all hosts are reachable with PMTUs of 1500 or
somewhat below, and as long as hosts generally do no attempt to send
out packets no longer than 1470 if they are not supporting RFC1191
PMTUD, I guess things generally work OK.

But this presents real difficulties for introducing any CES
architecture which uses encapsulation, assuming that these problems
with PTBs being dropped, or not generated in tunnels, continue and
affect the paths between ITRs and ETRs.

Hosts sending 1470 byte DF=0 packets is a much tougher nut to crack,
since a CES ITR simply cannot tunnel this in a single packet.  The
ITR can't tell the host to send shorter packets.  So each such packet
would have to travel as two packets in the ITR ETR tunnel.  This
would be extremely ugly.

If a CES scheme was introduced, it would probably need to be
accompanied by a BCP (and a pep-talk to Google) to the effect that it
really is not OK any more to send DF=0 packets which are longer than
can be handled by the encapsulation scheme.

In the long-term, I would prefer to see DF=0 packets no longer used -
and these problems of PTBs being dropped, or not generated in
tunnels, or ignored, fixed - rather than every host and application
having to use the messy and potentially error-prone RFC4821 approach
to PMTUD.

  - Robin

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 2 Feb 2010 22:00:18 -0500

On Thu, Jan 28, 2010 at 2:25 AM, Tony Li <[email-masked]> wrote:
> Robin Whittle wrote:
>
>>                 I argue against Fred Templin's position that
>>                 ordinary RFC1191 DF=1 Path MTU Discovery (and
>>                 therefore its RFC1981 IPv6 equivalent) is "busted".
>>
>>                 Where is the evidence that networks filtering out
>>                 PTB (Packet Too Big) messages is a significant
>>                 problem?
>
>
> This happens.  Consult some operator folks, privately and quietly.  Many
> enterprises blocked all inbound ICMP when DDoS attacks started happening.

yes, not just for 'dos attack' problems but because often people don't
understand what ICMP is there for :( the dreaded 'people can ping
you!!' apparently == 'deny icmp any any'

PMTUD is horrendously supported :(

-chris
(operatorish guy)

---

From: Xu Xiaohu <[email-masked]>
Date: Wed, 03 Feb 2010 11:15:03 +0800

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Christopher
> Morrow
> 发送时间: 2010年2月3日 11:00
> 收件人: Tony Li
> 抄送: RRG; Robin Whittle
> 主题: Re: [rrg] SEAL critique, PMTUD, RFC4821 = vapourware
> 

Yes, based on the above practice, some router vendors even give up the support for PMTUD in their router OSes.

Xiaohu

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 2 Feb 2010 22:21:24 -0500

> Yes, based on the above practice, some router vendors even give up the support for PMTUD in their router OSes.

that seems like a VERY BAD PLAN, being dumb and turning a knob off is
one thing, removing the knob is entirely different (and far worse).

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 2 Feb 2010 23:52:28 -0500

>  with the current messy situation.

the packet size at the DFZ can go to any larger size it wants... in
fact there are at least 2 large US carries (with global presence) that
have MTU on all core/internal links >9000. The problem, that I've
experienced/seen, is in edge connected places, and where tunnels
exist.

Often the tunnels crop up in v6 discussions, or in v4 vpn (ipsec)
scenarios. There's a reason that the MSS you see upon tcp setup to
most 'large' content provider hosts is < 1470 ...

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
23:50:18.826948 IP (tos 0x10, ttl 64, id 31860, offset 0, flags [DF],
proto TCP (6), length 60) my.ip.41646 > large.content.ip.http: S,
cksum 0xc950 (correct), 3573990180:3573990180(0) win 5840 <mss
1460,sackOK,timestamp 747355984 0,nop,wscale 7>
23:50:18.834402 IP (tos 0x0, ttl 53, id 54699, offset 0, flags [none],
proto TCP (6), length 60) large.content.ip.http > my.ip.41646: S,
cksum 0x9353 (correct), 686478021:686478021(0) ack 3573990181 win 5672
<mss 1430,sackOK,timestamp 2149169897 747355984,nop,wscale 6>

Note he squashes me to mss=1430, yes MSS != MTU, but... it's there
because raising it runs into MTU bottlenecks for a portion of users.

-Chris

---

From: Robin Whittle <[email-masked]>
Date: Thu, 04 Feb 2010 01:17:09 +1100

Short version:    Refining my understanding of Fred's SEAL
                  protocol for tunneling with PMTUD management.

Hi Fred,

I will include only parts of my previous message to which you are
replying.

> to fill in the MTU field in their PTB messages. 

OK - an iterative search strategy which only works downwards is not
"hunting"!

Are there really routers in use which don't include the 16 bit value?

Even if there theoretically are not, SEAL still has to cope with the
possibility of zero or unreasonable values in this MTU field.

> What I

OK - I understand that SEAL follows "an iterative searching strategy
that parallels (Section 5 of RFC1191)":

  http://tools.ietf.org/html/rfc1191#section-5

presumably the final part of this, which leads to:

  http://tools.ietf.org/html/rfc1191#section-7

I understand this means stepping downwards through the values in
Table 7-1.  However, I guess you would choose some new values to add
to this list

> a handle to support multipath traffic flow separation.

   http://en.wikipedia.org/wiki/Equal-cost_multi-path_routing

http://www.force10networks.com/CSPortal20/TechTips/0065_HowDoIConfigureLoadBalancing.aspx

As far as I know, these techniques are not something to consider with
the RANGER CES, or with LISP or Ivip.  If the routers can handle
ordinary traffic packets they can handle encapsulated packets too.  I
haven't read about these techniques in detail.  I guess that within
RANGER, beyond its use as a CES scalable routing solution, you may
want to support ECMP and LAG.

> probe per every N data packets).

I don't have a really clear idea of how SEAL sets the numeric window,
or what you mean by your second sentence above.  If you can give a
more complete description with an example, I would really appreciate
it - in part because I might want to use or adapt your technique for
Ivip, which currently uses nonces.

> minimum MTU is only 68 bytes per RFC791.

OK - I guess I wouldn't go this far for Ivip.  I will probably have
some guidance that if anyone has routers, tunnels or whatever with
PMTUs less than 1280 or some figure in that range, for both IPv4 or
IPv6, then they should make sure that ITRs or ETRs are not located so
that such low PMTU parts of the network are between the DFZ and these
ITRs or ETRs.

> another message on this soon.

Indeed you did - I had no idea things were this bad.

  http://www.ietf.org/mail-archive/web/rrg/current/msg05907.html
  http://www.ietf.org/mail-archive/web/rrg/current/msg05910.html

> RFC4821 and/or any functional equivalents.

OK.  I need to revisit all my thinking on PMTUD given the results of
your recent research.  But I haven't yet changed my thinking that RFC
4821 is messy and expensive, and that it would be better to
straighten out whatever it is within the network which is stopping
PMTUD from working correctly.

 - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Feb 2010 08:09:53 -0800

Hi Robin,

Thanks for your message; a few points for clarification below:

> Sent: Wednesday, February 03, 2010 6:17 AM
> To: Templin, Fred L
> Cc: RRG

SEAL explicitly turns off PMTUD and uses its own tunnel
endpoint-to-endpoint MTU determination, so in the normal
case it does not expect to receive any ICMP PTBs from
routers within the tunnel. SEAL *can* enable PMTUD for
certain "expendable" packets, and can benefit from any
ICMP PTBs coming from within the tunnel that contain
sufficient information. But, that would simply be an
optimization.

> want to support ECMP and LAG.

There has been a great deal of talk about taking care
of ECMP/LAG routers within the network that only
recognize common-case protocols (i.e., TCP and UDP),
which is why LISP has locked into using UDP encaps.

> Ivip, which currently uses nonces.

What I am asking for in SEAL is that the ITE sets the
SEAL_ID in each packet in monotonically-incrementing
fashion (modulo 32). Then, on every Nth packet (e.g.,
500th, 1000th, etc.) the ITE sets the "Acknowledgement
Requested" bit and the ETE (upon seeing the bit set)
sends back an explicit acknowledgement. The ITE can
then keep a window of outstanding SEAL_ID's by keeping
track of the most recently sent and most recently
acknowledged SEAL_IDs.

> that such low PMTU parts of the network are between the DFZ and these

I want to be able to use SEAL over truly constrained
links, such as certain wireless communications systems.
On such links, there may be a reason to set an
unusually small MTU.

> PMTUD from working correctly.

The good thing about RFC4821 is that only the source
host needs to implement it, so there is a natural path
for incremental deployment. I guess there are other
possibilities as well (e.g., similar schemes that do
PMTUD black hole detection) so as long as the method
chosen causes no harm the source host is welcome to
implement whatever it sees fit.

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Feb 2010 14:30:47 -0800

Robin,

Thanks for giving this a look, and see below:

> Sent: Tuesday, February 02, 2010 4:33 AM
> To: RRG
> Cc: Templin, Fred L
> Subject: Re: [rrg] RANGER and SEAL critique
>

Each enterprise network within a RANGER recursive hierarchy
has its own PRL, which is a list of ETRs that can advertise
"default" (called enterprise border gateways by VET). These
ETRs can be used to forward packets out of the enterprise.

In the RON, as you call it, none of the participating
routers advertise "default", but each of the participating
routers is considered a PRL router for the RON since each
will contain a full RIB of all VPs advertised in the RON.
So, a RON router can set up BGP peerings with other RON
routers by static configuration of RON router addresses,
by resolving a FQDN to get back a list of nearby RON routers,
by issuing a multicast "shout out" on the underlying network
to see if there are any other RON routers nearby, etc.

Let me know if you have a different viewpoint on how PRL
should be interpreted.

> "Enterprise" in this context means an ISP.  "Inter-enterprise" means

Not just ISPs, but also corporate enterprises, academic campuses,
major government agencies, etc. etc. If you want to categorize
them all as ISPs that is fine, but I choose to categorize them
as enterprises.

> of the term, an "interface" can't span the entire DFZ.

Sorry about that; I should have said "VET link". A VET
interface is a VET node's point of attachment to a VET
link, but the VET link that manifests the RON spans the
entire DFZ.

> since these are not ISP BRs.

The RON routers have to connect the ISP/Enterprise to the
RON. But, they need not be the same routers that connect
the ISP/Enterprise to the existing IPv4 Internet DFZ.

The RON routers need to have a global IPv4 address on an
interface that a VET interface (for connecting to the RON)
can be configured over. But, they need not participate in
global IPv4 BGP routing.

> other (where the packets go over the physical links anyway).

The tunnels are manifested by VET, and may traverse many
underlying physical links.

> router.  Also, their ETR function must somehow connect to edge

That is certainly one way to stand up a RON router, but
I do not thing the RON routers need to be on the same
physical platform as the DFZ routers. They can be built
up as a separate box instead that connects to the
ISP/Enterprise network on the "inside" and connects to
the public IPv4 Internet (and the RON) on the "outside".
                                                             Note
> Limit "scaling difficulties"?

Limit scaling to a manageable order-of-magnitude number of
VP entries in the RIB.

> are up to 10 million or so.

The RIB contains only highly-aggregated VPs, but the FIBs
of specific RON routers will contain separate prefixes
for each of the individual end-user "edge" EID prefixes.

Let's say that a RON router A configures a VP "BAA::/16".
A advertises BAA::/16 in the RON BGP instance and discovers
all other VPs used in the RON to build up its RIB. But, A
holds the more-specific edge EID prefixes corresponding to
customer allocations in its FIB. So, in A's RIB will be
only VP prefixes, and A's FIB will hold more-specifics
taken from BAA::/16, e.g.:

  BAA:0:0:0000::/56
  BAA:0:0:0100::/56
  BAA:0:0:0200::/56
  ...

So, if A's FIB gets to be too big because of lots of
allocations to customers, A's enterprise can simply
split its VP into smaller pieces (e.g., BAA::/17,
BAA:8000::/17), and use multiple RON routers to spread
the load.

> something totally different - reducing the contents of the RIB.

Well, the goal is to reduce the contents of the RIB for
all RON routers and require RON routers to carry FIB
entries only for those more-specific prefixes that it
cares about. So, if RON router A holds the VP BAA::/16,
then it must consider any active more-specifics taken
from that prefix in its FIB (e.g., BAA::/56). But, if
RON router B is not sending any traffic to hosts that
configure addresses from BAA::/56 then B does not have
to hold that prefix in its FIB.

> think of these ITR-ETR routers being separate from the underlying DFZ

RON routers forward packets by discovering the global
IPv4 addresses of other RON routers that are owners of
the EID VPs that cover the EID destination addresses
of packet. These global IPv4 addresses are nothing
more than RLOCs per the LISP/Ivip/RANGER nomenclature.
> able to forward the packet.

I think this helps me to spot out an issue in my
thinking. The VP's carried in the RON RIB also need
to be populated in each RON router's FIB. Otherwise,
if there was no more-specific route there would be no
matching route in the FIB and the packet would be
dropped. I do not want to require any special magic
here, so I will fix this.

> the next, but also to carry the TCP session for bidirectional BGP

Yes.

> - presumably, usually, not too far away.

Correct.

> routers) are implemented on DFZ routers but as separate entities?

They don't have to be implemented on existing RLOC-BGP
(aka DFZ) routers - they can be stood up as separate
boxes w/o disturbing the existing deployment.

> RIB if a packet arrives which needs a prefix not currently in the FIB.

What I am suggesting reduces both the RIB and FIB sizes.
Maybe I didn't take enough time to fully comprehend VA
and should just drop the VA terminology for the purpose
of what I'm trying to describe here?

OK.

> least in terms of the reference you provide, which does involve

Well, I can drop the VA comparisons if it would help.

> all these ITR-ETR routers.

Correct - not a full mesh the same as for the existing
RLOC-BGP peering arrangements. Hopefully each RON router
will peer with a limited set of other RON routers that
are close by.

> Which are these "selected" routers?

Both the RON router(s) that "own" the 42/16 prefix and any
RON router that is actively sending traffic to destinations
covered by the 42/16 prefix. But, no other RON routers.

> to all other ITR-ETR routers.

Yes.

OK.

> It is taking me many hours to try to understand what you wrote.

Sorry for that, but thanks for taking the time.

> so such "EID prefixes" which the Seattle router is responsible for.

Yes.

>    handles a given end-user network with a given "edge" (EID) prefix.

Actually, it means the RLOC address of the RON router(s)
that connect the ISPs/Enterprises in which the CE routers
reside.

>         of the CE router for this prefix.

No; for each FIB entry there is an RLOC address of another
RON router that sits in front of the CE router.

>         Seattle router, but in general, these CE routers are nowhere

But, it doesn't matter where the CE routers are, and they
can travel around the world if they choose to do so. As
long as they continue to tell the Seattle router where they
have moved to by telling the Seattle router about which
RON router(s) they are now in proximity of, the Seattle
router can know the next hop.

>         a router somewhere far from Seattle.

That is true I suppose, but I hadn't really thought about
the portability of VP prefixes like 42/16.

>         network "edge" EID prefixes to the Seattle router.

Right.

>         use BGP to manage their best paths for each such prefix.

Correct.

>         Only the Seattle router's FIB has this.

Correct.

>         no delays waiting for mapping.  Interesting . . .

Correct.

>              I don't see where you specified this.

I was actually thinking that the RON routers would only
advertise "default" into the local routing system, but
they could just as well advertise 42/16 if they wanted to.

>              DFZ, without any further tunneling?

In my way of thinking, there is a RON router A that connects
the ISP in which the host on the North Island of New Zeland
resides to the RON. A will consult its FIB (not RIB) to
find that 42/16 points to a tunnel to the Seattle RON
router B, and then tunnels the packet over the RON to B.
The Seattle router (B) now determines that it has a prefix
43.0.56.76 /30 in its FIB that points to a RON router C
that connects the ISP for destination address, and then
tunnels the packet over the RON to C.

>         reduce the number of packets each one has to handle.

That would be fine for me.

>         advertise a single single 42.0.0.0 /14.

That would be fine for me, too.

> ideally with diagrams and definitely with examples.

No, I think you have seen things clearly for the most part.

> router and the Seattle router.

Actually, I am expecting a single tunnel between the
Auckland router and the Seattle router - a single IP hop.

>   2 - Also, it creates a single point of failure.

I liked that you enumerated a number of alternatives
for distributing the RON router function above. Wouldn't
they help spread the load and/or remove single points
of failure?

>       excessively long paths, back and forth across the world.

This is where the RANGER route optimization comes in.
When RON router A sends a packet to RON router B that
holds the 42/16 prefix, B both forwards the packet to
RON router C and sends a Redirect message back to A.
Thereafter, A sends packets directly to C instead of
going through B (with appropriate locator liveness
checks to make sure that C is reachable).

OK.

> particular "edge" prefix with routers such as the one in Seattle.

That is spelled out in detail in VET.

>       multihoming service restoration.

This is the closest to what I had in mind.

> big reduction in total path length.

Sounds good.

> delays.  If I misunderstood you, then I just partially invented such

You are correctly understanding what I was meaning to convey.

> for each tunnel . . . I think it is not a good way to solve the

The piece you may be missing is the route optimization
piece. Again, the holders of VPs like 42/16 (or BAA::/16)
should in the normal case only see a few initial packets.
Thereafter, redirection will result in route optimization
and therefore reduce the load on the VP holders.

> them.

Well, if packets have to go over the core then we
are left with no choice but to handle them with the
RON routers. But with the RANGER recursive hierarchy,
we prefer to handle packets out toward the edges
without involving the core whenever possible. The
idea of RANGER is to keep as much of the churn as
possible pushed out to the outermost levels of
the recursive hierarchy and reduce the load for the
innermost levels of the recursive hierarchy.

> However, while you have a second BGP instance, the total

Right.

> new RON BGP control plane.

Right.

> described in your email.  RANGER can be used for many more things

But, the same organizational and operational principles
apply at all levels of the RANGER hierarchy.

> sort of "demand" is this?

I mean selected other RON routers. "On-demand" means that
the FIB entries get propagated by RON router A sending a
packet to VP-holding RON router B which sends back a
redirect telling that a more direct path can be taken
through RON router C.

> like I missed out on at last part of it.

This is well documented in VET, which RANGER points to.
I'm sorry for the fragmentation, but that's just the way
that it unfolded. I will try to clarify the RANGER text
so as to cause less astonishment.

> Please let me know how close I got to understanding what you have in

The only pieces you missed were the secure redirection
(for route optimization) and the secure prefix
registration (the piece I labeled as "astonishing"
above). Everything else closely matches what I was
trying to convey.

Thanks - Fred

---

From: Robin Whittle <[email-masked]>
Date: Mon, 08 Feb 2010 16:02:17 +1100

Hi Fred,

I am replying to your (msg05937).  I will soon write, in a different
thread, my understanding of SEAL - which I hope you will respond to
with any necessary corrections.

You wrote:

Regarding some IPv4 or IPv6 routers sending a PTB with a zero value
in the 16 bit field for MTU (IPv4) or the 32 bit field for MTU in IPv6:

> routers within the tunnel. 

My understanding is that this is only true for IPv4, because the SEAL
ITE (Ingress Tunnel Endpoint) sends packets with DF=0 to the ETE
(Egress Tunnel Endpoint).  For IPv6, the ITE can get PTBs from
routers in the tunnel since no packets are fragmentable.  So I think
it would not be true to state that SEAL ITE "turns off" the
traditional IPv6 RFC 1981 PMTUD mechanism when it tunnels packets to
the ITE.

> SEAL *can* enable PMTUD for certain "expendable" packets, 

I don't recall what these would be.

> and can benefit from any

Is there a mechanism for SEAL, in IPv4, to send these "expendable"
packets with DF=1?

> which is why LISP has locked into using UDP encaps.

Do you expect this to be the case for IRON?  If so, then I guess that
SEAL in IRON must always use this UDP header before the SEAL header -
since no ITE could know for sure whether ECMP/LAG is in use on the
path to the ETE.

If anyone can point me to good references on ECMP used with LAG, I
would really appreciate it.

Regarding how the ITE manages a sliding window in the (random
starting value, then monotonically incremented) 32 bit SEAL ID it
uses for each ETE - so it can recognise valid values in PTBs or
messages from the ETE, without having to cache each value used so
far, and while keeping the window small enough to make it hard for an

> track of the most recently sent and most recently

OK, except for the possibility that the packets arrive out of order.
For instance, let's say the random start point for the SEAL ID for
this ITE sending packets to this ETE was 3,000,000,000.  On the 500th
packet 3,000,000,499, the "Acknowledgement Requested" flag is set.
However, let's say it is received at the ETE before the packet with
SEAL-ID 3,000,000,498 is received.

By the time the ITE gets the acknowledgement packet from the ETE, it
has sent packets using SEAL IDs up to 3,000,000,510.

The ITE will then refuse to accept as valid any messages from the ITE
- or PTBs from routers in the tunnel path - with values in these
ranges, working backwards around the modulo 2^32 circle:

   Between 3,000,000,499 and 0 inclusive, or between
   4,294,967,295 and 3,000,000,511 inclusive.

But lets say the ETE then responds to the packet with SEAL ID
3,000,000,498, or that the ETE responded to this before sending the
Acknowledgement, but that the response was delivered to the ITE after
the acknowledgement.  Then this response will be considered invalid.

There are at least one reason for lowering the value of "500": To
reduce the proportion of random values in spoofed messages to the ITE
which it would accept.  500 (~2^9) already increases the chance of a
spoofed packet being accepted to 2^-23.

A reason for increasing it is to reduce the workload on the ITE and
ETE - though doing this once every 500 times doesn't seem excessive.

An alternative arrangement to the above would be is to implement in
the ITE a delay function, with a time-constant of a few seconds - say
3 seconds - so that at any instant, the ITE could find for a given
ETE what was the lowest SEAL ID it sent to this ETE within the last 3
seconds.  By using this and the last value used (which would already
be stored) then the ITE could maintain a relatively narrow window
without the ETE having to acknowledge anything, and without the
out-of-order packets problem I suggested above.

This "3 second timer" function doesn't have to be particularly
accurate.  If the effective time is between 3 and 4 seconds, then it
would suffice to take a snapshot of ETE IP addresses and the current
state of their SEAL ID counters, every second, and then stash the
snapshots in one of 4 locations on a circular basis.  Then for the
function which looks up the "SEAL ID last used ~3 seconds ago" simply
find that ETE's SEAL ID value in the snapshot taken 3 to 4 seconds ago.

> On such links, there may be a reason to set an

OK.  I will assume for IRON that SEAL wouldn't be required to perform
heroics - to try to work with PMTUs below ~1200 or so.  I certainly
don't want Ivip to have to cope with people putting ITRs and ETRs in
places where there are MTUs less than this.  Such MTUs shouldn't
exist in the DFZ - if anyone ISP is paying for such a link, they
should get another one.  Likewise, there shouldn't be such low MTU
restrictions in any self-respecting ISP or other network.

> implement whatever it sees fit.

It is up to whoever writes TCP packetization code in host stacks how
they want to set their packet length - and likewise application
developers who need to set their own packet lengths.

My view is that for IPv4, RFC 1191 PMTUD is an excellent system -
except that the PTB message should be made to follow the RFC 1981
requirement of sending back as much of the original packet as would
not make the PTB exceed 576 octets:

  http://tools.ietf.org/html/rfc1885#section-3.2

At present, the PTB spec in RFC 1191 (in 1990) is to only send back
the IPv4 header and the next 8 octets.  It needs to be like RFC 1981
so that when there are nested tunnels, a PTB sent by a router in the
innermost tunnel will contain enough of the original packet (IPv4
header = 8 octets) for the outermost tunnel entry router to craft a
valid PTB to the sending host, without having to cache anything from
the original traffic packet.  This would occur by a chain of events:

    A PTB from router in 3rd tunnel is recognised by the
    ITE of that tunnel as valid, since it contains enough bytes
    after the IPv4 header.  For instance, these bytes would include
    the SEAL ID or any other extra stuff the router put into the
    headers to enable it to accept only genuine PTBs and reject
    ones spoofed by an off-path attacker.  So the ITE at the start
    of the 3rd tunnel sends:

    A PTB to the ITE of the 2nd tunnel, which is long enough, as
    described above, to be validated and to contain enough of the
    tunneled packet for the ITE of the 2nd tunnel to send:

    A PTB to the ITE of the 1st (outer) tunnel.  This contains
    enough of the original traffic packet (at least the IPv4
    header plus 8 octets) for this ITE to be able to construct
    a valid PTB to the sending host, without the need to cache
    the initial part of the packet.  Likewise, the PTB, as
    mentioned above, must contain whatever this ITE needs to
    validate it as genuine.

If the RFC 1191 designers had correctly anticipated the need for one
or more levels of tunneling to support their PMTUD system, then I
think they would have altered the PTB requirements to be as long as
those for IPv6's RFC 1981.  Then we probably would have tunnels today
which properly support RFC 1191 PMTUD.

Also, I think that DF=0 packets should be deprecated - unless perhaps
they are shorter than some constant such as 1200 bytes or so.  I
think it would be bad to expect ITRs and ETRs and the whole CES
system to work over paths with MTUs below this.  People shouldn't use
such short PMTU links in the DFZ and shouldn't place their ITRs or
ETRs anywhere where there are such short PMTU links between them and
the DFZ.

My view is that for IPv6, RFC 1981 is an excellent system.

>From your research (msg05910), it seems that the current state of
PMTUD in IPv4 is a shambles - with some networks blocking PTBs, some
tunnels (or combinations of tunnels) not generating PTBs and with
some hosts ignoring PTBs, or not responding properly to them.  Also
some hosts send DF=0 packets of 1470 bytes (Google at least).

As far as I know, everything generally works because many hosts are
configured not to send packets long enough to run into PMTU problems.

>From the current basis, there's no way we can generally adopt
jumboframe paths in the DFZ as they appear.

Nor is there a way of introducing a tunneling-based CES architecture
which relies for its PMTUD on PTBs.  My IPTM approach and I think
your SEAL approach should be able to cope without relying on PTBs
from within the tunnel (but see my forthcoming message).  But what if
the ITRs (ITEs) can correctly sense the PMTU to the ETRs (ETEs) and
are unable to alter the sending host's packet lengths?

This could be due to:

  1  A PTB sent by the ITR is dropped by some filtering system
     before it can get to the SH.  This seems more likely if
     the ITR is outside the ISP or end-user network where the
     SH is located, than within it.

     If people filter PTBs from entering their system, or use an
     ISP which does the same, this is their own fault.

     The trouble is, they get away with it now, because the packets
     their hosts send are generally short enough not to run into MTU
     problems.  Unfortunately, such networks will perceive the
     difficulties resulting from their choices as being caused by
     sending packets to a host with an SPI ("edge") address in the
     CES architecture - and may not think it is their own filtering
     which is causing the trouble.

  2  The SH ignoring or responding incorrectly to the PTB.

     As above - they get away with it now, and would perceive the
     problem as being caused by the destination network which
     is using the CES system's "edge" space.

  3  The SH sends DF=0 packets which are too long, after
     encapsulation for some, many or all paths to ETRs.

     Again, as above, they get away with it now - but would blame
     the CES system, or rather the destination network which they
     may not know has adopted the "edge" space provided by the
     CES system.

     So does a CES system have to fragment every such packet?
     It seems so.

I think that to implement defensive, complex protocols such as RFC
4821 would be to accept and allow all these bad practices, and would
forever doom us to having to do extra work, and suffer extra
flakiness, just because of these bad practices.

RFC 4821 will always be a slower and less accurate method of
determining PMTU to a given host than RFC 1191 or RFC 1981.  It would
be subject to choosing a lower than proper value, if there was an
outage for a while and it interpreted this as a PMTU limitation.

 - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 8 Feb 2010 11:25:12 -0800

Hi Robin,

> Sent: Sunday, February 07, 2010 9:02 PM
> To: RRG
> Cc: Templin, Fred L

Yes, that's right. My mind has been so locked into the
IPv4 case that I forget that IPv6 does not allow
fragmentation in the network. So, you are right that
IPv6 as the outer protocol requires RFC1981 PMTUD
feedback from the network.

> I don't recall what these would be.

Out-of-band probes, e.g.

> Is there a mechanism for SEAL, in IPv4, to send these "expendable"

Yes; just set DF=1 in the outer IPv4 header and send it.

> since no ITE could know for sure whether ECMP/LAG is in use on the

Yes, I guess so.
> find that ETE's SEAL ID value in the snapshot taken 3 to 4 seconds ago.

OK, that sounds good on the ITE side but what about the
ETE side? If the ETE is going to be tracking the SEAL_ID
for this ITE, can't it similarly keep a sliding window
based on the packets received within the last ~3sec?

> not make the PTB exceed 576 octets:

How can a system with a going-in strategy of *throwing
away good data* be "excellent"?

> which properly support RFC 1191 PMTUD.

But, if any one of those tunnels uses IPsec encryption
or the like there is no opportunity for performing the
necessary translation function. So if there were a
decent segmentation and reassembly capability it seems
like IPsec implementations would be wise to use it.

> ETRs anywhere where there are such short PMTU links between them and

DF=0 has two benefits - it can allow good data to
get through in cases where DF=1 would have dropped
the data, and it can allows MTU indication through
to the ETE which can report back to the ITE.

> My view is that for IPv6, RFC 1981 is an excellent system.

How can a system that places blind faith in the network
be "excellent"?

> configured not to send packets long enough to run into PMTU problems.

Agree.

> jumboframe paths in the DFZ as they appear.

Also agree.

>      is using the CES system's "edge" space.

Cases 1 and 2 are a problem of the end site and not of
the ITE. If the ITE as an edge router of the site is
sending PTBs and the source host is not either not
getting them or not responding correctly then the end
site has to find the problems and fix them.

>      So does a CES system have to fragment every such packet?

The CES needs to select a "safe" size for performing inner
fragmentation while not choosing one so excessively small
as to invoke inner fragmentation very often.

> outage for a while and it interpreted this as a PMTU limitation.

My belief is that SEAL used correctly has a chance
to establish a minimum "Internet cell size" of 1500.
Then, if end systems adopt the strategy of "use
classic PMTUD for packets no larger than 1500 and
use RFC4821 or equivalent for packets larger than
1500" then we would have a path to an MTU-clean
Internet that can scale to any future packet sizes.

Fred

---

From: Robin Whittle <[email-masked]>
Date: Tue, 09 Feb 2010 13:34:01 +1100

Short version:   Discussion of SEAL and in particular of the
                 differing approaches Fred and I have about PMTUD.
                 Fred seems to support continued use of IPv4 DF=0
                 "fragmentable in the network" packets while I
                 think it should be deprecated.  (It is banned in
                 IPv6.)

                 Fred seems to support RFC 4821 PMTUD, at least for
                 packets longer than 1500 bytes.

                 I am opposed to RFC 4821 - because I think it is
                 over-defensively trying to cope with problems which
                 should be fixed, rather than tolerated and therefore
                 encouraged.

                 ETEs (ETRs) can't be expected to try to detect
                 packets arriving supposedly from legitimate ITEs
                 (ITRs) but which are sent by an attacker.

Hi Fred,

I plan to write a "IRON: SEAL summary V2" based on what I learnt from
your two recent on-list messages and one off-list message.

Here is my response to your first on-list message and some elements
of your off-list message.

>From your off-list message, I understand:

  * The longest IPv6 prefix length IRON/RANGER is intended to support
    is /56.  This sounds OK to me but at present I plan Ivip to work
    in integer units of /64.  Maybe that can be scaled back to /56
    nearer the time of deployment.

  * I will assume that the route redirect messages currently
    specified in RANGER (native ICMP IPv4 and IPv6 redirects)
    will be replaced by SEAL messages.  This will allow the
    inclusion of a caching time.

  * I understand that the North Island IRON router will first of
    all send the traffic packet to the Seattle router, based on
    its VP prefix 43.0.0.0 /16 in its FIB.

    The Seattle router somehow gets the packet to the correct IRON
    router in the South Island - and sends a redirect to the North
    Island IRON router.  That redirect causes the North Island IRON
    router to install a more-specific prefix in its FIB, for
    43.0.56.76 /30 with the path leading to the correct South
    Island router.  So subsequent traffic packets addressed to this
    prefix will be tunneled direct to the correct South Island router
    - this is RANGER's Route Optimization process.

  * I suggested, without any thought, a 10 minute time by which an
    IRON router would purge its FIB of any more-specific route
    for a particular EUN (End User Network) EID prefix if there
    was no traffic for it.  You suggest a 2 minute STALETIME - which
    seems fine to me.

    So a router which receives a SEAL redirect would maintain it in
    its FIB for the the caching time if packets keep arriving for it
    at intervals less than STALETIME or for the STALETIME if no
    packets arrive in that time.

  * I only have a rough idea how the EUN router creates "bubbles"
    with RANGER's (IPv6's?) RA (Router Advertisements) as a means of
    the one or more IRON routers of its ISPs securely registering
    themselves as the correct destination for packets whose
    destination address matches 43.0.56.76 /30.

    To what extent does this involve adding things to IPv6 - and
    to what extent is it practical, and with what additions, for
    IPv4?

  * I think we have very different goals regarding support for DF=0
    packets and for dealing with problems in the network, such as
    tunnels which don't support RFC 1191 / RFC 1981 PMTUD and
    filters which drop PTB packets.

    I think you support creating protocols which can cope with this
    stuff, including potentially very long DF=0 packets.

    I think these filtering and bad tunnel arrangements need to be
    fixed - and that DF=0 should be deprecated.

> feedback from the network.

I understand from what follows that you intend to rewrite SEAL to not
use the IPv6 Fragment Header, but to use an explicit SEAL header by
which the ITE can request the ETE acknowledge the receipt of the packet.

This would mean that if there were no PTBs arriving at the ITE due to
an MTU limit in a router in the ITE -> ETE path, that the ITE could
try various packet lengths until it found a length short enough to
avoid the lowest MTU limit.

>> I don't recall what these would be.

OK - I guess such as I just mentioned

OK.

>> since no ITE could know for sure whether ECMP/LAG is in use on the

OK but ...

>> would really appreciate it.

I am keen to read more about this.

Here you seem to agree with my suggestion that the "mid-layer
headers" are after the IPv4/6 header and before the SEAL header.

However, I see from the Figures 1 and 2 in:

that these "mid-layer" headers are after the SEAL header.

As far as I know, placing a UDP header after the SEAL header would
make it invisible to ECMP/LAG routers.  So I think that if the
packets have to be UDP packets to keep these ECMP/LAG routers happy,
the SEAL header and all that follows is part of the UDP payload.

Assuming you did this (and I am not suggesting you need to, since I
haven't yet read up on ECMP/LAG) and if you wanted to send IPv4 DF=1
packets in the tunnels, and if the MTU limiting router sent back a
PTB with only the IPv4 header and the next 8 bytes (the UDP header)
then the ITE could still authenticate the PTB by caching its 16 bit
UDP checksum.  This is because the UDP checksum would be affected by
the full 32 bit value of the SEAL_ID in the SEAL header, and the most
significant 16 bits of the SEAL_ID are in the returned IPv4 header's
Identification field anyway.

This might be handy for the IPv4 probing packets you mentioned above.

Regarding my suggestions about a timer-like algorithm by which the
ITE could decide which range of SEAL_IDs it had "recently" sent to an
ETE, for the purposes of authenticating messages from that ETE, or

> based on the packets received within the last ~3sec?

I don't recall any prior mention of the ETE attempting to decide
whether a packet apparently from an ITE was really from that ITE or not.

Maybe you could try this for ITE <-> ETE communications, but I think
it may be impossible for reasons similar or identical to the
arguments below.

Here is an argument about why it would be pointless or worse for the
ETE in an IRON/RANGER setting to try to use SEAL_ID to decide whether
or not to accept tunneled packets containing traffic packets.

 1 - Why it can't defend against an attacker.

     I assume the attacker's purpose is to get bogus packets to the
     Destination Host (DH).

     There's no need for an attacker to try to spoof a packet
     arriving from an active ITE - one which has recently
     tunneled traffic packets to this ETE.  If the attacker could
     get the ETE to accept them, then yes, the ETE would dutifully
     forward the packets towards the DH and the DH would get the
     bogus packet.

     However, the attacker doesn't need to do this in order to
     achieve his or her goal.  They can send a packet, tunneled
     just as if it were tunneled by some ITE which the ETE had
     not recently received packets from.  To the ETE, this would
     be a "new" ITE, and the SEAL_ID would be set to some random
     value by this ITE.

     The attacker could keep up this flow of packets and the
     ETE would keep accepting them.

     There's no point in making ETE's respond to such packets
     by sending a packet to the supposed ITE, and then by
     ignoring those packets if that ITE doesn't confirm it
     sent them.  This would lead to extra network traffic and
     the attacker could simply spoof the traffic packets
     themselves and allow them to be forwarded to a legitimate
     ITE.

 2 - Why it would be worse than useless.

     For the ETE to use SEAL_ID to accept or reject packets would
     lead to an easy DoS vulnerability.

     The attacker wants to clobber the ability of ITE X to tunnel
     packets to ETE Y, before ITE X has done so.  The attacker
     crafts a single packet which appears to ETE Y to have been
     sent by ITE X, with some random SEAL_ID value.  Then, the ETE
     would use this and reject any packets genuinely coming from
     ITE Y, because the real ITE Y would have chosen a different
     random starting point for its SEAL_ID.

Just as there is no way of fully preventing attackers sending packets
to any host now, with any source address they choose, nor is there
any way of preventing such problems with a CES system.

Furthermore, since in CES system, EUNs (End User Networks) using
"edge" addresses could be connected to any ISP, if these ISPs are
going to support these EUNs, then they need to allow the forwarding
of all packets from these EUNs which have source addresses matching
any "edge" address.

At present, ISPs can do their bit to stop spoofing by dropping
packets with source addresses not matching the prefix of the network
which they came from - but that can't be applied to packets with
"edge" addresses in a CES system, unless the ISP is prepared to be
extremely fussy and watch the mapping system to see which "edge"
prefixes are currently being mapped to an ETR which serves the
particular EUN.

> away good data* be "excellent"?

Because it is unreasonable of hosts or networks to emit some kinds of
packets - specifically any packet which is too long for the path to
the DH, and where the host or network expects the rest of the network
to fuss about chopping the packet into fragments, and then to carry
those fragments, and then for the DH to have to reassemble those
fragments - which is a complex task.

The Post-Office has maximum packet sizes and so does the IPv6
Internet.  I think DF=0 packets were always a mistake.  I guess it
made sense in the early days of very dumb hosts, but I think it is a
host responsibility to alter its behaviour so as to send packets
which are the right size for the network to deliver in a single piece.

If the network dutifully fragments DF=0 packets and attempts to
deliver them - as it does in IPv4 - then this allows and therefore
encourages hosts to send such packets which require this inefficient,
unfair and unreliable form of handling.

Likewise, if the network attempted to deliver packets which were too
big, by sending a PTB and then by fragmenting them so the final
packet could be assembled at the DH, this would allow and therefore
encourage SHs to keep sending such packets.  It would also involve
the DH getting some of the first packet in the second, since the SH
couldn't be sure that the longer packet was successfully reassembled
at the DH from its fragments.

The too-big packet should be thrown away - except for sending enough
of it back to the SH to enable the SH to authenticate the PTB.  Also,
since there can be multiple levels of tunnel, I think the PTB should
contain a few hundred bytes of the packet, so that the SH of an inner
tunnel, which is a router in the path of an outer tunnel, can
construct a PTB to its sending host (the ingress router of a still
further outer tunnel) which contains enough information not just for
authenticating the PTB, but to allow the final outer router of the
first tunnel to construct a PTB with sufficient length for the real
SH to authenticate.   Without this, ingress tunnel routers would need
to cache substantial parts of the packets they send, in order to be
able to generate a valid PTB.  I guess many IPv4 tunnels don't do
this, which is part of the reason for the lousy PMTUD situation today.

I think that DF=0 packets should be deprecated, and that the RFC 1191
PTB message should be revised to be the same as the ~540 bytes of
packet which are returned in an IPv6 RFC 1981 PTB.

I am opposed to what I think you are attempting with SEAL, in at
least some circumstances - of fragmenting or segmenting packets which
are too long, without an attempt to tell the sending host to send a
suitably shortened packet.

DF=0 packets do not allow the SH to be told this - so I think they
should be deprecated.  They were considered unworthy of inclusion in
IPv6 in the mid-1990s.  I think they should have been deprecated in
IPv4 from that time.

> like IPsec implementations would be wise to use it.

If an IPSEC ingress tunnel router can't decipher the initial part of
the packet returned in a PTB, then it needs to cache a copy of the
initial part of the packet it received as input to the tunnel, for
the purpose of constructing a PTB which would be recognised by a SH,
and furthermore which is long enough that if this tunnel is within
other tunnels, then that by the time this router's PTB contents have
been passed back up the line to the other ingress tunnel routers,
that the outer one still has enough of the original traffic packet to
be able to generate a valid PTB for the SH.

Without this, the IPSEC tunnel is not supporting RFC 1191 / RFC 1981
PMTUD.  I understand such support is mandatory, so how could a
self-respecting tunnel, IPSEC or not, not support it?

The world turns to rot if RFC 1191 and RFC 1981 PMTUD are not
supported.  People have to write messy things like RFC 4821 in an
effort to get around the mess caused by such tunnels, or by filtering
PTBs.

I think we should not be so defensive about stuff happening in the
network that we allow it and adapt to cope with it, when the
practices are fundamentally inefficient and do not support the best
way of doing things.

>> think it would be bad to expect ITRs and ETRs and thewhole CES

People pay to use services with tunnels.  If the tunnels screw up the
only efficient, practical, approach to PMTUD (RFC 1191 / RFC 1981)
then people shouldn't use such tunnels or pay for any service which
uses them.  If they do, then its all downhill from there - with
people fixing packet lengths to avoid trouble, and busying themselves
updating stacks and applications to no longer rely on the perfectly
good RFC 1191 / RFC 1981 PMTUD approach (if RFC 1191 had mandatory
~540 byte packet fragments).

> site has to find the problems and fix them.

I agree.

> as to invoke inner fragmentation very often.

Then you are always making things less efficient than you could be,
and ruling out the use of jumboframe paths in the DFZ until such time
that every path is jumboframe compatible - which may be never.

I am opposed to the CES scheme continually fragmenting packets if we
can possibly avoid it.  Maybe we have to for DF=0 packets which are
1470 bytes and the CES scheme can only get a little less than this
into each tunnel packet.  But this would be BAD considering how
Google sends out a lot of 1470 byte DF=0 packets.  I imagine Google
could be talked into lowering this or better still into using DF=1.

> to establish a minimum "Internet cell size" of 1500.

I can't see how you could do this, since there will always be 1500
limits in the DFZ, ISP and other networks for years to come, and
there will at times be tunneling, such as with PPPoE in DSL services.

> Internet that can scale to any future packet sizes.

I think this would be very messy.  Some hosts would be putting out 9k
byte packets and ignoring PTBs, just to see if they could get them to
the other host - and trying several times to make sure the failure
was due to a genuine MTU problem and not due to random packet loss.

 - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 9 Feb 2010 13:45:03 -0800

Robin,

More follow-up below:

> Sent: Monday, February 08, 2010 6:34 PM
> To: RRG
> Cc: Templin, Fred L

My understanding is that a router with multiple equal
cost routes toward the final destination can select
which next hop to use based on a hash of the packet's
(src,dst,sport,dpot,proto)-tuple (i.e., the packet's
"flow identifier"). I believe RFCs 2991 and 2992 may
have more to say on this.

> the SEAL header and all that follows is part of the UDP payload.

I am actually meaning to use UDP as an "outer"
encapsulation; not a "mid-layer" encapsulation.
So, the order of encapsulating headers on the wire
(left to right) would be:  IPvX/UDP/SEAL/IPvY.
So yes, the ECMP/LAG routers get to see the UDP
header in the clear and the SEAL header looks
like UDP data.

> Identification field anyway.

For IPv4, we will run with UDP checksums set to zero.
This just means that we may not be able to use the
"expendable packet" approach to probing when the UDP
header is required.

>      themselves and allow them to be forwarded to a legitimate

No; an attacker pretending to be an ITE first needs
to prove to the ETE that it is authorized to source
packets from a specific prefix. Secure Neighbor
Discovery (SEND) is the way that good ITE prove
their authorization to source packets.

> prefixes are currently being mapped to an ETR which serves the

What RANGER/VET/SEAL are providing is a way for the
ETE to do its own ingress filtering in case the ISPs
that contain end user networks are not doing ingress
filtering. This works when the ETE has a way of
authenticating that a specific ITE is authorized to
source packets from a given prefix.

> this, which is part of the reason for the lousy PMTUD situation today.

Having the PTBs contain enough of the too-big packet
may not be very helpful if some of the nested tunnels
use IP encryption. In that case, there may be no
opportunity to transcribe the PTB into a corresponding
message to send to the prior tunnel endpoint in the
nesting. This may be a minor point, however, since
the ITEs can still cache the new MTU for use when
later large packets arrive. So, I don't think there is
a need for the ITE to keep old packets around just a
PTB needs to be generated later.

> IPv6 in the mid-1990s.  I think they should have been deprecated in

I'm afraid I must continue to disagree. Your line of thinking
seems to be paralleling that of the Internet architects of the
1980's who rushed to proclaim "Fragmentation Considered Harmful".
IMHO, that was an all-or-nothing proclamation that did not take
into scenarios in which fragmentation is beneficial. AFAICT,
the community took the easy way out by seeking to abolish
fragmentation altogether. Instead, they could have done the
obvious, which was to *fix it* and *identify its use cases*.

> be able to generate a valid PTB for the SH.

But, it seems that tunnels are more and more taking
the "lazy" approach by simply discarding initial
PTBs silently then generating PTBs when subsequent
large packets arrive. The theory being that since
the network might drop an initial PTB anyway, so why
not just have the ITE drop the initial PTB on purpose?
With a willingness to accept such loss, there is no
need to cache copies of packets.

> self-respecting tunnel, IPSEC or not, not support it?

Based on what I have heard, implementations routinely
use rate limiting for the PTBs they send the same as
for any ICMP message. So, already today widely deployed
implementations drop some/many PTBs.

> ~540 byte packet fragments).

Your sudden faith in the network amazes me, but IMHO
it is unfounded. The Internet is a wild and wolly
place; not neat and orderly.

> could be talked into lowering this or better still into using DF=1.

I don't think there is a way to stop websites from setting
DF=0 if they want to. There are more website than just
google that do this.

I am not sure as to whether SEAL needs to be tweaked to
give more efficient treatment of inner fragmentation for
DF=0 packets. That seems to me to only be an issue when
IPv4 is used as EID space; when IPv6 is used as EID space,
DF=1 always.

> there will at times be tunneling, such as with PPPoE in DSL services.

Well, I have been told that links in the DFZ by and large
set an MTU of ~4KB or larger (I have no evidence of this).
So, if the core can handle 1500 w/o fragmentation then
we should be able to get the edges to also handle 1500
even if it requires a little bit of fragmentation.

Thanks - Fred

---

From: Robin Whittle <[email-masked]>
Date: Wed, 10 Feb 2010 12:40:19 +1100

Short version:      Tunnels ingress routers accepting a PTB packet
                    from within the tunnel and updating their MTU
                    for the tunnel - but not sending a PTB.  Then
                    they send a PTB for the second too-long traffic
                    packet.

                    Three nested tunnels like this means the SH has
                    to send four too-long packets before it gets
                    a PTB.

                    I doubt there is no such a thing as "a little
                    bit of fragmentation".

                    Fred suggests an ETE could decide whether or
                    not to accept a traffic packet tunneled by an
                    ITE based on the ETE determining whether or
                    not that ITE was authorised to handle packets
                    whose source address is that of the traffic
                    packets.  I can't imagine a practical way of
                    doing this.

Hi Fred,

Thanks for your continued conversation, from which I have learned a

> have more to say on this.

Thanks for these pointers.  I will check out:

  Multipath Issues in Unicast and Multicast Next-Hop Selection
  http://tools.ietf.org/html/rfc2991

  Analysis of an Equal-Cost Multi-Path Algorithm
  http://tools.ietf.org/html/rfc2992

Searching for "Link Aggregation Group" at the IETF site turns up
a number of documents.

> header in the clear and the SEAL header looks

OK.  I see after Figure 2 in:

the third dot point mentions UDP as an outer layer header.  I
suggest you add a note for why this would be needed, such as
to support ECMP/LAG or for any other reason.

OK.

> their authorization to source packets.

I don't understand this at all.  How can an ETE firstly know
that the tunneled packet it received really came from the ITE
whose IP address is in the outer source field?  This would
require it send a message to that IP address and get back a
response it could authenticate as resulting from its message.

This is not a practical thing to do when an ETE first gets a
packet.

> source packets from a given prefix.

I don't see a practical way an ITE could prove to any ETE
that it was authorised to send packets with a particular source address.

Theoretically, perhaps, the ITE might be able to establish that
it was located in a particular ISP's network or whatever and that
network was advertising a bunch of prefixes - so it would be OK for
the ITE to tunnel traffic packets to any ETE where the source address
of those traffic packets matched one of these ISP prefixes.

But what if the ISP's network includes one or more EUNs (end-user
networks) using "edge" space managed by IRON-RANGER?

These networks will be sending out packets with their own "edge"
addresses in the source field - and the ITE will be handling these
packets.  There could be thousands of such EUNs, each using one or
more "edge" prefixes - the total set of such prefixes would be large
and subject to considerable churn.

Even if the ITE could somehow "prove" all this, via some
PKI-compatible signed message, the message would be excessively long
and there's no way an ETE should hold onto a traffic packet and have
to check the authority of the ITE to send one with its particular
source address.

> use IP encryption. 

To support the proposal I made, if an IPSEC ingress tunnel router
(ITE in SEAL) can't decipher the first part of its own emitted packet
to create a valid PTB, then it should cache enough of the packet to
do so.

Below we discuss the idea of the ITE not sending a PTB, but updating
its MTU value, so that if (as would often be the case) the next
traffic packet exceeds that size, that it would generate a PTB from
that packet instead.

> In that case, there may be no

This "PTB on the second too-large-packet" arrangement is inexpensive
and in some ways appears to be adequate.

However, if there were three nested tunnels which did this, with a
single MTU limit in a router along the inner tunnel path, then it
would take several traffic packets being sent by the SH before the SH
would get a PTB:

  Packet 1        PTB-1 generated by router in inner tunnel path.

                  ITE of inner tunnel receives the packet, adjusts
                  its MTU value and does not generate a PTB.

  Packet 2        PTB-2 generated by ITE of inner tunnel.

                  ITE of middle tunnel receives the packet, adjusts
                  its MTU value and does not generate a PTB.

  Packet 3        PTB-3 generated by ITE of middle tunnel.

                  ITE of outer tunnel receives the packet, adjusts
                  its MTU value and does not generate a PTB.

  Packet 4        PTB-4 generated by ITE of outer tunnel - goes
                  to sending host, which adjusts its MTU.

  Packet 5        This has the correct length to get through to the
                  destination host.

So three tunnels which don't immediately send a valid PTB means three
more full length packets the SH has to send before it gets a PTB.

> obvious, which was to *fix it* and *identify its use cases*.

I am not expecting you to agree - I was just stating my beliefs.

I don't see how any attempt to fragment DF=0 packets could be
regarded as a long-term solution.  It should be the responsibility of
hosts not to send packets which the network can't handle efficiently
and reliably.  This is because it is a lot easier, overall, for the
host to do this small amount of work than to continually burden
routers with fragmentation tasks and then other routers with handling
more packets, which winds up being still more total data due to the
need for one or more extra headers.  Then there is the worsening of
reliability, and consequently the need to retransmit.

Can you suggest how you think DF=0 packets should be handled, in the
long term?

Other than a tunnel which segments and reassembles ordinary length
(~1500 byte) packets due to the particular link technology using very
small packet sizes, can you suggest when, in general, long DF=1
packets should be segmented - and how this is overall the most
efficient, robust, etc. approach compared to sending a PTB and
requiring the host to emit shorter packets?

> need to cache copies of packets.

Its a slippery slope which leads to the above scenario of nested
tunnels requiring a bunch of too-long packets before the SH gets a PTB.

I wonder if the tests done recently by Ben Stasiewicz:

might have been affected by this.  Matthew Luckie has written to the
RRG about this research.

> implementations drop some/many PTBs.

Yes - I think this PMTUD stuff is about as big a can of worms as
scalable routing, perhaps also deserving of a conference and
concerted research effort to decide what to do.

The trouble is, I believe the only practical scalable routing
proposals which don't require updates to all DFZ routers will require
tunneling, and such proposals (LISP, IRON-RANGER and Ivip) would run
straight into these PMTUD problems.   Even without a scalable routing
solution, the IPv4 and IPv6 Internets seem to be boxed into current
~<=1500 byte packet sizes, with no opportunity to break out of this
and use jumboframe ~9k MTU paths as they appear across the DFZ.

> place; not neat and orderly.

I don't have faith in it - I just think it would be easier to fix it
than to route around it.

I don't have complete faith in passenger jets or their ability to be
defended against terrorists.  (Though since flying to Australia from
England in a Boeing 707 in 1961 as a child, I feel happiest flying
one of your company's aircraft.)

No-one has complete faith in the ability to ensure passenger jets
from killing all on board.  However, collectively, we figure it is
easiest to do our best to make them safe and use them, rather than go
by bus, car or ship.

This is not argument by analogy - just demonstrating that I don't
have to think the Internet is perfectly orderly to decide that it is
easier and better for us all to work to make it more orderly, than to
take an overly defensive path around the disorder (and so allow it to
remain and grow), which is how I assess the task of converting all
stacks and application packetization layers to RFC 4821 PMTUD.

> DF=0 if they want to. There are more website than just

I think it will be easier overall to change this behaviour than to
have routers forever fragmenting packets and then have following
routers needing to carry two, with consequent inefficiencies and
greater chance of packet loss.

> IPv4 is used as EID space; when IPv6 is used as EID space,

I think it is well worth crafting protocols to work well with IPv4,
given it will be decades, if ever, before IPv6 or some other
alternative takes most of the traffic away from the IPv4 Internet.

> even if it requires a little bit of fragmentation.

I don't think there's such a thing as "a little bit of
fragmentation".  Splitting large numbers of traffic packets into two
packets, and then transporting and reassembling them is at least
twice the work, in many respects.

 - Robin