Subject: Re: [Int-area] I-D Action: draft-ietf-intarea-frag-fragile-06.txt

From: Ole Troan <[email-masked]>
Date: Fri, 1 Feb 2019 09:42:07 +0100

Tom,

> It does seem that Cisco configuration manuals and marketing materials
> are the best source of information on virtual reassembly.
> 
> https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_frre/configuration/xe-3s/frre-xe-3s-book/virt-frag-reassembly.html
> is a little more interesting in that it provides a few more details.
> In particular the requirement that all fragments must traverse the
> same intermediate device is mentioned:
> 
> "The reassembly process requires all fragments within an IP datagram.
> If fragments within an IP datagram are sent to different devices due
> to load balancing VFR may fail and fragments may be dropped.”

I can’t tell if you are actually interested in how it works, or if you are trying to make a point.
Assuming the former, here is an implementation:
https://github.com/FDio/vpp/blob/master/src/plugins/map/ip4_map.c#L517

if first fragment in chain
  found = lookup 4 tuple in reassenbly cache
  if found
     forward buffered packets
  else
    create session state entry in reassembly cache
    forward packet
else
  found = lookup 4 tuple in reassembly cache
  if found
    forward packet
  else
    buffer packet

Ole

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Fri, 1 Feb 2019 15:39:15 +0000

Hi Tom,

> -----Original Message-----
> From: Tom Herbert [mailto:[email-masked]]
> Sent: Thursday, January 31, 2019 5:17 PM
> To: Templin (US), Fred L <[email-masked]>
> Cc: Joe Touch <[email-masked]>; Ron Bonica <[email-masked]>; int-area <[email-masked]>
> Subject: Re: [Int-area] I-D Action: draft-ietf-intarea-frag-fragile-06.txt
> 
> On Thu, Jan 31, 2019 at 5:03 PM Templin (US), Fred L
> <[email-masked]> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Int-area [mailto:[email-masked]] On Behalf Of Tom Herbert
> > > Sent: Thursday, January 31, 2019 4:40 PM
> > > To: Joe Touch <[email-masked]>
> > > Cc: Ron Bonica <[email-masked]>; int-area <[email-masked]>
> > > Subject: Re: [Int-area] I-D Action: draft-ietf-intarea-frag-fragile-06.txt
> > >
> > > On Thu, Jan 31, 2019 at 3:10 PM Joe Touch <[email-masked]> wrote:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 2019-01-31 13:56, Tom Herbert wrote:
> > > >
> > > > On Thu, Jan 31, 2019 at 7:59 AM Joe Touch <[email-masked]> wrote:
> > > >
> > > >
> > > > The problem with dropping the entire paragraph is the section title - talking about stateless firewalls begs the question of
> stateful
> > > ones. This is reinforced later in the recommendations. The sentences you remove were the only thing that tied the two together,
> > > which IMO is important.
> > > >
> > > >
> > > > Joe,
> > > >
> > > > The term "Stateless firewalls" is unambiguous in this context. In a
> > > > stateless firewall, each packet is inspected and judge based solely on
> > > > it's content.
> > > >
> > > >
> > > > My statement above has no relation to any potential ambiguity in the term.
> > > >
> > > > ---
> > > >
> > > > However, the term stateless is inaccurate in a few places:
> > > >
> > > > (Sec 4.6) NAT is a stateful procedure for an otherwise stateless protocol as well. The same could be argued for load balancers
> that
> > > retain similar state through a connection for a flow (i.e., not just hashing the flow or tuple, but doing round-robin per-flow/tuple
> > > 'sticky' routing)
> > > >
> > > > (Sec 7.3) The problem is not just stateless middle boxes, but also certain stateful ones (e.g., NATs, some load balancers, etc.)
> > > >
> > > > ---
> > > >
> > > > Thus "stateful" actually is both ambiguous and inaccurate here.
> > > >
> > > > You appear to want to distinguish between the state needed for virtual reassembly and the state needed to maintain NAT
> > > translations or sticky round-robin load balancing, but there's no simple term that differentiates them. They're both content-
> > > dependent, context-dependent, and stateful.
> > > >
> > > >
> > > > Further, as you note there are no *specified* algorithms for virtual reassembly, nor are there any *specified* for NAT
> translation
> > > table maintenance or sticky load balancing. Everyone comes up with their own and the basic concept is well enough defined as to
> not
> > > need a specification.
> > > >
> > > In that case, if it's so obvious and well defined then there shouldn't
> > > be any issue in either providing a reference to a description or
> > > specifying it in the draft (if authors do choose discuss virtual
> > > reassembly in the draft).
> >
> > I have been telling everyone about virtual fragmentation reassembly for
> > years. You will find it in many of my docs. Here is what cisco says about it:
> >
> >
> https://www.cisco.com/c/en/us/td/docs/ios/sec_data_plane/configuration/guide/12_4/sec_data_plane_12_4_book/sec_virt_frag_
> reassm.html
> 
> Hi Fred,
> 
> to load balancing VFR may fail and fragments may be dropped."

Yes, of course the fragments all need to go through the same destination where
VFR is caching the fragments. That is assured due to the IP destination address
for unicast, but not necessarily for anycast. But, for anycast, the considerations
are no different for middleboxes than they are for anycast final destinations.

An analogy for VFR is that the middlebox receives the pieces of a puzzle from
the network. It checks each piece to make sure it is OK and holds them until
all pieces have arrived, but then instead of assembling the puzzle it forwards
the pieces on to the next hop destination, with the realization that there could
be NATs-within-NATs. Then, when the final destination receives all pieces it
assembles the puzzle.

I don' t think there needs to be a spec for this because the behavior is
inherently understood from the title (*Virtual* Fragment Reassembly).

Thanks - Fred

> Tom
> 
> >
> > Fred
> > >
> > > Tom
> > >
> > > > Joe
> > >
> > > _______________________________________________
> > > Int-area mailing list
> > > [email-masked]
> > > https://www.ietf.org/mailman/listinfo/int-area

---

From: Joe Touch <[email-masked]>
Date: Fri, 1 Feb 2019 07:44:50 -0800

Virtual reassembly means “forwards *as if* reassembled”, without actually reassembling. It’s actually not all that much different from the way NATs or unidirectional firewalls work for TCP. 

>  found = lookup 4 tuple in reassembly cache

The only addition to the pseudocode above is to timeout the cache entries based on the “expected reordering” (see RFC 6864). That timeout performs the same function as the TCP cache entry timeout in those NATS/firewalls.

For TCP, a FIN-ack or ack after FIN (depending on who closes the connection) can flush the entry before a timeout. For fragment reassembly, the middle box CAN keep track of the fragments seen and flush the entry when a complete “virtually reassembled” packet is seen, but that’s probably overkill vs. a simple timer (actually, a packet count can suffice).

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 1 Feb 2019 07:47:36 -0800

> On Feb 1, 2019, at 7:39 AM, Templin (US), Fred L <[email-masked]> wrote:
> 
>> https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_frre/configuration/xe-3s/frre-xe-3s-book/virt-frag-reassembly.html <https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_frre/configuration/xe-3s/frre-xe-3s-book/virt-frag-reassembly.html>
> VFR is caching the fragments. 

This isn’t any different from the fact that middle boxes don’t react well to multipathing of IP packets that ignores flow info, e.g., when all packets of a TCP connection or UDP flow don’t go through the same middle box.

As per my other message, the per-packet processing load isn’t all that different EXCEPT to queue fragments before the first-frag is seen.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Fri, 1 Feb 2019 08:15:24 -0800

> inherently understood from the title (*Virtual* Fragment Reassembly).
Fred,

I don't see this. It is not at all clear from three words that
everyone could draw the same conclusions on what the protocol is and
how it works. Your description above about the interaction between
multicast and unicast is not obvious from the title. Neither is it
obvious there is on only compliant way to implement the protocol
(there are at least three variants that have been implemented with
varying degrees of interoperability problems). I still maintain that
if virtual reassembly were to be used in a normative context, it needs
a normative definition. If nothing else, that ensures there is no
ambiguity in what the protocol is, how it works, and what
implementations need to do to be interoperable-- after all that is the
goal of formal protocol specifications.

Tom

---

From: Joe Touch <[email-masked]>
Date: Fri, 1 Feb 2019 08:23:00 -0800

This isn’t a formal protocol spec and there is no equivalent detailed specification for stateful firewalls or load balancers. 

> On Feb 1, 2019, at 8:15 AM, Tom Herbert <[email-masked]> wrote:
>

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Fri, 1 Feb 2019 16:25:27 +0000

Hi Tom,

> Sent: Friday, February 01, 2019 8:15 AM

I said "anycast"; not multicast. Multicast is different in that all nodes that
are members of the group will receive all fragments. With anycast, only
one node out of potentially many gets the fragments, and it may not
always be the same node for all fragments.

> goal of formal protocol specifications.

The only requirement is that the node receiving the fragments and doing
VFR internally has to emit them looking just like they did originally - albeit
with a potentially different IP destination address. What goes on inside the
box in terms of VFR is completely implementation-dependent, and not a
topic for standardization.

Thanks - Fred

---

From: Tom Herbert <[email-masked]>
Date: Fri, 1 Feb 2019 09:07:19 -0800

> topic for standardization.
Fred,
i
That is only true if the implementation is standards conformant and
doesn't break interoperability. We've already identified one instance
of non-conformance with virtual routing that breaks interoperability
(namely the aforementioned requirement that packets of flow must
traverse the same device). There is another variant of virtual routing
that has been deployed in which the implementation only tracks
reassembly state, but does not actually queue fragments. As long as
the first fragment is received by the device first, this works. This
probably works probably 99.9% of the time, and represents a huge cost
savings in terms of memory. There is even a simpler variant of that in
which an implementation just creates a state for <src, dst, id>,
allows non-first fragments to pass, and then destroys the state when
MF is not set in a fragment. This is even more savings in memory,
substantially reduces CPU overhead, and probably works 99.8% of the
time. Of course, IP does not require in order delivery, so like the
requirement above, such implementations are technically non-conformant
and inevitably will break interoperability for someone.

Tom

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Fri, 1 Feb 2019 18:08:14 +0000

Hi Tom,

We need to realize that there are two cases for middleboxes - transparent
and non-transparent.

For transparent middleboxes, the IP destination address *is not* the address
of the middlebox. Instead, IP routing inside of the middlebox determines the
next hop toward the destination. In the case of transparent middleboxes, unless
the network has been physically deployed to avoid any possibility of multiple
paths, you are right that some fragments might traverse middlebox A and
others traverse middlebox B. So, transparent middleboxes should provide a
configuration option to turn VFR on or off according to the deployment model. 

For non-transparent middleboxes, the IP destination address *is* the address
of the middlebox. In that case, state inside of the middlebox causes the IP
and possibly also higher layer headers to be rewritten with values that
correspond to either the next hop middlebox or the final destination.
So, in the case of non-transparent middleboxes, the node acts as a host
on its upstream interface and a router on its downstream interface. And,
hosts can be expected to receive all fragments of a fragmented IP packet
that are addressed to itself, therefore VFR is acceptable. The only time
that is not true is when the address of the middlebox is an anycast address.
In that case, VFR should be turned off.

Thanks - Fred

> Sent: Friday, February 01, 2019 9:07 AM
> > > > > > > table maintenance or sticky load balancing. Everyone comes up with their own and the basic concept is well enough
> defined as
> > > to
> > > > > not
> > > > > > > need a specification.
> > > > > > > >