Subject: [Int-area] WG Adoption Call: IP Fragmentation Considered Fragile

From: Wassim Haddad <[email-masked]>
Date: Tue, 24 Jul 2018 19:42:59 +0000

Dear all,

We would like to start a WG adoption call for draft-bonica-intarea-frag-fragile (“IP Fragmentation Considered Fragile”).

https://www.ietf.org/id/draft-bonica-intarea-frag-fragile-03.txt

Please indicate your preferences on the mailling list. The deadline is August 10th.

Thanks,

Juan & Wassim

---

From: "Black, David" <[email-masked]>
Date: Tue, 24 Jul 2018 19:57:47 +0000

I support intarea WG adoption of this draft - a BCP in this space is highly appropriate and likely to be quite useful.

Thanks, --David (as an individual, not as TSVWG chair)

> -----Original Message-----
> From: Int-area [mailto:[email-masked]] On Behalf Of Wassim
> Haddad
> Sent: Tuesday, July 24, 2018 3:43 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: [Int-area] WG Adoption Call: IP Fragmentation Considered Fragile
> 
> Dear all,
> 
> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Tue, 24 Jul 2018 22:54:04 +0000

I have an observation that I would like to see addressed in the document. Some applications
(e.g., 'iperf3' and others) actually leverage IP fragmentation to achieve higher data rates than
are possible using smaller (but unfragmented) whole packets.

Try it - by default, iperf3 sets an 8KB UDP packet size and allows packets to fragment across
paths that support only smaller MTUs. I have seen iperf3 exercise IP reassembly at line rates
on high-speed links, i.e., it shows that reassembly at high rates is feasible.

We know from RFC4963 that there are dangers for reassembly at high rates, but there are
applications such as iperf3 that ignore the "SHOULD NOT" and leverage IP fragmentation
anyway. So, should the "SHOULD NOT" have an asterisk?

Thanks - Fred

> From: Int-area [mailto:[email-masked]] On Behalf Of Wassim Haddad
> Sent: Tuesday, July 24, 2018 12:43 PM
> To: [email-masked] <[email-masked]>

---

From: Tom Herbert <[email-masked]>
Date: Tue, 24 Jul 2018 16:46:16 -0700

> anyway. So, should the "SHOULD NOT" have an asterisk?
Fred,

My reading of the draft is that IP fragmentation is fragile on the
open Internet and should be avoided for applications that run over the
Internet. That doesn't mean that fragmentation should be avoided in
all use cases. In particular, if fragmentation is used in a closed
network with low loss and has appropriate security measures in place,
then it can be beneficial. I suspect that describes the network that
your're running iperf in. If this interpretation of the draft's intent
is correct, maybe there could be some words to clarify that.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 25 Jul 2018 15:12:51 +1200

Those words are in RFC2119 already:

>> 4. SHOULD NOT   This phrase, or the phrase "NOT RECOMMENDED" mean that
>>    there may exist valid reasons in particular circumstances when the
>>    particular behavior is acceptable or even useful, but the full
>>    implications should be understood and the case carefully weighed
>>    before implementing any behavior described with this label.
   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 25 Jul 2018 15:37:48 +1200

+1 for adoption.

However, I am a bit concerned about this key recommendation:

>    Application developers SHOULD NOT develop applications that rely on
>    IPv6 fragmentation

It isn't obvious to me that this is an algorithmic requirement. If the
application runs over TCP, how does the developer ensure that TCP will
use an MSS that avoids the need for fragmentation? (The UDP situation
is a bit clearer, but RFC8085 introduces considerable complexity, unless
we accept that the Internet MTU for UDP is 576 or 1280.)

Regards
   Brian

---

From: "Eric Vyncke (evyncke)" <[email-masked]>
Date: Wed, 25 Jul 2018 08:16:52 +0000

+1 for WG adoption

The document will need some review though (cfr Brian's valid point below)

-éric

On 25/07/2018, 05:38, "Int-area on behalf of Brian E Carpenter" <[email-masked] on behalf of [email-masked]> wrote:

    +1 for adoption.

    
    
    Regards
       Brian

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Wed, 25 Jul 2018 14:34:26 +0000

Hi Tom,

You are right that I have been experimenting with iperf3 in a closed network, but
it would work just fine (and exhibit the same behavior) on the open Internet.

The thing about iperf3 is that it sends random data that is simply recorded for
performance metric purposes and then dropped. So, iperf3 doesn't care if it
is causing IP reassembly-related packet loss or even corruption.

So, there is a class of applications that knowingly engage IP fragmentation
without fear of what might go wrong during the subsequent reassembly. 
This class of applications might bear mention in the draft.

Thanks - Fred

> From: Tom Herbert [mailto:[email-masked]]
> Sent: Tuesday, July 24, 2018 4:46 PM
> To: Templin (US), Fred L <[email-masked]>
> Cc: Wassim Haddad <[email-masked]>; [email-masked] <[email-masked]>; [email-masked]
> Subject: Re: [Int-area] WG Adoption Call: IP Fragmentation Considered Fragile
>

---

From: Joe Touch <[email-masked]>
Date: Wed, 25 Jul 2018 19:14:42 -0700

Hi, all,

I still think it would be useful for this doc to describe how tunnels interact with fragmentation (per draft-ietf-intarea-tunnels), which seems to be something I’ve noted several times before.

I’m also still not thrilled with the title I’d be happier with “IP fragmentation still not supported per requirements”, and I’d have to see where this goes with final recommendations.

But I agree *some* statement is worthwhile here. My primary concern is that if we’re not careful, endorsing the status quo will only ensure nothing changes.

So I sincerely hope that some of the strongest recommendations here are that both direct IP devices and tunnel ingress/egress devices need to do a better job of supporting fragmentation, and that protocol/device designers SHOULD avoid mechanisms that are not compatible with fragmentation (e.g., NAT or DPI without doing reassembly first).

Joe

> On Jul 24, 2018, at 12:42 PM, Wassim Haddad <[email-masked]> wrote:
> 
> Dear all,
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 26 Jul 2018 09:01:51 +0200 (CEST)

On Tue, 24 Jul 2018, Templin (US), Fred L wrote:

> Try it - by default, iperf3 sets an 8KB UDP packet size and allows 
> packets to fragment across paths that support only smaller MTUs. I have 
> seen iperf3 exercise IP reassembly at line rates on high-speed links, 
> i.e., it shows that reassembly at high rates is feasible.
>
> We know from RFC4963 that there are dangers for reassembly at high 
> rates, but there are applications such as iperf3 that ignore the "SHOULD 
> NOT" and leverage IP fragmentation anyway. So, should the "SHOULD NOT" 
> have an asterisk?

The iperf3 usage of fragments for UDP testing seems to be platform 
specific, at least that's what I've seen. Behaviour has been different on 
MacOS compared to Linux.

Anyhow, I believe we should keep the "SHOULD NOT". This allows 
applications that feel they have a good reason to do fragmentation to do 
so, but doesn't disallow it.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 26 Jul 2018 09:03:25 +0200 (CEST)

> Please indicate your preferences on the mailling list. The deadline is 

I support the adoption of this document, I consider it an important piece 
of advice to application implementers.

---

From: "Black, David" <[email-masked]>
Date: Thu, 26 Jul 2018 14:27:32 +0000

> I still think it would be useful for this doc to describe how tunnels interact
> with fragmentation (per draft-ietf-intarea-tunnels), which seems to be
> something I’ve noted several times before.

I believe that Ron's already promised to add text on this ...

Thanks, --David

> From: Int-area [mailto:[email-masked]] On Behalf Of Joe Touch
> Sent: Wednesday, July 25, 2018 10:15 PM
> To: Wassim Haddad
> Cc: [email-masked]; [email-masked]
> I still think it would be useful for this doc to describe how tunnels interact
> with fragmentation (per draft-ietf-intarea-tunnels), which seems to be
> something I’ve noted several times before.
> 
> I’m also still not thrilled with the title I’d be happier with “IP fragmentation
> still not supported per requirements”, and I’d have to see where this goes
> with final recommendations.
> 
> But I agree *some* statement is worthwhile here. My primary concern is
> that if we’re not careful, endorsing the status quo will only ensure nothing
> changes.
> 
> So I sincerely hope that some of the strongest recommendations here are
> that both direct IP devices and tunnel ingress/egress devices need to do a
> better job of supporting fragmentation, and that protocol/device designers
> SHOULD avoid mechanisms that are not compatible with fragmentation (e.g.,
> NAT or DPI without doing reassembly first).
> 
> Joe
> 
> > We would like to start a WG adoption call for draft-bonica-intarea-frag-
> fragile (“IP Fragmentation Considered Fragile”).
> >

---

From: Tom Herbert <[email-masked]>
Date: Thu, 26 Jul 2018 07:56:48 -0700

On Thu, Jul 26, 2018 at 12:01 AM, Mikael Abrahamsson <[email-masked]> wrote:
>> Try it - by default, iperf3 sets an 8KB UDP packet size and allows packets
>> to fragment across paths that support only smaller MTUs. I have seen iperf3
>> exercise IP reassembly at line rates on high-speed links, i.e., it shows
>> that reassembly at high rates is feasible.
>>
>> We know from RFC4963 that there are dangers for reassembly at high rates,
>> but there are applications such as iperf3 that ignore the "SHOULD NOT" and
>> leverage IP fragmentation anyway. So, should the "SHOULD NOT" have an
>> asterisk?
>
>
> The iperf3 usage of fragments for UDP testing seems to be platform specific,
> at least that's what I've seen. Behaviour has been different on MacOS
> compared to Linux.
>
Mikael,

I don't understand what would be platform specific. Can you elaborate?
iperf3 is sending UDP packets that are exceeding MTU and packets are
being fragmented. For IPv4, this means that DF is not set and packets
are being fragmented somewhere in the network, for IPv6 they are being
fragmented at the source. I don't believe path MTU discovery is in use
by iperf3, so if packets do exceed MTU then the only way they can can
be delivered is if fragmented.

> Anyhow, I believe we should keep the "SHOULD NOT". This allows applications
> that feel they have a good reason to do fragmentation to do so, but doesn't
> disallow it.
>

The text is "Application developers SHOULD NOT develop applications
that rely on IPv6 fragmentation.". As Brian said, this requirement
does not describe the algorithm, nor is it obvious what it even means
for an application developer to knowingly develop an application that
relies on IPv6 fragmentation. Besides that, RFC8085, which is
referenced in section 5.2, already provides the recommendations that
applications should avoid fragmentation by limiting packer size or
doing PMTU discovery. I don't see how 7.1 of this draft adds anything
to that or is clarifying those recommendations.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Thu, 26 Jul 2018 08:28:04 -0700

> So I sincerely hope that some of the strongest recommendations here are that both direct IP devices and tunnel ingress/egress devices need to do a better job of supporting fragmentation, and that p…
I agree.

Specifically, I think there should be a requrement that intermediate
devices don't rely on doing DPI into transport layer, or if they need
it then they should do some sort of pseudo reassmbly as Joe alludes
to. Note that section 4.4 describes the problem of of fragmentation
going through a load balancing (e.g. ECMP) where transport ports are
used in the algorithm. This is solved in IPv6 by using flow label in
the hash instead of transport layer ports, so I think that use of flow
label for this purpose should be recommended somewhere in section 7.

Tom

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 26 Jul 2018 18:21:34 +0000

I agree with Tom and Joe. I don't think the document is far from being complete in
its current form, but I think addressing the points raised in these recent discussions
will bring about completion.

Thanks - Fred 

> From: Int-area [mailto:[email-masked]] On Behalf Of Tom Herbert
> Sent: Thursday, July 26, 2018 8:28 AM
> To: Joe Touch <[email-masked]>
> Cc: [email-masked] <[email-masked]>; [email-masked]
> > I still think it would be useful for this doc to describe how tunnels interact with fragmentation (per draft-ietf-intarea-tunnels), which
> seems to be something I’ve noted several times before.
> >
> > I’m also still not thrilled with the title I’d be happier with “IP fragmentation still not supported per requirements”, and I’d have to see
> where this goes with final recommendations.
> >
> > So I sincerely hope that some of the strongest recommendations here are that both direct IP devices and tunnel ingress/egress
> devices need to do a better job of supporting fragmentation, and that protocol/device designers SHOULD avoid mechanisms that are
> not compatible with fragmentation (e.g., NAT or DPI without doing reassembly first).
> >
> I agree.
>

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 26 Jul 2018 18:50:03 +0000

I also wonder if there is a new field of study that could stem from these
observations - maybe call it "Corruption-Tolerant Networking (CTN)"?

Fred

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 27 Jul 2018 10:21:20 +0200 (CEST)

> I don't understand what would be platform specific. Can you elaborate?

On one platform iperf3 sends MTU minus header sized UDP packet, without IP 
fragmentation (so ~1400 bytes UDP payload). On the other platform iperf3 
sends 8k UDP packets, fragmented by the IP stack if MTU is less than 8k.

I have no idea if this is an iperf3 version difference or platform 
specific, but this is what I see in my tcpdump:s.

---

From: Ole Troan <[email-masked]>
Date: Fri, 27 Jul 2018 14:37:34 +0200

Joe,

> I still think it would be useful for this doc to describe how tunnels interact with fragmentation (per draft-ietf-intarea-tunnels), which seems to be something I’ve noted several times before.

There's nothing intrinsic in tunnels that require network layer fragmentation. The current IP in IP / GRE etc tunnel implementations do, but in principle we could design tunnel fragmentation/reassembly above the network layer.

> I’m also still not thrilled with the title I’d be happier with “IP fragmentation still not supported per requirements”, and I’d have to see where this goes with final recommendations.

If we were to conclude that IP fragmentation is done at the wrong layer...

Cheers,
Ole

---

From: Fernando Gont <[email-masked]>
Date: Fri, 27 Jul 2018 14:38:26 +0200

Hi, Joe,

> But I agree *some* statement is worthwhile here. My primary concern is that if we’re not careful, endorsing the status quo will only ensure nothing changes.

FWIW, I see and understand your point. However, based on the operational
implications of fragmentation, it believe it is *extremely* unlikely
that the situation improves (if at all possible).

So my take is that this I-D rather than endorsing the status quo, is
essentially warning possible users of what it may happen with their
fragmented traffic.

Side coments:

It would all seem that there are two operating environments:
* Controlled environments, where you can somehow make all this work
* Public internet, which is more of a "fingers crossed" thing (if anything).

I'm not saying that I'm happy with the outcome, but rather that I think
that from an engenering point of view, it all looks like this ship has
sailed, and we should probably figure out how to deal with those cases
where fragmentation is actually needed.

Thanks!

Cheers,
-- 
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 27 Jul 2018 15:32:20 +0200 (CEST)

On Fri, 27 Jul 2018, Ole Troan wrote:

> There's nothing intrinsic in tunnels that require network layer 
> fragmentation. The current IP in IP / GRE etc tunnel implementations do, 
> but in principle we could design tunnel fragmentation/reassembly above 
> the network layer.

This is what I do with OpenVPN for instance. I tell it to application 
layer fragment at ~1400 bytes, so the resulting packets always are 
significantly below 1500 bytes. So even as the inside of the tunnel is 
9000 MTU clean, it never results in IP fragments to be transported.

---

From: Tom Herbert <[email-masked]>
Date: Fri, 27 Jul 2018 07:54:17 -0700

On Fri, Jul 27, 2018 at 6:32 AM, Mikael Abrahamsson <[email-masked]> wrote:
>> fragmentation. The current IP in IP / GRE etc tunnel implementations do, but
>> in principle we could design tunnel fragmentation/reassembly above the
>> network layer.

GUE defines that. Fragementation is done within the encapsulation
layer so that it is transparent to the network.
>
>
> This is what I do with OpenVPN for instance. I tell it to application layer
> fragment at ~1400 bytes, so the resulting packets always are significantly
> below 1500 bytes. So even as the inside of the tunnel is 9000 MTU clean, it
> never results in IP fragments to be transported.
>

I don't think that can be called a general approach. Setting MTU to
1400 bytes only avoids fragmentation if all the tunnel headers being
inserted are less than 100 bytes. If you start using more tunnel
options or the tunnel ingress inserts extension headers then that 100
byte budget may be exceeded may be exceeded. So a requirement to avoid
fragmentation in this manner would have to be that the MTU needs to be
low enough to account for all possible encapsulation overhead that may
be applied to a packet-- the emerging use of in-situ OAM, segment
routing, and even just switching from IPv4 to IPv6 for the underlay
puts downward pressure on MTUs. Lowering the MTU increase ratio of
overhead to user data thus making communications less efficient.

Fragmentation for tunneling is a special case since tunnels are often
used within a controlled network and precisley two fragments are
always generated. I know of at least one very large data center that
relies on fragmentation for tunneling. It seems to work fine in such
an environment and is preferable to lowering the MTU for everyone
(even non-tunnels) or turning on jumbo frames (complex to do at large
scale).

Tom

---

From: Tom Herbert <[email-masked]>
Date: Fri, 27 Jul 2018 08:15:39 -0700

> where fragmentation is actually needed.
Fernado,

Couldn't that same line of thinking be applied to several other
protocol features? So has the ship sailed for out ability to ever use
extension headers or any protocol other than TCP (and sometimes UDP)?
Maybe documents that recommend operational workarounds should
distinguish problems that inherent in the protocol from those that
have arisen because of non-conformant implementation. It's true that
calling implementations that probably won't help to fix what is out
there, but maybe this could help prevent new instances of
non-conformance.

Tom

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Fri, 27 Jul 2018 16:06:19 +0000

Hi Mikael,

> From: Mikael Abrahamsson [mailto:[email-masked]]
> Sent: Friday, July 27, 2018 1:21 AM
> To: Tom Herbert <[email-masked]>
> Cc: Templin (US), Fred L <[email-masked]>; [email-masked] <[email-masked]>; [email-masked]

Out of curiosity, what is the performance being reported by iperf3
on iOS? On linux, the performance for sending 1400 byte UDP datagrams
is less by an order of magnitude than sending 8K UDP datagrams even
though the 8K's incur fragmentation. Can you try boosting the UDP
datagram size to 8K on iOS (i.e., and let it fragment) and see what
performance is reported? Also, how does performance on iOS
compare to performance on linux for similar network configurations?

>From my testing, the reason for the big performance disparity on
linux is the size of the buffer being offered to the "sendto()" system
call. Offering small UDP datagrams means that only a small data
copy occurs per system call. Offering large UDP datagrams (and
therefore larger sendto() buffers) yields greater performance even
though IP fragmentation is exercised.

Thanks - Fred

---

From: Fernando Gont <[email-masked]>
Date: Fri, 27 Jul 2018 18:25:09 +0200

>> On 07/26/2018 04:14 AM, Joe Touch wrote:
[....]
> protocol features?

Yes, indeed.

> So has the ship sailed for out ability to ever use

It would seem that that's the case, yes.

> there, but maybe this could help prevent new instances of

I kind of agree with you. That path from spec to implementation is what
you'd call engineering (or what others might call "taking shortcuts", etc.).

For example, what happens with EHs has a lot to do with engineering:
they could be made to work (e.g., remove performance impact), but
devices would probably be more expensive. Folks buying gear wouldn't pay
for something that its not used in practice, and vendors wouldn't just
"improve" the boxes for free. -- yeah, you could argue that "hey, there
shouldn't be penalties for EHs, since they are part of the core IPv6
spec" -- but, while probably correct, that will not change reality.

Not that I like the situation, but... I think the least we can do is to
do a reality check wrt how things are supposed to work vs. how they
actually work.

For this particular case, this I-D makes that point for fragmentation: I
I think we all agree that fragmentation is fragile -- making that point
clear at least raises awareness of the problem, and might trigger some
action on the topic (whether to correct the issue, or to circumvent it).

Thanks,

---

From: Tom Herbert <[email-masked]>
Date: Fri, 27 Jul 2018 10:20:44 -0700

> spec" -- but, while probably correct, that will not change reality.
Fernando,

The irony is thatxtension headers (and alternative protocols as well),
including fragmentation, aren't supposed to have to "be made to work"
in intermediate devices. With the exception of Hop-by-Hop options they
are supposed to be ignored by design, and even in the case of
Hop-by-Hop options RFC8200 relaxed the requirements so that they can
be ignored. Extension headers are considered problematic only because
of ad hoc assumptions made for "value add", non-standard features
implemented in intermediate devices.

> action on the topic (whether to correct the issue, or to circumvent it).
Right, but I still think that we should be more clear about the root
origin of problems and blunt in requesting that non-conformant
implementations get fixed. For instance, as I mentioned the ECMP
hasing problem with fragmentation is entriely solvable if only
intermediate devices will use flow label instead of trying to find
ports for a hash. Fixing devices to support this reasonable and should
be low cost.  IMO, use of flow label for ECMP should be a strong
recommendation made in this draft.

Tom

---

From: Ole Troan <[email-masked]>
Date: Fri, 27 Jul 2018 20:24:12 +0200

> On 27 Jul 2018, at 19:20, Tom Herbert <[email-masked]> wrote:
> 
> implementations get fixed.

Barring bugs,  implementations work the way they do because customers have required them to have features that are used to parse deep into packets. 

IPv6 extension headers where to some extent designed to make it hard to parse in hardware. Not that, that helped. 

Yes, we agree on the root cause. But since these are legitimate features (from the perspective of whomever is operating the middlebox) this isn’t something that can be fixed by decree or by claiming it’s a bug. 

The tool we have in fighting ossification, is crypto. 

Cheers 
Ole

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 28 Jul 2018 08:12:32 +1200

>> On Fri, Jul 27, 2018 at 5:38 AM, Fernando Gont <[email-masked]> wrote:

...
> It would seem that that's the case, yes.

Thanks, another point we should cover in draft-carpenter-limited domains.

Fragmentation, (PL)PMTUD, extension headers, and innovative
L4 protocols are very possibly not viable on the open Internet.
At least, we can't assume that they will work on arbitrary paths.
Sad but apparently true.

   Brian

---

From: Ole Troan <[email-masked]>
Date: Fri, 27 Jul 2018 22:28:24 +0200

> On 27 Jul 2018, at 22:12, Brian E Carpenter <[email-masked]> wrote:
> 

Hasn’t this been discussed ad infinitum in the ossification work?
If you want to generalize, nothing is guaranteed to work across an arbitrary path in the Internet. 

So what? This is part of a tussle and it would be making a self fulfilling prophecy for us to take all policy based filtering or other brokenness into consideration when designing protocols. 

Ole

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 28 Jul 2018 08:51:32 +1200

> So what? This is part of a tussle and it would be making a self fulfilling prophecy for us to take all policy based filtering or other brokenness into consideration when designing protocols. 

So you'd prefer that we design protocols that can't possibly work?

Yes, it's a tussle. But sometimes it's better to know you're in a
fight; it gives you a better chance of winning ;-). Anyway, we're
off topic, sorry, my fault.

   Brian

---

From: Fernando Gont <[email-masked]>
Date: Fri, 27 Jul 2018 20:54:30 +0200

Tom,

On 07/27/2018 07:20 PM, Tom Herbert wrote:
[....]
> implemented in intermediate devices.

Please see:
<https://tools.ietf.org/html/draft-gont-v6ops-ipv6-ehs-packet-drops-03>

theory != practice. And no matter how right you might be, that doesn't
make the theory a reality.

> implementations get fixed.

That is certainly not going to happen. From the pov of the folks
operating the networks, there's nothing broken.

> For instance, as I mentioned the ECMP
> ports for a hash. 

Yes and no. There was at least a time in which the flow label wasn't set
at all, or even was mistaenly set (e value during 3WHS, a different
value afterwards). That means that you cannot really rely on it. If you
cannot rely on it, you need a back-up mechanism. And that mechanism is
inspecting the trasnport port numbers -- from that point of view,
there's not much sense in dong ECMP with the FL if you cannot rely on it
and somehow you'd have to be prepared to do it based on addresses and
port numbers.

> Fixing devices to support this reasonable and should

I wouldn't mind. However, that doesn't change the fact that
fragmentation is fragile.

That said, if you look at RFC7872, t all looks like anything that is EHs
is dropped to some extent (while not included in the RFC, I also mesured
IPsec, and you get similar numbers). So besides the issues that are
specific to fragmentation, anything that is EHs gets dropped here an
there. -- and ding ECMP with the FL will not change that.

(Again: not that I'm happy with the situation... but being unhappy will
not change reality anyway :-) )

Thanks,
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: Fernando Gont <[email-masked]>
Date: Fri, 27 Jul 2018 22:51:40 +0200

> So what? This is part of a tussle and it would be making a self fulfilling prophecy for us to take all policy based filtering or other brokenness into consideration when designing protocols. 

I see your point. However, how do you engineer something that "works" if
you ignore all brokenness etc.?

We do normally engineer protocols taking this things into account. e.g.

* PLPMTUD is a response to ICMP filtering
* ECN had a backup mechanism that would switch to non-ECN for cases
where e.g. firewalls were complaining about previously-unspecified bits
* Quic is most likely implemented over UDP to be able to survive NATs
and firewalls

Yes, and one hand is not nice to have to account for all types of
brokenness and filtering. OTOH, it would make any sense to enigneer a
protocol that only works on paper.

---

From: Ole Troan <[email-masked]>
Date: Fri, 27 Jul 2018 23:15:55 +0200

> On 27 Jul 2018, at 22:51, Fernando Gont <[email-masked]> wrote:
> 

To bring this back to the draft in question. Each type of “brokenness” needs to be considered on its own merits.

E.g brokenness as in network operator blocks all traffic from 1.0.0.0 is different from a poorly designed protocol. 

While fragmentation has issues because of network policy, the combination of PMTUD and fragmentation also exposes weaknesses in the design, which is something we should fix. 

Cheers 
Ole

---

From: Tom Herbert <[email-masked]>
Date: Fri, 27 Jul 2018 15:38:46 -0700

> and somehow you'd have to be prepared to do it based on addresses and

The fallback is to only hash over addresses. Hashing over ports is not
reliable and can be problematic in itself. For instance, wrt
fragmentation, some middleboxes will use ports in the first fragment
but fall back to hashing over addresses for rest of fragments. This
mean the first fragment almost always takes a different path which can
wreak havoc on down stream devices that aren't expecting that. So
maybe it's another recommendation in this draft that middleboxes don't
use port information from first fragment to do load balancing. This is
another relatively easy fix.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 28 Jul 2018 14:32:22 +1200

> The fallback is to only hash over addresses.

Hashing over addresses+flow-label is fine too. If the flow label
is zero, it's the same thing. If the flow label is set properly,
it's a better hash.

I believe this is covered in the various relevant RFCs already
(6437, 6438 and 7098).

Regards
   Brian

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 28 Jul 2018 08:13:52 +0200 (CEST)

On Fri, 27 Jul 2018, Tom Herbert wrote:

> I don't think that can be called a general approach. Setting MTU to 1400 
> bytes only avoids fragmentation if all the tunnel headers being inserted 
> are less than 100 bytes. If you start using more tunnel options or the 
> tunnel ingress inserts extension headers then that 100 byte budget may 
> be exceeded may be exceeded. So a requirement to avoid fragmentation in 
> this manner would have to be that the MTU needs to be low enough to 
> account for all possible encapsulation overhead that may be applied to a 
> packet-- the emerging use of in-situ OAM, segment routing, and even just 
> switching from IPv4 to IPv6 for the underlay puts downward pressure on 
> MTUs. Lowering the MTU increase ratio of overhead to user data thus 
> making communications less efficient.

A PLPMTUD mechanism for the tunnel as a complement to regular PMTUD would 
be enough. OpenVPN doesn't have this, if it did, I would use it.

> used within a controlled network and precisley two fragments are always 
> generated. I know of at least one very large data center that relies on 
> fragmentation for tunneling. It seems to work fine in such an 
> environment and is preferable to lowering the MTU for everyone (even 
> non-tunnels) or turning on jumbo frames (complex to do at large scale).

I run my tunnel over the Internet, using multiple ISPs. I prefer to avoid 
using IP fragments.

---

From: Joe Touch <[email-masked]>
Date: Sat, 28 Jul 2018 08:48:45 -0700

Hi, all,

Here’s the thing about fragmentation:

	1. all links have a maximum packet size
	2. all tunneling/encapsulation/layering increases payload size

1+2 implies there is always the need for fragmentation at some layer:

	3. fragmentation always splits info across packets

And there’s something important about layering:

	4. layering intends to isolate the behavior of one layer from another, such that
	it will always be impossible for an upper layer to know exactly what is going on below,
	i.e., to determine that limiting size across an entire path of possibly virtual tunnels

The next two are where we get into trouble:

	5. network devices increasingly WANT to inspect contents beyond the layer at which they are intended to operate
	6. inspecting contents ultimately means reassembly, at some level

Which brings us to the punchline:

	7. but network device vendors want to save money, so they don’t want to reassemble at any layer

So I agree, IP fragmentation has its flaws - but those flaws are created not only because it leaves out the transport port numbers, but also because DPI and NAT devices don’t reassemble. And they don’t because it’s cheaper to sell devices that say they run at 1 Gbps (e.g.) that don’t bother to reassemble.

I.e., it will never matter what layering we add to fix this - GRE, GUE, Aero, etc. - ultimately, we’re doomed to need fragmentation support down to IP exactly because:

	a. #1-4 mean we need frag/reassembly at any tunnel ingress
	b. vendors want to sell #5 at a price that is too low for them to support #6 (i.e., point #7)

So pushing this to another layer will never solve it. What will solve it will only be a compliance requirement for #6 - which could be done right now, and has to be done for ANY solution to work.

NOTE: even rewriting EVERY application won’t fix this, nor will deploying a new layer at any level. 

And yes, I do intend to add this to draft-ietf-tunnels, so it can be referred to elsewhere.

Joe

---

From: Joe Touch <[email-masked]>
Date: Sat, 28 Jul 2018 09:41:09 -0700

I thought that was just called the Internet...

> On Jul 26, 2018, at 11:50 AM, Templin (US), Fred L <[email-masked]> wrote:
>

---

From: Ole Troan <[email-masked]>
Date: Sat, 28 Jul 2018 20:24:15 +0200

> 1+2 implies there is always the need for fragmentation at some layer:

1 implies that.
There is enough head room designed in 1 to accommodate 2.

> 	5. network devices increasingly WANT to inspect contents beyond the layer at which they are intended to operate

not that network devices have an intent in themselves, but yes, it seems like network operators want to inspect content or are forced into it because of the necessity of IPv4 address sharing.

> 	6. inspecting contents ultimately means reassembly, at some level

_some_ content inspection would require that, but I don't think you can make that the general rule.
e.g. a NAT or an L4 ACL only needs access to the L4 header.

> 	7. but network device vendors want to save money, so they don’t want to reassemble at any layer

We'd all wish it to be that simple. Can you substantiate that claim?
You can easily make the speculation that customers don't want to pay what it costs to be able to do reassembly at terabit speeds...
Or accept that it's technically hard.

The implementations of e.g. NATs, IPv4 address sharing implementations I'm aware of do flavours of network layer reassembly.
However much money you throw at it, you can't reassemble fragments travelling on different paths, nor can you trivially make network layer reassembly not be an attack vector on those boxes.

> So I agree, IP fragmentation has its flaws - but those flaws are created not only because it leaves out the transport port numbers, but also because DPI and NAT devices don’t reassemble. And they d…

I don't agree with your conclusion.
NATs extend the network layer to include the L4 ports. NAT implementations of course do reassemble.

> So pushing this to another layer will never solve it. What will solve it will only be a compliance requirement for #6 - which could be done right now, and has to be done for ANY solution to work.

For IPv4 address sharing specifically removing network layer fragmentation would be a solution.

> NOTE: even rewriting EVERY application won’t fix this, nor will deploying a new layer at any level.

For some type of content inspection that would require reassembling the whole application context.
But that's quite different from IPv4 address sharing, which we have unfortunately made an integral part of the Internet architecture.

Ole

---

From: Tom Herbert <[email-masked]>
Date: Sat, 28 Jul 2018 11:48:21 -0700

> There is enough head room designed in 1 to accommodate 2.
Ole,

I'm not sure I follow what you're saying here. Ethernet MTU, the most
common value, is 1500 bytes. There's no reference to headroom for
that. If you're referring to the idea of artificially lowering MTUs to
account for potential overhead introduced in encapsulation that can be
done. However to avoid fragmentation _entirely_ one would need to
determine the maximum possible overhead ever added in encapsulation(s)
(plural in case of nested encapsulations). In a sprawling and dynamic
network that has different sub-domains and simultaneously uses
different encapsulation protocols, determining that specific magic
number might be infeasible. There is also the problem that some 0.01%
corner case of encapsulation might need extra large 100s of bytes of
overhead. Lowering the MTU for everyone just to avoid fragmentation
for that case is a poor tradeoff-- it's better to fragment for that
case.

Tom

---

From: Ole Troan <[email-masked]>
Date: Sat, 28 Jul 2018 23:08:59 +0200

Tom,

> On 28 Jul 2018, at 20:48, Tom Herbert <[email-masked]> wrote:
> 

I am talking of the headroom of 1500-1280 designed into IPv6. 

For restricted domains you can increase the headroom by increasing the link MTU.
And I am not saying there aren’t corner cases where fragmentation couldn’t be useful for tunnels.
 
Cheers 
Ole

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sun, 29 Jul 2018 08:24:59 +0200 (CEST)

On Sat, 28 Jul 2018, Joe Touch wrote:

> because DPI and NAT devices don’t reassemble. And they don’t because 
> it’s cheaper to sell devices that say they run at 1 Gbps (e.g.) that 
> don’t bother to reassemble.

Keeping lots of state is always more expensive than not keeping state, and 
customers like lower cost devices.

> So pushing this to another layer will never solve it. What will solve it 
> will only be a compliance requirement for #6 - which could be done right 
> now, and has to be done for ANY solution to work.

Where is that Internet Protocol Police when you need it? I appreciate your 
struggle, but I don't see how you will succeed in your struggle, in 
reality.

So I prefer to recommend not to rely on IP level fragmentation, and 
fragment at higher layers. It works better in reality.

---

From: Joe Touch <[email-masked]>
Date: Sun, 29 Jul 2018 08:27:16 -0700

> On Jul 28, 2018, at 11:24 PM, Mikael Abrahamsson <[email-masked]> wrote:
> 
>> because DPI and NAT devices don’t reassemble. And they don’t because it’s cheaper to sell devices that say they run at 1 Gbps (e.g.) that don’t bother to reassemble.
> 
> Keeping lots of state is always more expensive than not keeping state, and customers like lower cost devices.

Yes, but they need to be told that their device is “hobbled”.

> Where is that Internet Protocol Police when you need it? I appreciate your struggle, but I don't see how you will succeed in your struggle, in reality.
> 
> So I prefer to recommend not to rely on IP level fragmentation, and fragment at higher layers. It works better in reality.

Until it doesn’t, for exactly the same reason it isn’t working at IP.

You’re engaging in a game of escalation - whatever layer you add fragmentation will end up being a layer that a vendor puts a device that does DPI that fails.

Joe

---

From: Joe Touch <[email-masked]>
Date: Sun, 29 Jul 2018 08:38:56 -0700

> On Jul 28, 2018, at 11:24 AM, Ole Troan <[email-masked]> wrote:
> 

As was noted, there’s never enough headroom because you can’t control (or even know) how many layers of tunnels your traffic experiences.

> not that network devices have an intent in themselves, but yes, it seems like network operators want to inspect content or are forced into it because of the necessity of IPv4 address sharing.

They were “forced” into differentiating commercial from home customer pricing. NATs didn’t need to exist when they were first deployed.

That said, there’s no real problem with a NAT *IF* it acts as a host on the Internet
(see ouch, J: Middlebox Models Compatible with the Internet <>. USC/ISI (ISI-TR-711), 2016.)

> e.g. a NAT or an L4 ACL only needs access to the L4 header.

It’s a general rule; you merely refer to an instance that is relevant ONLY when the L3 header at which a router operates is followed (or expected to be followed) by an L4 header.

That may have been the Internet of 10 years ago, but it is less and less the Internet moving forward.

> Or accept that it's technically hard.

I’m not claiming it isn’t hard; I’m claiming that it’s always cheaper to *not* do something. 

My concern isn’t that reassembly isn’t being done; it’s that vendors sell devices that don’t make this clear - AND that they don’t pass on packets they don’t/can’t reassemble.

> The implementations of e.g. NATs, IPv4 address sharing implementations I'm aware of do flavours of network layer reassembly.

If they did so, we would not be here talking about considering IP fragmentation fragile.

> However much money you throw at it, you can't reassemble fragments travelling on different paths, nor can you trivially make network layer reassembly not be an attack vector on those boxes.

Agreed, but here’s the other point:

	Any device that inspects L4 content can do so ONLY as a proxy for the destination endpoint.

I.e., I know vendors WANT to sell devices they say can be deployed anywhere in the network, and operators believe that, but it’s wrong.

Basically, if you’re not at a place in the network where you represent that endpoint, you have no business acting as that endpoint - “full stop”.

> NATs extend the network layer to include the L4 ports. NAT implementations of course do reassemble.

See Touch, J: Middlebox Models Compatible with the Internet <>. USC/ISI (ISI-TR-711), 2016. 

NATs do not extend the network layer to include L4; they act as endpoints in the public net and routers in the private net - for the reason above (multipath), among others.

> For IPv4 address sharing specifically removing network layer fragmentation would be a solution.

For ONE problem there is ONE patch until a new device looks further (to do port filtering on UDP tunneled traffic, e.g.). Then we’re back here talking about fragmentation over GUE considered fragile.

> But that's quite different from IPv4 address sharing, which we have unfortunately made an integral part of the Internet architecture.

The only difference is that the impact of the current L4 problem is being currently discussed. If you consider why it’s happening, it might be more clear that this problem will cycle around again and cannot be solved by patching this way.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sun, 29 Jul 2018 09:11:13 -0700

On Sun, Jul 29, 2018 at 8:38 AM, Joe Touch <[email-masked]> wrote:
> As was noted, there’s never enough headroom because you can’t control (or
> even know) how many layers of tunnels your traffic experiences.
>
>
>
> not that network devices have an intent in themselves, but yes, it seems
> like network operators want to inspect content or are forced into it because
> of the necessity of IPv4 address sharing.
>
>
> They were “forced” into differentiating commercial from home customer
> pricing. NATs didn’t need to exist when they were first deployed.
>
> (see ouch, J: Middlebox Models Compatible with the Internet. USC/ISI
> (ISI-TR-711), 2016.)

Joe,

It's still a problem though. A NAT (or any stateful device in the
network) forces the requirement in network architecture that all
packets of a flow are routed through the same device. This has killed
our ability to use multi-homing and multi-path. The best way for an
intermediate devices to deal with transport layer state is to be an L4
proxy. The intermediate is a host endpoint for the proxy connections,
but then that has its own problems since it breaks E2E functionality
(like TCP auth). So the only real solution is to eliminate transport
state from the network. I'm still holding out hope that IPv6 will
start to obsolete use of NAT! FAST (draft-herbert-fast-02) is intended
to provide a viable alternative to stateful firewalls.

Tom

> It’s a general rule; you merely refer to an instance that is relevant ONLY
> when the L3 header at which a router operates is followed (or expected to be
> followed) by an L4 header.
>
> You can easily make the speculation that customers don't want to pay what it
> costs to be able to do reassembly at terabit speeds...
> My concern isn’t that reassembly isn’t being done; it’s that vendors sell
> devices that don’t make this clear - AND that they don’t pass on packets
> they don’t/can’t reassemble.
>
> The implementations of e.g. NATs, IPv4 address sharing implementations I'm
> aware of do flavours of network layer reassembly.
>
>
> However much money you throw at it, you can't reassemble fragments
> travelling on different paths, nor can you trivially make network layer
> reassembly not be an attack vector on those boxes.
>
>
> I.e., I know vendors WANT to sell devices they say can be deployed anywhere
> in the network, and operators believe that, but it’s wrong.
>
> Basically, if you’re not at a place in the network where you represent that
> endpoint, you have no business acting as that endpoint - “full stop”.
>
>
>
> So I agree, IP fragmentation has its flaws - but those flaws are created not
> only because it leaves out the transport port numbers, but also because DPI
> and NAT devices don’t reassemble. And they don’t because it’s cheaper to
> sell devices that say they run at 1 Gbps (e.g.) that don’t bother to
> reassemble.
>
>
> See Touch, J: Middlebox Models Compatible with the Internet. USC/ISI
> (ISI-TR-711), 2016.
>
> NATs do not extend the network layer to include L4; they act as endpoints in
> the public net and routers in the private net - for the reason above
> (multipath), among others.
>
>
> I.e., it will never matter what layering we add to fix this - GRE, GUE,
> Aero, etc. - ultimately, we’re doomed to need fragmentation support down to
> IP exactly because:
>
> For ONE problem there is ONE patch until a new device looks further (to do
> port filtering on UDP tunneled traffic, e.g.). Then we’re back here talking
> about fragmentation over GUE considered fragile.
>
>
> But that's quite different from IPv4 address sharing, which we have
> unfortunately made an integral part of the Internet architecture.
>
>
> The only difference is that the impact of the current L4 problem is being
> currently discussed. If you consider why it’s happening, it might be more
> clear that this problem will cycle around again and cannot be solved by
> patching this way.
>
> Joe
>

---

From: Joe Touch <[email-masked]>
Date: Sun, 29 Jul 2018 09:22:01 -0700

> On Jul 29, 2018, at 9:11 AM, Tom Herbert <[email-masked]> wrote:
> 
>> ...
>> 
> packets of a flow are routed through the same device.

I didn’t make that requirement. The Internet does - it’s what it *means* to have an IP address.

A NAT *has* the address of the packets it sources; if it isn’t the sink of that address, then it’s being used incorrectly. If it doesn’t reassemble those packets before translating them (i.e., by translating only unfragmented packets and dropping fragmented ones), then it is broken and ought to be returned for a refund.

> This has killed
> our ability to use multi-homing and multi-path.

No, the Internet supports multi path between two IP endpoints and allows multihoming for a single address when managed by a single endpoint (physical or virtual).

The disconnect is a failure to understand that a NAT *is* an IP endpoint. The term “middlebox” is wrong in that sense, at least it’s not a middle box to the Internet (it is to the device behind the NAT).

> The best way for an
> state from the network.

That would work only if the network didn’t look at or modify transport information - and it did work when that was the case.

> I'm still holding out hope that IPv6 will

Getting rid of NATs is only part of the problem. Anything that does DPI is a problem when it discards messages it can’t parse because they’re fragmented.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sun, 29 Jul 2018 09:43:39 -0700

On Sun, Jul 29, 2018 at 9:22 AM, Joe Touch <[email-masked]> wrote:
> A NAT *has* the address of the packets it sources; if it isn’t the sink of
> that address, then it’s being used incorrectly. If it doesn’t reassemble
> those packets before translating them (i.e., by translating only
> unfragmented packets and dropping fragmented ones), then it is broken and
> ought to be returned for a refund.
>
> This has killed
> No, the Internet supports multi path between two IP endpoints and allows
> multihoming for a single address when managed by a single endpoint (physical
> or virtual).
>
> The disconnect is a failure to understand that a NAT *is* an IP endpoint.
> The term “middlebox” is wrong in that sense, at least it’s not a middle box
> to the Internet (it is to the device behind the NAT).
>
> The best way for an
> That would work only if the network didn’t look at or modify transport
> information - and it did work when that was the case.
>
> Getting rid of NATs is only part of the problem. Anything that does DPI is a
> problem when it discards messages it can’t parse because they’re fragmented.
>
Right, that implies that we need to eliminate the motivation to do DPI
by providing a better way to get the same functionality. This is what
FAST proposes. Transport related information of interest to a firewall
is in Hop-by-Hop options. So middleboxes will look at HBH options
which come before fragment EH. No DPI needed, no issues with
fragmentation at middleboxes, and, as a bonus, HBH options are the
only mechanism defined in IP protocol that allows data to be changed
in flight (this will be important property as in-situ OAM starts to
get traction).

Tom

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 30 Jul 2018 07:29:11 +0200 (CEST)

On Sun, 29 Jul 2018, Joe Touch wrote:

> You’re engaging in a game of escalation - whatever layer you add 
> fragmentation will end up being a layer that a vendor puts a device that 
> does DPI that fails.

Yes, but I can filter those UDP packets by looking in the UDP header, 
that's all the DPI I need in that box. It doesn't need to understand the 
upper-protocol level fragmentation, because I do not require it to 
understand that protocol at all. I just need for it to understand that 
it's UDP and look at the UDP port number.

The biggest mistake of TCP and UDP combined with IP level fragmentation is 
that the port information isn't available in every packet.

---

From: Ole Troan <[email-masked]>
Date: Mon, 30 Jul 2018 09:33:45 +0200

Joe,

> Basically, if you’re not at a place in the network where you represent that endpoint, you have no business acting as that endpoint - “full stop”.

I understand you want it to fit in your model, but it doesn't.
Take A+P (RFC6346), and it's instantiations through e.g. MAP-E (RFC7597). That's essentially normal longest match forwarding on addresses and ports.

With regards to your point about reassembly at higher layers, crypto is the answer to that.

Ole

---

From: Joe Touch <[email-masked]>
Date: Mon, 30 Jul 2018 06:36:16 -0700

> On Jul 29, 2018, at 10:29 PM, Mikael Abrahamsson <[email-masked]> wrote:
> 
> Yes, but I can filter those UDP packets by looking in the UDP header, that's all the DPI I need in that box. It doesn't need to understand the upper-protocol level fragmentation, because I do not require it to understand that protocol at all. I just need for it to understand that it's UDP and look at the UDP port number.

Right. You need just UDP ports right now for YOUR DPI.

Others need to look at the payload (the D in DPI).

> 
> The biggest mistake of TCP and UDP combined with IP level fragmentation is that the port information isn't available in every packet.

The biggest mistake of protocol X with X-1 level fragmentation is that the entire headers of X aren’t available in every X-1 packet.

Replace X with your favorite protocol and you’ll see how and why this can’t continue to work. The packets would eventually burst with all the headers.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 30 Jul 2018 06:44:17 -0700

> On Jul 30, 2018, at 12:33 AM, Ole Troan <[email-masked]> wrote:
> 
> Joe,
> 
> I understand you want it to fit in your model, but it doesn’t.

My model describes the rules under which translation devices can operate correctly and predictably in the Internet model.

There are only a few alternatives for devices not explained by either model:
	1- the Internet and my model are incomplete
		in that case, you’re welcome to provide one for the new device
	2- the Internet and/or my model are incorrect
		in that case, you’re welcome to explain why
	3- the device should be considered incorrect and itself corrected

Un-doing fragmentation at IP is an attempt to jump to a solution for #1 without explaining WHY, other than “we need to do this to fix the Internet to support these new devices”.

I don’t think we should break known models to adapt to devices whose behavior might never be correctly accommodated.

> Take A+P (RFC6346), and it's instantiations through e.g. MAP-E (RFC7597). That's essentially normal longest match forwarding on addresses and ports.

So? Any device that sources packets with addresses it owns IS an endpoint on the Internet. Nothing changes based on how it translates those devices to the private side.

> With regards to your point about reassembly at higher layers, crypto is the answer to that.

Crypto will just end up putting a halt to all this nonsense - but not quickly enough, IMO.

Then again, I would not be surprised to be back here discussing a doc on “transport crypto considered fragile” in a decade...

Joe

---

From: Ole Troan <[email-masked]>
Date: Mon, 30 Jul 2018 15:58:00 +0200

Joe,

> So? Any device that sources packets with addresses it owns IS an endpoint on the Internet. Nothing changes based on how it translates those devices to the private side.

Could you please read those documents and explain how A+P fits in your model?
Note an A+P router does not translate, it forwards based on address and port. And as a normal router those addresses (and ports) are not identifying interfaces on the router, but on some end-system further away.

Best regards,
Ole

---

From: Tom Herbert <[email-masked]>
Date: Mon, 30 Jul 2018 08:11:49 -0700

> I didn’t make that requirement. The Internet does - it’s what it *means* to
It's not a requirement of the Internet and certainly not a core IETF
requirement that packets always follow the same path. It's an ad hoc
requirement imposed by _some_ solutions that have been deployed.

> ought to be returned for a refund.
You are only considering the return path. Packets sent from the client
origin to the remote server need to always follow the same path to hit
the same device that has the NAT state for the flow. The destination
address is not the address of the NAT device, so the only way this
works is if the routing in the internal network consistently routes
packets through the right device. If routing changes, packets could be
sent to a different NAT device that doesn't have the state to handle
the packet so it will just drop it. This is a fundamental problem.
Vendors have mostly gotten away with it because NAT and firewalls are
often used in very simple networks, like home networks, that only have
one default router. But in a more complex network with multiple egress
points, including increasing use of multi-homing in home networks and
personal devices, getting consistent routing to satisfy the
requirements of stateful network devices is a major issue.

Fragmentation exacerbates the problem because fragments must be routed
precisely the same way that non-fragments are in order to hit the same
egress device. ECMP routing non-fragments using ports and fragments
without using ports means that they take different paths. Using flow
label instead of ports for ECMP is the best way to ensure all packets
(fragments and non-fragments) of a flow follow the same route (or at
least will produce the same hash for load balancing and packet
steering algorithms).

> multihoming for a single address when managed by a single endpoint (physical
See above explanation.

Tom

---

From: Joe Touch <[email-masked]>
Date: Mon, 30 Jul 2018 10:34:34 -0700

On 2018-07-30 08:11, Tom Herbert wrote:

>> That said, there's no real problem with a NAT *IF* it acts as a host on the
>> Internet
>> I didn't make that requirement. The Internet does - it's what it *means* to
>> have an IP address.

1122 requires that devices shouldn't be sourcing IP addresses they don't
own. 

And any device behind a NAT would have a similar requirement - that the
private side is setup such that any translation appears as a single
device -- which means either that each host on the private side forwards
all its traffic to one egress (as a stub net) or that the multiple
egresses coordinate state to look like one egress on the private side
and one host on the public side. 

I.e., the Internet architecture imposes exactly the restrictions you
note - these aren't ad-hoc, they're derived from the Internet
requirements. 

>> A NAT *has* the address of the packets it sources; if it isn't the sink of
>> that address, then it's being used incorrectly. If it doesn't reassemble
> You are only considering the return path.

Only for that rule; for the private side, the rule is that the network
is a stub behind the translator (or emulates that). 

...

> our ability to use multi-homing and multi-path.

It didn't kill it. It never existed *for this sort of mechanism* -
exactly because of requirements of the Internet architecture. 

If there is only one device that represents a private node on the public
net, it can work (using the model I describe). If there isn't, then it's
the Internet architecture that makes the device inherently broken - we
shouldn't go around believing we can patch the protocols to "make it
work again".  

Joe

---

From: Tom Herbert <[email-masked]>
Date: Mon, 30 Jul 2018 11:16:21 -0700

On Mon, Jul 30, 2018 at 10:34 AM, Joe Touch <[email-masked]> wrote:
> private side is setup such that any translation appears as a single device
> -- which means either that each host on the private side forwards all its
> traffic to one egress (as a stub net) or that the multiple egresses
> coordinate state to look like one egress on the private side and one host on
> the public side.
>
> I.e., the Internet architecture imposes exactly the restrictions you note -
> these aren't ad-hoc, they're derived from the Internet requirements.
>
IP is packet switched not circuit switched. Other than the source and
destination nodes, no two packets sent on a flow are required to
traverse the same nodes, nor does there need to be any symmetry in the
path between two directions of communication. If this is an incorrect
interpretation of the requirements then please reference the RFC that
states otherwise.

> Only for that rule; for the private side, the rule is that the network is a
> stub behind the translator (or emulates that).
>
> ...
>
> This has killed
> It didn't kill it. It never existed *for this sort of mechanism* - exactly
> because of requirements of the Internet architecture.
>
The mechanism (specifically the requirement that packets traverse
intermediate nodes that are not addressed) breaks multi-homing. All
the hacks and hoops we've had to jump through over the years to make
NAT, stateful firewalls, and L4 load balancers work attest to this
being a real problem.

Tom

> net, it can work (using the model I describe). If there isn't, then it's the
> Internet architecture that makes the device inherently broken - we shouldn't
> go around believing we can patch the protocols to "make it work again".
>
> Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 30 Jul 2018 11:50:34 -0700

> interpretation of the requirements then please reference the RFC that

RFC1122 says that an address belongs to one interface. 

So if you have a system that treats internet addresses as belonging to
more than one device, that's on you to make sure they ACT like one
device. 

You can have a host on the private side use multiple egress NATs for
different connections, but the reverse traffic has to treat the two as
if they were one device. If putting those devices in the middle of the
network means you can't see the traffic to make that happen, then you've
deployed the devices incorrectly or insufficiently configured them to
act as a single unit. 

> NAT, stateful firewalls, and L4 load balancers work attest to this

*traversing* intermediate nodes is not an issue. It's changing addresses
and ports that is. 

Once you do either thing, you're a host - ONE logical host for every IP
address you do this with. If you deploy a device that intends to act
this way that can't see all the traffic it should be receiving - or
deploy a device behind it that isn't represented by exactly one such
device - then it is your device and/or its deployment that is broken. 

Oh, and I have no debate that people deploying badly designed devices
and/or doing so in incorrect ways is a real problem. It's just not MY
problem, nor do I think it should be the IETF's. 

Joe

---

From: Tom Herbert <[email-masked]>
Date: Mon, 30 Jul 2018 12:13:05 -0700

On Mon, Jul 30, 2018 at 11:50 AM, Joe Touch <[email-masked]> wrote:
> So if you have a system that treats internet addresses as belonging to more
> than one device, that's on you to make sure they ACT like one device.
>
> different connections, but the reverse traffic has to treat the two as if
> they were one device. If putting those devices in the middle of the network
> means you can't see the traffic to make that happen, then you've deployed
> the devices incorrectly or insufficiently configured them to act as a single
> unit.
>
>
> ...
> *traversing* intermediate nodes is not an issue. It's changing addresses and
> ports that is.
>
> address you do this with. If you deploy a device that intends to act this
> way that can't see all the traffic it should be receiving - or deploy a
> device behind it that isn't represented by exactly one such device - then it
> is your device and/or its deployment that is broken.
>
> Oh, and I have no debate that people deploying badly designed devices and/or
> doing so in incorrect ways is a real problem. It's just not MY problem, nor
> do I think it should be the IETF's.

Joe,

I agree, except that a BCP like this is making practical (and current)
recommendations that take into account the protocols have been
commonly implemented regardless of whether they are conformant. From
that perspective, it does seem to be IETF's problem. I think this is
reasonable assuming that it's clear why such recommendations are being
made. If they're being made in order to workaround non-confomant
implementations then that should be highlighted as such. In no way
should such recommendations be interpreted as acceptance of
non-conformance. The need for fragmentation cannot be completely
eliminated and we do need it to work. Devices that do things to
prevent correct operation still need to be fixed, and it would be
productive for the draft to include statements on how some of the
sub-problems problems can be fixed (like using flow label for ECMP
instead of ports).

Tom

---

From: Joe Touch <[email-masked]>
Date: Mon, 30 Jul 2018 17:42:53 -0700

> On Jul 30, 2018, at 12:13 PM, Tom Herbert <[email-masked]> wrote:
> 
> On Mon, Jul 30, 2018 at 11:50 AM, Joe Touch <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> 
>> 
>> 
> commonly implemented regardless of whether they are conformant.

I don’t want to codify implementation and/or deployment errors in a way that has a long-term bad impact on the Internet as a whole.

> From
> implementations then that should be highlighted as such.

Yes, and the hazard is that we can’t make protocols (inherently “rules”) to deal with people or systems that don’t follow rules (i.e., laws for the lawless are not productive).

> In no way

That indeed is the trick.

> The need for fragmentation cannot be completely

On that we agree. That’s my key concern - how to do this in a way that doesn’t effectively kill off IP fragmentation for the rest of us.

Joe

---

From: Ole Troan <[email-masked]>
Date: Tue, 31 Jul 2018 14:27:59 +0200

Joe,

> On that we agree. That’s my key concern - how to do this in a way that doesn’t effectively kill off IP fragmentation for the rest of us.

For IPv4 it’s an unfortunate side-effect of the fact that we are out of IPv4 addresses. As we continue to increase the ratio of users per IPv4 address, IPv4 fragmentation, or any protocol that isn’t TCP or UDP are not going to work well in the public IPv4 Internet.
IPv4 Internet Architecture is evolving as a consequence of address run-out. I think we’ve pretty much explored the solution space for IPv4 sharing mechanisms, so I think you just have to accept this new and unimproved (sic) IPv4 Internet architecture.

Cheers,
Ole

---

From: Tom Herbert <[email-masked]>
Date: Tue, 31 Jul 2018 08:25:34 -0700

On Tue, Jul 31, 2018, 5:28 AM Ole Troan <[email-masked]> wrote:

> > On that we agree. That’s my key concern - how to do this in a way that
> doesn’t effectively kill off IP fragmentation for the rest of us.
>
> For IPv4 it’s an unfortunate side-effect of the fact that we are out of
> IPv4 addresses. As we continue to increase the ratio of users per IPv4
> address, IPv4 fragmentation, or any protocol that isn’t TCP or UDP are not
> going to work well in the public IPv4 Internet.
> IPv4 Internet Architecture is evolving as a consequence of address
> run-out. I think we’ve pretty much explored the solution space for IPv4
> sharing mechanisms, so I think you just have to accept this new and
> unimproved (sic) IPv4 Internet architecture.
>

Ole,

How is this story going to be different for IPv6? How do we ensure that
non-conformant implementation for IPv4 isn't just carried over so that
fragmentation, alternative protocols, and extension headers are viable on
the IPv6 Internet?

Tom

---

From: Ole Troan <[email-masked]>
Date: Tue, 31 Jul 2018 23:21:42 +0200

Tom,

> How is this story going to be different for IPv6? How do we ensure that non-conformant implementation for IPv4 isn't just carried over so that fragmentation, alternative protocols, and extension headers are viable on the IPv6 Internet?

I don’t think the IPv4 implementations are non-conformant.
(With regards to the implications of A+P on the IPv4 architecture).

For IPv6 one would fear that the same pressures that has led to IPv4 ossification applies.
Well, what can we do? Apart from crypto, ensure that popular applications use the features, so they cannot be shut down?

Cheers,
Ole

---

From: Tom Herbert <[email-masked]>
Date: Tue, 31 Jul 2018 16:29:41 -0700

Ole,

That's the "use it or lose it" model of protocol features. TCP options
are firmly established as a required part of TCP protocol so there is
no way they could be obsoleted by external implemenation; IP options
on the other were never really required for IP operation so they are
considered expendable. The problem is that protocol features are often
defined before the application that would use them is built, so the
motivation to support all the features from the start isn't there.
This seems to be the case with extension headers, since only now does
there seem to be some serious proposals to use that functionality long
after the mechanism was first defined and IPv6 was deployed. In
reality, support of protocol features in the Internet is hardly ever
binary. Plain TCP/IPv4 packets are probably the only combination of
protocols that is guaranteed to work with probability approaching
100%, however pretty much anything else works with some varying of
probability greater than 0% but less than 100% (like EH success rates
in RFC7872). To that end, I am wondering if the idea of Happy Eyeballs
could somehow be generalized to work with these other "non-standard"
features.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 1 Aug 2018 13:50:13 +1200

> could somehow be generalized to work with these other "non-standard"

Generalised probing might be an answer, but (especially when there
are multiple address pairs to probe) it can significantly impact
start-up latency for a session. (Some dead work on that topic is at
https://tools.ietf.org/html/draft-naderi-ipv6-probing .)

    Brian