Subject: Re: [tsvwg] UDP Options: on forcing the use of UDP CS=0 in connection with FRAG+LITE

From: "C. M. Heard" <[email-masked]>
Date: Mon, 1 Jul 2019 12:16:00 -0700

On Sat, Jun 29, 2019 at 7:58 AM Joe Touch wrote:
>
> See sec 8.1, which incorporates rfc 6936:
>
>          As an exception to the default behavior, protocols that use UDP
>          as a tunnel encapsulation may enable zero-checksum mode for a
>          specific port (or set of ports) for sending and/or receiving.
>          Any node implementing zero-checksum mode must follow the
>          requirements specified in "Applicability Statement for the Use
>          of IPv6 UDP Datagrams with Zero Checksums" [RFC6936].
>
> Frag is a kind of such a tunnel because it adds its own reassembly CS.

I'll agree that FRAG+LITE with UDP CS=0 does conform to the requirements
in RFC 6936 Sections 4 and 5 because it includes a reassembly CS.

> But that also means middleboxes should allow relaying these since 6936
> was passed.

Yes. middleboxes are ***supposed*** to do that. But the empirical data
that we have says that in a significant number of cases (affecting 26%-36%
of the paths) they do not.

It's possible that the situation will improve in the future, but given
that it has not improved enough to make UDP CS=0 useable in other than
controlled environments, I don't think it's a good idea to propose its
use when we have an alternative that avoid UDP CS=0 that is no more
complex to implement.

Mike Heard

---

From: Joe Touch <[email-masked]>
Date: Mon, 1 Jul 2019 12:26:38 -0700

> of the paths) they do not.

And a similar number of net people aren’t likely aware this is incorrect behavior,IMO.

But...

> use when we have an alternative that avoid UDP CS=0 that is no more

You haven’t addressed the data copying issue. If the proposed alternative can be achieved with fixed, small amounts of data movement, it might be viable.  But if we need to move the whole fragment, it’s not. Data coping overheads cannot be mitigated; bugs in other people’s implementations can.

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 1 Jul 2019 12:49:00 -0700

> You haven’t addressed the data copying issue.

Yes, I have, in
https://mailarchive.ietf.org/arch/msg/tsvwg/RJBWi_tiW6M_phfc22zKuVn_vh4

Can you please respond to that message and explain exactly why you think
that the proposal in the thread "DP Options: how to do FRAG without LITE
and forced UDP CS=0" imposes data movement requirements for packet
reassembly beyond those required by FRAG+LITE (or FRAG without LITE) in
in draft-ietf-tsvwg-udp-options-07? I sure don't see it.

Mike

---

From: Joe Touch <[email-masked]>
Date: Mon, 1 Jul 2019 13:18:37 -0700

> On Jul 1, 2019, at 12:49 PM, C. M. Heard <[email-masked]> wrote:
> 

When I’m done moving a small, fixed number of bytes, the existing FRAG+LITE option leaves the fragment at the head of the packet.

When you’re done, the frag is after a variable length string of options.

Yes, both need to be gathered together. No, they’re not the same - some UDP hardware and software implementations treat the offset where they expect the data to start as “special”; yours could be anywhere.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 1 Jul 2019 13:45:32 -0700

To be a little more specific:

> On Jul 1, 2019, at 1:18 PM, Joe Touch <[email-masked]> wrote:
> 
> 
> 

Consider mbufs chains where headers are in separate mbufs than the body.

In that case, the existing FRAG+LITE would need to just chain the bodies together and alter their length

In the new proposal, there would be a potentially variable offset from the head of each mbuf in the chain.

Those might have very different overheads…

Joe

---

From: Tom Herbert <[email-masked]>
Date: Mon, 1 Jul 2019 14:56:14 -0700

> in RFC 6936 Sections 4 and 5 because it includes a reassembly CS.
Mike,

The reason that UDPv6 checksums were required in the first place is
that IPv6, unlike IPv4, has no header checksum. The pseudo header in
the UDP checksum includes the IP addresses which provides protection
against misdelivery when addresses are corrupted. The UDP options
reassembly checksum doesn't include the pseudo header so it provides
no protection of the IP addresses, and hence I don't believe it
satisfies the requirements of RFC6936 for using a zero UDP checksum.

> of the paths) they do not.
Thanks for the numbers, that is very enlightening.

> use when we have an alternative that avoid UDP CS=0 that is no more

Agreed. I'd also point out that the very reason to force UDP checksum
to zero is that some implementations incorrectly compute the UDP
checksum based on length of the packet as opposed to using UDP length.
Unlike expecting devices to accept UDPv6 zero checksums when prior to
RFC6936 was strictly prohibited for all cases, basing UDP checksum on
packet length was _never_ allowed. IMO,  forcing zero checksums is at
best trading off one source of brokenness for another, but the numbers
you're reporting don't seem justify the tradeoff is worth it.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Mon, 1 Jul 2019 15:08:02 -0700

> Yes, both need to be gathered together. No, they’re not the same - some UDP hardware and software implementations treat the offset where they expect the data to start as “special”; yours could be a…

Joe,

If we need to perform the reassembly checksum in the host CPU then any
performance gains your seeing trying to keep the data early in the
packet would be eclipsed by all the cache misses we'll be taking for
the calculation. We have a much better chance to be able to offload
the checksum if it's per packet instead of being over the reassembled
packet (the merits of per packet checksums have been previously
discussed).

Tom

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 1 Jul 2019 15:42:33 -0700

On Mon, Jul 1, 2019 at 1:19 PM Joe Touch wrote:
> On Jul 1, 2019, at 12:49 PM, C. M. Heard wrote:
> > On Mon, Jul 1, 2019 at 12:27 PM Joe Touch  wrote
> When I’m done moving a small, fixed number of bytes, the existing
> FRAG+LITE option leaves the fragment at the head of the packet.
>
> Yes, both need to be gathered together. No, they’re not the same - some
> UDP hardware and software implementations treat the offset where they
> expect the data to start as “special”; yours could be anywhere.
...
> Those might have very different overheads…

YMMV, but I find that argument to be unconvincing. Any not conspicuously
deficient mbuf design needs to be able to easily trim off variable length
headers from the start of a packet (e.g., by having a starting offset), or
else it would not be able to efficiently accommodate IP options/extension
headers, TCP options, tunnel encapsulation headers, and so on.

Mike

---

From: Joe Touch <[email-masked]>
Date: Mon, 1 Jul 2019 17:06:03 -0700

> headers, TCP options, tunnel encapsulation headers, and so on.

That correlates with what I’ve seen in performance exactly though. 

The common case is fast; other cases not so much. Even the mbuf start offset from malloc start matters a bit (per measurements I did with MD5 in memory). 

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 1 Jul 2019 17:08:03 -0700

> On Jul 1, 2019, at 2:56 PM, Tom Herbert <[email-masked]> wrote:
> 
> The pseudo header in
> against misdelivery when addresses are corrupted

I’d wager NAT boxes corrupt more addresses than anything else, but they fix the checksum. 

So why would not including these fields matter?

Joe

---

From: Tom Herbert <[email-masked]>
Date: Mon, 1 Jul 2019 17:35:29 -0700

> So why would not including these fields matter?

Because it matters to UDPv6. Again the reason very reason why RFC8200
requires the UDPv6 is that there is no header checksum to protect the
IPv6 addresses. RFC6936 is only applicable to tunnel encapsulations
where either the risks of corruption are controlled or there is
alternative protection that is applied to the addresses. The
requirements of RFC6936 are really quite narrow, and even to enable
UDPv6 checksums in standard tunneling protocols required much more
text (see RFC7510 section 3.1, and RFC8086 section 6.2).

Tom

---

From: Joe Touch <[email-masked]>
Date: Mon, 1 Jul 2019 17:40:27 -0700

.
On Jul 1, 2019, at 5:35 PM, Tom Herbert <[email-masked]> wrote:

> Because it matters to UDPv6.

In a world with NATs that recalc CSs, it doesn’t.  It just checks that the NAT sw implemented the update. 

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 1 Jul 2019 17:42:26 -0700

On Mon, Jul 1, 2019 at 5:06 PM Joe Touch <[email-masked]> wrote:
> > On Mon, Jul 1, 2019 at 12:27 PM Joe Touch  wrote:
...
> The common case is fast; other cases not so much. Even the mbuf start
> offset from malloc start matters a bit (per measurements I did with MD5
> in memory).

The original complaint was:

> The solution below doesn't appear to avoid copying/moving large amounts
> of data.
>
> At a minimum, can you address #2? If we can do this without moving
> around a lot of data, it might be viable.

I believe I've addressed that point. There are implementation strategies
that avoid copying/moving large amounts of data.

The complaint now is that (under some implementation strategies, at least)
FRAG+LITE and CS=0 may have a modest performance advantage ("matters a bit")
compared to the new proposal with a fully compensated checksum. Is the
purported performance advantage enough to outweigh the fact that CS=0
suffers a significantly higher drop rate than a fully compensated checksum?

Mike

---

From: Tom Herbert <[email-masked]>
Date: Mon, 1 Jul 2019 18:12:07 -0700

> In a world with NATs that recalc CSs, it doesn’t.  It just checks that the NAT sw implemented the update.
Is that the same world where 24-36% of paths drop UDPv6 packets with
zero checksums as Mike reported?

Tom

---

From: Joe Touch <[email-masked]>
Date: Mon, 1 Jul 2019 18:55:02 -0700

> On Jul 1, 2019, at 5:42 PM, C. M. Heard <[email-masked]> wrote:
> 
> I believe I've addressed that point.

The assumption was ‘to place the fragments in order using existing efficiencies,’ which assumed the data starts at the beginning of the conventional UDP area in each fragment. That’s what LITE+FRAG does.. You need to copy to achieve that.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 1 Jul 2019 18:57:40 -0700

> zero checksums as Mike reported?

Yes. I find it ironic that you care so much to ensure that the checksum validates addresses and ports that nats change and here you’ve found a true error that you want to cover up instead of fix.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 1 Jul 2019 19:05:02 -0700

> FRAG+LITE and CS=0 may have a modest performance advantage ("matters a bit")

That bit can be significant, depending on how maids are laid out.

> suffers a significantly higher drop rate than a fully compensated checksum?

Again, wer also talking about per frag checksums vs. reassembled packet checksum. The latter is safer, due to potential reassembly errors.

The issue is what kind of errors should we catch? What should we be robust to?

IMO,  detecting address or port errors is very unlikely, given how NATS overwrite CSs.
Detecting reasssemblu errrors is very important, esp. at high data rates.
Tolerating implementation errors that are already a problem for others is very low on the list - those paths should be debugged and fixed, especially when tolerating them means either disabling reassembly checks or doing double the work.

IMO, we should design for the long term, not the transient.

But it’d be useful to hear from as many others as possible on these issues to determine where consensus is.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 1 Jul 2019 19:16:59 -0700

Oh, I should mention why this was an assumption - it allows RDMA-style direct data placement and zero-copy reassembly in the fast-path (happy order) case.

That’s not possible without copies otherwise.

Further, in the most typical expected case of two fragments, it enables zero-copy for the first fragment, which could more than halve the copies even if later fragments are put in holding areas.

Joe

> On Jul 1, 2019, at 6:55 PM, Joe Touch <[email-masked]> wrote:
> 
> 
>

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 1 Jul 2019 19:27:13 -0700

> issues to determine where consensus is.

I strongly agree with that.

Mike

---

From: Tom Herbert <[email-masked]>
Date: Mon, 1 Jul 2019 20:00:42 -0700

On Mon, Jul 1, 2019, 6:57 PM Joe Touch <[email-masked]> wrote:

> Yes. I find it ironic that you care so much to ensure that the checksum
> validates addresses and ports that nats change and here you’ve found a true
> error that you want to cover up instead of fix.
>

What is the "true error" to which you're referring?

---

From: Joe Touch <[email-masked]>
Date: Mon, 1 Jul 2019 20:40:31 -0700

> What is the "true error" to which you're referring?

Intermediate devices dropping cs=0.

Joe

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 2 Jul 2019 04:04:27 +0000 (UTC)

That's too low.

I'd want more zero-checksum UDP packets to be dropped at AS boundaries.

No checksum, no entry.

Lloyd Wood [email-masked] http://about.me/lloydwood

On Tuesday, 2 July 2019, 11:12:44 GMT+10, Tom Herbert <[email-masked]> wrote: 

Tom

---

From: Tom Herbert <[email-masked]>
Date: Mon, 1 Jul 2019 21:47:18 -0700

> Intermediate devices dropping cs=0.
Section 4.3 of RFC6935 is pertinent:

"Applicability Statement for the Use of IPv6 UDP Datagrams with Zero
Checksums" [RFC6936] specifies requirements for middleboxes and
tunnels that need to traverse middleboxes.  Tunnel protocols intending
to use a zero UDP checksum need to ensure that they have defined a
method for handling cases when a middlebox prevents the path between
the tunnel ingress and egress from supporting transmission of
datagrams with a zero UDP checksum.  This is especially important as
middleboxes that conform to RFC 2460 are likely to discard datagrams
with a zero UDP checksum."

---

From: [email-masked]
Date: Tue, 2 Jul 2019 09:26:23 +0000 (UTC)

Dropping a packet with zero checksum to prevent possible port pollution of another service due to misdelivery (which a checksum would have caught) is, to my mind, the best action possible.
The checksum is a proof-of-work minimum criterion; you don't do the work on the checksum, I don't do the work to carry you across my network.
RFC6935 was written from the perspective of tunnelers who benefit, not from the perspective of the network as a whole. We speak not for the factory, but for the trees. Prevent pollution.
Lloyd [email-masked]

On Tuesday, July 2, 2019, 2:47 pm, Tom Herbert <[email-masked]> wrote:

---

From: "C. M. Heard" <[email-masked]>
Date: Tue, 2 Jul 2019 10:33:04 -0700

On Mon, Jul 1, 2019 at 2:56 PM Tom Herbert wrote:
> On Mon, Jul 1, 2019 at 12:16 PM C. M. Heard  wrote:

Tom, after a more careful review of the relevant specifications I am
coming around to your point of view on this matter.

The normative requirements in RFC 6936 are in Section 4 (which defines
requirements and recommendations on the implementation of IPv6 nodes
that support the use of a zero UDP checksum) and in Section 5 (which
defines requirements and recommendations for protocols and tunnel
encapsulations that are transported over IPv6 without performing a UDP
checksum calculation). Section 3.41 of RFC 8085
<https://tools.ietf.org/html/rfc8085#section-3.4.1> reiterates these
requirements and levies another requirement of its own, which is stated
in the last bulleted paragraph below:

   o  Use of the UDP checksum with IPv6 MUST be the default
      configuration for all implementations [RFC6935].  The receiving
      endpoint MUST only allow the use of UDP zero-checksum mode for
      IPv6 on a UDP destination port that is specifically enabled.

   o  An application that supports a checksum different than that in
      [RFC2460] MUST comply with all implementation requirements
      specified in Section 4 of [RFC6936]
<https://tools.ietf.org/html/rfc6936#section-4> and with the usage
      requirements specified in Section 5 of [RFC6936]
<https://tools.ietf.org/html/rfc6936#section-5>.

   o  A UDP application MUST check that the source and destination IPv6
      addresses are valid for any packets with a UDP zero-checksum and
      MUST discard any packet for which this check fails.  To protect
      from misdelivery, new encapsulation designs SHOULD include an
      integrity check at the transport layer that includes at least the
      IPv6 header, the UDP header and the shim header for the
      encapsulation, if any [RFC6936].

So, to fully comply with existing requirements, FRAG+LITE with forced
CS=0 over IPv6 would have to conform with the following, even when
the user leaves the UDP checksum enabled for non-fragmented packets:

1.) The implementation would have to default to disabling FRAG+LITE,
i.e., unless the application using a particular port specifically
requests it, do not transmit datagrams fragmented in this manner and
do not accept datagrams fragmented in this manner (RFC 6936 Section 4,
requirements 2-5, RFC 6936 Section 5 requirement 1, and first bulleted
paragraph above). Ideally, this would be an API switch distinct from
those that allow UDP CS=0 in general, so that allowing it for FRAG+LITE
would not open up the floodgates for arbitrary UDP CS=0 datagrams. This
would require updates to Section 9 of draft-ietf-tsvwg-udp-options-07
<https://tools.ietf.org/html/draft-ietf-tsvwg-udp-options-07#section-9>.

2.) A transport layer checksum must be provided. The reassembly CS in
draft-ietf-tsvwg-udp-options-07 satisfies this requirement.

3.) A UDP application, or the transport layer working on its behalf,
MUST verify that the source and destination IPv6 addresses are valid
in any received packet that uses FRAG+LITE with CS=0 (RFC 8085 Section
3.4.1, last bulleted paragraph above). This is possible for the client
end of simple request-response protocols such as DNS or (with suitable
initial handshakes that don't rely on CS=0) for both ends of a UDP
application that sets up long-lived sessions between peers. It is not
possible for a general server that does not know (and cannot establish
during session setup) the source from which it will receive a packet.

4.) In order to cope with middleboxes that discard UDP datagrams with
CS=0, a UDP application that uses FRAG+LITE with forced CS=0 SHOULD
have a means to detect paths with high loss rates (RFC 6936 Section 5,
objectives 6 and 7). This is possible only for applications that use
long-lived flows.

>From my perspective, (1) and (2) are pretty much a wash between
FRAG+LITE with CS=0 and the new alternative proposed in
https://mailarchive.ietf.org/arch/msg/tsvwg/Wv--BLVMPAX6g5umok9BQsXAEGg:
the new proposal provides a transport checksum of equivalent strength,
and we still want application control over whether UDP fragmentation is
enabled (which IMO should be on a per-port basis and should not happen
by default). The new proposal does not disable the UDP checksum and so
is not subject to (3). That is a big deal in my book, because (3) is a
major burden. And (4) does not apply to the new proposal unless the
application explicitly requests UDP CS=0, in which case it applies anyway.

Mike

---

From: Tom Herbert <[email-masked]>
Date: Tue, 2 Jul 2019 11:07:14 -0700

On Tue, Jul 2, 2019 at 10:33 AM C. M. Heard <[email-masked]> wrote:
> checksum calculation). Section 3.41 of RFC 8085 reiterates these
>       specified in Section 4 of [RFC6936] and with the usage
>       requirements specified in Section 5 of [RFC6936].
>

Mike,

Right. Conceptually the reassembly checksum could meet the criteria if
the IP addresses were included, but that would break in the presence
of NAT since no NAT devices are going update a new checksum field
(with the possible exception that NAT makes a checksum neutral changes
as described in RFC6296).

So the best approach is to not touch the UDPv6 checksum and to ensure
that the surplus area sums to zero (i.e. the UDP checksum effectively
covers both the UDP payload and surplus area). As I've mentioned
previously, this motivates putting a required checksum field in the
surplus space not just in an option like OCS. A per packet checksum in
surplus area has other benefits beyond making UDP checksum compatible
with middleboxes.

Tom

> would require updates to Section 9 of draft-ietf-tsvwg-udp-options-07.
>

---

From: "C. M. Heard" <[email-masked]>
Date: Tue, 2 Jul 2019 16:50:37 -0700

> surplus area has other benefits beyond making UDP checksum compatible

In https://mailarchive.ietf.org/arch/msg/tsvwg/_RKlp1O4JgfCUoWBr2L5VKLCj_M
I proposed something very similar to that:

(a') OCS must be included by a sender whenever the UDP checksum is enabled
     (I agree that OCS seems pointless when the user has requested CS=0)
(b') OCS, when present, should provide coverage from the start of the
     options area (before LITE is moved on transmit or after it is moved on
     receive) to the end of the packet

That indeed does provide advantages beyond middlebox traversal, not the
least of which is that OCS can be verified ***before*** one loops through
all the option TLVs.

Being agnostic on whether anything after the end of the UDP options (if
marked by an EOL option) should be allowed, I see no particular advantage
to having a fixed header as described in draft-herbert-udp-space-hdr as
opposed to the OCS and NOP options, assuming that OCS has an implicit
length of two. If we require two-byte or four-byte alignment, then the
latter combination will consume less space than the fixed header when the
regular UDP payload ends on a byte boundary that is 1 mod 2 or 1 mod 4
(respectively), and if we don't constrain the placement of OCS, it may
consume less space in other cases as well. It will always consume less
space if we don't levy an alignment requirement.

Also, I'm of the opinion that an optional-to-implement capability for the
user to specify UDP CS=0 for a particular port (in conformance with
Section 4 of RFC 6936) should be retained when UDP options are enabled.
If (as Joe and I argue) OCS is pointless in that case, we might as well
save space by omitting it, which not possible with a fixed header.

Mike