Subject: [tsvwg] A review of draft-ietf-tsvwg-udp-options-12

From: "C. M. Heard" <[email-masked]>
Date: Thu, 3 Jun 2021 19:22:02 -0700

TSVWG,

This message is a detailed review of draft-ietf-tsvwg-udp-options-12. My
apologies for the length delay in getting it out. My opinion is that the
draft is NOT READY for WG last call; it needs at least one more pass to
correct some significant editorial and technical errors. Please note that
there will be a follow-on message dealing exclusively with the FRAG option;
I think it would be helpful if responses concerning that option were sent
as part of that thread.

Here is a summary of major issues that IMO still need to be addressed.
Details (and some less important issues) will follow.

   1. Minimum option lengths: there is contradictory text between Section
   4, which requires that the entire option area be dropped if an option
   length is too small, and section 5, which states that receivers cannot treat
   unexpected option lengths as invalid.
   2. Extended length format: there is contradictory text (also between
   Sections 4 and 5) as to whether the extended length format is required in
   all cases for all options that may need it, or whether the encoding
   should be the simplest possible. Recommendation: go with the latter
   (default format when possible, extended format only when the option
   instance occupies 255 bytes or more).
   3. Required options: I have no objection to requiring an implementation
   to recognize all of EOL, NOP, OCS, ACS, MSS, MRSS, and FRAG. I do,
   however, strongly object to requiring that an implementation be required to
   support generation and reassembly of fragments. I think it is very likely
   too high a bar for many implementers, and as much as I like the FRAG
   option, I would rather see the FRAG option removed altogether than see
   uptake of the specification impeded by a requirement to actually implement
   fragmentation and reassembly. There is a distinction between "must support"
   and "must recognize," and I think the spec needs to reflect this nuance.
   4. EOL option: the -12 draft requires that trailing padding
   following NOP be zeroes and requires that the receiver check that this is
   the case. I point out that this defeats the whole raison d'être for this
   option, and that if this requirement is maintained we might as well
   eliminate EOL and use NOP instead. Recommendation: keep the requirement
   that all padding past EOL be all zeroes, but do not require the receiver to
   check this. Correct decisions on this matter require careful thinking about
   the threat model and specifically on how covert channels would actually be
   used and countered by security equipment (hint: the behavior of a benign
   receiver has no role in countering typical threats).
   5. NOP option: why a limit of three? Why not allow seven NOPs for
   8-octet alignment? (Note: an arbitrary number at the end would need to be
   allowed if EOL is deleted).
   6. FRAG: the spec in the -12 draft is not coherent as it stands
   (specifically, OFFSET is not defined). At a minimum that needs to be
   corrected and the text edited to make the narrative coherent; however,
   there is a more economical way to design FRAG, and my subsequent message
   presents that alternative.
   7. MRSS: if the WG agrees that the capability to reassemble fragments
   should be optional, then there should be a stipulation that inclusion of
   the MRSS option indicates that the sender is capable of reassembling UDP
   fragments up to the limit indicated by MRSS. This would be an ideal way to
   enable the use of FRAG with DNS, a prime use case.
   8. UNSAFE option: what is propsed will work, but it is much less
   efficient than allocating a single bit in the option kind for this purpose.
   I'm aware that this was introduced several months ago, in the -09 draft,
   but as far as I can tell, there has been no discussion of the WG about this
   point. At a minimum I would like to see that discussion take place.
   9. AE: it is unclear to me why this is not considered to be an unsafe
   option, especially when the datagram is encrypted. There is also a
   per-fragment vs per-datagram ambiguity that needs to be resolved.
   Per-datagram is more in line with TCP-AO (in which case making it UNSAFE is
   correct).

Below is a section-by-section review covering the above points in more
detail, as well as a few relatively minor issues.

*Section 4, Page 7:*

   >> Any UDP option whose length is only smaller than 255 MUST always
   use the UDP option default format shown in Figure 4, excepting only
   EOL and NOP.

*Changing "is only smaller" to "is always smaller" would make this read
better. Also, the list of exceptions should include OCS (and FRAG, if my
forthcoming proposal is accepted), since that option (those options) have
no length field.*

   >> Any UDP option whose length can be larger than 254 MUST always
   use the UDP option extended default format shown in Figure 5,
   including UNSAFE and EXP.

   I.e., a UDP option always uses only the default format or the
   extended default format, depending on whether its length is only
   ever smaller than 255 or not.

*That is contradicted in Section 5 on page 9 by:*

   >> Options with fixed lengths MUST use the default option format.

   >> Options with variable lengths MUST use the default option format
   where their total length is 254 bytes or less.

   >> Options using the extended option format MUST indicate extended
   lengths of 255 or higher; smaller extended length values MUST be
   treated as an error.

*That contradiction must be resolved, preferably by accepting the language
in Section 5 and rewriting the earlier language (i.e., always use the
format that produces the shortest encoding). The language in the UNSAFE and
EXP sections needs to be reconciled also.*

   >> UDP options MUST be interpreted in the order in which they occur
   in the UDP option area.

*An exception should be allowed for OCS. It can be processed by computing
the Internet checksum over the surplus area conceptually prefixed by its
length, before attempting to parse any TLVs in the surplus area. This
should be a permissible implementation option (and IMO should be actively
encouraged).*

*Section 5, Page 8:*

   >> An endpoint supporting UDP options MUST support those marked with
   a "*" above: EOL, NOP, OCS, ACS, FRAG, MSS, MRSS, and UNSAFE. This
   includes both recognizing and being able to generate these options
   if configured to do so. These are called "must-support" options.

*I do not object that implementations should be required to recognize these
options, but IMO it is a BAD IDEA to REQUIRE implementation of reassembly
of fragments. It should be sufficient that implementations of UDP options
recognize FRAG, be able to process a self-contained terminal fragment that
is also an initial fragment, and be able to recognize and discard sequences
of fragments if they are not able to reassemble them. As a corollary, it is
also necessary that implementations be able to indicate when they do
support reassembly, and I propose that inclusion of MRSS in a packet so
indicate -- thus, there should be no need to send MRSS unless reassembly of
UDP fragments is supported, nor to process MRSS unless generation of UDP
fragments is supported. So, in summary, I propose:*

- All implementations of UDP options must support EOL, NOP, OCS, ACS (using
CRC32c), and MSS.
- All implementations must be able to recognize and process a
self-contained terminal fragment that is also an initial fragment.
- Support for generation and reassembly of UDP fragments is OPTIONAL (and
an implementation may support one of these capabilities without supporting
the other).
- If an implementation supports reassembly of UDP fragments, it must be
able to generate the MRSS option.
- If an implementation supports generation of UDP fragments, it must be
able to honor the MRSS option.
- All implementations must be able to recognize FRAG in both terminal and
non-terminal forms and to discard packets with non-terminal fragments if
reassembly is not supported.
- All implementations must be able to recognize UNSAFE options and discard
packets that contain UNSAFE options that are unknown or that occur outside
of a UDP fragment.

*Section 5, Page 9:*

   Receivers cannot treat unexpected option lengths as invalid, as this
   would unnecessarily limit future revision of options (e.g., defining
   a new ACS that is defined by having a different length).

*That is contradicted in Section 4 on page 7 by*

   >> Option Lengths (or Extended Lengths, where applicable) smaller
   than the minimum for the corresponding Kind and default format MUST
   be treated as an error. Such errors call into question the remainder
   of the option area and thus MUST result in all UDP options being
   silently discarded.

*That contradiction must be resolved, preferably by accepting the language
in Section 5 and deleting the above paragraph from Section 4 (i.e., always
treat a length that is not understood by skipping the option, except in the
case of ACS, where it should be treated as a checksum failure). A minimum
length dependent on the option Kind could not possibly work for any options
to be defined in the future, since their minimum lengths are unknown to old
implementations, and we need to ensure that new options can be correctly
processed by those old implementations. If the objective is to treat as a
parse error a default format length less than 2 or an extended format
length less than 255 (or 4 if we don't enforce a smallest encoding rule),
then that is what this paragraph should say (while noting also that the
rule does not apply to the mandatory-to-recognize options defined in this
spec that have an implied length).*

   >> "Must-support" options other than NOP and EOL MUST come before
   other options.

*This text does not properly take into account the interaction with FRAG
(either as specified in the -12 draft or in my forthcoming
counter-proposal).*

*Section 5.1, Page 9:*

   The End of Options List (EOL) option indicates that there are no
   more options. It is used to indicate the end of the list of options
   without needing to pad the options to fill all available option
   space.

*That last phrase ("without needing to pad") is contradicted in Section 5.1
on page 10 by:*

   >> All bytes in the surplus area after EOL MUST be zero. If these
   bytes are non-zero, the entire surplus area MUST be silently ignored
   and only the UDP data passed to the user with an adjusted UDP length
   to indicate that no options were present.

   Requiring the post-option surplus area to be zero prevents side-
   channel uses of this area, requiring instead that all use of the
   surplus area be UDP options supported by both endpoints. It is
   useful to allow for such padding to increase the packet length
   without affecting the payload length, e.g., for UDP DPLPMTUD [Fa21].

*There is NO POINT in having an EOL option if the entire data area
following the EOL contains data that must be checked by the receiver. You
might as well just fill the remaining surplus area with NOPs, which would
force all use of the surplus area to be supported by both endpoints … if,
that is, they both support UDP options, and modulo the "SHOULD NOT" in Sec.
5.2 (which could be adjusted). But I think that is the wrong direction to
go.*

*Allow me to point out that malicious actors who might use the surplus area
to exfiltrate information (e.g., stolen credit card data) in violation of a
security policy can do so NOW unless intermediate systems are present to
monitor the potentially illegitimate uses and filter the packets containing
them. My recommendation would be to say that the sender MUST fill the area
past EOL with zeroes so that network security devices can easily detect
illegitimate uses, but to levy NO requirements on the receiver to verify
this. (For a good explanation of data exfiltration using DNS, see
https://blogs.akamai.com/2017/09/introduction-to-dns-data-exfiltration.html
<https://blogs.akamai.com/2017/09/introduction-to-dns-data-exfiltration.html>;
certainly in that scenario, the well-behaved receivers have no role to play
in prevention of illegitimate exfiltration of data).*

*Section 5.2, Page 10:*

   >> Segments SHOULD NOT use more than three consecutive NOPs. NOPs
   are intended to assist with alignment, not other padding or fill.

*Assuming that the comments on Section 5.1 do not result in a need for more
extensive surgery, the question arises why it would not be legitimate to
allow for seven consecutive NOPs so as to achieve eight-octet alignment.*

*Section 5.3, Page 11:*

   >> When present, the OCS SHOULD occur as early as possible, preceded
   by only NOP options for alignment and the FRAG option if present.

*This language will need to be adjusted if the forthcoming  FRAG
counter-proposal is accepted.*

   This Internet checksum is computed over the surplus area (including
   EOL, if present)

*s/EOL, if present/EOL and any trailing zeroes, if present/ (unless EOL is
eliminated based on previous comments).*

   As a reminder, use of the UDP checksum is optional when the UDP
   checksum is zero. When not used, the OCS is assumed to be "correct"
   for the purpose of accepting UDP packets at a receiver (see Section
   7).

*s/UDP checksum is optional/option checksum is optional/*

*Section 5.4, Page 13:*

   >> Packets with incorrect ACS checksums MUST be passed to the
   application by default, e.g., with a flag indicating ACS failure.

   Like all non-UNSAFE UDP options, ACS need to be silently ignored
   when failing. Although all UDP option-aware endpoints support ACS
   (being in the required set), this silently-ignored behavior ensures
   that option-aware receivers operate the same as legacy receivers
   unless overridden.

*There should be no prohibition on having a per-socket option to silently
discard datagrams with incorrect ACS, so: **s/ACS need to be silently
ignored when failing/ACS needs to be silently ignored when failing, unless
different behavior has been configured/. Also, please add the following (cf
Section 5, Page 9):*

   >> Packets with unrecognized ACS lengths MUST be receive the same

   treatment as packets with incorrect ACS checksums.

*Section 5.5, Pages 13-16:*

   The Fragmentation option (FRAG) combines properties of IP
   fragmentation and the UDP Lite transport protocol [RFC3828].

[ . . . ]

   >> When FRAG is present, it MUST come first in the UDP options list.

   >> When FRAG is present, the UDP payload MUST be empty. If the
   payload is not empty, all UDP options MUST be silently ignored and
   the payload received to the user.

[ . . . ]

                   +--------+--------+--------+--------+
                   | Kind=4 | Len=10 |      Offset     |
                   +--------+--------+--------+--------+
                   |           Identification          |
                   +--------+--------+--------+--------+
                   |  Frag. Offset   |
                   +--------+--------+

              Figure 11   UDP non-terminal FRAG option format

[ . . . ]

                   +--------+--------+--------+--------+
                   | Kind=4 | Len=12 |      Offset     |
                   +--------+--------+--------+--------+
                   |           Identification          |
                   +--------+--------+--------+--------+
                   |  Frag. Offset   | Reassy. Checksum|
                   +--------+--------+--------+--------+

                Figure 12   UDP terminal FRAG option format

*The Offset field, and the other provisions quoted above, were introduced
in going from -08 to -09. Unfortunately, there is no new text that contains
the definition and usage of the Offset field. I can guess that "Offset" is
intended to mean the offset to the next option (i.e., the number of octets
of fragment data), but this should not be left for the reader to
infer. Without this being clearly specified, it is not possible for an
implementer to know for sure what to implement. The statement in the first
paragraph of the section that partial checksum coverage is provided is, as
far as I can tell, no longer true; with the updated OCS specification (and
specifically the requirement to include OCS whenever the UDP checksum is
present), there is no real possibility of partial checksum coverage (and
indeed, a partial checksum would defeat the middlebox traversal properties
that the updated OCS was designed to provide). Additionally, there is no
longer any need for a reassembly checksum: given that the OCS option covers
the entire surplus area (and is mandatory whenever the UDP checksum is
non-zero), it provides complete segment-by-segment checksum coverage. That
provides coverage for the reassembled datagram payload that is equivalent
to that provided by TCP for an equivalent amount of data. The analogy is
essentially perfect because the fragment offset (in the context of a given
Identification value) is exactly analogous to the TCP sequence number. The
only errors that would be detected by a separate reassembly checksum that
would not be detected by OCS would be two sets of errors, one in the
options overhead and the other in the fragment data, that together happen
to have a zero checksum. That is possible, but it is unlikely, and it is
something that TCP has dealt with successfully over the years. ACS is
available for situations when a stronger checksum for the reassembled data
is desired.*

*In a forthcoming message I will make an alternative proposal that includes
full replacement text for Section 5.2. The basic idea originally appeared
in https://mailarchive.ietf.org/arch/msg/tsvwg/Wv--BLVMPAX6g5umok9BQsXAEGg/
<https://mailarchive.ietf.org/arch/msg/tsvwg/Wv--BLVMPAX6g5umok9BQsXAEGg/>.
The** main features are:*

*- Eliminate the Offset field and the requirement that FRAG appear first.
Instead, require that the FRAG option appear LAST and consume all following
surplus data.*
*- Eliminate the Reassy. Checksum field and replace Len (which would
otherwise have a fixed value of 8) by a More Fragments (MF) flag.*

*Please respond to that message to criticize the alternate FRAG proposal.
It's fair to reply to this message to rebut my critique of the existing
text in Section 5.2.*

*Section 5.7, Page 17:*

   The Maximum Reassembled Segment Size (MRSS, Kind=6) option is a 16-
   bit indicator of the largest reassembled UDP segment that can be
   received. MRSS is the UDP equivalent of IP's EMTU_R but the two are
   not related [RFC1122]. Using the FRAG option (Section 5.5), UDP
   segments can be transmitted as fragments in multiple IP datagrams
   and be reassembled larger than the IP layer allows.

*s/multiple IP datagrams/multiple complete IP datagrams/ ... the current
text agrees with the terminology in RFC 1122, but I think the addition
makes the text a bit clearer.*

*Also, if the capability to reassemble fragments is made optional, a
sentence should be added to this section stipulating that the presence of
the MRSS option indicates that the sender has that capability. This will
allow (e.g.) a DNS client to safely signal to a DNS server that a response
may be sent using UDP fragmentation, and it will tell the server how large
a reply using that mechanism could be. A server that does not have the
capability to send fragmented UDP datagrams -- including a server that does
not support UDP options at all -- would send a conventional UDP response.*

*Section 5.8, Pages 17-18:*

   UNSAFE options are an extended option space, with its own additional
   option types. These are indicated in the first byte after the option
   Kind as shown in Figure 15, which is followed by the Length. Length
   is 1 byte for UKinds whose total length (including Kind, UKind, and
   Length fields) is less than 255 or 2 bytes for larger lengths (in
   the similar style as the extended option format).

                        +--------+--------+--------+
                        | Kind=7 | UKind  | Length |...
                        +--------+--------+--------+
                          1 byte   1 byte  1-2 bytes

                   Figure 15   UDP UNSAFE option format

*IMO it would have been preferable to use a bit (e.g., the MSB) in the
option kind for this purpose. The UNSAFE option as specified will
accomplish the purpose, but it has a more complex encoding with no obvious
advantage. In any case, shouldn't the length be either 1 byte or 3 bytes,
as outlined in  Section 4?*

   >> UNSAFE options MUST be used only as part of UDP fragments, used
   either per-fragment or after reassembly.

*This appears to contradict the wording in  Section 5.5 that "UDP options
MUST NOT follow the FRAG header in non-terminal fragments." It may be
useful to allow per-fragment options other than NOP, OCS, AE, and FRAG, but
I must admit that I cannot think of an example offhand.*

*Section 5.10, Pages 19-20:*

   The Authentication and Encryption (AE) option is intended to allow
   UDP to provide a similar type of authentication as the TCP
   Authentication Option (TCP-AO) [RFC5925]. AE the conventional UDP

   payload and may also cover the surplus area, depending on
   configuration.

*s/AE the/AE covers the/*

   >> Packets with incorrect AE HMACs MUST be passed to the application
   by default, e.g., with a flag indicating AE failure.

   Like all non-UNSAFE UDP options, AE needs to be silently ignored
   when failing. This silently-ignored behavior ensures that option-
   aware receivers operate the same as legacy receivers unless
   overridden.

*Why isn't AE an UNSAFE option? Ignoring the option would lead to accepting
unauthenticated data or accepting cipher text as plaintext. IMO neither of
these outcomes is acceptable, and both are avoidable. In addition, this
section needs to specify whether this option is per-fragment, per complete
UDP datagram, or either, depending on configuration. My sense is that **AE
is most naturally a per-fragment option, and that a per-datagram non-UNSAFE
version is not really needed. That most closely follows the TCP-AO model.*

*Section 5.12, Page 21:*

               +----------+----------+----------+----------+
               | Kind=254 |   Len    |      UDP ExID       |
               +----------+----------+----------+----------+
               |  (option contents, as defined)...         |
               +----------+----------+----------+----------+

                     Figure 19   UDP EXP option format

*Don't we also need to document this format too?*

               +----------+----------+----------+----------+
               | Kind=254 |   255    |         Len         |
               +----------+----------+----------+----------+
               |      UDP ExID       |(cont.,as defined)...|
               +----------+----------+----------+----------+

                     Figure 20   UDP EXP option format

*Section 6, Pages 22-23:*

   The UDP option Kind space allows for the definition of new options,
   however the currently defined options do not allow for arbitrary new
   options. For example, FRAG needs to come first if present; new
   options cannot declare that they need to precede it. The following
   is a summary of rules for new options and their rationales:

*Allowance should be made for OCS and NOP to precede FRAG if the alternate
FRAG proposal is adopted.*

   >> New options MUST NOT depend on option space content, excepting
   only those contained within the UNSAFE option. Only OCS and AE
   depend on the content of the options themselves and their order is
   fixed (on transmission, AE is computed first using a zero-checksum
   OCS if present, and OCS is computed last before transmission, over
   the entire option area, including AE).

   >> UNSAFE options can both depend on and vary option space content
   because they are contained only inside UDP fragments and thus are
   processed only by UDP option capable receivers.

*UNSAFE options cannot depend on option or modify option content if AE and
OCS are to work predictably. IMO the complete ban on such dependence
(except for OCS and AE) that appears in previous drafts is the right way to
go and should be reinstated.*

   Note that only certain of the initially defined options violate
   these rules:

   o  >> FRAG MUST be first, if present, and MUST be processed when
      encountered (e.g., even before security options).

   o  >> Only FRAG and UNSAFE options are permitted to modify the UDP
      body or option areas.

   o  >> OCS SHOULD be the first option, except in the presence of
      FRAG, in which case it SHOULD be the first option after FRAG.

*See above.*

*Section 7, Page 23:*

   >> All options MUST be processed by receivers in the order
   encountered in the options list.

*It is possible to process OCS before the options area is parsed by
calculating the Internet checksum of the surplus area with the surplus
length pseudo-header. That does not in any way depend on where the OCS
option is located. A check of OCS before processing the surplus area should
be both allowed and encouraged. I personally would not countenance an
implementation that did anything else.*

*Section 10, Page 26:*

   The UDP FRAG options option supports a similar service to UDP-Lite.

*The statement that a service like UDP-Lite is offered has to go, it is
untrue. I recommend deleting this entire section as it has been overtaken
by events.*

*Section 15, Page 29:*

   ROHC compresses UDP headers only when this match succeeds. It does
   not prohibit UDP headers where the match fails; in those cases, ROHC
   default rules (Section 5.10) would cause the UDP header to remain
   uncompressed. Upon receivep of a compressed UDP header, Section
   A.1.3 of that document indicates that the UDP length is "INFERRED";
   in uncompressed packets, it would simply be explicitly provided.

*s/receivep/receipt/ (typo).*

*Section 17, Page 30:*

   [ . . . ] Transport security is provided in TCP by the TCP
   Authentication Option (TCP-AO [RFC5925]) or in UDP by the
   Authentication Extension option (Section 5.10). Transport headers
   are also protected as payload when using IP security (IPsec)
   [RFC4301].

*s/Authentication Extension option/Authentication and Encryption option/*

*Thanks for reading to the end. Please look at the upcoming missive on an
alternative version of FRAG if that subject is of interest.*

*Mike Heard*

---

From: Joseph Touch <[email-masked]>
Date: Thu, 10 Jun 2021 21:07:49 -0700

Hi, Mike,

I’ve been waiting for others to participate, but given it’s been a week…

First, I want to thank you for the deep review - it’s sincerely appreciated.

I’ll also be the first to admit there are more than a few places where it needs another pair of eyes and may have inconsistencies. So let’s start by assuming that those are dealt with separately. At a minimum, assume that most of the obvious corrections will be included in the next update.

I will respond  to the revised LITE option in reply to your other post.

On the primary issues, below summarizes my perspective. I hope others on the list will weigh in.

Joe

-----

- regarding fragment/reassembly as being required
	This is one of the primary current intended use cases and motivations for UDP options. 
	That’s why it’s currently a MUST support and why I do not think that should change.

- regarding removing EOL
	Removing EOL raises several questions: can other options use that data? Do we care if it’s a covert channel? etc.
	It’s simpler to declare it zeroes and let EOL be the trigger to use a loop with a branch predictor hint (likely zero).
	At a minimum, everything past EOL MUST be covered by OCS (otherwise OCS would need to parse all options to find EOL).
	
- regarding increasing NOPs
	There was a concern in letting these be unlimited; we can increase to 7, 15, or even 31 (high enough for future-proof seems
	to open a DOS problem; IMO, DOS detection should just check to see if this is a persistent load, not a specific number)

- regarding UNSAFE option format
	Yes, we could pick a range rather than making them longer, but IMO they aren’t likely except as already accounted for, so 
	allocating space from the main range is a waste.
	However, if we did allocate from the range, we should NOT do so with a single bit; that gives the false impression
	that a given option value has two variants - safe and unsafe, and that’s not the case.

- regarding AE being safe
	This is not considered unsafe for two reasons:
	1. A(authentication) isn’t unsafe
	2. Encryption should only be used when sending packets to a party that is keyed; that can/should be known or checked before use anyway.
	i.e., there’s never a case where you send encrypted text to a party you don’t know should be ready.

----

---

From: <[email-masked]>
Date: Fri, 11 Jun 2021 11:43:11 +0000

Hi Joe, all,

Regarding the FRAG option, I'm not sure there was a consensus this is a MUST to implement/support. 

Unless I'm mistaken, I didn't even remember a conclusion for one of the main threads when FRAG and other "design assumptions" were discussed back in 07/2019.

As indicated in https://mailarchive.ietf.org/arch/msg/tsvwg/rfh1wBYzlIHIzeVQruT5BVd8BRA/, I do still think that we would better focus on key foundations and build over them. FRAG can be defined properly in a separate document. It can be tagged as unsafe but it does not need to be mandatory to implement. 

Likewise, I do still think it is weird to assign codepoints for REQ/RES, but no details about the processing of these options is provided. These options are better covered in Gorry's draft and kinds assigned there. BTW, I'm still hoping a call for adoption can be issued for it. 

Cheers,
Med

> -----Message d'origine-----
> De : tsvwg [mailto:[email-masked]] De la part de Joseph
> Touch
> Envoyé : vendredi 11 juin 2021 06:08
> À : C. M. Heard <[email-masked]>
> Cc : TSVWG <[email-masked]>
> Objet : Re: [tsvwg] A review of draft-ietf-tsvwg-udp-options-12
> 
> Hi, Mike,
> 
> I’ll also be the first to admit there are more than a few places
> where it needs another pair of eyes and may have inconsistencies. So
> let’s start by assuming that those are dealt with separately. At a
> minimum, assume that most of the obvious corrections will be
> included in the next update.
> 
> 	Removing EOL raises several questions: can other options use
> that data? Do we care if it’s a covert channel? etc.
> 	It’s simpler to declare it zeroes and let EOL be the trigger
> to use a loop with a branch predictor hint (likely zero).
> 	At a minimum, everything past EOL MUST be covered by OCS
> (otherwise OCS would need to parse all options to find EOL).
> 
> 	There was a concern in letting these be unlimited; we can
> increase to 7, 15, or even 31 (high enough for future-proof seems
> 	to open a DOS problem; IMO, DOS detection should just check to
> see if this is a persistent load, not a specific number)
> 
> 	Yes, we could pick a range rather than making them longer, but
> IMO they aren’t likely except as already accounted for, so
> 	However, if we did allocate from the range, we should NOT do
> so with a single bit; that gives the false impression
> 	2. Encryption should only be used when sending packets to a
> party that is keyed; that can/should be known or checked before use
> anyway.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 11 Jun 2021 15:37:03 +0100

Hi all,

> Unless I'm mistaken, I didn't even remember a conclusion for one of the main threads when FRAG and other "design assumptions" were discussed back in 07/2019.
As I recall, this is the last fragmentation is the last thing we need to 
nail-down. I'm  unsure
> As indicated in https://mailarchive.ietf.org/arch/msg/tsvwg/rfh1wBYzlIHIzeVQruT5BVd8BRA/, I do still think that we would better focus on key foundations and build over them. FRAG can be defined pro…

On the one hand: I can ses it would be nice to have many things in the 
common standard and immediately available to use.

On the other hand: There seems to me to be singificantly more complexity 
and choice in managing reassembly than in the other aspects of UDP 
options- I can see how this could vary depending on usage: how much 
memory to use for reassembly, and what size is considered acceptable. To 
me, that makes it seems reasonable to ask whether a minimal implemention 
needs to offer this.

I know when someone in Aberdeen University implemented an earlier rev. 
of UDP-Options, this was the part that wasn't done.

> Likewise, I do still think it is weird to assign codepoints for REQ/RES, but no details about the processing of these options is provided. These options are better covered in Gorry's draft and kind…

I'd also like to see adoption:-). I think it would be nice if the 
UDP-Opt and UDP-DPLPMTUD could be published together with consecutive 
RFC numbers!

One way to avoid fragmentation can be to discover and cache a larger 
PMTU:-).

Gorry (as an individual tsvwg'er)

---

From: Joseph Touch <[email-masked]>
Date: Fri, 11 Jun 2021 09:39:56 -0700

Hi, all,

I agree with everyone that FRAG support shouldn’t be open-ended. Maybe we can say MUST implement at least 2-fragment reassembly, SHOULD implement up to 9K (that’s the upper bound on most MTUs that aren’t 64K AFAICT).

I understand it’s the last thing to be implemented because it’s hard, but that alone isn’t a reason to make it optional. Maybe we should just skip RDMA support as unnecessary; RDMA perhaps should tune to MTU rather than rely on long streams of UDP fragments. 

Also, I agree that REQ/RES needs to be explained; the I shared Gorry’s assumption that these two docs would go in together.

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 13 Jun 2021 10:08:24 -0700

> I’ve been waiting for others to participate, but given it’s been a week…
Joe, I share your desire to see greater WG participation. Silence is not a
good sign :-(

> On the primary issues, below summarizes my perspective. I hope others on
I will give a brief answer to each point and encourage others to weigh in.
These comments are mainly to clarify my position

>         That’s why it’s currently a MUST support and why I do not think
There have been some other comments on this (thank you, Med and Gorry) and
I will respond separately.

> - regarding removing EOL
>         Removing EOL raises several questions: can other options use that
> data? Do we care if it’s a covert channel? etc.
>         It’s simpler to declare it zeroes and let EOL be the trigger to
> use a loop with a branch predictor hint (likely zero).
I would be satisfied if the sender MUST set all data past EOL to zero and a
provision that the receiver MAY check this.

IOW my major objection is requiring that the receiver check this. If the
receiver is required to check, the fill may as well be NOPs.

I agree that there is no need to make provision for other uses of this area
and that it MUST be covered by OCS.

>         There was a concern in letting these be unlimited; we can increase
> to 7, 15, or even 31 (high enough for future-proof seems
>         to open a DOS problem; IMO, DOS detection should just check to see
> if this is a persistent load, not a specific number)
>
>
OK by me. All I asked for was to allow eight-octet alignment.

>         Yes, we could pick a range rather than making them longer, but IMO
> they aren’t likely except as already accounted for, so
>         However, if we did allocate from the range, we should NOT do so
> with a single bit; that gives the false impression
First, let me say that the method proposed in the -12 draft will work. I
just don't think it's an efficient way to go. If there is consensus in the
WG for what's in the -12 draft I can live with it.

>         2. Encryption should only be used when sending packets to a party
> that is keyed; that can/should be known or checked before use anyway.
I do not agree that this behaviour is optimum ... I'm of the opinion that
authentication should be a contract between the two ends. TC-AO could not
have done anything else because unknown TCP options are ignored. UDP-AE can
do better. That being said, if there is consensus in the WG for what's in
the -12 draft I can live with it.

Mike Heard

---

From: Joseph Touch <[email-masked]>
Date: Sun, 13 Jun 2021 10:21:14 -0700

Hi, Mike,

> On Jun 13, 2021, at 10:08 AM, C. M. Heard <[email-masked]> wrote:
> 
> I do not agree that this behaviour is optimum ... I'm of the opinion that authentication should be a contract between the two ends. TC-AO could not have done anything else because unknown TCP options are ignored. UDP-AE can do better. That being said, if there is consensus in the WG for what's in the -12 draft I can live with it.

FWIW, we don’t have any stateful handshake inside of UDP options - that’s deliberate. We COULD add that sort of thing - in the spirit of a zero-byte packet with an option check list (a new option where you list the option codepoints you want to support), but we’ve resisted that because we didn’t want to bake statefulness into the options themselves. In a sense, that state is at the endpoint user layer in how it interprets options only.

It might be sufficient to say that:

	- when AE is used as auth, it has no other restrictions
	- when AE is used as encryption, it MUST be used as an unsafe option, i.e., only as a post-reassembly option on FRAG-buried data

That way users who don’t support encryption wouldn’t get bad data.

Joe

---

From: Joseph Touch <[email-masked]>
Date: Sun, 13 Jun 2021 10:21:14 -0700

Hi, Mike,

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 13 Jun 2021 10:25:30 -0700

On Sun, Jun 13, 2021 at 10:21 AM Joseph Touch wrote:

> have done anything else because unknown TCP options are ignored. UDP-AE
> can do better. That being said, if there is consensus in the WG for
> what's in the -12 draft I can live with it.
>
>
> FWIW, we don’t have any stateful handshake inside of UDP options - that’s
> deliberate. We COULD add that sort of thing - in the spirit of a zero-byte
> packet with an option check list (a new option where you list the option
> codepoints you want to support), but we’ve resisted that because we didn’t
> want to bake statefulness into the options themselves. In a sense, that
> state is at the endpoint user layer in how it interprets options only.
>
> - when AE is used as encryption, it MUST be used as an unsafe option,
> i.e., only as a post-reassembly option on FRAG-buried data
>

That would work for me.

Mike

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 13 Jun 2021 12:28:39 -0700

On Fri, Jun 11, 2021 at 4:43 AM  Mohamed Boucadair wrote:

> Unless I'm mistaken, I didn't even remember a conclusion for one of the
> main threads when FRAG and other "design assumptions" were discussed back
> in 07/2019.
>
> As indicated in
> https://mailarchive.ietf.org/arch/msg/tsvwg/rfh1wBYzlIHIzeVQruT5BVd8BRA/,
> I do still think that we would better focus on key foundations and build
> over them. FRAG can be defined properly in a separate document. It can be
> tagged as unsafe but it does not need to be mandatory to implement.

Indeed, as far as I can tell, that discussion never did achieve consensus
on the matter of whether fragmentation is a requirement. So let's discuss.
Here's my position:

While I agree that it is not appropriate to require every implementation of
UDP options to support reassembly of fragments, I do believe that the FRAG
mechanism needs to be in the base specification of UDP options if we are
going to have it at all. It interacts with other options, and thus should
not be deferred to a future document.

> of UDP-Options, this was the part that wasn't done.
I believe that the references are:
https://datatracker.ietf.org/meeting/101/materials/slides-101-tsvwg-sessa-53-tom-jones-udp-options-implementation-02
https://datatracker.ietf.org/meeting/101/materials/minutes-101-tsvwg-02

On Fri, Jun 11, 2021 at 9:40 AM Joseph Touch wrote:

> I agree with everyone that FRAG support shouldn’t be open-ended. Maybe we
> can say MUST implement at least 2-fragment reassembly, SHOULD implement up
> to 9K (that’s the upper bound on most MTUs that aren’t 64K AFAICT).
>
> I understand it’s the last thing to be implemented because it’s hard, but
> that alone isn’t a reason to make it optional. Maybe we should just skip
> RDMA support as unnecessary; RDMA perhaps should tune to MTU rather than
> rely on long streams of UDP fragments.

As I stated in the review, I believe that all implementations of UDP
options should be required to recognize the FRAG option, but I believe that
it is a bridge too far to require actual reassembly of fragments. It would
IMO be fair to require implementations of UDP options to be able to process
and accept a datagram that consists of a single fragment that is both an
initial fragment and a terminal fragment. That is needed to support UNSAFE
options.  It is also straightforward and stateless; actual reassembly of
fragments is neither.

I also advocate that sending a MRSS (Maximum Reassembled Segment Size)
option be an unambiguous indication that an implementation supports
reassembly of fragments. My take is that the number of fragments that can
be accepted is an implementation-defined limit and thus a Quality of
Implementation matter and does not need to be standardized. A sender that
has received both the MSS and MRSS options from a particular destination
will have a very strong hint on when and how to fragment a datagram sent to
that destination.

FWIW, I think that the MRSS option is a valuable addition to the spec. I
wish I'd thought of it!

Thanks and regards,

Mike Heard

---

From: Joseph Touch <[email-masked]>
Date: Sun, 13 Jun 2021 12:55:26 -0700

> On Jun 13, 2021, at 12:28 PM, C. M. Heard <[email-masked]> wrote:
> 
> As indicated in https://mailarchive.ietf.org/arch/msg/tsvwg/rfh1wBYzlIHIzeVQruT5BVd8BRA/ <https://mailarchive.ietf.org/arch/msg/tsvwg/rfh1wBYzlIHIzeVQruT5BVd8BRA/>, I do still think that we would better focus on key foundations and build over them. FRAG can be defined properly in a separate document. It can be tagged as unsafe but it does not need to be mandatory to implement. 
> 
>  
> Indeed, as far as I can tell, that discussion never did achieve consensus on the matter of whether fragmentation is a requirement.

Agreed, but that’s because support for fragmentation was a design goal from the start (e.g., to support DNSSEC that spans multiple UDP packets).

If we remove that requirement, what is the driving use case?

I agree with you that we need all UDP-option-capable variants to recognize FRAG, which means single-frag as a minimum.

However, if we do not require reassembly beyond that, we’ll likely be stuck. To drive use of this option for the only currently known use case, we need to support reassembly of at least 2 fragments.

That effectively limits us to between 1K and 3K.

We already know that a reasonable upper bound would be 9K anyway. That argues for between 6 and 18 segments.

However, I think we could knock this down to either 1500B - the reassembly minimum for IPv6 anyway.

That would suggest:
	- RMSS should never be below 1500B
	- reassembly MUST support as many fragments as that requires, e.g., either 2 (in IPv6 stacks) or 4 (IPv4 stacks - supporting the IPv4 and some UDP headers).

Requiring support for reassembling 4 doesn’t seem like a huge burden. Any endpoint that can’t do that, IMO, should not be supporting UDP options.

And no, I do not think we should rely on RMSS to indicate that this is available. Note that a receiver would have to indicate RMSS and know the path MTU to know how many fragments it would actually take; we don’t require path MTU discovery for UDP options, so that’s an undue burden. Besides again injecting state into UDP.

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 13 Jun 2021 15:00:35 -0700

Responding to just this one specific point ...

On Sun, Jun 13, 2021 at 12:55 PM Joseph Touch wrote:

> And no, I do not think we should rely on RMSS to indicate that this is
> available. Note that a receiver would have to indicate RMSS and know the
> path MTU to know how many fragments it would actually take; we don’t
> require path MTU discovery for UDP options, so that’s an undue burden.
> Besides again injecting state into UDP.
>
>
For IPv6 -- where fragmentation has the greatest issues -- we are
guaranteed a minimum MTU of 1280 bytes. A DNS server operating over IPv6
would be able to serve up a fragmented response with the right number of
1280 byte fragments (or else set the TC bit) if it received the MRSS option
in the request from the client.

Mike Heard

---

From: Joseph Touch <[email-masked]>
Date: Sun, 13 Jun 2021 15:53:28 -0700

> On Jun 13, 2021, at 3:00 PM, C. M. Heard <[email-masked]> wrote:
> 
> For IPv6 -- where fragmentation has the greatest issues

Why?

Effectively, both IPv4 and IPv6 currently support only source fragmentation. On-path has lots of problems.

> -- we are guaranteed a minimum MTU of 1280 bytes. A DNS server operating over IPv6 would be able to serve up a fragmented response with the right number of 1280 byte fragments (or else set the TC bit) if it received the MRSS option in the request from the client.

I’m speaking of a DNSSEC response, not just DNS.

Sure, in IPv6 the max number of fragments would be MRSS/(1280 - IP and UDP headers and options).

In IPv4, it arguably could be MRSS/(576 - headers).

But in both cases, it could be different for larger MSS.

Joe

---

From: Paul Vixie <[email-masked]>
Date: Sun, 13 Jun 2021 22:56:40 +0000

> Responding to just this one specific point ...

same.

> > path MTU to know how many fragments it would actually take; we don???t
> > require path MTU discovery for UDP options, so that???s an undue burden.

computing UDP payload size by subtraction requires either that one be inside
the network stack (sometimes this means "in the kernel") so as to know the
interface MTU and perhaps the path MTU, as well as the IP6 header and option
sizes. when this information is unavailable, for example when running inside
a DNS responder, pessimistic assumptions have to be made, such as a payload
maximum for DNS data of 1232 or even 1220 octets. this is not just pessimistic
but also pessimal, and ought to be avoided.

i have high hopes for a non-pessimal future to be reached via:

https://www.rfc-editor.org/rfc/rfc8899.html

for which assumptions such as 1280, 1232, or 1220 will be "of last resort".
i think this may mean i disagree with joe's estimation of "undue" above. when
commodity (end user) hardware reaches terabit speed, we are going to wish we
were not shackled to the MTU of 10-megabit ethernet for wide area packets. in
fact i already wish this, but in the future the wish will be universal. it was
not too early to plan and develop for this future starting ~20 years ago, and
it is definitely not too early today.

-- 
Paul Vixie

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 13 Jun 2021 16:29:09 -0700

> For IPv6 -- where fragmentation has the greatest issues

See Geoff Huston's blog (for which Google is your friend). And yes,
though I didn't mention DNSSEC, that indeed is where the problems arise..

--cmh

---

From: Joseph Touch <[email-masked]>
Date: Sun, 13 Jun 2021 17:04:03 -0700

> On Jun 13, 2021, at 3:56 PM, Paul Vixie <[email-masked]> wrote:
> 
> sizes.

That’s correct; that’s exactly how TCP MSS works. The UDP stack can make this information available to the user.

> when this information is unavailable, for example when running inside
> a DNS responder,

A DNS responder would not be creating UDP option packets; it creates user that it passes to UDP. The same way TCP keeps ithis info on a per-socket basis, UDP is expected to.

Joe

---

From: Joseph Touch <[email-masked]>
Date: Sun, 13 Jun 2021 19:12:44 -0700

So, for everyone, getting back to the list and in summary, AFAICT (please let me know if I miss anything):

> On Jun 10, 2021, at 9:07 PM, Joseph Touch <[email-masked]> wrote:
> 

Mike and I (at least) appear to agree that FRAG is required and at least single-frag messages MUST be supported.

I hope we can at least require reassembly of at least 2, if not 4 fragments.

> - regarding removing EOL

I think we agree to keep EOL and:
	- transmitter MUST set everything post EOL as zero
	- receiver MAY check those zeros
	- those zeroes MUST be covered by OCS

> - regarding increasing NOPs

I think we still want a limit, but it MUST NOT be smaller than 7 in a row.

IMO, checks for excessive use of NOPs as DOS should be optional.

> - regarding UNSAFE option format

I think it is probably sufficient to reserve a block of option values for this purpose, BUT not a single bit that would imply options could be “unsafe or not, depending on the flag”. I’ll look at the number space to see what might work (probably at most 25%)

> - regarding AE being safe

Revise so that:
- when used as AUTH, it is safe and can be used wherever
- when used as ENCRYPT (modifying payload data), it MUST be used only as post-FRAG to be safe

Finally, regarding FRAG:
- we need to decide whether we want to support zero-copy (probably a better way of putting it than RDMA)
	if so, then we should also consider whether to allow options after non-terminal fragments
	that would operate on the fragment only (this would help reduce the amount of data that needs to be moved
	to allow zero copy)

- either way
	- non-terminal fragments MUST use FRAG last
	- terminal fragments include post-FRAG options that operate on the entire reassembled fragment

If we DO NOT support zero-copy, the format would be:

	Nonterminal format, where OPTLEN = 8
		KIND/8, OPTLEN/8, FRAGOFFSET/16
		FRAGID/32

	Terminal format, where OPTLEN = 10
		KIND/8, OPTLEN/8, FRAGOFFSET/16
		FRAGID/32
		THISFRAGLEN/16

	FRAGOFFSET = where to place the fragment in the reassembled user data 
	THISFRAGLEN = length of the terminal fragment, i.e., indicating where the post-frag options resume

	(note - this is what we have in the current draft, version -12, except it swaps FRAGOFFSET and THISFRAGLEN
	and omits it from non-terminal fragments)

If we DO support zero-copy and thus want to allow non-terminal fragments to have post-fragoption options that operate on each fragment, then we would add THISFRAGLEN to the nonterminal format and issue different KIND numbers to nonterminal/terminal fragment.

---

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 13 Jun 2021 19:20:33 -0700

On Sun, Jun 13, 2021 at 7:12 PM Joseph Touch <[email-masked]> wrote:

>         that would operate on the fragment only (this would help reduce
> the amount of data that needs to be moved
>         to allow zero copy)
>
> - either way
>         (note - this is what we have in the current draft, version -12,
> except it swaps FRAGOFFSET and THISFRAGLEN
> If we DO support zero-copy and thus want to allow non-terminal fragments
> to have post-fragoption options that operate on each fragment, then we
> would add THISFRAGLEN to the nonterminal format and issue different KIND
> numbers to nonterminal/terminal fragment.
>

I for one would appreciate further discussion of these last points. I admit
that I have failed to grasp Joe's message on the RDMA thread, and I would
appreciate some time to think about it.

Thanks

Mike

---

From: Joseph Touch <[email-masked]>
Date: Sun, 13 Jun 2021 21:31:10 -0700

> On Jun 13, 2021, at 7:20 PM, C. M. Heard <[email-masked]> wrote:
> 
> I for one would appreciate further discussion of these last points. I admit that I have failed to grasp Joe's message on the RDMA thread, and I would appreciate some time to think about it.

Sure - here’s how it all works. Note that this is relevant mostly for long transfers with persistent UDP fragmentation; if that is assumed to be ‘adjusted’ at the app layer (as QUIC does), then we don’t need zero-copy support...

- right now, UDP data can be zero-copied when received into user space, starting with the user data
- if we add options, UDP data can still be zero-copied because it hasn’t moved (it still begins the payload
- however, fragments are different because (esp given the merging of frag and lite) they don’t start at the beginning of data
	- they always start after OCS (which I think we should make fit the uniform KIND/LEN/OCS format of 4 bytes)
	- if the FRAG comes next, then we can move the frag content around a little and still support zero-copy

		notably, we move the first 10 bytes of the fragment to the end
			4 for OCS
			6 for FRAG (assuming FRAG includes KIND/OPTLEN/FRAGOFFSET/ID/FRAGLEN)
		that way we can zero-copy the frag packet into place, then just copy those last 8 bytes over OCS and the FRAG header

This method assumes that we try to keep FRAG early in the packet - preferably right after OCS. The later it comes, the more additional bytes we need to move to “fix” the copy (beyond the 8 bytes noted above).

—

This method is the only reason we would want to allow options after non-terminal fragments - basically to keep the fragment toward the front of the packet, using the rule that post-noninitial frag options still operate on the fragment, rather than waiting for reassembly. The exception is the terminal fragment, where post-terminal fragment options operate on the reassembled packet. 

Joe

---

From: Joseph Touch <[email-masked]>
Date: Sun, 13 Jun 2021 21:34:56 -0700

PS - I should note that I don’t care strongly that we keep this zero-copy support, It’s not important for individual packets; it’s useful only for streams, IMO anyway. If omitting it is fine, then we can make FRAG processing simpler.

The argument for omitting it is:
- zero-copy is more useful for streams where apps should adjust MTU anyway and thus wouldn’t need FRAG for MTU adjustment
- zero-copy works fine when the data isn’t UNSAFE (in the body)
- when data is otherwise UNSAFE, it’s because it is modified (encrypted, compressed, etc.), where zero-copy is not relevant either

Joe

> On Jun 13, 2021, at 9:31 PM, Joseph Touch <[email-masked]> wrote:
> 
> 
> 
>> On Jun 13, 2021, at 7:20 PM, C. M. Heard <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 14 Jun 2021 12:00:05 +0100

<snip>
On 14/06/2021 03:12, Joseph Touch wrote:
>> On Jun 10, 2021, at 9:07 PM, Joseph Touch<[email-masked]>  wrote:
>>
<snip>

I can believe that 2 fragment reassembly could be a useful general 
compromise to think more on:

* Two fragments can often be significantly more helpful than no 
fragments: where the PMTU is reduced by a small size (e.g., to include a 
tunnel header) and datagrams need to be fragmented.

* Endpoints concerned about resources, can examine the second received 
fragment and immedaitely determine whether the oiriginal datagram can be 
reassembled. There are fewer opportunities for overlapping/small 
fragments to cause mischief. It might even be possible to set a receiver 
"reordering window" outside of which unreassembled Fragments are 
silently dropped.

Gorry

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 14 Jun 2021 12:03:09 +0100

Tom, do you recall why you hated the mangling of byte order in the 
UDP-Option fragment proposal?

Gorry

On 14/06/2021 05:34, Joseph Touch wrote:
> PS - I should note that I don’t care strongly that we keep this 
> zero-copy support, It’s not important for individual packets; it’s 
> useful only for streams, IMO anyway. If omitting it is fine, then we 
> can make FRAG processing simpler.
>
> - zero-copy is more useful for streams where apps should adjust MTU 
> anyway and thus wouldn’t need FRAG for MTU adjustment
> - when data is otherwise UNSAFE, it’s because it is modified 
> (encrypted, compressed, etc.), where zero-copy is not relevant either
>
> Joe

>
>> On Jun 13, 2021, at 9:31 PM, Joseph Touch <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>>
>>
>>>     If we DO support zero-copy and thus want to allow non-terminal
>>>     fragments to have post-fragoption options that operate on each
>>>     fragment, then we would add THISFRAGLEN to the nonterminal
>>>     format and issue different KIND numbers to nonterminal/terminal
>>>     fragment.
>>>
>>>
>>>
>>> I for one would appreciate further discussion of these last points. 
>>> I admit that I have failed to grasp Joe's message on the RDMA 
>>> thread, and I would appreciate some time to think about it.
>>
>> Sure - here’s how it all works. Note that this is relevant mostly for 
>> long transfers with persistent UDP fragmentation; if that is assumed 
>> to be ‘adjusted’ at the app layer (as QUIC does), then we don’t need 
>> zero-copy support...
>>
>> - if we add options, UDP data can still be zero-copied because it 
>> hasn’t moved (it still begins the payload
>> - however, fragments are different because (esp given the merging of 
>> frag and lite) they don’t start at the beginning of data
>> that way we can zero-copy the frag packet into place, then just copy 
>> those last 8 bytes over OCS and the FRAG header
>>
>> This method assumes that we try to keep FRAG early in the packet - 
>> preferably right after OCS. The later it comes, the more additional 
>> bytes we need to move to “fix” the copy (beyond the 8 bytes noted above).
>>
>> —
>>
>> This method is the only reason we would want to allow options after 
>> non-terminal fragments - basically to keep the fragment toward the 
>> front of the packet, using the rule that post-noninitial frag options 
>> still operate on the fragment, rather than waiting for reassembly. 
>> The exception is the terminal fragment, where post-terminal fragment 
>> options operate on the reassembled packet.
>>
>> Joe
>>
>>
>>
>>
>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 14 Jun 2021 09:17:50 -0700

> - right now, UDP data can be zero-copied when received into user space, starting with the user data

Only if the device supports header/data split where the headers are in
one buffer and UDP data is in aligned buffer.

> that way we can zero-copy the frag packet into place, then just copy those last 8 bytes over OCS and the FRAG header

An obvious feature we'd want is NIC hardware to do UDP options
fragementation and reassembly, analogous to existing UDP Fragmentation
Offload (UFO) which performs IP fragmentation of UDP packets. The
impediment with supporting this is that hardware devices would need to
perform protocol processing on trailers as opposed to headers. Nearly
all hardware devices, including switches and NICs, are optimized to
process protocol headers and in modern devices they are quite
programmable in that regard. However, they typically rely on a parsing
buffer that holds the first N bytes of the packet and assume that all
the protocol headers lie within that. They wouldn't process data after
that header in the fast path at least, and almost certainly would have
capability to process protocol headers at that end of a large packet.
I am doubtful we'll ever see hardware support for trailer protocols,
and hence it's unlikely we'd see accelerations for UDP options like we
have for TCP.

Tom

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 14 Jun 2021 17:31:49 +0100

> and hence it's unlikely we'd see accelerations for UDP options like we

OK.... Is there any way that we could design to enable this?

I'm "fishing" for ideas because I know you've talked about the various 
offload methods.

So for options in the trailer, this is clearly an impediment.

For UDP-Opt fragmentation, I understand there is no standard UDP payload,

.... only an option containing a fragment, so the Fragment information 
would actually be in the" first N bytes of the packet".

So, what do you think  could be most likely helpful to enable fastpath 
accelleration for the fragments?

Gorry

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 14 Jun 2021 09:41:35 -0700

> accelleration for the fragments?

The proposal that I floated a few days ago in
https://mailarchive.ietf.org/arch/msg/tsvwg/N1fmoBxQ5iOL5X7o34oVSA8Ayvw/
would move all of the fragment data to the end of the packet, with UDP
header and UDP options preceding. The very last option would be FRAG
itself, and it would consume all trailing data.

Joe was displeased by replacing the length byte by a More Fragments flag
and suggested instead using two Option Kind codepoints to distinguish
terminal and non-terminal fragments. That would be fine with me and would
not change the proposal in any fundamental way.

Mike Heard

---

From: Tom Herbert <[email-masked]>
Date: Mon, 14 Jun 2021 10:02:29 -0700

> I'm "fishing" for ideas because I know you've talked about the various

Gorry,

My suggestion was to place UDP options after the UDP header. Instead
of just placing fragment header after the UDP header, place all the
UDP options there and then follow that by the Payload. So packet looks
like:

+-------------------+
|   UDP header  |
+-------------------+
|  UDP options  |
+-------------------+
|     Payload      |
+-------------------+

Now this looks a lot like a TCP packet and other variable length
headers which we know how to handle. For zero copy we can do
header/split by programming emerging smart devices to split through
UDP options in one buffer and payload in another thereby also
eliminating any need to move headers or data around.

Tom

---

From: Joe Touch <[email-masked]>
Date: Mon, 14 Jun 2021 10:15:44 -0700

FYI that’s what fragments look like. We can’t do this for non fragments. 

> On Jun 14, 2021, at 10:03 AM, Tom Herbert <[email-masked]> wrote:
>

---

From: Joe Touch <[email-masked]>
Date: Mon, 14 Jun 2021 10:17:10 -0700

> On Jun 14, 2021, at 9:42 AM, C. M. Heard <[email-masked]> wrote:
> 
> The proposal that I floated a few days ago in https://mailarchive.ietf.org/arch/msg/tsvwg/N1fmoBxQ5iOL5X7o34oVSA8Ayvw/ would move all of the fragment data to the end of the packet, with UDP header and UDP options preceding. The very last option would be FRAG itself, and it would consume all trailing data.
> 
> Joe was displeased by replacing the length byte by a More Fragments flag and suggested instead using two Option Kind codepoints to distinguish terminal and non-terminal fragments. That would be fine with me and would not change the proposal in any fundamental way.

Yes, but we still need the fraglen field in the last frag too. 

See the format I posted this am. 

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 14 Jun 2021 10:20:38 -0700

Ps - we need an option length field to make fragments look like tcp. I can put that in - do we want that in OCS? Or independent?

> On Jun 14, 2021, at 10:16 AM, Joe Touch <[email-masked]> wrote:
>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 14 Jun 2021 10:32:23 -0700

Joe,

I suggest that the UDP options should be preceded by a four byte
header consisting of one byte type, one byte length, and two byte
checksum. As I've mentioned previously, making the checksum optional
is inherently problematic because it cannot protect against a
corrupted type field for the optional checksum. e.g. a single bit flip
in the type field for the checksum could turn the checksum option into
some other type and there is no way to detect that.

Tom

Tom

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 14 Jun 2021 18:33:39 +0100

> FYI that’s what fragments look like. We can’t do this for non fragments.

OK, we might be on the same page here!

If Tom is saying that might allow Frags to be "easily" reassembled in 
offload, I'm interested.

Gorry

---

From: Joseph Touch <[email-masked]>
Date: Mon, 14 Jun 2021 10:43:55 -0700

Hi, Tom,

OCS has been required since -08 (Sept. 2019). Here’s the relevant text:
   >> The OCS MUST be included when the UDP checksum is nonzero and UDP
   options are present.
Joe

> On Jun 14, 2021, at 10:32 AM, Tom Herbert <[email-masked]> wrote:
> 
> Joe,
>

---

From: Joseph Touch <[email-masked]>
Date: Mon, 14 Jun 2021 10:46:00 -0700

Note that this also means that non-frags might not have offload support unless offload engines evolve to support trailers.

(And, as noted earlier, we need an option-length equivalent to TCP’s data offset in a sense - do we add that to OCS or as an independent option?)

Note, though, that offload engines already do lots of things that are in direct violation of specs, so we shouldn’t assume they set the tone for where we go either.

Joe

> On Jun 14, 2021, at 10:33 AM, Gorry Fairhurst <[email-masked]> wrote:
> 
> If Tom is saying that might allow Frags to be "easily" reassembled in offload, I'm interested.
> 
> Gorry
> 
>>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 14 Jun 2021 10:57:32 -0700

Joe,

Yes, but in the case when that's not true the problem of corrupted
option type still exists. Also, this rule requires that the
implementation needs to find the checksum option in a list of TLVS,
that makes implementation harder and leads to some strange scenarios--
like a receiver may process twenty options only to get to the 21st
which is checksum and if the checksum cannot be validated then the
node just wasted cycles processing TLVs for no reason. In TCP, for
instance, we validate the checksum before processing any options so
this scenario cannot occur.

Tom

---

From: Joseph Touch <[email-masked]>
Date: Mon, 14 Jun 2021 11:04:00 -0700

> On Jun 14, 2021, at 10:57 AM, Tom Herbert <[email-masked]> wrote:
> 
> option type still exists.

Not under the conditions where UDP checksum can be zero, as per RFC 8200. In that case, the entire UDP packet becomes a payload in another frame, so it would already be covered. The reason to allow this is to avoid double-protection effort for tunnels. This is no different.

> Also, this rule requires that the

If we don’t need to reorder bytes for DMA (which we don’t now), we can require it to be first if present, preceded only by NOPs for alignment (we can remove that too if needed).

Joe

---

From: Joseph Touch <[email-masked]>
Date: Mon, 14 Jun 2021 11:09:10 -0700

FYI, here’s where this appears to be going for fragments:

- OCS always comes first if present (it’s still optional in the same cases where UDP checksum is optional, e.g., when the entire UDP frame is a payload that is protected by some other layer, e.g., in a tunnel)

- FRAG can always come next if present (OCS still optional, per above), with the following format (moving things around a little to keep 32-bit alignment where possible):

	Kind/8, len = 10 or 12/8
	FRAGSTART/16	= location where the options end and frag data starts
	FRAGID/32
	FRAGOFFSET/16
	(For last frag) FRAGEND/16 = location where options continue after the frag for the final fragment

Other options that apply to that fragment come after this option but before where FRAGSTART points. This effectively puts the info needed for offload in one of two fixed locations (with and without OCS).

The frag data is not scrambled or reordered; it just comes in order as intended. Frag data always goes to the end of the IP packet data area UNLESS its the terminal fragment, in which case FRAGEND points to where post-reassembly UDP options occur.

Would this work?

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 14 Jun 2021 11:14:58 -0700

> Yes, but we still need the fraglen field in the last frag too.

No, it is not necessary, if every option precedes the FRAG header.

I did read it. If I correctly understand what you wrote, there is an
underlying ASSUMPTION that options that apply per-fragment need to precede
the FRAG header and those that apply to the reassembled datagram have to
appear afterward. To be clear, I do not dispute that such a design will
work; it will. What I dispute is that it is necessary.

In the design that I presented earlier, I also made an implicit assumption
that whether an option applies to an individual fragment or to the
reassembled datagram can be determined from the Option Kind (or UKind or
ExID). Going down the list:

EOL - not applicable (FRAG consumes all trailing data)
NOP - per fragment
OCS - per fragment
ACS - per datagram
FRAG  - per fragment (obviously)
UNSAFE - depends on Ukind
TIME - per datagram
AE - per fragment (see below)
REQ, RES - per datagram
EXP - depends on UDP ExID

I will grant that defining AE as per-fragment or per-datagram is a design
choice. Either can be made to work, but if it's designed to be per-fragment
then the description in the draft of how it works needs no modifications --
in particular, this:

   >> Only the OCS and AE options depend on the contents of the option
   area. AE is always computed as if the AE hash and OCS checksum are
   zero; OCS is always computed as if the OCS checksum is zero and
   after the AE hash has been computed.

I'm still struggling to wrap my head around how zero copy interacts with
all this, but I hope to post a response to that within the next 12 hours.

Thanks and regards,

Mike Heard

---

From: Tom Herbert <[email-masked]>
Date: Mon, 14 Jun 2021 11:44:11 -0700

> If we don’t need to reorder bytes for DMA (which we don’t now), we can require it to be first if present, preceded only by NOPs for alignment (we can remove that too if needed).

If it's required and it's first then it's no longer an option, it's
just a two-byte field in a header that precedes the option list. This
is much better for efficient processing.

---

From: Joseph Touch <[email-masked]>
Date: Mon, 14 Jun 2021 11:50:50 -0700

> On Jun 14, 2021, at 11:14 AM, C. M. Heard <[email-masked]> wrote:
> 
> On Mon, Jun 14, 2021 at 10:17 AM Joe Touch <[email-masked] <mailto:[email-masked]>> wrote:
> I did read it. If I correctly understand what you wrote, there is an underlying ASSUMPTION that options that apply per-fragment need to precede the FRAG header and those that apply to the reassembled datagram have to appear afterward.

Yes, it was explicit that this is the distinction. That’s because, ultimately, we don’t know how people will define future options or apply them.

> To be clear, I do not dispute that such a design will work; it will. What I dispute is that it is necessary.
> 
> In the design that I presented earlier, I also made an implicit assumption that whether an option applies to an individual fragment or to the reassembled datagram can be determined from the Option Kind (or UKind or ExID). Going down the list:

First, you can’t go down a list that includes options to be defined, unless we further fracture the option space into pre/post reassembly.

Second, many of the examples you give as “known” are not:
	NOP can be needed for padding in either place.
	UNSAFE can’t be known in advance
	TIME can be per frag (for path RTT) or per segment (aggregate delay)
	REQ/RES can be either as well.
	AE for sure - it’s a trade between per-fragment safety and per-datagram efficiency

Regarding zero copy, none of this matters if we put the FRAG header up front as I just proposed. What was missing was the pointer to where the FRAG option was; instead, I put the option up front and point to the data in the option. Same number of fields, simpler offload processing.

Joe

---

From: Joseph Touch <[email-masked]>
Date: Mon, 14 Jun 2021 11:52:13 -0700

> On Jun 14, 2021, at 11:44 AM, Tom Herbert <[email-masked]> wrote:
> 

It isn’t required. It’s required to be first *if present*. It’s still optional if the UDP checksum is zero.

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 14 Jun 2021 12:51:22 -0700

> is much better for efficient processing.

That idea will work, but if it is adopted, I would like to make sure that
the following points are covered:

1.) Many of us want it to be possible to include "safe" UDP options in a
trailer for compatibility with legacy receivers who don't support UDP
options.

2.) OCS needs to be present in that case too, and it must cover the entire
trailer, in order to ensure middlebox traversal (as discovered by Tom
Jones).

Thus, if there is a UDP options header that contains OCS, it would need to
be at the front end of the options trailer. If that header included a total
options length, EOL would no longer serve a useful purpose. Everything from
the first byte pointed to by the option length to the end of the packet
would be padding, required to be set to zeroes (but not required to be
checked), and would be covered by the OCS.

Honestly, though, I fail to see a compelling advantage in having a UDP
options header, since an implementation of UDP options is required to loop
through all options in the packet. To paraphrase RFC 791, what is optional
is not the requirement to inspect options when present, but the requirement
that they be in any particular packet. Given that, I don't see how an
options header would be better for fragmentation offload than what I
proposed in
https://mailarchive.ietf.org/arch/msg/tsvwg/N1fmoBxQ5iOL5X7o34oVSA8Ayvw/.

> optional if the UDP checksum is zero.

For the record, it actually does not matter where OCS is placed when it is
required to be present (UDP CS<>0), for it is possible to check OCS without
ever parsing the options at all: just calculate the Internet checksum over
the trailer (using the appropriate pseudo-header, of course). That
comprehensively covers the case where UDP CS<>0, which IMO is the important
one, as CS<>0 is required by operational considerations in most situations.

For sockets where UDP CS=0 is allowed, OCS is optional unless the receiver
is configured otherwise. In that case it is indeed advantageous to require
that OCS be first (preceded only by a handful of NOPs) if it is present.
It's not hard to implement optional OCS with UDP CS=0, but IMO parsing a
list of TLVs without first doing an integrity check is a very risky
business, and I question whether it adds any value. (But I won't squawk if
it stays.)

Mike Heard

---

From: Paul Vixie <[email-masked]>
Date: Mon, 14 Jun 2021 21:49:30 +0000

On Sun, Jun 13, 2021 at 05:04:03PM -0700, Joseph Touch wrote:
> > ...
> > computing UDP payload size by subtraction requires either that one be
> > inside the network stack (sometimes this means "in the kernel") so as to
> > know the interface MTU and perhaps the path MTU, as well as the IP6 header
> > and option sizes.
> 
> That???s correct; that???s exactly how TCP MSS works. The UDP stack can make
> this information available to the user.

TCP MSS is different, because TCP segments are of elastic size, because
message boundaries are not retained between the TCP user and the network.

no UDP stack has ever made payload size (per interface, per destination)
information available in a portable way. at best we can learn the first-hop
MTU or perhaps the PMTU and then we begin subtracting, pessimistically.

the difference is, a UDP user currently knows that a message will be a packet,
and can ask to shut off source fragmentation to assure this. in the UDP
Options world, one UDP message from a user might become more than one
packet on the network. and that's fine, but it's not what i'm asking about.

> A DNS responder would not be creating UDP option packets;
> it creates user that it passes to UDP. The same way TCP
> keeps ithis info on a per-socket basis, UDP is expected to.

we will need a portable kernel/user API that allows a UDP user such as DNS
to learn the maximum message size tolerated by that socket. if fragmentation
is available then the size might be close to "64K" which is fine. but the
payload budget left over after IP6 or IP4 headers and options, and after UDP
options, is something the UDP user should not try to guess, using subtraction,
with pessimism as to how much overhead each (invisible) layer will occupy.

-- 
Paul Vixie

---

From: Joseph Touch <[email-masked]>
Date: Mon, 14 Jun 2021 15:10:39 -0700

Hi, Paul,

> On Jun 14, 2021, at 2:49 PM, Paul Vixie <[email-masked]> wrote:
> 

Yes, that’s what is expected - per RFC 1122, Sec 4.1.4 GET_MAXSIZES().

> and can ask to shut off source fragmentation to assure this.

In a stack supporting UDP fragmentation, this would be the same as shutting off IP source fragmentation but allowing UDP fragmentation.

> in the UDP
> to learn the maximum message size tolerated by that socket.

Why is that not GET_MAXSIZES() as per RFC 4.1.3? That’s supposed to be a requirement. 

> if fragmentation
> is available then the size might be close to "64K" which is fine.

It would be limited by RMSS in UDP options (the UDP receiver max reassembly size).

> but the

Agreed - that’s what the interface in RFC 1122 is supposed to provide, for exactly that reason.

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 14 Jun 2021 22:10:00 -0700

On Sun, Jun 13, 2021 at 9:31 PM Joseph Touch wrote:

> On Jun 13, 2021, at 7:20 PM, C. M. Heard wrote:
>
>> I for one would appreciate further discussion of these last points. I
>> admit that I have failed to grasp Joe's message on the RDMA thread, and I
>> would appreciate some time to think about it .
>>
>
> Sure - here’s how it all works. Note that this is relevant mostly for long
> transfers with persistent UDP fragmentation; if that is assumed to be
> ‘adjusted’ at the app layer (as QUIC does), then we don’t need zero-copy
> support...
>
> - however, fragments are different because (esp given the merging of frag
> and lite) they don’t start at the beginning of data
> that way we can zero-copy the frag packet into place, then just copy those
> last 8 bytes over OCS and the FRAG header
>
> preferably right after OCS. The later it comes, the more additional bytes
> we need to move to “fix” the copy (beyond the 8 bytes noted above).
>
> —
>
> non-terminal fragments - basically to keep the fragment toward the front of
> the packet, using the rule that post-noninitial frag options still operate
> on the fragment, rather than waiting for reassembly. The exception is the
> terminal fragment, where post-terminal fragment options operate on the
> reassembled packet.
>

I'm not understanding this AT ALL, and I apologize if there is well-known
stuff of which I am embarrassingly ignorant. That being said:

EVERY description of a zero-copy receive describes something involving MTUs
and highly constrained header length that allow the user data in a TCP
segment or UDP packet to be mapped to one or more kernel pages. Here is one
example:

PATH to TCP 4K MTU and RX zerocopy
<https://netdevconf.info/0x14/pub/slides/62/Implementing%20TCP%20RX%20zero%20copy.pdf>

In every case that I have found, the solutions apply only to a highly
constrained environment, such as a data center, and not over the Internet
writ large. Some even involve requiring the application to process the
transport headers, which is surely not an outcome that we wish in general.

If I am wrong -- and it would most assuredly not be the first time -- I am
eager to be disabused, preferably with a complete and open description of a
zero-copy technology without such shortcomings.

But if my conclusions are substantially correct, I don't think that TSVWG
should expend effort on zero copy for UDP fragment reassembly. Transport
options for UDP need to apply across the general Internet.

NOTE: the unfavorable conclusions that I make about zero-copy do NOT apply
to checksum offload; the advantages and applicability of that technology
(especially with OCS now defined to be an equivalent to the CCO proposal)
are readily apparent, even though they are not realizable in every
implementation.

Thanks

Mike Heard

---

From: Joseph Touch <[email-masked]>
Date: Tue, 15 Jun 2021 08:11:56 -0700

Mike,

Zero-copy networking has been around since roughly 1991 (James Sterbenz PhD dissertation on “Axon” was arguably one of the first, the IBM 360 had zero copy for memory storage transfers back in the 1960s), and it has been used in networking stacks since that time. Yes, it’s more prevalent in datacenter and HPC (high performance computing) environments, where RDMA is one variant, but it can be supported in at least some network cards that offload packet processing.

It requires few per-packet decisions to confirm support, which is why some simple decisions we can make - the don’t constrain us - help.Those below include making OCS come first and enabling FRAG to come immediately after, so the entire TLV chain between the two doesn’t need to be walked. I agree we should not design in ways that complicate processing for non-zerocopy endpoints, but if the work is either small (as it was previously) or effectively none (as it would be with the structure below), there’s no utility in ignoring it.

It might be useful to note that zero copy support exists in most modern OSes, including Linux, macOS, and Windows. 

Joe

> On Jun 14, 2021, at 10:10 PM, C. M. Heard <[email-masked]> wrote:
> 
>> I for one would appreciate further discussion of these last points. I admit that I have failed to grasp Joe's message on the RDMA thread, and I would appreciate some time to think about it .
> 
> I'm not understanding this AT ALL, and I apologize if there is well-known stuff of which I am embarrassingly ignorant. That being said:
> 
> EVERY description of a zero-copy receive describes something involving MTUs and highly constrained header length that allow the user data in a TCP segment or UDP packet to be mapped to one or more kernel pages. Here is one example:
> 
> PATH to TCP 4K MTU and RX zerocopy <https://netdevconf.info/0x14/pub/slides/62/Implementing%20TCP%20RX%20zero%20copy.pdf>
> 
> In every case that I have found, the solutions apply only to a highly constrained environment, such as a data center, and not over the Internet writ large. Some even involve requiring the application to process the transport headers, which is surely not an outcome that we wish in general.
> 
> If I am wrong -- and it would most assuredly not be the first time -- I am eager to be disabused, preferably with a complete and open description of a zero-copy technology without such shortcomings.
> 
> But if my conclusions are substantially correct, I don't think that TSVWG should expend effort on zero copy for UDP fragment reassembly. Transport options for UDP need to apply across the general Internet.
> 
> NOTE: the unfavorable conclusions that I make about zero-copy do NOT apply to checksum offload; the advantages and applicability of that technology (especially with OCS now defined to be an equivalent to the CCO proposal) are readily apparent, even though they are not realizable in every implementation.
> 
> Thanks
> 
> Mike Heard
> 
> 
> 
> 
>  
>

---

From: Tom Herbert <[email-masked]>
Date: Tue, 15 Jun 2021 08:56:24 -0700

Joe,

Please see the paper that Mike referred to, that describes concisely
how zero copy receive is supported in Linux. Receive zero was not easy
to pull off and does require a constrained environment. Zero copy send
was easier as we have splice and MSG_ZEROCOPY has been added as the
deck describes.

IMO, if you want to make UDP options seamlessly integrate and get
ubiquitous support of UDP options in the OS and hardware (leverage
checksum offload, zero copy, hardware fragmentation/reassembly,
offload authentication/crypto, header/data split, etc.), the best
direction would be to make UDP with options look like TCP as much as
possible at least with regards to packet format. i.e. use the same
option format, have a header length field, use headers instead of
trailers, etc.

Tom

---

From: Joseph Touch <[email-masked]>
Date: Tue, 15 Jun 2021 09:32:50 -0700

Hi, Tom,

There’s no debating that zero-copy networking is challenging - moreso when dealing with an OS that is “different for the sake of being different” (Linux, largely for copyright reasons).

To your points:
> offload authentication/crypto, header/data split, etc.),

Seamlessness is not a goal; we want only to not get in the way.

> the best
> possible at least with regards to packet format.

QUIC doesn’t and UDP can’t.

> i.e. use the same

We do use the same basic format (TLV). We do have a length field where it would help zero-copy (FRAG), but not elsewhere. 

And we do use headers for FRAG but not elsewhere.

Again, this is UDP. We can’t redefine it in a way that lets incorrect data be received at legacy endpoints.

And - this is key - we also do not want to make the mistakes TCP made, even if that makes UDP options require its own code. E.g., we don’t want to limit UDP header length to 40 bytes.

So the best we can do (and are doing) is *enable* new code for UDP zero-copy. For FRAG, that means putting the length where it can be found without TLV-chain parsing. But a true UDP zero-copy stack that supports non-frag messages would need to know how to deal with trailers rather than headers.

And, FWIW, trailers are just as easy if not *EASIER* for zero-copy to handle; they just DMA the user data first then the UDP options.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Tue, 15 Jun 2021 09:58:55 -0700

Joe,

I am dubious of that statement. You'll have to show running code and
performance results. Software stacks and hardware have long been
designed around the fact that networking protocols are in headers; UDP
options is an outlier in that regard. While it is certainly possible
to make a functional stack to handle trailers, it is going to be hard
if not impossible to leverage many implementation optimizations and
techniques we've done over the years.

Tom

---

From: Joseph Touch <[email-masked]>
Date: Tue, 15 Jun 2021 10:54:14 -0700

> On Jun 15, 2021, at 9:58 AM, Tom Herbert <[email-masked]> wrote:
> 
> I am dubious of that statement.

Zero-copy matters for the body, not the options.

For TCP, you have to start your DMA after the variable-length header.

For UDP with FRAG, the same thing happens. 
For UDP without FRAG, you start the DMA 8 bytes after the IP header.

> You'll have to show running code and
> performance results.

If the above won’t convince you, a demo won’t help.

> Software stacks and hardware have long been
> options is an outlier in that regard.

Again, not for FRAG; FRAG works exactly the same way.

And again, doing otherwise for non-FRAG is not a viable path forward due to legacy constraints.

> While it is certainly possible

*That* statement really needs proof by running code and performance, IMO.

Either you can do DMA from an known offset for a known length or you can’t. 

Joe

---

From: Paul Vixie <[email-masked]>
Date: Tue, 15 Jun 2021 17:56:51 +0000

On Tue, Jun 15, 2021 at 09:58:55AM -0700, Tom Herbert wrote:
> > And, FWIW, trailers are just as easy if not *EASIER* for zero-copy to
> > handle; they just DMA the user data first then the UDP options.

> techniques we've done over the years.

just as ip trailers were a temporary crutch to help the VAX BSD networking
stack use page-aligned mbufs and led to a great improvement in performance,
so it is that there will always be some implementation detail which will
attract our protocol design attention.

however, we must recognize that all such attractions are temporary. just as
the current linux networking stack has had several decades in which to
optimize for its operating conditions, so it is that other networking stacks
will eventually supplant it and take a commanding market share.

the zero-copy feature joe is working on should last 50 to 100 years. linux
will almost certainly be revised if not replaced on that time scale. let's
make the best protocol we can, so that both current _and future_
implementations can get maximum benefit from our efforts.

640KB was a bad architectural assumption, as was 16 bits, as is 1500 octets.
let us, please, treat any current operating condition as likely to be wrong
as a guidepost. we must think and act more fundamentally and more eternally.

-- 
Paul Vixie

---

From: "C. M. Heard" <[email-masked]>
Date: Sat, 26 Jun 2021 16:53:30 -0700

On Mon, Jun 14, 2021 at 11:51 AM Joseph Touch  wrote:

> On Mon, Jun 14, 2021 at 11:14 AM C. M. Heard wrote:
>
>> On Mon, Jun 14, 2021 at 10:17 AM Joe Touch wrote:
>>
>> In the design that I presented earlier, I also made an implicit
>> assumption that whether an option applies to an individual fragment or to
>> the reassembled datagram can be determined from the Option Kind (or UKind
>> or ExID). Going down the list:
>>
> First, you can’t go down a list that includes options to be defined,
> unless we further fracture the option space into pre/post reassembly.
>

What I had in mind was that the definition of each option that the higher
layer could request would specify whether it applied to the entire
reassembled datagram (thus appearing only in the terminal fragment) or to
each individual fragment (thus appearing in each). Options that are
produced and consumed entirely at the transport layer would be per-fragment.

Logically, this is similar to the IPv4 Copied on Fragmentation flag, which
is specified for each IPv4 option, but with two important differences:

1. Since there is no such thing as in-flight UDP fragmentation -- it occurs
only at the source -- there is no need for the fragmentation engine to
worry about options it does not understand; the higher layer simply cannot
request these options.

2. In IPv4 EOL and NOP are not copied on fragmentation, but they also
cannot be explicitly requested by the higher layer. They exist solely for
the convenience of the layer where they are used. On transmit, they are
inserted as needed; on receive, they are consumed. These options can appear
in any fragment, but they do not make their way to the higher layer layer
with the payload of the received datagram.

In the cases -- rare, IMO -- where either behavior would be appropriate, it
can be distinguished by Option Kind or UKind.

> AE for sure - it’s a trade between per-fragment safety and per-datagram

I think I explained NOP and EOL. I already said that the behavior of any
UNSAFE options would depend on UKIND, i.e., the definition of the
particular unsafe option would state whether it applies per-datagram or
per-fragment.

If TIME can be requested by the upper layer, what is the meaning if the
option is requested for a datagram that gets split into multiple fragments,
and the option is replicated in every fragment? I can see the per-fragment
argument but only when the option is produced and consumed at the UDP layer
-- as is the case for the corresponding TCP option (which appears on every
segment if it is negotiated). If it's intended to be produced and consumed
by the higher layer, it should be per-datagram.

The REQ and RES options exist for the benefit of DPLPMTUD and are intended
to be sent on a PLPMTU Probe Packet, which itself must not be further
fragmented (the probe packet will not serve its purpose if it is fragmented
to make it get to its destination). Ideally, these options are used on
packets with padding only and no user data; such packets would be generated
and consumed entirely within the transport layer or a shim. As they are not
subject to fragmentation, they can be considered complete datagrams. In
addition, it seems to me that there is little or no value in taking a REQ
option from the user and replicating it on every fragment of a datagram
that needs to be fragmented; this causes duplicate REQ tokens. Gorry, any
thoughts? (*)

AE (now AUTH and ENCR, as of -13) can protect  (AUTH) or depend on (ENCR)
all the options, if so configured. Certainly in that case it needs to be
per segment. It's possible to specify per-datagram versions -- possibly
including only per-datagram options, in some normalized form after
reassembly -- but this seriously complicates the specification. Do you
really want to do that?

(*) As an aside I see that draft-fairhurst-tsvwg-udp-options-dplpmtud-04
allows one packet contain RES options for multiple REQ datagrams; this
currently is not allowed in draft-ietf-tsvwg-udp-options-13:

   >> Except for NOP, each option SHOULD NOT occur more than once in a
   single UDP datagram. If an option other than NOP occurs more than
   once, a receiver MUST interpret only the first instance of that
   option and MUST ignore all others.

This is an inconsistency that needs to be resolved.

Mike Heard