Subject: Re: [tsvwg] UDP options and header-data split (zero copy)

From: Tom Herbert <[email-masked]>
Date: Sun, 1 Aug 2021 10:03:47 -0700

On Sat, Jul 31, 2021 at 2:17 PM Joseph Touch <[email-masked]> wrote:
>
> FWIW, there’s another option here, which could be considered unnecessarily complex if there’s no need for per-reassembled datagram options for most UDP tunnels….
>
> ----
>
> Right now, the terminal fragment has a header as follows:
>
>                    +--------+--------+--------+--------+
>                    | Kind=4 | Len=12 |   Frag. Start   |
>                    +--------+--------+--------+--------+
>                    |           Identification          |
>                    +--------+--------+--------+--------+
>                    |  Frag. Offset   |    Frag. End    |
>                    +--------+--------+--------+--------+
>
>                 Figure 12   UDP terminal FRAG option format
>
> As to whether to put the per-reassembled options before or after the terminal fragment, we could let the user decide (again, consistent with the idea of UDP), as follows.

Joe,

I suggest that the per-reassembled options be in headers in the first
fragment. This would work by specifying that options preceding the
fragment option are per-packet, options following the fragment option
in the first fragment apply to reassembled packets, and options
following a fragment option in non-first fragment would be ignored.
So, in this manner fragments don't have any protocol trailers and
therefore Frag. End isn't needed and can be removed from the fragment
option saving two bytes (the end of the fragment would coincide with
the end of the packet). As I mentioned previously, there are other
advantages to placing the options that apply to the reassembled packet
in the first fragment instead of the last.

In the tunnel case we'd also cases where UDP options might be sent
that are not fragments￼, however we still need to know where the
payload begins. In the fragment option this is the Frag. Start field,
but sending a whole fragment option just to convey these two bytes of
information would be way overkill. IMO a better approach is to
indicate the length of the options in a fixed header preceding the
options since the information is always required.
draft-herbert-udp-space-hdr-00 defines such a fixed header. In that
format, the options length is expressed in one byte with a multiplier
of four, so that would allow up to 1020 bytes. I know that the does
set a limit, however consider that (AFAIK) no one has ever proposed an
option or set of options in datapath protocol that would come close
needing that much space, the length of the options is already limited
by MTU where 1020 bytes already exceeds the Min MTU for IPv4 and is a
significant portion of IPv6 Min MTU, and also we know that allowing
unlimited options creates problems in itself (i.e. why we need to
limit IPv6 HbH and Dest Ops like in draft-herbert-6man-eh-limits).

Tom

>
> The code cost is small; this way an implementation could decide when it can use a single DMA or no. The only hazard is that the transmitter decides - but for the UDP tunnel case, given both sides already need configuration, it should be simple enough to encourage transmitters to use the preferred case.
>
> ----
>
> We could consider “Frag. End” to really just point to the start of the per-reassembled datagram options, so let’s instead call that field “Reassembled datagram option start”, or RDOS for short. Let’s assume there’s a “RDOE” (end pointer) as well, which we’ll compute.
>
> If so, then we could allow those options to be either before the start of the fragment (where RDOS < Frag Start) or at the end of the packet (where RDOS > Frag Start).
>
> If we allow that, then the code needs a condition to decide where the end of the packet is:
>
> if (RDOS < frag start)
> then  // single DMA case
> frag end = IP packet end
> RDOE = frag start - 1
> else  // receive processing same as for non-terminals
> frag end = RDOS-1
> RDOE = IP packet end
>
> At that point, the per-reassembled options are from RDOS to RDOE and the fragment is from frag start to frag end.
>
> ----
>
>
> On Jul 31, 2021, at 2:08 PM, Joseph Touch <[email-masked]> wrote:
>
> Right - per fragment options would be OK, not per reassembled datagram.
>
> Would that suffice?
>
> Joe
>
> On Jul 31, 2021, at 1:42 PM, Tom Herbert <[email-masked]> wrote:
>
> On Sat, Jul 31, 2021 at 12:42 PM Joe Touch <[email-masked]> wrote:
>
>
>
>
> On Jul 31, 2021, at 11:58 AM, Tom Herbert <[email-masked]> wrote:
>
> There’s no way that endpoint can ensure it never sees non-fragment UDP options, so it cannot support UDP options.
>
> For a tunnel, use of UDP options would just be another configuration
> parameter at the end points; we don't need negotiation to send
> non-legacy packets. If there's a mismatch in endpoint configuration
> then the operator simply detects it and fixes it like any other tunnel
> configuration problem.
>
>
> Ok, the simply don’t set any per packet options and I’ll be fine.
>
> I think you might mean the other way around. Per packet options are
> already defined in the draft to be in the headers, including the
> fragment option, so there's no issue concerning protocol trailers.
> There would only be a problem from options that apply to the
> reassembled packet which would be in trailers of the last fragment.
>
> Tom
>
> Joe
>
>
>
>

---

From: Joseph Touch <[email-masked]>
Date: Sun, 1 Aug 2021 10:48:22 -0700

> On Aug 1, 2021, at 10:03 AM, Tom Herbert <[email-masked]> wrote:
> 

Again, that means that the FRAG option itself would float far inside the TLV list, which complicates DMA.

Also, the trailing variant allows per-reassembled options to be arbitrarily long (limited by the reassembled length), rather than requiring them to fit inside a single fragment.

That’s a key reason for using trailing options - no limit on length, even with fragmentation. That’s why we need a version that allows that, otherwise the options would be limited to what will fit in exactly one option.

> the end of the packet).

But then we need at least one byte back for the one bit to indicate last-fragment.

> As I mentioned previously, there are other

We need to keep a way to allow arbitrary option lengths; every fixed header approach to that ends up hitting a limit that later needs a workaround (see the work on extending the TCP header).

> options since the information is always required.

We’ve been down the path of fixed headers before. It wastes space for some uses to conserve it for others. E.g., in tunnel cases where UDP CS==0, it would waste space for OCS when not needed.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sun, 1 Aug 2021 12:03:41 -0700

> Again, that means that the FRAG option itself would float far inside the TLV list, which complicates DMA.
I don't understand your concerns. The method I am describing is
functionally equivalent to how fragmentation is done in IPv6. There is
a fragment extension header that "floats" inside the IPv6 header
chain, options that are before the fragment extension header are
applied per packet and those after the extension header are applied to
the reassembled packet. This is no issue with DMA since there are no
trailers and we can use header-data split with fragments.

> That’s a key reason for using trailing options - no limit on length, even with fragmentation. That’s why we need a version that allows that, otherwise the options would be limited to what will fit…
Deployment experience has shown arbitrarily long lists of TLVs in a
datapath protocol is more a problem than a benefit. Implementations
have a lot of trouble with these and to date the only use case for
this is DoS attack.

> We need to keep a way to allow arbitrary option lengths;

Why? Can you provide a *specific* example of an option or set of
options that needs hundreds of bytes of options?

> every fixed header approach to that ends up hitting a limit that later needs a workaround (see the work on extending the TCP header).
>
> We’ve been down the path of fixed headers before.

And we've been down the path of protocols that allow arbitrarily long
lists of TLVs. See IPv6 Hop-by-Hop options and Destination options.
There is about no deployment of these and a major reason is that
implementations efficiently implement necessary support.

> It wastes space for some uses to conserve it for others. E.g., in tunnel cases where UDP CS==0, it would waste space for OCS when not needed.

Somehow the whole purpose of having a checksum in protocol is lost in
this draft. The point of a checksum is that it's an integrity check of
the protocol. Frankly, the requirements described in the draft seem
more concerned with making sure the checksum is set properly to
packets to get through some misbehaving routers on the Internet. The
argument that UDP Options doesn't need a checksum is disingenuous
given that UDP doesn't have options in the first place so making the
checksum required in UDP to cover options; however the other major
transport protocol of IETF, namely TCP, has a checksum that always
covers TCP options. Similarly, the argument that the IPV6 UDP tunnels
can have a zero checksum is disingenuous. If you read RFC6935 and
RFC6936 you'll find that the conditions for which a zero checksum can
be safely set are quite narrow. One of those conditions is if the
encapsulated protocol has an integrity check of its own.

IMO, a checksum should always be required over the surplus area. The
value we get from an integrity far exceeds the overhead which is a
whole two bytes. It's pointless to spend time quibbling rather than a
fixed overhead of two bytes is better or worse than a protocol that
saves those two bytes in some cases, but increases overhead to three
bytes.

Tom

>I

---

From: Joseph Touch <[email-masked]>
Date: Sun, 1 Aug 2021 12:42:25 -0700

> On Aug 1, 2021, at 12:03 PM, Tom Herbert <[email-masked]> wrote:
>> ...

We moved the FRAG header to the front due to previously expressed concerns - notably long TLV processing prior to DMA.

> have a lot of trouble with these and to date the only use case for

The issue isn’t just whether it’s arbitrarily long; it’s also that it’s restricted to a single fragment, which could be quite small.
>> ...

Had we limited the option length as a few suggested when this work started, we would not have FRAG.

We don’t know what others are, but we also don’t know that the first frag will have hundreds of bytes of available space either.

...
> implementations efficiently implement necessary support.

RFC8504 discusses these; they are all MAY limits via receiver configuration, notably that the default is that it accepts both sets of options of any length.

Your suggestion would limit options by design, not receiver configuration.

> encapsulated protocol has an integrity check of its own.

Yes, but that’s actually the dominant case where Mike’s optimization fails, i.e., where UDP CS==0 but OCS is optional. In that case, if OCS fails, the TLV chain still needs to be checked to decide whether to drop the packet. It’s only when it succeeds that it need not be checked.

> IMO, a checksum should always be required over the surplus area.

This, like the proposal for fixed headers, has already been discussed.

New discussions would be useful; rehashing previous ones can probably be left to the interim.

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 1 Aug 2021 13:32:11 -0700

On Sun, Aug 1, 2021 at 10:48 AM Joseph Touch wrote:

> Also, the trailing variant allows per-reassembled options to be
> arbitrarily long (limited by the reassembled length), rather than requiring
> them to fit inside a single fragment.
>

If that is the intent, then draft-ietf-tsvwg-udp-options-13#section-5.5
needs significant clarification. What I read there tells me that the
per-reassembled-datagram options must all appear in the terminal fragment,
following the user data. If the terminal fragment user data length is zero,
that does allow for a long options field, but it is limited by the maximum
fragment length, not the reassembled datagram length.

We’ve been down the path of fixed headers before. It wastes space for some
> uses to conserve it for others. E.g., in tunnel cases where UDP CS==0, it
> would waste space for OCS when not needed.
>

Indeed we have, but AFAICT a definitive consensus has not been reached.

As you know, recently I have been advocating a fragment format along the
following lines:

                   +--------+--------+--------+--------+
                   |  Source Port    |   Dest. Port    |

                   +--------+--------+--------+--------+
                   |   UDP Len=8     |  UDP Checksum   |

                   +--------+--------+--------+--------+
                   | OCS=2  | LEN=4  | Option Checksum |
                   +--------+--------+--------+--------+
                   |       ... Other Options ...       |
                   +--------+--------+--------+--------+
                   | FRAG=X | LEN=8  |  Frag. Offset   |
                   +--------+--------+--------+--------+
                   |          Identification           |
                   +--------+--------+--------+--------+
                   |       ... Fragment Data ...       |
                   +--------+--------+--------+--------+

where X is one of two codepoints, depending on whether the fragment is a
terminal or non-terminal fragment.

This would achieve the goal of pushing all user data in self-contained
fragments to the end of the packet and would thereby allow for checksum
offload of encapsulated packets on commonly available hardware. However,
there's one thing that Tom's header would give us that a naked stack of
TLVs would not: it would enable a NIC to perform header-data split without
parsing a long stack of TLVs. We still don't get around the need to loop
through all those TLVs at some point, but that does not have to be handled
in hardware that is not well-suited for it.

In https://mailarchive.ietf.org/arch/msg/tsvwg/XZxL29UA-95ReA72mxv5-kEytK0/
I floated the following variant of Tom's proposal"

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\
   |        Source port            |      Destination port         | |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -Base Hdr
   |        UDP Length = 8         |   UDP Header Checksum         | |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+/
   |        Payload Offset         |  Option+Payload Checksum      |\
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
   |                                                               | |
   ~                           UDP Options                         ~ -Ext Hdr
   |                                                               | |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+/
   |                                                               |
   ~                         Payload Data                          ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

In this case the fixed header just replaces the currently-defined OCS,
occupying the exact same amount of space while using the same algorithm.
And it is neutral as to whether FRAG is defined as in
draft-ietf-tsvwg-udp-options-13, or as in my counterproposal, or in some
other way, as long as FRAG data is contained in the option space.

For the record:

1. I do not think that it is useful to allow an arbitrary UDP option to be
encoded either as a per-fragment or per-datagram option (or perhaps both).
As I have stated previously, that usage should be spelled out in the option
definition. I believe that there also needs to be a better specification of
what the application protocol running on top of UDP will be able to request
of the UDP layer regarding options. This consideration informs how the FRAG
option needs to be designed.

2. I do not agree with the contention that there is any good reason for
reassembled UDP datagrams with options to wind up in the same format as the
wire format for legacy UDP datagrams with an options trailer. That is an
implementation detail, generally invisible to the outside world. In common
existing implementations options and user data generally follow different
paths to get between the user and the protocol stack, and it is not at all
obvious that the machinations to force these two things into the same mold
provide any benefit to the implementation. As a corollary, I see no
compelling reason why the legacy format cannot remain as is, with a fixed
header replacing OCS in the case where UDP Len = 0.

3, I do not agree that consenting endpoints should be disallowed from using
just the legacy format with a trailer, or just the format with UDP Len = 0.
We would not expect this of a fully conforming implementation, but if it is
useful in specific use cases, I can't see the downside in allowing it, no
matter how hard I squint.  Unlike some other limited domain proposals that
have been floated, this is not something that is likely to cause harm to
the general internet if it escapes.

I expect strong disagreement on all these points. It would be nice to hear
from folks who have been silent in this latest round of discussions.

Thanks and regards,

Mike Heard

---

From: Tom Herbert <[email-masked]>
Date: Sun, 1 Aug 2021 14:19:48 -0700

On Sun, Aug 1, 2021, 12:42 PM Joseph Touch <[email-masked]> wrote:

> We moved the FRAG header to the front due to previously expressed concerns
> - notably long TLV processing prior to DMA.
>
> That’s a key reason for using trailing options - no limit on length, even
> with fragmentation. That’s why we need a version that allows that,
> otherwise the options would be limited to what will fit in exactly one
> option.
>
> The issue isn’t just whether it’s arbitrarily long; it’s also that it’s
> restricted to a single fragment, which could be quite small.
>
> ...
> We don’t know what others are, but we also don’t know that the first frag
> will have hundreds of bytes of available space either.
>

Actually, we do know that. The minimum MTU in IPv6 is 1280 and the minimum
MTU for IPv4 is 576. If someone we're so inclined they could fill up the
first fragment packet with nothing but options and start the payload in the
second. That means you'll have at least 520 bytes for options. But all this
is academic since there's no use case other than DoS attack that would need
anything close to that much space.

> RFC8504 discusses these; they are all MAY limits via receiver
> configuration, notably that the default is that it accepts both sets of
> options of any length.
>
> It wastes space for some uses to conserve it for others. E.g., in tunnel
> cases where UDP CS==0, it would waste space for OCS when not needed.
>
>
> Yes, but that’s actually the dominant case where Mike’s optimization
> fails, i.e., where UDP CS==0 but OCS is optional. In that case, if OCS
> fails, the TLV chain still needs to be checked to decide whether to drop
> the packet. It’s only when it succeeds that it need not be checked.
>

---

From: Joseph Touch <[email-masked]>
Date: Sun, 1 Aug 2021 14:24:20 -0700

> On Aug 1, 2021, at 2:19 PM, Tom Herbert <[email-masked]> wrote:
> 
> 
> 
> On Sun, Aug 1, 2021, 12:42 PM Joseph Touch <[email-masked] <mailto:[email-masked]>> wrote:
> 
> ...
> Actually, we do know that. The minimum MTU in IPv6 is 1280 and the minimum MTU for IPv4 is 576.

The min MTU for IPv4 is 68.

> If someone we're so inclined they could fill up the first fragment packet with nothing but options and start the payload in the second. That means you'll have at least 520 bytes for options.

That includes BOTH per-fragment and per-reassembled datagram options.

> But all this is academic since there's no use case other than DoS attack that would need anything close to that much space.

That was true until FRAG too. 

Again, this is a new decision - to limit the option space.

Joe

---

From: Joseph Touch <[email-masked]>
Date: Sun, 1 Aug 2021 14:33:01 -0700

> On Aug 1, 2021, at 1:32 PM, C. M. Heard <[email-masked]> wrote:
> 
> If that is the intent, then draft-ietf-tsvwg-udp-options-13#section-5.5 needs significant clarification.

Yes; this is pending.

> Indeed we have, but AFAICT a definitive consensus has not been reached. 

What we have so far are three of us largely debating the issues, agree.

> As you know, recently I have been advocating a fragment format along the following lines:
> 
> where X is one of two codepoints, depending on whether the fragment is a terminal or non-terminal fragment.

With that version, the DMA can’t happen until the whole TLV is parsed.

> This would achieve the goal of pushing all user data in self-contained fragments to the end of the packet and would thereby allow for checksum offload of encapsulated packets on commonly available hardware. However, there's one thing that Tom's header would give us that a naked stack of TLVs would not: it would enable a NIC to perform header-data split without parsing a long stack of TLVs.

We haven’t proposed a long stack of TLVs; the current text puts the FRAG field right after the optional OCS, specifically for this reason,

HOWEVER, I keep hearing that IPv6 “got it right”, yet a long stack of TLVs is exactly what it uses for frag and reassy too - not a fixed header.

> We still don't get around the need to loop through all those TLVs at some point, but that does not have to be handled in hardware that is not well-suited for it.
> 
> In https://mailarchive.ietf.org/arch/msg/tsvwg/XZxL29UA-95ReA72mxv5-kEytK0/ <https://mailarchive.ietf.org/arch/msg/tsvwg/XZxL29UA-95ReA72mxv5-kEytK0/> I floated the following variant of Tom's proposal"
> 
>     0                   1                   2                   3
> In this case the fixed header just replaces the currently-defined OCS, occupying the exact same amount of space while using the same algorithm. And it is neutral as to whether FRAG is defined as in draft-ietf-tsvwg-udp-options-13, or as in my counterproposal, or in some other way, as long as FRAG data is contained in the option space.

But the format above doesn’t make sense for non-FRAGs because the payload comes earlier; if it were used, it would waste space when OCS isn’t used in addition to wasting space on the payload offset.

We’ve discussed using fixed headers before, but the issue is that UDP, like IPv6 (the most modern example of options), does have a fixed header and then a set of TLVS. There’s no pointer like you assume above to where the options end and data starts.

So if hardware supports frag/reassy for IPv6, why wouldn’t it work for UDP?

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sun, 1 Aug 2021 14:43:16 -0700

On Sun, Aug 1, 2021 at 2:24 PM Joseph Touch <[email-masked]> wrote:
> The min MTU for IPv4 is 68..

Per RFC791 it's 576 bytes.

> Again, this is a new decision - to limit the option space.

I don't know what that means.

---

From: Joe Touch <[email-masked]>
Date: Sun, 1 Aug 2021 14:50:02 -0700

> On Aug 1, 2021, at 2:43 PM, Tom Herbert <[email-masked]> wrote:
> 
>> 

You’re assuming after reassembly by IP?

If so, yes - but again that would impose a limit we currently don’t have. 

Joe

---

From: Joe Touch <[email-masked]>
Date: Sun, 1 Aug 2021 14:56:45 -0700

> On Aug 1, 2021, at 2:33 PM, Joseph Touch <[email-masked]> wrote:
> 

Fwiw I was assuming that field pointed to the start of the trailer options measured from the start of the UDP data (I might have forget that nuance in the -13 text) If that field points inside this fragment, it can help the split. If not, then the per reasssembled options could be split out only after reassembly. But they don’t need to fit in a single fragment. 

Joe

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 2 Aug 2021 00:04:13 +0200

> On 1. Aug 2021, at 23:43, Tom Herbert <[email-masked]> wrote:
> 
RFC 791 makes a requirement on the receiver, not on the links.

Page 25 of RFC 791 reads:

    Every internet module must be able to forward a datagram of 68
    octets without further fragmentation.  This is because an internet
    header may be up to 60 octets, and the minimum fragment is 8 octets.

Therefore, the min MTU for IPv4 is 68.

Best regards
Michael

---

From: Joe Touch <[email-masked]>
Date: Sun, 1 Aug 2021 15:14:57 -0700

Right, but for UDP the receive MTU is the one we care about. 

Joe 

> On Aug 1, 2021, at 3:05 PM, Michael Tuexen <[email-masked]> wrote:
> 
> 
>>

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 2 Aug 2021 00:22:09 +0200

> On 2. Aug 2021, at 00:14, Joe Touch <[email-masked]> wrote:
> 
Ahh, OK.

Best regards
Michael

---

From: Tom Herbert <[email-masked]>
Date: Sun, 1 Aug 2021 15:39:14 -0700

There is also RFC9000:

"QUIC assumes a minimum IP packet size of at least 1280 bytes.  This
is the IPv6 minimum size [IPv6] and is also supported by most modern
IPv4 networks.  Assuming the minimum IP header size of 40 bytes for
IPv6 and 20 bytes for IPv4 and a UDP header size of 8 bytes, this
results in a maximum datagram size of 1232 bytes for IPv6 and 1252
bytes for IPv4.  Thus, modern IPv4 and all IPv6 network paths are
expected to be able to support QUIC."

and

"This requirement to support a UDP payload of 1200 bytes limits the
space available for IPv6 extension headers to 32  bytes or IPv4
options to 52 bytes if the path only supports the IPv6 minimum MTU of
1280 bytes.  This affects Initial packets and path validation."

If UDP options were used with QUIC, these limits would be applicable
to UDP options as well, i.e. 32 bytes of UDP options for IPv6, 52
bytes for IPv4 at least for initial packets and path validation.

---

From: Joseph Touch <[email-masked]>
Date: Sun, 1 Aug 2021 15:48:09 -0700

Hi, Tom,

> On Aug 1, 2021, at 3:39 PM, Tom Herbert <[email-masked]> wrote:
> 
> There is also RFC9000:
> 
> IPv4 networks.  

Hmm. Seems like they completely overlooked IP source fragmentation support and 1500B reassembly...

> Assuming the minimum IP header size of 40 bytes for

QUIC over UDP with fragmentation wouldn’t be bothered by these limits at all; it could send a 1200B QUIC payload with up to 63KB of additional combined IP and UDP headers - not that it should, but it could.

By the time QUIC sees the packet, it would be both IP reassembled (at least 1500B) and, with UDP options, UDP reassembled (up to a total of 64K).

> 1280 bytes.  This affects Initial packets and path validation."

See above.

> bytes for IPv4 at least for initial packets and path validation.

I think they missed something significant, even if just for IPv6 at 1500B.

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 1 Aug 2021 15:48:16 -0700

On Sun, Aug 1, 2021 at 2:56 PM Joe Touch wrote:
> > On Aug 1, 2021, at 2:33 PM, Joseph Touch wrote:
> > > > Also, the trailing variant allows per-reassembled options
> > > > to be arbitrarily long (limited by the reassembled length),
> > > > rather than requiring them to fit inside a single fragment.
> > >
> > >
> > > If that is the intent, then
> > > draft-ietf-tsvwg-udp-options-13#section-5.5 needs significant
clarification.
> >
> > Yes; this is pending.
>
> Fwiw I was assuming that field pointed to the start of the trailer
> options measured from the start of the UDP data (I might have [forgotten]
> that nuance in the -13 text) If that field points inside this
> fragment, it can help the split. If not, then the per reassembled
> options could be split out only after reassembly. But they don’t need
> to fit in a single fragment.

I have to say that I was not altogether pleased with what I thought the
draft said, but I find this level of complexity to be downright
alarming, and I don't see a need for it. It's reminiscent of IPv6
fragmentation, which has stubbornly resisted correct specification (witness
the verified errata posted against RFC 8200). UDP fragmentation is
important to have largely because IP fragmentation, and IPv6 fragmentation
in particular, have not been fully successful. Let's please not repeat
those mistakes. Keep it simple, please.

Mike Heard

---

From: Joseph Touch <[email-masked]>
Date: Sun, 1 Aug 2021 15:56:28 -0700

> On Aug 1, 2021, at 3:48 PM, C. M. Heard <[email-masked]> wrote:
> 
> > > > draft-ietf-tsvwg-udp-options-13#section-5.5 needs significant clarification.
> > >
> > > Yes; this is pending.
> >
> I have to say that I was not altogether pleased with what I thought the draft said, but I find this level of complexity to be downright alarming, and I don't see a need for it.

The value in the terminal fragment is just the value that would have been in the original IP length before fragmentation, less the IP and UDP headers.

That’s very simple. 

Whether it can be used to split off the trailing options requires a simple comparison - if FRAG_END > FRAG_OFFSET, then the trailing options are entirely in this fragment. If not, then not.

The fact that per-reassembled datagram options look exactly like legacy options it similar in its simplicity, as is using TLVs for everything. It’s all about fewer different ways to do the same thing.

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 1 Aug 2021 16:22:38 -0700

On Sun, Aug 1, 2021 at 3:56 PM Joseph Touch <wrote:

> The value in the terminal fragment is just the value that would have been
> in the original IP length before fragmentation, less the IP and UDP headers.
>
> That’s very simple.

That's highly dependent on the details of the implementation,

> Whether it can be used to split off the trailing options requires a simple
> comparison - if FRAG_END > FRAG_OFFSET, then the trailing options are
> entirely in this fragment. If not, then not.
>
> The fact that per-reassembled datagram options look exactly like legacy
> options it similar in its simplicity, as is using TLVs for everything. It’s
> all about fewer different ways to do the same thing.
>

Based on my own experience, I do not agree that this necessarily makes for
a simpler implementation. But let's see what other folks think.

Mike Heard

---

From: Tom Herbert <[email-masked]>
Date: Sun, 1 Aug 2021 17:41:24 -0700

> Hmm. Seems like they completely overlooked IP source fragmentation support

No IP fragmentation in QUIC. DF bit is always set.

> QUIC over UDP with fragmentation wouldn’t be bothered by these limits at
> all; it could send a 1200B QUIC payload with up to 63KB of additional
> combined IP and UDP headers - not that it should, but it could.
>
> By the time QUIC sees the packet, it would be both IP reassembled (at
> least 1500B) and, with UDP options, UDP reassembled (up to a total of 64K).
>
> > and
> >

---

From: Joseph Touch <[email-masked]>
Date: Sun, 1 Aug 2021 18:12:53 -0700

> On Aug 1, 2021, at 5:41 PM, Tom Herbert <[email-masked]> wrote:
> 
> 
> 
> On Sun, Aug 1, 2021, 3:48 PM Joseph Touch <[email-masked] <mailto:[email-masked]>> wrote:
> Hi, Tom,
> 
> > On Aug 1, 2021, at 3:39 PM, Tom Herbert <[email-masked] <mailto:[email-masked]>> wrote:
> > 
> > There is also RFC9000:
> > 

RFC 9000 talks about DF being set for IPv4, but that’s the default for IPv6 (no on-path fragmentation).

It does say "no fragmentation at the IP layer”, which presumably also prohibits IP source fragmentation, though it’s not at all clear why, other than to avoid black-holing through NAT. DF being set in IPv4 is critical to RFC1191 path MTU discovery, but that does not prohibit use of source fragmentation as long as DF is set. 

However, that does not prevent use of UDP fragmentation - which would both be opaque to avoiding IP fragmentation and QUIC, as well as would traverse NATs because of the replicated UDP header in UDP fragments.

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 1 Aug 2021 18:16:22 -0700

On Sun, Aug 1, 2021 at 6:13 PM Joseph Touch wrote:

> However, that does not prevent use of UDP fragmentation - which would both
> be opaque to avoiding IP fragmentation and QUIC, as well as would traverse
> NATs because of the replicated UDP header in UDP fragments.
>

True, but QUIC has its own solutions to that problem. From recollection,
the comments I've seen from QUIC proponents is that there is no interest in
that quarter in UDP options.

Mike

---

From: Tom Herbert <[email-masked]>
Date: Sun, 1 Aug 2021 18:45:19 -0700

On Sun, Aug 1, 2021, 6:13 PM Joseph Touch <[email-masked]> wrote:

> It does say "no fragmentation at the IP layer”, which presumably also
> prohibits IP source fragmentation, though it’s not at all clear why, other
> than to avoid black-holing through NAT. DF being set in IPv4 is critical to
> RFC1191 path MTU discovery, but that does not prohibit use of source
> fragmentation as long as DF is set.
>

Path MTU discovery doesn't work if the source fragments. For instance, if a
host fragments a 1500 byte packet into three packets of 500 bytes then
we're probing the path with 500 bytes not 1500 which is what the transport
protocol expects.

> NATs because of the replicated UDP header in UDP fragments.

UDP fragmentation similarly breaks pmtud.

---

From: Tom Herbert <[email-masked]>
Date: Sun, 1 Aug 2021 18:55:03 -0700

On Sun, Aug 1, 2021, 6:16 PM C. M. Heard <[email-masked]> wrote:

>> However, that does not prevent use of UDP fragmentation - which would
>> both be opaque to avoiding IP fragmentation and QUIC, as well as would
>> traverse NATs because of the replicated UDP header in UDP fragments.
>>
>

They might not have interest, but that is not the same thing as saying UDP
options won't work with QUIC. I don't believe there's anything in the draft
that limits the use cases of UDP options. Besides that, QUIC is just an
anecdote, other protocols will force limits on UDP options (like UDP
tunnels can't user trailers).

---

From: Joseph Touch <[email-masked]>
Date: Sun, 1 Aug 2021 18:56:21 -0700

> On Aug 1, 2021, at 6:45 PM, Tom Herbert <[email-masked]> wrote:
> 
> 
> 
> On Sun, Aug 1, 2021, 6:13 PM Joseph Touch <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Path MTU discovery doesn't work if the source fragments.

It does - it depends on what the transport wants to do.

> For instance, if a host fragments a 1500 byte packet into three packets of 500 bytes then we're probing the path with 500 bytes not 1500 which is what the transport protocol expects.

And if the packets get through, then there’s no issue. If they fail, the source will get back a PTB with the size that works at the IP layer, which informs transport.

So it will work fine - IF the user wants to let IP do source fragmentation. The only requirement is that the IP set DF, which is always implicit for IPv6.

> UDP fragmentation similarly breaks pmtud.

It would be a layer that adapts to it in lieu of the user - if that’s what the user wants (i.e., if they ask for UDP fragmentation to be used).

Joe

---

From: Joseph Touch <[email-masked]>
Date: Sun, 1 Aug 2021 18:57:57 -0700

> On Aug 1, 2021, at 6:16 PM, C. M. Heard <[email-masked]> wrote:
> 
> True, but QUIC has its own solutions to that problem. From recollection, the comments I've seen from QUIC proponents is that there is no interest in that quarter in UDP options.

Perhaps, but they have no solution to protecting the transport layer header - if that’s something users want. Nor do they have a solution to using more IP option space. Those can’t be solved inside QUIC.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Mon, 2 Aug 2021 07:38:28 -0700

On Sun, Aug 1, 2021 at 4:23 PM C. M. Heard <[email-masked]> wrote:
> Based on my own experience, I do not agree that this necessarily makes for a simpler implementation. But let's see what other folks think.
>
Mike,

No, it's not simpler for an implementation. In the current design
options are in trailers for legacy mode and headers in non-legacy mode
so by definition we need divergent code paths code paths for those two
case. In the non-legacy mode case we need to support both options in
headers and options in trailers within the same packet which
contradicts the principle of having fewer ways to do the same thing.
In any case, having divergent code paths isn't a problem for
implemenations, the real complication to implemenations would be the
requirement to process protocol trailers.

Tom

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 2 Aug 2021 11:12:28 -0700

On Mon, Aug 2, 2021 at 7:38 AM Tom Herbert wrote:
> implementations, the real complication to implementations would be the

Actually, in non-legacy mode options may appear BOTH in headers (for
per-fragment options) AND in trailers (for the per-datagram options),
and in the current draft the latter can fill a complete fragment. An
upcoming revision is pending that will change this to allow the options
trailer to span multiple fragments.

As I understand it, the procedure (in concept) is to create a datagram
in legacy format with its per-datagram options, and then split that up
into fragments, each of which can have its own per-fragment options.
The per-fragment options precede the corresponding fragment data; in
the current draft, the  per-datagram options follow the data in the
final fragment.

The upcoming revision will change this a bit ... FRAG END in the
terminal FRAG option will be replaced with USER DATA LENGTH. In this
case the per-datagram options will be carried within the fragment
data. It won't be possible, in general, to tell where the user data
ends and the options begin until the reassembly process is complete.

Joe: please offer corrections if I have misrepresented your intent in
the thumbnail sketch above.

WG: I understand Joe's point about conceptual simplicity in a common
format for all cases, but I have to agree with Tom's point that having
divergent code paths isn't the main problem for implementations,
overall complexity is, and I remain concerned about what I perceive as
the ever-growing complexity of the fragmentation and reassembly
specification for UDP options.

Mike Heard

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 2 Aug 2021 20:55:13 +0200

Mmmh, I have  a quick question, actually two.

a) who is the intended audience of the fragmentation feature? And why is IP or application level "fragmentation? not enough?

b) instead of a trailer just add a bit to each option to denote per fragment or per aggregate, sure each option gets one bit longer, but on the flip side no trailer...

Regards
	Sebastian

> On Aug 2, 2021, at 20:12, C. M. Heard <[email-masked]> wrote:
>

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 2 Aug 2021 12:12:08 -0700

On Mon, Aug 2, 2021 at 11:55 AM Sebastian Moeller wrote:

> a) who is the intended audience of the fragmentation feature? And why is
> IP or application level "fragmentation? not enough?
>

One of the principal use cases would be DNS, where IPv6 fragmentation is
notoriously unreliable since it uses extension headers.

b) instead of a trailer just add a bit to each option to denote per
> fragment or per aggregate, sure each option gets one bit longer, but on the
> flip side no trailer...
>

Or it cuts the encoding space in half. That's a possibility. Or just use
different code points for option variants that can be per-frag or
per-datagram. Most options would not need both forms, IMO, Mr Touch has a
different opinion.

Note that the trailer is absolutely required in order to send options to
legacy receivers; legacy receivers ignore the trailer, as it is in the data
that comes after UDP Length. Hence those options must be safe to ignore.

If you need further explanation and don't find it in the draft, feel free
to contact me off-list.

Mike Heard

---

From: Joe Touch <[email-masked]>
Date: Mon, 2 Aug 2021 12:31:06 -0700

> On Aug 2, 2021, at 11:56 AM, Sebastian Moeller <[email-masked]> wrote:
> 

Any protocol using UDP where IP fragmentation won’t work - the most notable example is NAT traversal.  IP doesn’t copy the UDP ports into each fragment but UDP option fragmentation does. 

> b) instead of a trailer just add a bit to each option to denote per fragment or per aggregate, sure each option gets one bit longer, but on the flip side no trailer...

At a minimum, because there are options that could be used in either place, eg. OCS, AUTH, and ENCR at least. 

As noted, this won’t avoid the trailer for legacy mode. 

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 2 Aug 2021 12:36:28 -0700

> On Aug 2, 2021, at 11:13 AM, C. M. Heard <[email-masked]> wrote:
> 
> ends and the options begin until the reassembly process is complete

Yup.  

There are special cases, eg where the trailing options fit in the terminal frag and the data gram is split to make that happen. 

The design has been intended so the sender ‘makes the legacy-style datagram first, then splits into fragments and add per frag options. The receiver would do that in reverse. That’s generally required logically because the trailing options can depend on the whole datagram and the option values in some cases (auth). 

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 2 Aug 2021 15:13:14 -0700

> place, eg. OCS, AUTH, and ENCR at least.

The per-fragment OCS icovers both the fragment data and the options. Is
there really a justification for a per-datagram version? I think not ...
though of course opinions vary ...

Mike Heard

---

From: Joe Touch <[email-masked]>
Date: Mon, 2 Aug 2021 15:54:27 -0700

> On Aug 2, 2021, at 3:13 PM, C. M. Heard <[email-masked]> wrote:
> 
> 
> The per-fragment OCS icovers both the fragment data and the options. Is there really a justification for a per-datagram version? I think not ... though of course opinions vary ...

It would be a reassembly check, if that were desired. 

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 2 Aug 2021 16:00:18 -0700

> It would be a reassembly check, if that were desired.

We went through this discussion regarding a reassembly checksum for the
data and decided NOT to have one of those that uses the same checksum
algorithm as the one that covers the fragments. An extension of ACS to
cover reassembled per-datagram options (or another option similar to it but
covering just the options) would be a better choice, I think.

Mike

---

From: Joe Touch <[email-masked]>
Date: Mon, 2 Aug 2021 16:15:05 -0700

> On Aug 2, 2021, at 4:01 PM, C. M. Heard <[email-masked]> wrote:
> 
> 
> We went through this discussion regarding a reassembly checksum for the data and decided NOT to have one of those that uses the same checksum algorithm as the one that covers the fragments. An extension of ACS to cover reassembled per-datagram options (or another option similar to it but covering just the options) would be a better choice, I think.

Right - thanks. 

Joe.