Subject: [Masque] HTTP Datagrams and MTU handling

From: Magnus Westerlund <[email-masked]>
Date: Tue, 13 Jul 2021 12:29:51 +0000

Hi,

In the work of writing up our Connect-IP proposal
(https://datatracker.ietf.org/doc/draft-kuehlewind-masque-connect-ip/) we
looked into how to deal with MTU issues effectively. Our conclusion was that
this is going to be a general problem for any user of HTTP Datagrams. Thus,
we like to propose that MTU handling is done within HTTP datagram. 

This email will start explaining what we see as requirements for a MTU
signaling solution for HTTP datagram, then propose a potential solution. 

Lets start with a figure that provides us with a framework to discuss the
requirements:

+--------+ Path#1A +--------+        +-------+        +--------+

|Client A|<------->| HTTP   | Path#2 | MASQUE| Path#3 |        |

+--------+         | Inter- |<------>| Server|<------>| Target |

+--------+ Path#1B | mediary|        |(proxy)|        |        |

|Client B|<------->|        |        |       |        |        |

+--------+         +--------+        +-------+        +--------+

I think what makes this a bit more complex is the fact that we need to
consider HTTP intermediaries, such as a front end load balancer that
terminates a first QUIC and HTTP/3 connection between the client and that
intermediary. From that intermediary another HTTP connection is used towards
the HTTP server that consumes and produces the HTTP datagrams, and in the
case of CONNECT-UDP and CONNECT-IP it also have a third path towards the
target to consider. This figure includes two clients to remind us to
consider that the HTTP intermediary may actually aggregate the HTTP request
and response and HTTP datagram over one HTTP/2 or HTTP/3 connection over
Path #2. In the case of MTU this complicates things more as the proxy cant
assume that all HTTP requests have the same MTU for its datagrams on a
HTTP/3 connection. And the Intermediary will be the entity that have direct
knowledge of the client facing as well as the next hop MTU over the HTTP
connections that may all differ in MTU. 

We also have to consider the fact that the underlying transport connection
may at any time be subject to a IP MTU change due to route change for the
path between the nodes. In addition if one have enabled PMTUD in TCP or QUIC
a larger MTU on the individual path could be made available and in some case
desirable to use. Thus, we need to consider dynamic changes during the HTTP
connections life time and each HTTP request response pairs usage of HTTP
datagram. 

So when using HTTP/3 datagram there are a strict MTU limit on the individual
datagrams for it to be possible to be sent as QUIC datagrams, and not being
forced to be encapsulated as CAPSULES over the reliable stream. This is
clearly a possibility but results in that the datagrams are sent reliable
and in order for each HTTP request, i.e. Connect-UDP or CONNECT-IP request.
Also in case some end-to-end payloads fit in HTTP datagrams and others don't
there is potential for reordering among the payloads. Thus, to avoid this
the client and the proxy needs to determine what the lowest currently
supported HTTP datagram size on the path. 

For each QUIC connection the end point will know what the initial MTU value
is for this path when the HTTP/3 connection has been successful completed.
However, that knowledge will not be available if one attempt to construct
and send the HTTP request prior to connection establishment has concluded.

So the requirements we see for an MTU handling solution for HTTP Datagrams
are the following.

1.	Hop-by-hop signaling across the HTTP entities of the lowest MTU of
any sub-path
2.	Needs to be associated to a particular HTTP request or end-to-end
path to support aggregation by HTTP intermediaries
3.	Endpoints needs to be able to initiate update of the MTU value upon
detection of any changes during HTTP Datagram streams lifetime.
4.	HTTP Intermediaries needs to be able to initiate updates of the MTU
value upon detecting MTU changes from the individual HTTP connections.

Solution proposal

==============

A new HTTP Datagram Capsule is defined for MTU value exchange. This one is
intended for the HTTP intermediary that needs to interpret, update or
initiate sending of it. Thus, it needs to be a fixed registered type so it
can be easily processed. It can also be exchanged in parallel to the
Register_Datagram_* capsules as at that point the underlying HTTP connection
is established and initial HTTP Datagram values will be known. A capsule
will also travel all the way to the end. And an intermediary can initiate
one in each direction for request paths. The only downside I see of this is
that one is required per open stream when MTU changes occurs. Maybe someone
have an idea of how to handle signaling when aggregating multiple endpoints
streams onto one HTTP/3 connection. 

To make it more efficient, rather than sending one MTU capsule per stream,
an MTU capsule could list all streams it is applicable to. That way the
number of MTU capsules would be no more than the number of end-to-end paths
actually used. 

Cheers

Magnus Westerlund

---

From: Ben Schwartz <[email-masked]>
Date: Tue, 13 Jul 2021 11:12:41 -0400

I agree that MTU discovery is valuable, and I think we should try to
provide a workable solution.  However, I think the solution you've sketched
sounds very complicated, and I have some concerns about whether it would be
implemented correctly.

Here's an alternative proposal:

An HTTP intermediary routes each datagram from an "incoming" link to an
"outgoing" link.  When the outgoing link is DATAGRAM-capable,
intermediaries SHOULD NOT convert DATAGRAMs to CAPSULEs, and SHOULD NOT
fragment (or permit fragmentation of) the corresponding QUIC packets.
DATAGRAMs that are too large for any segment's MTU will therefore be
dropped silently, and will not reach their destination.

This is sufficient to enable end-to-end PMTUD over MASQUE.  I think this
arrangement is likely preferable: it avoids adding complexity to
intermediaries, and places responsibility for PMTUD at the endpoints, who
have the clearest incentive to make it work well.

Endpoints who need to send packets larger than the MTU, or who need to send
large packets before PMTUD converges, can use CAPSULEs at their discretion.

On Tue, Jul 13, 2021 at 8:30 AM Magnus Westerlund <magnus.westerlund=
[email-masked]> wrote:

> Hi,
>
>
>
> In the work of writing up our Connect-IP proposal (
> https://datatracker.ietf.org/doc/draft-kuehlewind-masque-connect-ip/) we
> looked into how to deal with MTU issues effectively. Our conclusion was
> that this is going to be a general problem for any user of HTTP Datagrams.
> Thus, we like to propose that MTU handling is done within HTTP datagram.
>
>
>
> intermediary. From that intermediary another HTTP connection is used
> towards the HTTP server that consumes and produces the HTTP datagrams, and
> in the case of CONNECT-UDP and CONNECT-IP it also have a third path towards
> the target to consider. This figure includes two clients to remind us to
> path between the nodes. In addition if one have enabled PMTUD in TCP or
> QUIC a larger MTU on the individual path could be made available and in
> some case desirable to use. Thus, we need to consider dynamic changes
> during the HTTP connections life time and each HTTP request response pairs
> usage of HTTP datagram.
>
>
>
> So when using HTTP/3 datagram there are a strict MTU limit on the
> individual datagrams for it to be possible to be sent as QUIC datagrams,
> and not being forced to be encapsulated as CAPSULES over the reliable
> stream. This is clearly a possibility but results in that the datagrams are
> sent reliable and in order for each HTTP request, i.e. Connect-UDP or
> CONNECT-IP request. Also in case some end-to-end payloads fit in HTTP
> datagrams and others don’t there is potential for reordering among the
> payloads. Thus, to avoid this the client and the proxy needs to determine
> what the lowest currently supported HTTP datagram size on the path.
>
>
>
> For each QUIC connection the end point will know what the initial MTU
> value is for this path when the HTTP/3 connection has been successful
> completed. However, that knowledge will not be available if one attempt to
> construct and send the HTTP request prior to connection establishment has
> concluded.
>
>
>
> Register_Datagram_* capsules as at that point the underlying HTTP
> connection is established and initial HTTP Datagram values will be known. A
> capsule will also travel all the way to the end. And an intermediary can
> initiate one in each direction for request paths. The only downside I see
> of this is that one is required per open stream when MTU changes occurs.
> Maybe someone have an idea of how to handle signaling when aggregating
> multiple endpoints streams onto one HTTP/3 connection.
>
>
>
> --
> Masque mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/masque
>

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 13 Jul 2021 15:22:56 +0000

Hi Ben,

On Tue, 2021-07-13 at 11:12 -0400, Ben Schwartz wrote:
> provide a workable solution.  However, I think the solution you've
> sketched sounds very complicated, and I have some concerns about
> whether it would be implemented correctly.
> An HTTP intermediary routes each datagram from an "incoming" link to
> an "outgoing" link.  When the outgoing link is DATAGRAM-capable,
> intermediaries SHOULD NOT convert DATAGRAMs to CAPSULEs, and SHOULD
> NOT fragment (or permit fragmentation of) the corresponding QUIC
> packets.  DATAGRAMs that are too large for any segment's MTU will
> therefore be dropped silently, and will not reach their destination.
> 
> 
> This is sufficient to enable end-to-end PMTUD over MASQUE.  I think
> this arrangement is likely preferable: it avoids adding complexity to
> intermediaries, and places responsibility for PMTUD at the endpoints,
> who have the clearest incentive to make it work well.
> 
> Endpoints who need to send packets larger than the MTU, or who need
> to send large packets before PMTUD converges, can use CAPSULEs at
> their discretion.

Yes, that is simple but problematic in the context of Connect-IP. If
the HTTP Datagram MTU would be below 1280, then CONNECT-IP needs to
know this so that it can encapsualet instead to maintain the minimal
IPv6 packet size. That was the reason I was looking into actually
discovering the currently supported MTU for HTTP Datagram so that the
CONNECT-IP application can do the right thing to select between
enforcing an acceptable MTU or be forced to encapsulate.
Cheers

Magnus Westerlund

---

From: Ben Schwartz <[email-masked]>
Date: Tue, 13 Jul 2021 11:42:14 -0400

On Tue, Jul 13, 2021 at 11:23 AM Magnus Westerlund <magnus.westerlund=
[email-masked]> wrote:

> Endpoints who need to send packets larger than the MTU, or who need to
> send large packets before PMTUD converges, can use CAPSULEs at their
> discretion.
>
>
> Yes, that is simple but problematic in the context of Connect-IP. If the
> HTTP Datagram MTU would be below 1280, then CONNECT-IP needs to know this
> so that it can encapsualet instead to maintain the minimal IPv6 packet size.
>

I think the question is: who needs to know this, and when?  The endpoints
can detect this situation via PMTUD, and encapsulate if appropriate.

If there's a compelling reason why that is not sufficient, we could
consider a (method-independent) Datagram-Min-MTU request and response
header, which would be read (but not modified) by intermediaries.
Intermediaries would encapsulate if necessary to deliver any DATAGRAM of
this size or less.

> That was the reason I was looking into actually discovering the currently
> supported MTU for HTTP Datagram so that the CONNECT-IP application can do
> the right thing to select between enforcing an acceptable MTU or be forced
> to encapsulate.
>
> Cheers
>
> Magnus Westerlund
> --
> Masque mailing list
> [email-masked]

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 14 Jul 2021 10:22:49 +1000

It seems to me like Ben's approach allows for progressive enhancement.  Endpoints can use PLPMTUD to discover the "true" MTU with this.  Enhancements might be encapsulation techniques (to cram larger datagrams into smaller tunnels at the cost of latency and loss exposure, which might be discretionary or explicitly requested as Ben suggests) and explicit MTU-related signaling (which might need to be advisory, like ICMP PTB).  The baseline could be quite simple though and that seems like a better starting point.

On Wed, Jul 14, 2021, at 01:42, Ben Schwartz wrote:
> 
> 
> On Tue, Jul 13, 2021 at 11:23 AM Magnus Westerlund 
> <magnus.westerlund=[email-masked]> wrote:
> > Hi Ben,
> > 
> >> I agree that MTU discovery is valuable, and I think we should try to provide a workable solution.  However, I think the solution you've sketched sounds very complicated, and I have some concerns about whether it would be implemented correctly.
> >> 
> >> An HTTP intermediary routes each datagram from an "incoming" link to an "outgoing" link.  When the outgoing link is DATAGRAM-capable, intermediaries SHOULD NOT convert DATAGRAMs to CAPSULEs, and SHOULD NOT fragment (or permit fragmentation of) the corresponding QUIC packets.  DATAGRAMs that are too large for any segment's MTU will therefore be dropped silently, and will not reach their destination.
> >> 
> >> This is sufficient to enable end-to-end PMTUD over MASQUE.  I think this arrangement is likely preferable: it avoids adding complexity to intermediaries, and places responsibility for PMTUD at the endpoints, who have the clearest incentive to make it work well.
> >> 
> >> Endpoints who need to send packets larger than the MTU, or who need to send large packets before PMTUD converges, can use CAPSULEs at their discretion.
> > 
> > Yes, that is simple but problematic in the context of Connect-IP. If the HTTP Datagram MTU would be below 1280, then CONNECT-IP needs to know this so that it can encapsualet instead to maintain the minimal IPv6 packet size.
> 
> I think the question is: who needs to know this, and when?  The 
> endpoints can detect this situation via PMTUD, and encapsulate if 
> appropriate.
> 
> Intermediaries would encapsulate if necessary to deliver any DATAGRAM 
> of this size or less.
>  
> > That was the reason I was looking into actually discovering the currently supported MTU for HTTP Datagram so that the CONNECT-IP application can do the right thing to select between enforcing an acceptable MTU or be forced to encapsulate.
> > 
> > Cheers
> > 
> > Magnus Westerlund
> > -- 
> > Masque mailing list
> > [email-masked]
> Attachments:
> * smime.p7s

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 14 Jul 2021 08:35:00 +0000

Hi,

I will no disagree that the middle point injection is likely a complexity
that is desirable to avoid.

But more ideas are great and I will think through this one.

Cheers

Magnus Westerlund

> -----Original Message-----
> From: Masque <[email-masked]> On Behalf Of Martin Thomson
> Sent: den 14 juli 2021 02:23
> To: [email-masked]
> Subject: Re: [Masque] HTTP Datagrams and MTU handling
> 
> It seems to me like Ben's approach allows for progressive enhancement.
> Endpoints can use PLPMTUD to discover the "true" MTU with this.
> Enhancements might be encapsulation techniques (to cram larger datagrams
> into smaller tunnels at the cost of latency and loss exposure, which might
be
> discretionary or explicitly requested as Ben suggests) and explicit MTU-
> related signaling (which might need to be advisory, like ICMP PTB).  The
> baseline could be quite simple though and that seems like a better
starting
> point.
sketched
> sounds very complicated, and I have some concerns about whether it would
> be implemented correctly.
> > >>
> NOT fragment (or permit fragmentation of) the corresponding QUIC packets.
> to send large packets before PMTUD converges, can use CAPSULEs at their
> discretion.
> > >
the
> HTTP Datagram MTU would be below 1280, then CONNECT-IP needs to know
> this so that it can encapsualet instead to maintain the minimal IPv6
packet
> size.
> supported MTU for HTTP Datagram so that the CONNECT-IP application can
> do the right thing to select between enforcing an acceptable MTU or be
> forced to encapsulate.
> > >
> > > Cheers
> > >
> > > Magnus Westerlund
> > > --
> > > Masque mailing list
> > > [email-masked]

---

From: David Schinazi <[email-masked]>
Date: Wed, 14 Jul 2021 08:21:28 -0700

I agree with Ben and MT here. Simplicity generally works better, and
putting the
burden on the nodes who care sets us up for success.

I'll note that RFC 9000 already states:
   UDP datagrams MUST NOT be fragmented at the IP layer.  In IPv4,
   the Don't Fragment (DF) bit MUST be set if possible, to prevent
   fragmentation on the path.

So all we need to do is tell HTTP Datagram intermediaries that they
SHOULD NOT convert DATAGRAM frames to capsules.

I've written this up as a PR:
https://github.com/ietf-wg-masque/draft-ietf-masque-h3-datagram/pull/69

David

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 15 Jul 2021 08:44:14 +0000

Hi,
To keep the mailing list in the loop also. 
I don't think just consistently dropping HTTP Datagrams that don't fit
the QUIC Datagram MTU of the next hop is sufficient. It is good to be
able to determine the end-to-end MTU. However, the issue here is that
the applications we are discussing in this group the IP and UDP
tunneling do have some requirements on them supporting some minimal
packet sizes.
For IPv6 the CONNECT-IP method must be able to carry a minimal requires
size IPv6 packet of 1280 bytes between the MASQUE client and server.
That does not work without encapsulation if the QUIC connection runs
over a 1280 bytes MTU. And due to HTTP intermediares the endpoints only
know the QUIC Datagram MTU of the first hop.
Similarly I think we can discuss if we should ensure that QUIC initial
packets actually can make it over CONNECT-UDP in all cases, or if we
are okay with failure in those cases where tunneling QUIC over QUIC
over an IP MTU of 1280 results in that the smallest allowed initial
packets will not make it over the HTTP datagram.
Therefore I think we need to take one more step of either determine the
MTU end-to-end across the HTTP hops between the MASQUE client and
server, or have a way of telling the HTTP intermediaries to send the
HTTP datagram over the reliable stream instead if the packet is below a
certain threhshold that is required to be supported.
Cheers
Magnus
On Wed, 2021-07-14 at 08:21 -0700, David Schinazi wrote:
> putting theburden on the nodes who care sets us up for success.
> 
> 
> David
> 
> On Wed, Jul 14, 2021 at 1:35 AM Magnus Westerlund <
> magnus.westerlund=[email-masked]> wrote:
> > Hi,
> > 
> > 
> > 
> > packets.
> > their
> > > > consider a (method-independent) Datagram-Min-MTU request and
> > 
> > > response
> > 
> > > > > That was the reason I was looking into actually discovering
> > the
> > 
> > currently
> >

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 19 Jul 2021 07:23:39 +0000

Hi,

I want to clarify why I think the HTTP Datagram simply can’t drop over size MTU without some type of signal that it has done this, or alternatively in case the HTTP Datagram sender knows what the MTU is so that it can act on it. 

For CONNECT-IP we are going to support IPv6. That means that CONNECT-IP will act as an IPv6 router and thus have to fulfill the IPv6 layer requirements from the IPv6 node requirements (RFC8504), and we are also defining a IPv6 link layer in some sense. Thus, we need to support:

First at least 1280 bytes IPv6 packet sizes. If we can’t send that size as atomic datagrams then we need a fragmentation and reassembly mechanism. We have one in the CAPSULE transmission of Datagrams, however we need to know which HTTP datagrams to do this to. 

Secondly, RFC 8504 requires support for ICMPv6 and thus we need to ensure that this works. If an HTTP intermediary drops an packet to big and no signal is making it back that this was dropped, then we can’t fulfill the requirement to send ICMPv6 PTBs. Either a signal on drop, or the CONNECT-IP ingress needs to know that this will be dropped and can drop it and send the PTB. 

I hope this makes it clear that we can’t simply off load this to higher layer.

Cheers

Magnus Westerlund

From: Masque <[email-masked]> On Behalf Of Magnus Westerlund
Sent: den 15 juli 2021 10:44
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [Masque] HTTP Datagrams and MTU handling

Hi,

I don't think just consistently dropping HTTP Datagrams that don't fit the QUIC Datagram MTU of the next hop is sufficient. It is good to be able to determine the end-to-end MTU. However, the issue here is that the applications we are discussing in this group the IP and UDP tunneling do have some requirements on them supporting some minimal packet sizes.

For IPv6 the CONNECT-IP method must be able to carry a minimal requires size IPv6 packet of 1280 bytes between the MASQUE client and server. That does not work without encapsulation if the QUIC connection runs over a 1280 bytes MTU. And due to HTTP intermediares the endpoints only know the QUIC Datagram MTU of the first hop.

Similarly I think we can discuss if we should ensure that QUIC initial packets actually can make it over CONNECT-UDP in all cases, or if we are okay with failure in those cases where tunneling QUIC over QUIC over an IP MTU of 1280 results in that the smallest allowed initial packets will not make it over the HTTP datagram.

Therefore I think we need to take one more step of either determine the MTU end-to-end across the HTTP hops between the MASQUE client and server, or have a way of telling the HTTP intermediaries to send the HTTP datagram over the reliable stream instead if the packet is below a certain threhshold that is required to be supported.

Cheers

Magnus

I agree with Ben and MT here. Simplicity generally works better, and putting the

https://github.com/ietf-wg-masque/draft-ietf-masque-h3-datagram/pull/69 <https://protect2.fireeye.com/v1/url?k=17838980-4818b0bc-1783c91b-86b1886cfa64-df5dea70ef0a9e22&q=1&e=798725cb-4263-4ece-b4d4-d5fb3dbda473&u=https%3A%2F%2Fgithub.com%2Fietf-wg-masque%2Fdraft-ietf-masque-h3-datagram%2Fpull%2F69> 

David

On Wed, Jul 14, 2021 at 1:35 AM Magnus Westerlund <magnus.westerlund=[email-masked] <mailto:[email-masked]> > wrote:

Hi,

Cheers

Magnus Westerlund

> From: Masque <[email-masked] <mailto:[email-masked]> > On Behalf Of Martin Thomson
> To: [email-masked] <mailto:[email-masked]> 
be
starting
> > <magnus.westerlund=[email-masked] <mailto:[email-masked]> > wrote:
> > > Hi Ben,
> > >
sketched
the
packet
> > > [email-masked] <mailto:[email-masked]> 
> > [email-masked] <mailto:[email-masked]> 
> [email-masked] <mailto:[email-masked]>

---

From: David Schinazi <[email-masked]>
Date: Mon, 19 Jul 2021 13:27:44 -0700

Hi Magnus,

While I see the relevance of your use-case, I don't think that we
absolutely need to support IPv6 over CONNECT-IP when the QUIC PMTU makes
the maximum datagram size below 1280. It's reasonable to fail the
CONNECT-IP request in that scenario. Assuming that we have an effective MTU
inside the tunnel of at least 1280, then the tunnel can carry ICMP PTB so
the signal is available. I don't think an explicit signal at the HTTP
Datagram layer is required.

David

On Mon, Jul 19, 2021 at 12:23 AM Magnus Westerlund <
[email-masked]> wrote:

> Hi,
>
>
>
> I want to clarify why I think the HTTP Datagram simply can’t drop over
> size MTU without some type of signal that it has done this, or
> alternatively in case the HTTP Datagram sender knows what the MTU is so
> that it can act on it.
>
>
>
> For CONNECT-IP we are going to support IPv6. That means that CONNECT-IP
> will act as an IPv6 router and thus have to fulfill the IPv6 layer
> requirements from the IPv6 node requirements (RFC8504), and we are also
> defining a IPv6 link layer in some sense. Thus, we need to support:
>
>
>
> First at least 1280 bytes IPv6 packet sizes. If we can’t send that size as
> atomic datagrams then we need a fragmentation and reassembly mechanism. We
> have one in the CAPSULE transmission of Datagrams, however we need to know
> which HTTP datagrams to do this to.
>
>
>
> Secondly, RFC 8504 requires support for ICMPv6 and thus we need to ensure
> that this works. If an HTTP intermediary drops an packet to big and no
> signal is making it back that this was dropped, then we can’t fulfill the
> requirement to send ICMPv6 PTBs. Either a signal on drop, or the CONNECT-IP
> ingress needs to know that this will be dropped and can drop it and send
> the PTB.
>
>
>
> *From:* Masque <[email-masked]> *On Behalf Of *Magnus Westerlund
> *Sent:* den 15 juli 2021 10:44
> *To:* [email-masked]
> *Cc:* [email-masked]; [email-masked]
> *Subject:* Re: [Masque] HTTP Datagrams and MTU handling
>
>
>
> Hi,
>
>
>
> I don't think just consistently dropping HTTP Datagrams that don't fit the
> QUIC Datagram MTU of the next hop is sufficient. It is good to be able to
> determine the end-to-end MTU. However, the issue here is that the
> applications we are discussing in this group the IP and UDP tunneling do
> have some requirements on them supporting some minimal packet sizes.
>
>
>
> size IPv6 packet of 1280 bytes between the MASQUE client and server. That
> does not work without encapsulation if the QUIC connection runs over a 1280
> bytes MTU. And due to HTTP intermediares the endpoints only know the QUIC
> Datagram MTU of the first hop.
>
>
>
> packets actually can make it over CONNECT-UDP in all cases, or if we are
> okay with failure in those cases where tunneling QUIC over QUIC over an IP
> MTU of 1280 results in that the smallest allowed initial packets will not
> make it over the HTTP datagram.
>
>
>
> MTU end-to-end across the HTTP hops between the MASQUE client and server,
> or have a way of telling the HTTP intermediaries to send the HTTP datagram
> over the reliable stream instead if the packet is below a certain
> threhshold that is required to be supported.
>
>
>
> Cheers
>
>
>
> Magnus
>
>
>
> <https://protect2.fireeye.com/v1/url?k=17838980-4818b0bc-1783c91b-86b1886cfa64-df5dea70ef0a9e22&q=1&e=798725cb-4263-4ece-b4d4-d5fb3dbda473&u=https%3A%2F%2Fgithub.com%2Fietf-wg-masque%2Fdraft-ietf-masque-h3-datagram%2Fpull%2F69>
>
>
>
> David
>
>
>

---

From: Alex Chernyakhovsky <[email-masked]>
Date: Mon, 19 Jul 2021 16:43:38 -0400

Hi David,

I think this is also an example of when we would likely want/need to fall
back to the reliable data transport concept, which should work over both H2
and H3 for these small-MTU networks.

Sincerely,
-Alex

On Mon, Jul 19, 2021 at 4:28 PM David Schinazi <[email-masked]>
wrote:

>> First at least 1280 bytes IPv6 packet sizes. If we can’t send that size
>> as atomic datagrams then we need a fragmentation and reassembly mechanism.
>> We have one in the CAPSULE transmission of Datagrams, however we need to
>> know which HTTP datagrams to do this to.
>>
>>
>>
>> the QUIC Datagram MTU of the next hop is sufficient. It is good to be able
>> to determine the end-to-end MTU. However, the issue here is that the
>> who

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 20 Jul 2021 08:52:16 +0000

Hi David,
How will the MASQUE Server know that it can't support CONNECT-IP if the
restricted QUIC session is between HTTP intermediaries and not between
either MASQUE endpoints?
By the way, has the people involved in WebTrans considered if they are
okay with HTTP datagrams just disappearing in case they are to large.
Are you expecting every webapp to implement PLPMTUD to determine if it
can send unreliable datagrams for its protocol or need to use
encapsualtion? 
Cheers
Magnus

On Mon, 2021-07-19 at 13:27 -0700, David Schinazi wrote:
> absolutely need to support IPv6 over CONNECT-IP when the QUIC PMTU
> makes the maximum datagram size below 1280. It's reasonable to fail
> the CONNECT-IP request in that scenario. Assuming that we have an
> effective MTU inside the tunnel of at least 1280, then the tunnel can
> carry ICMP PTB so the signal is available. I don't think an explicit
> signal at the HTTP Datagram layer is required.
> 
> David
> > I want to clarify why I think the HTTP Datagram simply can’t drop
> > over size MTU without some type of signal that it has done this, or
> > alternatively in case the HTTP Datagram sender knows what the MTU
> > is so that it can act on it. 
> >  
> > For CONNECT-IP we are going to support IPv6. That means that
> > CONNECT-IP will act as an IPv6 router and thus have to fulfill the
> > IPv6 layer requirements from the IPv6 node requirements (RFC8504),
> > and we are also defining a IPv6 link layer in some sense. Thus, we
> > need to support:
> >  
> > First at least 1280 bytes IPv6 packet sizes. If we can’t send that
> > size as atomic datagrams then we need a fragmentation and
> > reassembly mechanism. We have one in the CAPSULE transmission of
> > Datagrams, however we need to know which HTTP datagrams to do this
> > to. 
> >  
> > Secondly, RFC 8504 requires support for ICMPv6 and thus we need to
> > ensure that this works. If an HTTP intermediary drops an packet to
> > big and no signal is making it back that this was dropped, then we
> > can’t fulfill the requirement to send ICMPv6 PTBs. Either a signal
> > on drop, or the CONNECT-IP ingress needs to know that this will be
> > dropped and can drop it and send the PTB. 
> >  
> > Subject: Re: [Masque] HTTP Datagrams and MTU handling Hi, To keep
> > the mailing list in the loop also.  I don't think just consistently
> > dropping HTTP Datagrams that don't fit the QUIC Datagram MTU of the
> > next hop is sufficient. It is good to be able to determine the end-
> > to-end MTU. However, the issue here is that the applications we are
> > discussing in this group the IP and UDP tunneling do have some
> > requirements on them supporting some minimal packet sizes. For IPv6
> > the CONNECT-IP method must be able to carry a minimal requires size
> > IPv6 packet of 1280 bytes between the MASQUE client and server.
> > That does not work without encapsulation if the QUIC connection
> > runs over a 1280 bytes MTU. And due to HTTP intermediares the
> > endpoints only know the QUIC Datagram MTU of the first
> > hop. Similarly I think we can discuss if we should ensure that QUIC
> > initial packets actually can make it over CONNECT-UDP in all cases,
> > or if we are okay with failure in those cases where tunneling QUIC
> > over QUIC over an IP MTU of 1280 results in that the smallest
> > allowed initial packets will not make it over the HTTP
> > datagram. Therefore I think we need to take one more step of either
> > determine the MTU end-to-end across the HTTP hops between the
> > MASQUE client and server, or have a way of telling the HTTP
> > intermediaries to send the HTTP datagram over the reliable stream
> > instead if the packet is below a certain threhshold that is
> > required to be supported. Cheers Magnus On Wed, 2021-07-14 at 08:21
> > -0700, David Schinazi wrote:
> > > > > > > That was the reason I was looking into actually
> > > > discovering the
> > > > currently

---

From: David Schinazi <[email-masked]>
Date: Tue, 20 Jul 2021 11:56:37 -0700

> restricted QUIC session is between HTTP intermediaries and not between

The MASQUE endpoints could perform PMTUD by sending ICMP echo requests
through the tunnel.

By the way, has the people involved in WebTrans considered if they are okay
> with HTTP datagrams just disappearing in case they are to large. Are you
> expecting every webapp to implement PLPMTUD to determine if it can send
> unreliable datagrams for its protocol or need to use encapsualtion?
>

Indeed, this isn't required for WebTransport. Since QUIC will fail if it
can't get 1200 byte packets across, you're guaranteed to be able to send
datagrams of at least 1150 bytes. That's plenty for most applications. If
they need more they can implement PMTUD.

To be clear: I think PMTUD sounds like a useful extension to HTTP
Datagrams, I just haven't seen any reason to make it a requirement of the
base protocol.

David

Cheers

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 23 Jul 2021 08:32:14 +0000

On Tue, 2021-07-20 at 11:56 -0700, David Schinazi wrote:
> > How will the MASQUE Server know that it can't support CONNECT-IP if
> > the restricted QUIC session is between HTTP intermediaries and not
> > between either MASQUE endpoints?
> 
> The MASQUE endpoints could perform PMTUD by sending ICMP echo
> requests through the tunnel.

Yes, that is a possibility, but not something I like for the following
reasons.
1. Forces a choice between awaiting the verification or provide an
inconsistent behavior in regards to MTU if the tunnel drops oversized
and then on later packets behave consistently.
2. In case the tunnel MTU changes during the sessions lifetime then we
have no method other than continously run PMTUD over the HTTP datagrams
to verify that the minimal size is still working. 
I would definitely prefer a mechanism where the HTTP implementation
sent back a message indicating it dropped the HTTP datagram as oversize
and provided the current MTU. 
> > By the way, has the people involved in WebTrans considered if they
> > are okay with HTTP datagrams just disappearing in case they are to
> > large. Are you expecting every webapp to implement PLPMTUD to
> > determine if it can send unreliable datagrams for its protocol or
> > need to use encapsualtion?
> 
> Indeed, this isn't required for WebTransport. Since QUIC will fail if
> it can't get 1200 byte packets across, you're guaranteed to be able
> to send datagrams of at least 1150 bytes. That's plenty for most
> applications. If they need more they can implement PMTUD.
> 
> Datagrams, I just haven't seen any reason to make it a requirement of
> the base protocol.

I think a stream lined mechanism that gives feedback and that can
handle dynamic external events changing the tunnel MTU for the HTTP
datagram is the best solution. By not requiring support of such
functionality from the start you definitely increase the risk that it
will not work or be present when it is needed.
Cheers
Magnus

---

From: Ben Schwartz <[email-masked]>
Date: Mon, 26 Jul 2021 11:26:45 -0400

On Fri, Jul 23, 2021 at 4:32 AM Magnus Westerlund <magnus.westerlund=
[email-masked]> wrote:

> inconsistent behavior in regards to MTU if the tunnel drops oversized and
> then on later packets behave consistently.
>
> have no method other than continously run PMTUD over the HTTP datagrams to
> verify that the minimal size is still working.
>
> I would definitely prefer a mechanism where the HTTP implementation sent
> back a message indicating it dropped the HTTP datagram as oversize and
> provided the current MTU.
>

I don't understand.  That would also result in problems 1 and 2.  If you
want to avoid problems 1 and 2, you need an equivalent to "forward
fragmentation" (like a "Datagram-Min-MTU" header).

Alternatively, we can simply declare that the DATAGRAM path does not
provide an MTU guarantee, and if you need such a guarantee, you have to use
the DATAGRAM CAPSULE path.

> okay with HTTP datagrams just disappearing in case they are to large. Are
> you expecting every webapp to implement PLPMTUD to determine if it can send
> I think a stream lined mechanism that gives feedback and that can handle
> dynamic external events changing the tunnel MTU for the HTTP datagram is
> the best solution. By not requiring support of such functionality from the
> start you definitely increase the risk that it will not work or be present
> when it is needed.
>
> Cheers
>
> Magnus
>
> --
> Masque mailing list
> [email-masked]

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 26 Jul 2021 19:34:45 +0000

Hi Ben,

My point is that if we can mimic the ICMP PTB mechanism that is sufficient. 
Yes, it will result in a few packets being dropped from the change until the 
ingress side of the tunnel gets the PTB message and can do something about it, 
either encapsulate if that is required to meet minimal MTU sizes, or 
alternatively from this point generate an ICMP or API indication to higher 
layer.

But it prevents a MTU black hole and allows support IPv6 minimal MTU.

Cheers

Magnus Westerlund

From: Ben Schwartz <bemasc=[email-masked]>
Sent: den 26 juli 2021 17:27
To: Magnus Westerlund <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Masque] HTTP Datagrams and MTU handling

On Fri, Jul 23, 2021 at 4:32 AM Magnus Westerlund 
<magnus.westerlund=[email-masked] 
<mailto:[email-masked]> > wrote:

On Tue, Jul 20, 2021 at 1:52 AM Magnus Westerlund 
<[email-masked] <mailto:[email-masked]> > 
wrote:

Hi David,

How will the MASQUE Server know that it can't support CONNECT-IP if the 
restricted QUIC session is between HTTP intermediaries and not between either 
MASQUE endpoints?

The MASQUE endpoints could perform PMTUD by sending ICMP echo requests through 
the tunnel.

1. Forces a choice between awaiting the verification or provide an 
inconsistent behavior in regards to MTU if the tunnel drops oversized and then 
on later packets behave consistently.

2. In case the tunnel MTU changes during the sessions lifetime then we have no 
method other than continously run PMTUD over the HTTP datagrams to verify that 
the minimal size is still working.

I would definitely prefer a mechanism where the HTTP implementation sent back 
a message indicating it dropped the HTTP datagram as oversize and provided the 
current MTU.

I don't understand.  That would also result in problems 1 and 2.  If you want 
to avoid problems 1 and 2, you need an equivalent to "forward fragmentation" 
(like a "Datagram-Min-MTU" header).

Alternatively, we can simply declare that the DATAGRAM path does not provide 
an MTU guarantee, and if you need such a guarantee, you have to use the 
DATAGRAM CAPSULE path.