Subject: Update to draft QUIC DPLPMTUD text i draft-ietf-tsvwg-datagram-plpmtud

From: Magnus Westerlund <[email-masked]>
Date: Tue, 12 May 2020 08:49:30 +0000

QUIC WG,

In response to the IESG evaluation of
https://datatracker.ietf.org/doc/draft-ietf-tsvwg-datagram-plpmtud/ some
changes was made to the QUIC chapter. 

The current text is included below. And a Diff for the changes in this
section -19 to -21 is here:

QUIC related section is 6.3:

https://www.ietf.org/rfcdiff?url1=draft-ietf-tsvwg-datagram-plpmtud-19
<https://www.ietf.org/rfcdiff?url1=draft-ietf-tsvwg-datagram-plpmtud-19&url2
=draft-ietf-tsvwg-datagram-plpmtud-21>
&url2=draft-ietf-tsvwg-datagram-plpmtud-21

My plan is to approve this document on Friday after 12:00 (CEST) if not
anyone yells. 

6.3.  DPLPMTUD for QUIC

   QUIC [I-D.ietf-quic-transport] is a UDP-based transport that provides

   reception feedback.  The UDP payload includes the QUIC packet header,

   protected payload, and any authentication fields.  QUIC depends on a

   PMTU of at least 1280 bytes.

   Section 14 of [I-D.ietf-quic-transport] describes the path

   considerations when sending QUIC packets.  It recommends the use of

   PADDING frames to build the probe packet.  Pure probe-only packets

   are constructed with PADDING frames and PING frames to create a

   padding only packet that will elicit an acknowledgment.  Such padding

   only packets enable probing without affecting the transfer of other

   QUIC frames.

   The recommendation for QUIC endpoints implementing DPLPMTUD is that a

   MPS is maintained for each combination of local and remote IP

   addresses [I-D.ietf-quic-transport].  If a QUIC endpoint determines

   that the PMTU between any pair of local and remote IP addresses has

   fallen below the size required for an acceptable MPS, it immediately

   ceases to send QUIC packets on the affected path.  This could result

   in termination of the connection if an alternative path cannot be

   found [I-D.ietf-quic-transport].

6.3.1.  Initial Connectivity

   The base protocol is specified in [I-D.ietf-quic-transport].  This

   provides an acknowledged PL.  A sender can therefore enter the BASE

   state as soon as connectivity has been confirmed.

   QUIC provides an acknowledged PL, a sender can therefore enter the

   BASE state as soon as the connection handshake has been completed and

   the endpoint has an 1-RTT key established.

6.3.2.  Sending QUIC Probe Packets

   Probe packets consist of a QUIC Header and a payload containing a

   PING Frame and multiple PADDING Frames.  A PADDING Frame is

   represented by a single octet (0x00).  Several PADDING Frames are

   used together to control the length of the probe packet.  The PING

   Frame is used to trigger generation of an acknowledgement.

   The current specification of QUIC sets the following:

   *  BASE_PLPMTU: A QUIC sender pads initial packets to confirm the

      path can support packets of the required size, which sets the

      BASE_PLPMTU and MIN_PLPMTU.

   *  MIN_PLPMTU: A QUIC sender that determines the MIN_PLPMTU has

      fallen MUST immediately stop sending on the affected path.

6.3.3.  Validating the Path with QUIC

   QUIC provides an acknowledged PL, therefore a sender does not

   implement the CONFIRMATION_TIMER while in the SEARCH_COMPLETE state.

6.3.4.  Handling of PTB Messages by QUIC

   QUIC validates ICMP PTB messages.  In addition to UDP Port

  validation, QUIC can validate an ICMP message by using other PL

   information (e.g., validation of connection identifiers (CIDs) in the

   quoted packet of any received ICMP message).

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 12 May 2020 14:00:03 +0000

Magnus,

Section "6.3.  DPLPMTUD for QUIC", quotes a very old version of the QUIC Transport draft.  The current text in QUIC transport says:

"
If a QUIC endpoint determines that the PMTU between any pair of local and remote IP addresses has fallen below the size needed to support the smallest allowed maximum packet size, it MUST immediately cease sending QUIC packets, except for PMTU probe packets, on the affected path. An endpoint MAY terminate the connection if an alternative path cannot be found.
"

  *   Igor

From: Magnus Westerlund <magnus.westerlund=[email-masked]>
Sent: Tuesday, May 12, 2020 4:50 AM
To: [email-masked]
Cc: [email-masked]
Subject: Update to draft QUIC DPLPMTUD text i draft-ietf-tsvwg-datagram-plpmtud

QUIC WG,

In response to the IESG evaluation of https://datatracker.ietf.org/doc/draft-ietf-tsvwg-datagram-plpmtud/ some changes was made to the QUIC chapter.

The current text is included below. And a Diff for the changes in this section -19 to -21 is here:

QUIC related section is 6.3:
https://www.ietf..org/rfcdiff?url1=draft-ietf-tsvwg-datagram-plpmtud-19&url2=draft-ietf-tsvwg-datagram-plpmtud-21<https://www.ietf.org/rfcdiff?url1=draft-ietf-tsvwg-datagram-plpmtud-19&url2=draft-ietf-tsvwg-datagram-plpmtud-21>

My plan is to approve this document on Friday after 12:00 (CEST) if not anyone yells.

6.3.  DPLPMTUD for QUIC

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 13 May 2020 16:18:38 +0000

Hi Igor,

Thanks for finding this issue. 

On Tue, 2020-05-12 at 14:00 +0000, Lubashev, Igor wrote:
> Magnus,
>  
> Section “6.3.  DPLPMTUD for QUIC”, quotes a very old version of the QUIC
> Transport draft.  The current text in QUIC transport says:

So you are specific talking about the third paragraph in Section 6.3. I think
that can be addressed by the authors. 

>  
> “
> If a QUIC endpoint determines that the PMTU between any pair of local and
> remote IP addresses has fallen below the size needed to support the smallest
> allowed maximum packet size, it MUST immediately cease sending QUIC packets,
> except for PMTU probe packets, on the affected path. An endpoint MAY terminate
> the connection if an alternative path cannot be found.
> “

I will note that this type of issues may still creep up and will have to be
addressed in AUTH48. That was why I did raise the question if we should actually
move all of the text in section 6.3 into quic-transport to avoid the normative
reference to quic-transport. 

Cheers

Magnus Westerlund 

----------------------------------------------------------------------
Networks, Ericsson Research
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Torshamnsgatan 23           | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: Martin Duke <[email-masked]>
Date: Thu, 14 May 2020 10:20:14 -0700

I don’t understand the MIN_PLPMTU language in 6.3.2. This is a constant set
based on the IP version. I don’t understand how it can change on a path?

On Tue, May 12, 2020 at 1:49 AM Magnus Westerlund <
[email-masked]> wrote:

> https://www.ietf.org/rfcdiff?url1=draft-ietf-tsvwg-datagram-plpmtud-19&url2=draft-ietf-tsvwg-datagram-plpmtud-21
>
>
>

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 14 May 2020 20:50:51 +0200

> On 14. May 2020, at 19:20, Martin Duke <[email-masked]> wrote:
> 
> I don’t understand the MIN_PLPMTU language in 6.3.2. This is a constant set based on the IP version. I don’t understand how it can change on a path?
I guess it should state that if the PLPMTU falls below MIN_PLPMTU, you stop sending packets.

Best regards
Michael
> 
> On Tue, May 12, 2020 at 1:49 AM Magnus Westerlund <[email-masked]> wrote:
> QUIC WG,
> 
>  
>

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 14 May 2020 20:19:39 +0100

To me, this is a PL choice, and likley a constant for a PL.

The PL simply configures as an endpoint what size it needs. If you need 
to work smaller than that size, then the PL has to utilise IP 
fragmentation, etc. The DPLPMTUD alg won't probe for smaller sizes.

I expect most apps will actually use a MIN value that is irrespective of 
IP version, such as 1200B, but the app could choose a smaller number - 
which is permitted for IPv4, but in reality 1200B is likely good. For 
QUIC, I'd expect ~1200B and no IP Frag - but that's QUIC's decision. 
BASE_PLPMTU and MIN_PLPMTU can be set the same.

Gorry

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 15 May 2020 15:56:33 +0000

Hi,

So what Gorry says appear correct. And it is a constant but its value could in
worst case be not only interface but actually path dependent. The reason being
that if the lower layers adds headers in specific cases. 

Although use of IPv6 extension headers are not problem free there might be cases
where such a header is added for some paths but not others. Thus, specifying the
MIN_PLPMTU correctly to be stable acrosss interfaces and paths could be
problematic. But for QUIC as Gorry says the stake is already in the ground and
it is in fact reasonable maybe to be more precise to say that a default
MIN_PLPMTU to use can be 1200 bytes as we don't expect QUIC to work over paths
that don't support this. 

We might get some interesting questions on the padded handshake packets when
some want to run QUIC as IOT transport over radios such as what LPWAN WG
discusses where it is very costly to send 1200 bytes packets. The radio blocks
are more in the range of 32-64 bytes if I remember correctly. 

So I think this discussion is pointing to that we should actually move Section
6.3 from draft-ietf-tsvwg-datagram-plpmtud into draft-ietf-quic-transport so
that the details can still be polished. 

Cheers

Magnus Westerlund

On Thu, 2020-05-14 at 20:19 +0100, Gorry Fairhurst wrote:
> > > I don’t understand the MIN_PLPMTU language in 6.3.2. This is a constant
> > > set based on the IP version. I don’t understand how it can change on a
> > > path?
> > 
-- 
Cheers

Magnus Westerlund

---

From: Christian Huitema <[email-masked]>
Date: Fri, 15 May 2020 10:31:17 -0700

> MIN_PLPMTU to use can be 1200 bytes as we don't expect QUIC to work over paths

Quic servers will only accept an Initial packet if it is at least 1200
bytes long. This pretty much enforces a min MTU requirement of 1200
bytes. Quic will not work on a network path does not support that.

> are more in the range of 32-64 bytes if I remember correctly. 

It is pretty hard to design an end-to-end transport that works well with
a very small end-to-end MTU. There are just too many trade-offs. For
example, the 1200 minimum MTU in Quic was chosen to mitigate DDOS
attacks. That's the kind of trade-off mandated by the requirement to
work end-to-end over the Internet. I assume that IOT devices with
limited networking capability will also have limited connectivity, for
example to a local relay.

> that the details can still be polished. 

I am comparing the specification in the paper with my early
implementation of PMTU discovery in Quic (Picoquic). They are pretty
much aligned, except for one important point regarding the sending of
probes. As the draft says, the Picoquic implementation will send probes
to discover the Path MTU, "by selecting step sizes from a table of
common PMTU sizes". But there is an additional step in the Picoquic
algorithm, a decision to probe or not as a trade-off between potential
gains in efficiency and potential costs in probing.

End-to-end probes consume resource. Sending a 1500 bytes probe consumes
about the same resource as sending 1500 bytes of data using the already
discovered packet size. Nodes send such probes because they expect a
gain in efficiency. If they discover that MTU size is 1500 bytes instead
of 1200 bytes, they can send a 1.5 MB file in 1000 packets instead of
1250. There will be 25% fewer packet headers, 25% fewer packet
processing, 25% fewer packet acknowledgements. That seems "worth the
try". But these gains depends on how many bytes the nodes expect to send
during the course of the connection.

Suppose now that instead of 1.5MB of data the node only anticipates
sending 15KB. Successful probe means sending 10 packets of 1500 bytes
instead of 13 packets of 1200 bytes. The typical overhead per Quic
packet would be about 60 bytes, between link header, IPv6 header, UDP
header and Quic header. Sending larger packets will save 180 bytes in
packet overhead, which is less than the 1500 bytes of sending a probe.
The node will achieve better throughput by not bothering with PMTU probes.

Of course, the numbers in the examples above are arbitrary. But there is
a principle there: at a given time, nodes send probes because they
anticipate a gain in efficiency over the reminder of the connection, if
the probe is successful. This gives us a simple test: only send probes
if the anticipated gain is larger than the cost of sending the probe:

    if (anticipated_gain * probability_of_success > cost-of_probe)

        then: send probe

        else: keep current MTU

That inequality informs the decision to send a probe or not. For
example, suppose that discovery so far has assessed that the PMTU is at
least 1400 bytes but lower than 1500. The table of step sizes says that
the most likely value between 1400 and 1500 is 1430. Is it a good idea
to try 1430? Maybe, but the efficiency gain will be just over 2%. In
case of success, the cost of probing will only be amortized after
sending 50 packets. If the probability of success is about 50%, the node
should only send the probe if it anticipates sending at least 100 packets.

The inequality also informs the decision of which probe to send. For
example, assume that the discovery so far has assessed that PMTU is at
least 1200 bytes. Will the node probe for 1500 bytes or 1400 bytes? To
make the decision, the node can assess the potential gains and the
probability of success. 1400 bytes might be a safer bet if the node
anticipates just sending a small amount of data, 1500 would provide a
higher gain if the node anticipates sending a large amount. Trying 8K
might be even more attractive if both peers support that size, as the
lower probability of success is compensated by a much higher gain.

The inequality also informs the decision on whether to repeat a probe.
Suppose that a node tried a 1500 bytes probe and failed. There are two
explanations: either the path MTU is lower than 1500 bytes, or the probe
was lost for other reasons. Applying Bayesian logic, the node can
reevaluate the probability of success of a 1500 bytes probe. It may well
be that sending a 1400 bytes probe will now be more interesting than
trying 1500 bytes again.

It may be that this discussion is too much detail for the plpmtud draft.
I could of course submit a draft discussing the probabilistic approach
to PMTUD. But it still might be a good idea to add a general
consideration in the description of the algorithm, asking implementation
to take into account probability of success and expected gains before
sending a probe.

-- Christian Huitema

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 15 May 2020 19:27:13 +0100

Thanks Christian, 

I think the search algorithm is interesting, I also think there are probably multiple ways to organise the search - and I like your analysis of the cost before probing higher, there is also a need to decide what to choose to probe when black holes are detected and there is inconsistent results, and i don’t know if  the performance of all the algorithms might be highly dependent on path properties ... we tried intentionally to avoid going beyond what we were sure about, expecting that if the framework worked and could be deployed in simple ways for robust black hole detection, we can determine good algorithms.

Actually, I think your thought is really cool, and I know a couple of people who would be really keen to work on the selection of algorithms (including how to deal with really awkward cases). Thinking and measuring could stimulate some useful research. I have huge hopes that we come around to some great algorithm, or set of algorithms, and we might then be able to publish an RFC on that part too.

Sent from my iPad

> On 15 May 2020, at 18:40, Christian Huitema <[email-masked]> wrote:
> 
>

---

From: Timo Völker <[email-masked]>
Date: Tue, 26 May 2020 12:32:43 +0000

Hi Christian,

thanks for sharing your experiences. Especially for the in-equation.

I wonder how to determine the values.

anticipated_gain
It is easy if we know how much data we are going to send over a connection, but how can QUIC figures that out in advance?
Also, a discovered PMTU could be reused in the next connection to the same destination. How to anticipate whether and how often we will reconnect in a short term?

probability_of_success
I think, measurements can tell (or have told?) us.

cost-of_probe
What do you mean with cost? The extra load for the network or the consequences for the endpoints?
It seems to me, the costs for a sender is quite high in QUIC. If we follow the spec, a sent probe packet consumes cwnd and, if it gets lost (for whatever reason), it reduces the (New Reno) cwnd, right? Did you include exceptions for probe packets?

Timo

> On 15. May 2020, at 19:31, Christian Huitema <[email-masked]> wrote:
> 
>

---

From: Christian Huitema <[email-masked]>
Date: Tue, 26 May 2020 11:47:12 -0700

> It is easy if we know how much data we are going to send over a connection, but how can QUIC figures that out in advance?

This tends to depend on the application running on top of QUIC. In the
case of HTTP3, the server knows which documents have been requested by
the client, and the size of these documents. That gives an indication of
the "committed load". For other applications, the server has to make an
educated guess based on application profile, statistics, etc. In my
implementation of DNS over QUIC, for example, I just turn off PMTUD
because it is seldom useful for the DNS application.

> Also, a discovered PMTU could be reused in the next connection to the same destination. How to anticipate whether and how often we will reconnect in a short term?
That's another can of worms. I suppose this could be incorporated in the
general framework. It is not clear that two connections to the same
destination will encounter the same PMTU -- routing could have changed.
I guess that recent tests on the same source-destination path will
inform the "probability of success" of trying a specific MTU, but that
if tests are old it is better to assume some regression to average. On
the benefit side, measured frequency of past connections to the
destination could inform the potential benefits.
> I think, measurements can tell (or have told?) us.

Yes. Big services with millions of users can get lots of measurements,
and so have an advantage there. It would be very nice if there were some
publicly available measurements, for the benefit of smaller deployments.

> It seems to me, the costs for a sender is quite high in QUIC. If we follow the spec, a sent probe packet consumes cwnd and, if it gets lost (for whatever reason), it reduces the (New Reno) cwnd, ri…

In my implementation, I just use the number of octets to be sent, and I
do have an exception for probe packets. Other implementations might have
a different interpretation of what the cost is.

-- Christian Huitema