Subject: [Last-Call] Opsdir last call review of draft-ietf-quic-manageability-14

From: Al Morton via Datatracker <[email-masked]>
Date: Sat, 05 Feb 2022 14:11:10 -0800

Reviewer: Al Morton
Review result: Has Issues

Hi Mirja and Brian,

This is the OPSDIR review of

              Manageability of the QUIC Transport Protocol
                    draft-ietf-quic-manageability-14

Thanks for preparing this draft. I think it will succeed to inform your
intended audience. I found that it filled-in some gaps for me.

Likewise, I found some areas where I make suggestions or comment. I would be
happy to discuss any of these areas further. Your familarity with
related-work/papers-to-cite could minimize your efforts in response. Protocols
are evolving, and so are Networks.

Editorially, the doc is in great shape, except for the use of different terms
for when TCP takes-over: fail over, fallback, and fail-over are all used.

regards,
Al

Abstract

   This document discusses manageability of the QUIC transport protocol,
   focusing on the implications of QUIC's design and wire image on
   network operations involving QUIC traffic.  It is intended as a
   "user's manual" for the wire image, providing guidance for network
   operators and equipment vendors who rely on the use of transport-
   aware network functions.

...

2.4.  The QUIC Handshake

...

   Client                                    Server
     |                                          |
     +----Client Initial----------------------->|
     +----(zero or more 0RTT)------------------>|
     |                                          |
     |<-----------------------Server Initial----+
     |<---------(1RTT encrypted data starts)----+
     |                                          |
     +----Client Completion-------------------->|
     +----(1RTT encrypted data starts)--------->|
     |                                          |
     |<--------------------Server Completion----+
     |                                          |

   Figure 1: General communication pattern visible in the QUIC handshake

   As shown here, the client can send 0-RTT data as soon as it has sent
   its Client Hello, and the server can send 1-RTT data as soon as it
   has sent its Server Hello.  The Client Completion flight contains at
   least one Handshake packet and could also include an Initial packet.
   QUIC packets in separate contexts during the handshake can be
   coalesced (see Section 2.2) in order to reduce the number of UDP
   datagrams sent during the handshake.  QUIC packets can be lost and
   reordered, so packets within a flight might not be sent close in
   time, though the sequence of the flights will not change, because one
   flight depends upon the peer's previous flight.

[acm]
It's great to add some Not-Sunny-Day info in the description, thanks!
But can you add a little more?  For example:
Is it possible that network reordering can cause the handshake to fail?
What rerodering extent (yes, that's a metric) would be required to cause
failure or unnecessary retransmission? Lost packets would result in time-outs
and retransmission, so what are the default time-outs? Is there a paper where
some/all of the above have been investigated, that you could reference to save
some work?

...

2.8.  Version Negotiation and Greasing

...
   QUIC is expected to evolve rapidly, so new versions, both
   experimental and IETF standard versions, will be deployed on the
   Internet more often than with traditional Internet- and transport-
   layer protocols.  Using a particular version number to recognize
   valid QUIC traffic is likely to persistently miss a fraction of QUIC
   flows and completely fail in the near future, and is therefore not
   recommended.
[acm] Where "valid traffic" is the focus, I agree, let it flow.
But the Operator's focus may instead be "admissible traffic", where
experimental traffic is not wanted or allowed. IOW, only traffic that is
understood to conform to <RFC list> shall pass, because "Active Attacks are
also Pervasive", to put a different spin on 7258. [acm] See also the comment in
3.4.1.

   In addition, due to the speed of evolution of the
   protocol, devices that attempt to distinguish QUIC traffic from non-
   QUIC traffic for purposes of network admission control should admit
   all QUIC traffic regardless of version.

[acm] I was hoping to see a description of fallback to TCP (I see that fallback
is mentioned briefly at the end of section 4.2., and later, fail over and
failover. pick one...)

How can Network Operators observe when a QUIC setup has failed, and the
corresponding TCP fallback connection(s) succeeded?

Is there a reference available with this info, to save effort here?

...

3.4.1.  Extracting Server Name Indication (SNI) Information

...

   Note that proprietary QUIC versions, that have been deployed before
   standardization, might not set the first bit in a QUIC long header
   packet to 1.  However, it is expected that these versions will
   gradually disappear over time.
[acm]
And some networks may prefer not to admit experimental traffic. The goal of the
experiment may be problematic for the network operator and/or their
subscribers. I think this is legitimate operator behavior, and worth a few more
words in the draft.

...

3.8.1.  Measuring Initial RTT

...

   Handshake RTT can be measured by adding the client-to-observer and
   observer-to-server RTT components together.  This measurement
   necessarily includes any transport- and application-layer delay at
   both endpoints.
[acm] suggest s/any/all/

3.8.2.  Using the Spin Bit for Passive RTT Measurement

...

   Note that this measurement, as with passive RTT measurement for TCP,
   includes any transport protocol delay (e.g., delayed sending of
[acm] suggest s/any/all/

...

   Since the spin bit logic at each endpoint considers only samples from
   packets that advance the largest packet number, signal generation
   itself is resistant to reordering.  However, reordering can cause
   problems at an observer by causing spurious edge detection and
   therefore inaccurate (i.e., lower) RTT estimates, if reordering
   occurs across a spin-bit flip in the stream.
[acm] thanks for mentioning this!

...

   Raw RTT samples generated using these techniques can be processed in
   various ways to generate useful network performance metrics.  A
   simple linear smoothing or moving minimum filter can be applied to
   the stream of RTT samples to get a more stable estimate of
   application-experienced RTT.  RTT samples measured from the spin bit
   can also be used to generate RTT distribution information, including
   minimum RTT (which approximates network RTT over longer time windows)
   and RTT variance (which approximates jitter as seen by the
   application).
[acm]   (let's avoid the clocky term "jitter", and clarify)
Suggest: (which over-estimates one-way packet delay variance as seen by an
application end-point).

4.  Specific Network Management Tasks
...

4.2.  Stateful Treatment of QUIC Traffic

   Stateful treatment of QUIC traffic (e.g., at a firewall or NAT
   middlebox) is possible through QUIC traffic and version
   identification (Section 3.1) and observation of the handshake for
   connection confirmation (Section 3.2).  The lack of any visible end-
   of-flow signal (Section 3.6) means that this state must be purged
   either through timers or through least-recently-used eviction,
   depending on application requirements.
[acm] Comment: It suddenly struck me that this might be similar to the scenario
that dkg frequently cited during QUIC development: His ISP would terminate idle
TCP connections after many hours. See the citation of RFC5382 below. Don't
expect QUIC connections to stay-up forever! The next Purge will occur in 3, 2,
1, ...

   While QUIC has no clear network-visible end-of-flow signal and
   therefore does require timer-based state removal, the QUIC handshake
   indicates confirmation by both ends of a valid bidirectional
   transmission.  As soon as the handshake completed, timers should be
   set long enough to also allow for short idle time during a valid
   transmission.

   [RFC4787] requires a network state timeout that is not less than 2
   minutes for most UDP traffic.  However, in practice, a QUIC endpoint
   can experience lower timeouts, in the range of 30 to 60 seconds
   [QUIC-TIMEOUT].

   In contrast, [RFC5382] recommends a state timeout of more than 2
   hours for TCP, given that TCP is a connection-oriented protocol with
   well- defined closure semantics.  Even though QUIC has explicitly
   been designed to tolerate NAT rebindings, decreasing the NAT timeout
   is not recommended, as it may negatively impact application
   performance or incentivize endpoints to send very frequent keep-alive
   packets.

   The recommendation is therefore that, even when lower state timeouts
   are used for other UDP traffic, a state timeout of at least two
   minutes ought to be used for QUIC traffic.
[acm]
2 minutes, not hours. got it.

...

4.5.  Filtering Behavior

   [RFC4787] describes possible packet filtering behaviors that relate
   to NATs but is often also used is other scenarios where packet
   filtering is desired.  Though the guidance there holds, a
   particularly unwise behavior admits a handful of UDP packets and then
   makes a decision to whether or not filter later packets in the same
   connection.  QUIC applications are encouraged to fail over to TCP if
[acm]
is "fail over" or "fallback" the preferred term?
(using only one will help)

   early packets do not arrive at their destination
   [QUIC-APPLICABILITY], as QUIC is based on UDP and there are known
   blocks of UDP traffic (see Section 4.6).  Admitting a few packets
   allows the QUIC endpoint to determine that the path accepts QUIC.
   Sudden drops afterwards will result in slow and costly timeouts
   before abandoning the connection.

4.6.  UDP Blocking, Throttling, and NAT Binding

...
   Further, if UDP traffic is desired to be throttled, it is recommended
   to block individual QUIC flows entirely rather than dropping packets
   indiscriminately.  When the handshake is blocked, QUIC-capable
   applications may fail over to TCP.  However, blocking a random
[acm]
is "fail over" or "fallback" the preferred term?
(using only one will help)

   fraction of QUIC packets across 4-tuples will allow many QUIC
   handshakes to complete, preventing a TCP failover, but these
[acm] ... or "failover" preferred?

   connections will suffer from severe packet loss (see also
   Section 4.5).  Therefore, UDP throttling should be realized by per-
   flow policing, as opposed to per-packet policing.  Note that this
   per-flow policing should be stateless to avoid problems with stateful
   treatment of QUIC flows (see Section 4.2), for example blocking a
   portion of the space of values of a hash function over the addresses
   and ports in the UDP datagram.  While QUIC endpoints are often able
   to survive address changes, e.g. by NAT rebindings, blocking a
   portion of the traffic based on 5-tuple hashing increases the risk of
   black-holing an active connection when the address changes.
...

4.8.  Quality of Service Handling and ECMP Routing

   It is expected that any QoS handling in the network, e.g. based on
   use of DiffServ Code Points (DSCPs) [RFC2475] as well as Equal-Cost
   Multi-Path (ECMP) routing, is applied on a per flow-basis (and not
   per-packet) and as such that all packets belonging to the same active
   QUIC connection get uniform treatment.
[acm] Comment: so networks should continue their *extra* efforts for datagrams,
like maintaining order, while the datagram streams take away as much info as
they can. got it...

Done.

---

From: "Martin Thomson" <[email-masked]>
Date: Mon, 07 Feb 2022 09:19:58 +1100

Hi Al,

On Sun, Feb 6, 2022, at 09:11, Al Morton via Datatracker wrote:
> failure or unnecessary retransmission? 

For my benefit: Is "rerodering" a typo, or a clever joke?

As for the question, I believe that QUIC implementations will deal with any amount of reordering within their timeout period (which varies, but it might be 10s or 30s in common cases).

There are some narrow cases where packets might be discarded, but that would be down to specific constraints on the deployment, not the protocol itself.  For example, a server might choose not to take on arbitrary state based on the first packet, so it might discard any data that arrives out of order.  This is still likely to resolve as the client is required to retransmit.  Worst case, it takes N round trips to resolve, where N is the number of packets in the flight.  If that ends up taking more than the timeout (on either endpoint), then it will fail.  N == 1 in most current cases, so this is more of a theoretical problem than a practical one.

This scenario was extensively tested. interop.seemann.io has a bunch of tests of handshake robustness, which sometimes produces a lot of reordering.  You can see there that quant, which has the above behaviour, fails the L1 and C1 tests in some cases as a result.

That's probably more detail than the doc needs though.

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 7 Feb 2022 13:59:37 +0000

> That's probably more detail than the doc needs though.

The transport uses pretty normal standard transport stuff: Reordering 
*can* impact loss detection and hence lead to spurious retransmission 
(DUPack threshold is at least 3 by default for Reno - used thresholds 
depend on the sender implementation). Spurious retransmission can 
potentially lead to spurious CC reaction (mitigated/avoided when if 
original data is ACked as received).

Gorry

---

From: "MORTON JR., AL" <[email-masked]>
Date: Mon, 7 Feb 2022 15:04:24 +0000

> -----Original Message-----
> From: Gorry Fairhurst <[email-masked]>
> Sent: Monday, February 7, 2022 9:00 AM
> To: Martin Thomson <[email-masked]>; MORTON JR., AL <[email-masked]>;
> [email-masked]
> Cc: [email-masked]; [email-masked];
> [email-masked]
> Subject: Re: Opsdir last call review of draft-ietf-quic-manageability-14
> 
[acm] Typo this time, clever joke next time 😊
My original joke idea was that the reordering RFC # should illustrate reordering with single-digit seq numbers. Instead, we were assigned 4737, which has loss (5,6), duplication (7), and reordering (3). Not quite the clear example I was looking for...

> >
> > As for the question, I believe that QUIC implementations will deal with any
> amount of reordering within their timeout period (which varies, but it might
> be 10s or 30s in common cases).
> >
> > There are some narrow cases where packets might be discarded, but that would
> be down to specific constraints on the deployment, not the protocol itself.
> For example, a server might choose not to take on arbitrary state based on the
> first packet, so it might discard any data that arrives out of order.  This is
> still likely to resolve as the client is required to retransmit.  Worst case,
> it takes N round trips to resolve, where N is the number of packets in the
> flight.  If that ends up taking more than the timeout (on either endpoint),
> then it will fail.  N == 1 in most current cases, so this is more of a
> theoretical problem than a practical one.
> >
> > This scenario was extensively tested. interop.seemann.io has a bunch of
> tests of handshake robustness, which sometimes produces a lot of reordering.
> You can see there that quant, which has the above behaviour, fails the L1 and
> C1 tests in some cases as a result.
> >
[acm] 
Thanks Martin and Gorry, there is plenty to work with above if the authors agree to add some more detail.

Al

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Mon, 7 Feb 2022 09:35:56 -0600

I haven't seen anyone else respond to Al's email on this point, so I
thought I'd share an opinion.

>                     draft-ietf-quic-manageability-14

Snip, down to

>    black-holing an active connection when the address changes.

In my mind,

   - "fallback" makes more sense if we are talking about falling back
   within a single protocol (for example, attempting to use an extension,
   discovering that the other host doesn't support that extension, and
   retrying without the extension - or,, also within a single protocol,
   attempting to use version 9, discovering that the other host doesn't
   support that extension, and retrying with a different version), and
   - "failover" makes more sense if we are talking about starting with one
   protocol (QUIC, in this case) and if that doesn't work, switching to a
   different protocol (TCP, in this case).

I know we've used both terms somewhat interchangeably during discussions
about QUIC (and not just discussions about this document), but if one term
is to be chosen (Al's suggestion, which I agree with), I think what we're
talking about here is "failover".

Other people may have thoughts, of course.

Best,

Spencer

---

From: Lucas Pardue <[email-masked]>
Date: Mon, 7 Feb 2022 16:24:51 +0000

Hey,

On the topic of fallback or failover or something else. My 2c with no hats.

On Mon, 7 Feb 2022, 15:36 Spencer Dawkins at IETF, <
[email-masked]> wrote:

>    - "failover" makes more sense if we are talking about starting with
>    one protocol (QUIC, in this case) and if that doesn't work, switching to a

To me this is really an implementation detail.

In HTTP/3's case you might learn of a it via Alt-Svc over TCP and try yo
swtich to it. That could fail and you might "fallback" or really just
ignored a suggestion that turned out to be bogus. Nothing fell over or
failed except an opportunistic connection attempt.

Now add in the alternative discovery mechanism of SVCB and HTTPS record.
Clients would learn the availability of multiple HTTP versions and are
likely to implement different strategies for selecting on or the other.
Imagine a client which learns that on some kinds of network they block
QUIC. Maybe it would just stop bothering to try QUIC on that network
despite any other information available.

There are, of course, other application protocols that run over QUIC. Those
things might have no TCP equivalent. The network operators should not be
trying to introspect such things as application mapping.

All of these things are examples where client agency gives no hope for
others to guess their behaviour. And that's fine. IMO it suffices to say
that a clients can switch to any transport, at any time, for any reason.
That is neither failover or fallback, just choice.

Trying to diagnose network issues based on choice is futile and, if done
naively, could false diagnose problems that do not exist.

Cheers
Lucas

---

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 11 Feb 2022 17:20:50 +0000

Hi Al,

thanks for your review. Glad to hear that you find the draft useful. I think your input is very valuable for this draft as you are part of the intended audience, however, not sure how/if to address many of your comments. But let me try provide some explanations. Please see inline marked with [MK].

Mirja

On 05.02.22, 23:11, "QUIC on behalf of Al Morton via Datatracker" <[email-masked] on behalf of [email-masked]> wrote:

    Reviewer: Al Morton

    Hi Mirja and Brian,

[MK] This we were able to address. Thanks for pointing this out. See PR here if you want: https://github.com/quicwg/ops-drafts/pull/457/files 

    regards,
    Al

    Abstract

    ...

    2.4.  The QUIC Handshake

    ...

       Client                                    Server
         |                                          |

    [acm]

[MK] I don't think I have a good answer to these questions. However, I don't think these questions are necessarily QUIC specific. Reordering in itself is not a problem for the QUIC handshake, however, if some packets are also delayed a lot and therefore detected as loss, it might get the handshake to fail. How long you keep state to wait for delayed packets in mostly implementation specific, however, as I said this is also not a unique problem to the QUIC handshake.

[MK] Further, this section in the draft really only explains what you have to expect when you see QUIC traffic as a passive obverser. The reason why we mention loss and reorder in this section, is really just to say, if you e.g. want to detect the QUIC handshake as a passive on-path observer, you should not expect it to always look exactly like this (as here might be reordering or loss earlier in the network between the client and your observation point).

    ...

    ...

[MK] This is not about experimentation. The expectation is that QUIC versions will change often, e.g. we already have a draft for a new version adopted in the group and there might be another RFC some time this year. So if you "manually" have to allow for new versions in all your equipment that will delay deployment of new versions (or even hinder them because there is always one box that doesn't get updated). Therefore we strongly recommend to not use the version to filter QUIC traffic. Is that not clear enough in the text?

[MK] There is no unified way how and if fallback is implemented. However, why do you think a network operator would need that information?

[MK] As I said this is rather implementation specific, so I would say no.

    ...

    ...

[MK] To be honest I don't understand this point. How would an operator even know if an experiment would be problematic or no? QUIC is fully encrypted. Versioning is only one extension mechanism. So basically even if you see the same version number, the QUIC behind that could behave very differently depending on which extensions are used and because of the encryption, there is no chance for the operator to know about this. Is this not clear in the document? Do we need to state this more clearly?

    ...

    ...

[MK] Done!

    ...

[MK] Done!

    ...

    ...

[MK] Done!

[MK] QUIC has the connection ID mainly because time-outs for UDP are often short. So I think this is a known problem. Or is there anything you think we should add to this document?

    ...

    4.5.  Filtering Behavior

    ...

[MK] I don't think networks should put in extra effort to reordering, especially as reordering usually causes delays. Actually QUIC, as well as TCP with certain extensions, can be quite robust to re-ordering but that's implementation specific, and therefore you never know as passive observer.  The ask is rather, as you do today for TCP, to avoid reordering in the first place if possible, e.g. use the full 5-tuple for ECMP. So I think the ask is actually rather to keep thing running as they are right now, than doing anything special for QUIC. Again do we need to make that message more clear?

[MK] Thanks again! Mirja

    Done.

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 11 Feb 2022 13:38:22 -0600

Just one observation, that's relevant to an Al question. Snipping down to

On Fri, Feb 11, 2022 at 11:21 AM Mirja Kuehlewind <mirja.kuehlewind=
[email-masked]> wrote:

> Hi Al,
>
> thanks for your review. Glad to hear that you find the draft useful. I
> think your input is very valuable for this draft as you are part of the
> intended audience, however, not sure how/if to address many of your
> comments. But let me try provide some explanations. Please see inline
> marked with [MK].
>
> Mirja
>
>
> On 05.02.22, 23:11, "QUIC on behalf of Al Morton via Datatracker" <
> [email-masked] on behalf of [email-masked]> wrote:
>
> [MK] There is no unified way how and if fallback is implemented. However,
> why do you think a network operator would need that information?
>

 On the left hand side of Mirja's response, we spent a LOT of time talking
about what QUIC (more properly, HTTP) would do if UDP blocking or severe
rate limiting prevented a path from being usable for HTTP/3.

In the HTTP case, giving up on HTTP/3 over QUIC over UDP is obvious
(because the path isn't usable for HTTP/3). If you insist on continuing to
use HTTP, you'll use HTTP/1 or HTTP/1.1 or HTTP/2, all over TCP, which
won't be affected by network treatment of UDP.

I remember having a conversation with Brian about what it means for the
Internet architecture that some applications that might make use of QUIC do
have obvious strategies when QUIC (or UDP) blocking happens, and other
applications do not.

There's quite a bit of discussion about this in
https://www.ietf.org/archive/id/draft-ietf-quic-applicability-14.html#name-the-necessity-of-fallback
.

If I'm tracking the discussion in this thread, I wonder if the
manageability draft, which already points to the applicability draft, needs
to say more than

   - A passive observer can't consistently tell whether QUIC handshakes
   succeed or fail, and
   - A passive observer can't know what QUIC applications will do next in
   the general case if their handshakes fail.

But I think the answer to the right hand side of Mirja's response is needed
for us to Do The Right Thing.

Best,

Spencer