Subject: WGLC review of draft-ietf-quic-recovery-29

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 8 Jul 2020 21:29:59 +0000

This is a WGLC review of draft-ietf-quic-recovery-29. Mostly editorial but some issues.

Section 2
" are not acknowledged, declared lost, or abandoned along with old keys." -->  are not acknowledged, declared lost, or discarded along with old keys."
Discarding is the term used in the rest of the doc notably in section 2.4. I see some reference to "abandon packets" in the transport draft as well. Would be good to use the same term for consistency everywhere.

Section 4.4.
 "does not allow any acked packet to be reneged" --> "does not allow any acknowledged packet to be reneged"

Section 4.7
"QUIC uses a probe timeout (see Section 6.2)," --> "QUIC uses a Probe Timeout (PTO) (see Section 6.2),"

Section 5.2 "If the path's actual RTT increases, the min_rtt will not adapt to it, allowing future RTT samples that are smaller than the  new RTT be included in smoothed_rtt."
Would it safer to reset min_rtt after persistent congestion is declared. This is important so we can adapt to a consistent spike in RTT as a result of a path change.

Section 6.1.1

"Algorithms that increase the reordering threshold after spuriously detecting losses, such as TCP-NCR [RFC4653], have proven to be useful in TCP and are expected to at least as useful in QUIC."
It might be better to refer to RACK here which has an optional feature to increase reorder threshold in time. I don't know of any major implementations that do TCP-NCR but maybe I am mistaken.

Section 6.2.1
"time threshold Section 6.1.2" --> time threshold (Section 6.1.2)

"The life of a connection that is experiencing consecutive PTOs is limited by the endpoint's idle timeout."
Is there any specification of recommended values for idle timeout? Any values lower than 9 seconds could adversely impact preserving of connectivity during VM live migration.

Section 6.2.4
"If the sender wants to elicit a faster acknowledgement on PTO, it can skip a packet number to eliminate the ack delay."
This adds some unnecessary complexity. Given that endpoint MAY send two full-sized datagrams, isn't that a better technique than skipping packet numbers?

Section 7.8
"The congestion period is calculated as the time between the oldest and newest lost packets: (3 - 0) = 3." --> "The congestion period is calculated as the sent time difference between the oldest and newest lost packets: (3 - 0) = 3."
It might be worth noting that persistent congestion can be declared without sending a PTO. Particularly because the example assumes PTOs. In the case where app kept on periodically sending new data and no ACKs came back for the threshold and then an ACK came back and declared all packets sent before it as lost.

Section 7.9
"Implementations MUST either use pacing or another method to limit such bursts to the initial congestion window; see Section 7.2."
This seems to preclude use of segmentation offload of sizes greater than IW. In datacenters we routinely send bursts that are higher without causing loss. The MUST here seems unnecessary. It also conflicts with the RECOMMENDED in an earlier sentence.

Section 8.2
"Endpoints can use PADDING frames or bundle acknowledgments with other frames to reduce leaked information."
A reminder should be added here that PADDING counts towards in-flight so there is a possible impact on performance.

Section B.8
The function AreAllPacketsLost does not have any pseudocode.

Thanks

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 9 Jul 2020 09:39:27 +0100

Hi Praveen,

On 08/07/2020 22:29, Praveen Balasubramanian wrote:
>
> Section 7.9
>
> "Implementations MUST either use pacing or another method to limit 
> such bursts to the initial congestion window; see Section 7.2."
>
> This seems to preclude use of segmentation offload of sizes greater 
> than IW. In datacenters we routinely send bursts that are higher 
> without causing loss.
>
Is that within data centres? or from data centres to Internet 
destinations? ... if the latter, how can the sender be sure this is not 
impacting other traffic?
>
> The MUST here seems unnecessary. It also conflicts with the 
> RECOMMENDED in an earlier sentence.
>
I actually don't see this as conflicting. The recommendation in this 
section is to pace. The requirement is that sender limits burst, using 
some method, to the size of IW.

Gorry

---

From: Praveen Balasubramanian <[email-masked]>
Date: Thu, 9 Jul 2020 16:11:33 +0000

Within data center mainly. To hit 40 Gbps line rate you effectively have to batch as much as you can to minimize the CPU processing costs. The burst size today is capped to 64k in most cases for TCP. Now that we have segmentation offload for UDP the same should apply. Even for Internet facing traffic, the bursts primarily impact the first hop within the datacenter because subsequently you get statistical multiplexing from ToR switch handling traffic forwarding for many flows and ports.

Ok understood about there being no conflict. But I still contend that the MUST regarding burst size is way too restrictive. We should allow for experimentation here. Or present data that this is universally bad on all networks. Current production use of TCP shows there is no major problems with bursting higher. In fact for low latency same rack scenarios this will boost receive performance as well due to better opportunity for LRO etc.

From: Gorry Fairhurst <[email-masked]>
Sent: Thursday, July 9, 2020 1:39 AM
To: Praveen Balasubramanian <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: [EXTERNAL] Re: WGLC review of draft-ietf-quic-recovery-29

Hi Praveen,
On 08/07/2020 22:29, Praveen Balasubramanian wrote:
Section 7.9
"Implementations MUST either use pacing or another method to limit such bursts to the initial congestion window; see Section 7.2."
This seems to preclude use of segmentation offload of sizes greater than IW. In datacenters we routinely send bursts that are higher without causing loss.
Is that within data centres? or from data centres to Internet destinations? ... if the latter, how can the sender be sure this is not impacting other traffic?

The MUST here seems unnecessary. It also conflicts with the RECOMMENDED in an earlier sentence.

I actually don't see this as conflicting. The recommendation in this section is to pace. The requirement is that sender limits burst, using some method, to the size of IW.

Gorry

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 10 Jul 2020 09:21:00 +0100

Thanks Praveen,

I'd like to talk more, see below:

On 09/07/2020 17:11, Praveen Balasubramanian wrote:
>
> Within data center mainly. To hit 40 Gbps line rate you effectively 
> have to batch as much as you can to minimize the CPU processing costs. 
> The burst size today is capped to 64k in most cases for TCP. Now that 
> we have segmentation offload for UDP the same should apply. Even for 
> Internet facing traffic, the bursts primarily impact the first hop 
> within the datacenter because subsequently you get statistical 
> multiplexing from ToR switch handling traffic forwarding for many 
> flows and ports.
>
I get this for this use-case and comments on high-speed NICs. I agree 
offload is important.

> Ok understood about there being no conflict. But I still contend that 
> the MUST regarding burst size is way too restrictive. We should allow 
> for experimentation here.
>
I am not sure. The IW spec for TCP was based on evidence back around 
2010, and the IETF agreed in RFC6928 that a burst size of 10 was 
acceptable as an EXP spec for TCP.

I think we can safely refer to that - even perhaps take the 
leap-of-faith that EXP for TCP is equivalent to PS for QUIC, given 
subsequent experience.  I think the IW  **is** the burst size that the 
IETF should specify (because it's dictated by buffering that is expected 
on the path).

I see opportunities here for experimentation and measurement 
opportunities, and new specs can be proposed. Or the Specs can be recast 
with more detail. I also know that practical experience is that many 
people have configured (probably tested, I hope) much larger IW's - and 
I'd hope they also make a reasonable effort to monitor for possible 
interactions with other Internet applications and services (as per 
section 12 of RFC6928).

> Or present data that this is universally bad on all networks.
>
I think you touch on something that the IETF hasn't been great at 
talking about:

I could myself indeed likely produce data this shows this is "bad" in 
some networks that I value, and also show side-effects of collateral 
damage to other flows and adverse effects of line-rate bursts into 
equipment with small buffers. I don't know how useful that would be at all.

The IETF has typically defined a spec for endpoint. The way a modern 
endpoint is implemented can be quite different in a DC from a single 
device connected to the Internet. In a DC, I understand it is common  to 
distribute transport functions to other parts of the network. e.g.: DDOS 
protection could be done on a server; or by a box in front of it - 
similarly what matters to the network CC is the  bursts of traffic that 
leaves a DC within a flow to a local receiver, not what leaves a 
particular server within the DC network - I think in future we may do 
well to consider these differences?

> Current production use of TCP shows there is no major problems with 
> bursting higher. In fact for low latency same rack scenarios this will 
> boost receive performance as well due to better opportunity for LRO etc.
>
It's also possible that QUIC LRO (or whatever) includes some notion of 
per-flow burst-mitigation. Is that impossible?

However, I personally would not hold-up QUIC, to arrive at a new consensus.

Gorry

> *From:* Gorry Fairhurst <[email-masked]>
> *Sent:* Thursday, July 9, 2020 1:39 AM
> *To:* Praveen Balasubramanian <[email-masked]>; IETF QUIC WG 
> <[email-masked]>
> *Subject:* [EXTERNAL] Re: WGLC review of draft-ietf-quic-recovery-29
>
> Hi Praveen,
>
>     This seems to preclude use of segmentation offload of sizes
>     greater than IW. In datacenters we routinely send bursts that are
>     higher without causing loss.
>
> destinations? ... if the latter, how can the sender be sure this is 
> not impacting other traffic?
>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 10 Jul 2020 17:06:19 +0000

Thanks Neal and Yuchung. +quic@ietf and Gorry

From: Neal Cardwell <[email-masked]>
Sent: Friday, July 10, 2020 7:51 AM
To: Praveen Balasubramanian <[email-masked]>
Cc: Yuchung Cheng <[email-masked]>
Subject: Re: FW: [EXTERNAL] Re: WGLC review of draft-ietf-quic-recovery-29

Yes, a 64k TSO qualify as a burst higher than 10 packets.

I agree with Praveen's remark that "the MUST regarding burst size is way too restrictive".

I'd suggest something like:

   Implementations MUST either use pacing or another method to limit such bursts.

   It is RECOMMENDED that traffic over the public Internet limit bursts to the initial

   congestion window; see Section 7.2<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-quic-recovery-29%23section-7.2&data=02%7C01%7Cpravb%40microsoft.com%7C9df3e5f4f0ca4e586f0908d824e0a3ad%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637299894526657906&sdata=ze%2F7SpKUbrdgwI6m%2BS6uEGSBgc2NapVtPn9vkOYFjVs%3D&reserved=0>.

The current Linux TSO sizing algorithm would be compatible with such a recommendation up to rates of 10 * 1500 * 8 / .001 / 10^6 = 120Mbps.

neal

On Thu, Jul 9, 2020 at 1:26 PM Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>> wrote:
Yes but TSO to NICs that do it in hardware effectively means a burst because AFAIK hardware doesn’t pace those out. So wouldn’t 64k TSO qualify as a burst higher than 10 packets. I agree a SHOULD is much better.

From: Yuchung Cheng <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, July 9, 2020 9:59 AM
To: Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>
Cc: Neal Cardwell <[email-masked]<mailto:[email-masked]>>
Subject: Re: FW: [EXTERNAL] Re: WGLC review of draft-ietf-quic-recovery-29

oh I read the sentence too fast. I read it as "use pacing or another method (that both) to limit bursts to the initial window".

It seems the draft is saying, without pacing, the burst should be capped. I think that's from the old BSD "max_burst" config to 3 == old_IW. For that I'd say IW10 setting is reasonable. Although MUST is too strong. I agree SHOULD is more appropriate

On Thu, Jul 9, 2020 at 9:54 AM Yuchung Cheng <[email-masked]<mailto:[email-masked]>> wrote:
Linux sizes TSO (upt to 64KB) based on the socket pacing rate. >.5G reaches 64KB. ~64Mbps produces 16KB > IW10, regardless of the destination (e.g. DC or not).

The artificial IW10 TSO cap makes no sense IMO in the draft. I'd challenge the authors to justify with data. If anything the TSO needs to be even bigger than 64KB on higher speed NICs.

On Thu, Jul 9, 2020 at 9:16 AM Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>> wrote:
Does Linux never send a TSO of size more than 10 packets under any circumstances?

Thanks

From: Praveen Balasubramanian
Sent: Thursday, July 9, 2020 9:12 AM
To: Gorry Fairhurst <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: RE: [EXTERNAL] Re: WGLC review of draft-ietf-quic-recovery-29

From: Gorry Fairhurst <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, July 9, 2020 1:39 AM
To: Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: [EXTERNAL] Re: WGLC review of draft-ietf-quic-recovery-29

Hi Praveen,

Gorry

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 10 Jul 2020 17:11:27 +0000

Precluding bursts higher than 10 packets is simply not viable or practical at really high data dates and is a major regression from the way TCP works today. QUIC is a general purpose transport and not scoped to the Internet only. On a side thread around Linux TCP behavior Neal suggested some alternate text to scope this to Internet facing traffic but my contention still holds that there are many networks where higher bursts are not a problem and we should not restrict those use cases.

From: Gorry Fairhurst <[email-masked]>
Sent: Friday, July 10, 2020 1:21 AM
To: Praveen Balasubramanian <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: [EXTERNAL] Re: WGLC review of draft-ietf-quic-recovery-29

Thanks Praveen,

I get this for this use-case and comments on high-speed NICs. I agree offload is important.
Ok understood about there being no conflict. But I still contend that the MUST regarding burst size is way too restrictive. We should allow for experimentation here.

I am not sure. The IW spec for TCP was based on evidence back around 2010, and the IETF agreed in RFC6928 that a burst size of 10 was acceptable as an EXP spec for TCP.

I think we can safely refer to that - even perhaps take the leap-of-faith that EXP for TCP is equivalent to PS for QUIC, given subsequent experience.  I think the IW  **is** the burst size that the IETF should specify (because it's dictated by buffering that is expected on the path).

I see opportunities here for experimentation and measurement opportunities, and new specs can be proposed. Or the Specs can be recast with more detail. I also know that practical experience is that many people have configured (probably tested, I hope) much larger IW's - and I'd hope they also make a reasonable effort to monitor for possible interactions with other Internet applications and services (as per section 12 of RFC6928).
Or present data that this is universally bad on all networks.

I think you touch on something that the IETF hasn't been great at talking about:

I could myself indeed likely produce data this shows this is "bad" in some networks that I value, and also show side-effects of collateral damage to other flows and adverse effects of line-rate bursts into equipment with small buffers. I don't know how useful that would be at all.

The IETF has typically defined a spec for endpoint. The way a modern endpoint is implemented can be quite different in a DC from a single device connected to the Internet. In a DC, I understand it is common  to distribute transport functions to other parts of the network. e.g.: DDOS protection could be done on a server; or by a box in front of it - similarly what matters to the network CC is the  bursts of traffic that leaves a DC within a flow to a local receiver, not what leaves a particular server within the DC network - I think in future we may do well to consider these differences?
Current production use of TCP shows there is no major problems with bursting higher. In fact for low latency same rack scenarios this will boost receive performance as well due to better opportunity for LRO etc.

It's also possible that QUIC LRO (or whatever) includes some notion of per-flow burst-mitigation. Is that impossible?

Gorry

From: Gorry Fairhurst <[email-masked]><mailto:[email-masked]>
Sent: Thursday, July 9, 2020 1:39 AM
To: Praveen Balasubramanian <[email-masked]><mailto:[email-masked]>; IETF QUIC WG <[email-masked]><mailto:[email-masked]>
Subject: [EXTERNAL] Re: WGLC review of draft-ietf-quic-recovery-29

Hi Praveen,

Gorry

---

From: Ian Swett <[email-masked]>
Date: Thu, 16 Jul 2020 15:22:24 -0400

Thanks for the comments, I filed #3918
<https://github.com/quicwg/base-drafts/issues/3918> and #3919
<https://github.com/quicwg/base-drafts/issues/3919>

For my background, are you aware other transport RFCs have normative
statements that only apply to the public internet?  Also, if you're that
well buffered and the bursts are that large, do you still start with IW10,
or do you start with a larger window?

Thanks, Ian

On Fri, Jul 10, 2020 at 1:11 PM Praveen Balasubramanian <pravb=
[email-masked]> wrote:

> Precluding bursts higher than 10 packets is simply not viable or practical
> at really high data dates and is a major regression from the way TCP works
> today. QUIC is a general purpose transport and not scoped to the Internet
> only. On a side thread around Linux TCP behavior Neal suggested some
> alternate text to scope this to Internet facing traffic but my contention
> still holds that there are many networks where higher bursts are not a
> problem and we should not restrict those use cases.
>
>
>
> *Sent:* Friday, July 10, 2020 1:21 AM
> *To:* Praveen Balasubramanian <[email-masked]>; IETF QUIC WG <
> [email-masked]>
> *Subject:* Re: [EXTERNAL] Re: WGLC review of draft-ietf-quic-recovery-29
>
>
>
> Thanks Praveen,
>
> Within data center mainly. To hit 40 Gbps line rate you effectively have
> to batch as much as you can to minimize the CPU processing costs. The burst
> size today is capped to 64k in most cases for TCP. Now that we have
> segmentation offload for UDP the same should apply. Even for Internet
> facing traffic, the bursts primarily impact the first hop within the
> datacenter because subsequently you get statistical multiplexing from ToR
> switch handling traffic forwarding for many flows and ports.
>
>
>
> Ok understood about there being no conflict. But I still contend that the
> MUST regarding burst size is way too restrictive. We should allow for
> experimentation here.
>
> I am not sure. The IW spec for TCP was based on evidence back around 2010,
> and the IETF agreed in RFC6928 that a burst size of 10 was acceptable as an
> EXP spec for TCP.
>
> I think we can safely refer to that - even perhaps take the leap-of-faith
> that EXP for TCP is equivalent to PS for QUIC, given subsequent
> experience.  I think the IW  **is** the burst size that the IETF should
> specify (because it's dictated by buffering that is expected on the path).
>
> with more detail. I also know that practical experience is that many people
> have configured (probably tested, I hope) much larger IW's - and I'd hope
> they also make a reasonable effort to monitor for possible interactions
> with other Internet applications and services (as per section 12 of
> RFC6928).
>
> I could myself indeed likely produce data this shows this is "bad" in some
> networks that I value, and also show side-effects of collateral damage to
> other flows and adverse effects of line-rate bursts into equipment with
> small buffers. I don't know how useful that would be at all.
>
> endpoint is implemented can be quite different in a DC from a single device
> connected to the Internet. In a DC, I understand it is common  to
> leaves a DC within a flow to a local receiver, not what leaves a particular
> server within the DC network - I think in future we may do well to consider
> these differences?
>
> *From:* Gorry Fairhurst <[email-masked]> <[email-masked]>
> *To:* Praveen Balasubramanian <[email-masked]> <[email-masked]>;
> IETF QUIC WG <[email-masked]> <[email-masked]>
> "Implementations MUST either use pacing or another method to limit such
> bursts to the initial congestion window; see Section 7.2."
>
> This seems to preclude use of segmentation offload of sizes greater than
> IW. In datacenters we routinely send bursts that are higher without causing
> loss.
>
> section is to pace. The requirement is that sender limits burst, using some
> method, to the size of IW.
>
> Gorry
>
>
>
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 16 Jul 2020 16:17:58 -0500

Hi, Ian,

On Thu, Jul 16, 2020 at 2:22 PM Ian Swett <ianswett=
[email-masked]> wrote:

> <https://github..com/quicwg/base-drafts/issues/3918> and #3919
> statements that only apply to the public internet?
>

Tunnels are transport, right? :-)

The earliest RFC I was involved with, that made distinctions about what you
can get away with on the public internet versus in a managed network, was
https://tools.ietf.org/html/rfc7510#section-5. This was negotiated after a
fairly vigorous back-and-forth between transport people and non-transport
people during IETF Last Call.

We put together a design team with a few people to try to work through this
for the myriad other protocols that people wanted to tunnel over UDP, once
and for all.

https://tools.ietf.org/html/rfc8084 was published as a BCP.

I may be forgetting other stuff, of course.

Best,

Spencer

> Also, if you're that well buffered and the bursts are that large, do you
> still start with IW10, or do you start with a larger window?
>
> Thanks, Ian
>
>>

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 16 Jul 2020 18:30:10 -0700

At a high level, the recovery doc is _not_ written with DC environments in
mind. We can make that clear in the document, if that helps. There are
other things as well that won't work well in a DC environment, and I
imagine a separate doc would be helpful here that specifies how QUIC ought
to be tuned for DCs.

On Thu, Jul 16, 2020 at 2:18 PM Spencer Dawkins at IETF <
[email-masked]> wrote:

> The earliest RFC I was involved with, that made distinctions about what
> you can get away with on the public internet versus in a managed network,
> was https://tools.ietf.org/html/rfc7510#section-5. This was negotiated
> after a fairly vigorous back-and-forth between transport people and
> non-transport people during IETF Last Call.
>
> We put together a design team with a few people to try to work through
> this for the myriad other protocols that people wanted to tunnel over UDP,
> once and for all.
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 17 Jul 2020 07:21:08 -0500

Hi, Jana,

> imagine a separate doc would be helpful here that specifies how QUIC ought

Two things.

My response to Ian was just about the history of what TSV has come to
expect for encapsulation protocol behaviors in the past half-dozen years
("if you are doing stuff that's not safe on an unmanaged network, this is
what you should do; if you're doing the same thing on a managed network,
this is what you can probably get away with, and please point to your
circuit breaker functionality in case what you're doing turns out to be a
Really Bad Idea" - paraphrasing over a lot of subtilty).

Those documents tended to have both cases in a single document, along with
the rest of the encapsulation protocol, in many cases because the
"unmanaged networks" considerations were being added to a single document
that already described "managed networks" considerations.

Pretty much every conversation I was involved in at QUIC charter time
assumed that we would want to swap recovery functionality in and out over
time without reopening draft-ietf-quic-transport, so having a separate
recovery document was the right thing to do.

Because QUIC already has a separate "safe for unmanaged networks" recovery
document, I think putting other recovery strategies in separate documents
is a fine plan.

Does that make sense?

Best,

Spencer

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 17 Jul 2020 16:41:51 +0000

AFAICT there is no RFC for TCP that limits burst sizes in general. I don’t think we need to special case Internet and DC networks. Changing the language from a MUST to a SHOULD is sufficient here. It leaves enough wiggle room for deployments and experimentation.

I don’t see too many other changes needed in recovery draft for DC environment particularly now that there is no min rto. Jana, what other changes do you have in mind?

From: Spencer Dawkins at IETF <[email-masked]>
Sent: Friday, July 17, 2020 5:21 AM
To: Jana Iyengar <[email-masked]>
Cc: Ian Swett <[email-masked]>; Praveen Balasubramanian <[email-masked]>; Gorry Fairhurst <[email-masked]>; IETF QUIC WG <[email-masked]>; Neal Cardwell <[email-masked]>; Yuchung Cheng <[email-masked]>
Subject: Re: [EXTERNAL] Re: WGLC review of draft-ietf-quic-recovery-29

Hi, Jana,

On Thu, Jul 16, 2020 at 8:30 PM Jana Iyengar <jri.ietf@gmail..com<mailto:[email-masked]>> wrote:
At a high level, the recovery doc is _not_ written with DC environments in mind. We can make that clear in the document, if that helps. There are other things as well that won't work well in a DC environment, and I imagine a separate doc would be helpful here that specifies how QUIC ought to be tuned for DCs.

Two things.

My response to Ian was just about the history of what TSV has come to expect for encapsulation protocol behaviors in the past half-dozen years ("if you are doing stuff that's not safe on an unmanaged network, this is what you should do; if you're doing the same thing on a managed network, this is what you can probably get away with, and please point to your circuit breaker functionality in case what you're doing turns out to be a Really Bad Idea" - paraphrasing over a lot of subtilty).

Those documents tended to have both cases in a single document, along with the rest of the encapsulation protocol, in many cases because the "unmanaged networks" considerations were being added to a single document that already described "managed networks" considerations.

Pretty much every conversation I was involved in at QUIC charter time assumed that we would want to swap recovery functionality in and out over time without reopening draft-ietf-quic-transport, so having a separate recovery document was the right thing to do.

Because QUIC already has a separate "safe for unmanaged networks" recovery document, I think putting other recovery strategies in separate documents is a fine plan.

Does that make sense?

Best,

Spencer

On Thu, Jul 16, 2020 at 2:18 PM Spencer Dawkins at IETF <[email-masked]<mailto:spencerdawkins.ietf@gmail..com>> wrote:
Hi, Ian,

On Thu, Jul 16, 2020 at 2:22 PM Ian Swett <ianswett=[email-masked]<mailto:[email-masked]>> wrote:
Thanks for the comments, I filed #3918<https://github..com/quicwg/base-drafts/issues/3918> and #3919<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fquicwg%2Fbase-drafts%2Fissues%2F3919&data=02%7C01%7Cpravb%40microsoft.com%7C3aec4cda750d49b8a7ef08d82a4bf551%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637305853016570346&sdata=akzKemBS1w2ASNuqfhCfXjV0PjkmtWJnD3NKm%2FWL%2BJg%3D&reserved=0>

For my background, are you aware other transport RFCs have normative statements that only apply to the public internet?

The earliest RFC I was involved with, that made distinctions about what you can get away with on the public internet versus in a managed network, was https://tools.ietf.org/html/rfc7510#section-5<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc7510%23section-5&data=02%7C01%7Cpravb%40microsoft.com%7C3aec4cda750d49b8a7ef08d82a4bf551%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637305853016580341&sdata=yS2XCONR3lZx10mfajWbppCDytE2DM1Y5Qc4r5uIyv0%3D&reserved=0>. This was negotiated after a fairly vigorous back-and-forth between transport people and non-transport people during IETF Last Call.

We put together a design team with a few people to try to work through this for the myriad other protocols that people wanted to tunnel over UDP, once and for all.

https://tools.ietf.org/html/rfc8084<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc8084&data=02%7C01%7Cpravb%40microsoft.com%7C3aec4cda750d49b8a7ef08d82a4bf551%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637305853016580341&sdata=GOTJobJC74eepsoltLYDssFu4%2FE8M1C97m173JHwdH0%3D&reserved=0> was published as a BCP.

Best,

Spencer

Also, if you're that well buffered and the bursts are that large, do you still start with IW10, or do you start with a larger window?

Thanks, Ian

---

From: Ian Swett <[email-masked]>
Date: Fri, 17 Jul 2020 16:20:53 -0400

Thanks for the follow up comments and the excellent example Spencer.

Praveen, I agree that QUIC shouldn't require as many DC specific changes as
TCP, given it explicitly communicates max_ack_delay and there's no min RTO.

It seems there at least 4 options
1) Do nothing
2) Do nothing, but add a note that if you're on a managed network, some
requirements might not hold.
3) Change the MUST to a SHOULD
4) Use Neal's text or some variation thereof: "Implementations MUST either
use pacing or another method to limit such bursts. It is RECOMMENDED that
traffic over the public Internet limit bursts to the initial congestion
window; see Section 7.2."

I support softening this text some, given Linux today will burst well more
than 10 packets at once if the bandwidth is high enough.  Realistically, to
hit 1Gbps on a 5G network with QUIC, one might have to have bursts larger
than 10 packets, so I'm not sure this is only a DC issue.

I don't like the idea of saying you can change anything in the recovery doc
if you're on a managed network.  Obviously one can, but I'd be concerned
what might result.  Changing the MUST to a SHOULD is easy, but doesn't
provide any guidance on when it's ok to violate.  I'm hesitant to make a
distinction between public and private networks, partially because I do
think there are or will be valid cases for larger bursts on the public
internet.

I'm leaning towards saying you MUST limit bursts to IW unless the pacing
system can't provide enough sending opportunities to send at the rate
specified by the congestion controller.  Thoughts?

On Fri, Jul 17, 2020 at 12:41 PM Praveen Balasubramanian <
[email-masked]> wrote:

> AFAICT there is no RFC for TCP that limits burst sizes in general. I don’t
> think we need to special case Internet and DC networks. Changing the
> language from a MUST to a SHOULD is sufficient here. It leaves enough
> wiggle room for deployments and experimentation.
>
>
>
> I don’t see too many other changes needed in recovery draft for DC
> environment particularly now that there is no min rto. Jana, what other
> changes do you have in mind?
>
>
>
> *From:* Spencer Dawkins at IETF <[email-masked]>
> *Sent:* Friday, July 17, 2020 5:21 AM
> *To:* Jana Iyengar <[email-masked]>
> *Cc:* Ian Swett <[email-masked]>; Praveen Balasubramanian <
> [email-masked]>; Gorry Fairhurst <[email-masked]>; IETF QUIC
> WG <[email-masked]>; Neal Cardwell <[email-masked]>; Yuchung Cheng <
> [email-masked]>
> On Thu, Jul 16, 2020 at 8:30 PM Jana Iyengar <jri.ietf@gmail..com
> <[email-masked]>> wrote:
>
> [email-masked] <spencerdawkins.ietf@gmail..com>> wrote:
>
> Hi, Ian,
>
>
>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fquicwg%2Fbase-drafts%2Fissues%2F3919&data=02%7C01%7Cpravb%40microsoft.com%7C3aec4cda750d49b8a7ef08d82a4bf551%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637305853016570346&sdata=akzKemBS1w2ASNuqfhCfXjV0PjkmtWJnD3NKm%2FWL%2BJg%3D&reserved=0>
>
>
>
>
> was https://tools.ietf.org/html/rfc7510#section-5
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc7510%23section-5&data=02%7C01%7Cpravb%40microsoft.com%7C3aec4cda750d49b8a7ef08d82a4bf551%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637305853016580341&sdata=yS2XCONR3lZx10mfajWbppCDytE2DM1Y5Qc4r5uIyv0%3D&reserved=0>.
> This was negotiated after a fairly vigorous back-and-forth between
> transport people and non-transport people during IETF Last Call.
>
>
>
> https://tools.ietf.org/html/rfc8084
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc8084&data=02%7C01%7Cpravb%40microsoft.com%7C3aec4cda750d49b8a7ef08d82a4bf551%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637305853016580341&sdata=GOTJobJC74eepsoltLYDssFu4%2FE8M1C97m173JHwdH0%3D&reserved=0>
> was published as a BCP.
>
>
>

---

From: Jeremy Harris <[email-masked]>
Date: Fri, 17 Jul 2020 23:06:56 +0100

On 08/07/2020 22:29, Praveen Balasubramanian wrote:
> This seems to preclude use of segmentation offload of sizes greater than IW.. In datacenters we routinely send bursts that are higher without causing loss. The MUST here seems unnecessary. It also conflicts with the RECOMMENDED in an earlier sentence.

There's a way of satisfying both desires: have the NIC handle the
pacing.
-- 
Cheers,
  Jeremy

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 17 Jul 2020 19:17:58 -0700

There's a protocol question and there's a question of endpoint behavior. In
terms of the protocol itself, yes, there's no real need to distinguish
between Internet and DC environments; we've tried to ensure that the
protocol can be used broadly. My point was that the constants in the spec
were based on what we believe to be true for the public Internet, and not
for DC environments.

That said, perhaps I was a bit too hasty. IW10 and InitialRTT values are
the others I was thinking about, but those are recommendations in the spec,
not requirements. And as Ian notes, there's no minimum timeout anymore.

Perhaps we can have a principle here: recommendations that are specific for
Internet use are just that, and we use SHOULDs for those. IW10 makes sense
based on this, and I would then also be fine with changing the MUST to a
SHOULD. Perhaps we can state this principle upfront.

Ian, I share your hesitation that we don't want to make a distinction
between private and public networks, but we already allow for
implementations to do that with a different IW and Initial RTT. Is it
different when talking about burst limits?

On Fri, Jul 17, 2020 at 3:07 PM Jeremy Harris <[email-masked]> wrote:

> IW.. In datacenters we routinely send bursts that are higher without
> causing loss. The MUST here seems unnecessary. It also conflicts with the

---

From: Praveen Balasubramanian <[email-masked]>
Date: Sun, 19 Jul 2020 20:07:32 +0000

>> There's a way of satisfying both desires: have the NIC handle the pacing.
Yes letting the NIC handle pacing will make improve CPU efficiency and improve accuracy due to fine grained hardware timers. But that’s in the future. Today’s NICs don’t pace large send offloads for TCP or UDP.

>> Perhaps we can have a principle here: recommendations that are specific for Internet use are just that, and we use SHOULDs for those. IW10 makes sense based on this, and I would then also be fine with changing the MUST to a SHOULD. Perhaps we can state this principle upfront.
I like the idea of stating that principle up front. SHOULD would be sufficient resolution for the burst size issue.

From: QUIC <[email-masked]> On Behalf Of Jana Iyengar
Sent: Friday, July 17, 2020 7:18 PM
To: QUIC WG <[email-masked]>
Subject: [EXTERNAL] Re: WGLC review of draft-ietf-quic-recovery-29

There's a protocol question and there's a question of endpoint behavior. In terms of the protocol itself, yes, there's no real need to distinguish between Internet and DC environments; we've tried to ensure that the protocol can be used broadly. My point was that the constants in the spec were based on what we believe to be true for the public Internet, and not for DC environments.

That said, perhaps I was a bit too hasty. IW10 and InitialRTT values are the others I was thinking about, but those are recommendations in the spec, not requirements. And as Ian notes, there's no minimum timeout anymore.

Perhaps we can have a principle here: recommendations that are specific for Internet use are just that, and we use SHOULDs for those. IW10 makes sense based on this, and I would then also be fine with changing the MUST to a SHOULD. Perhaps we can state this principle upfront.

Ian, I share your hesitation that we don't want to make a distinction between private and public networks, but we already allow for implementations to do that with a different IW and Initial RTT. Is it different when talking about burst limits?

On Fri, Jul 17, 2020 at 3:07 PM Jeremy Harris <[email-masked]<mailto:[email-masked]>> wrote:

---

From: Ian Swett <[email-masked]>
Date: Mon, 20 Jul 2020 12:48:37 -0400

I think it's ok to include burst limits in the set of things we expect may
be different in DCs, but if we do that, I'd prefer to use the RECOMMENDED
rather than SHOULD, since there are a number of SHOULDs which I
believe apply both to the public internet and to DCs.  Then we could add a
note at the top about how there are some values which are RECOMMENDED,
including X, Y and Z, and those recommendations are expected to be good
choices for most, but not all environments.

In terms of text, I'd suggest taking Neal's suggestion and dropping an
explicit mention of the public internet:

It is RECOMMENDED that implementations limit bursts to the initial
congestion window; see Section 7.2
<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-quic-recovery-29%23section-7.2&data=02%7C01%7Cpravb%40microsoft.com%7C9df3e5f4f0ca4e586f0908d824e0a3ad%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637299894526657906&sdata=ze%2F7SpKUbrdgwI6m%2BS6uEGSBgc2NapVtPn9vkOYFjVs%3D&reserved=0>.

On Sun, Jul 19, 2020 at 4:08 PM Praveen Balasubramanian <pravb=
[email-masked]> wrote:

> Yes letting the NIC handle pacing will make improve CPU efficiency and
> improve accuracy due to fine grained hardware timers. But that’s in the
> future. Today’s NICs don’t pace large send offloads for TCP or UDP.
>
>
>
> >> Perhaps we can have a principle here: recommendations that are specific
> for Internet use are just that, and we use SHOULDs for those. IW10 makes
> sense based on this, and I would then also be fine with changing the MUST
> to a SHOULD. Perhaps we can state this principle upfront.
>
> I like the idea of stating that principle up front. SHOULD would be
> sufficient resolution for the burst size issue.
>
>
>
> *From:* QUIC <[email-masked]> *On Behalf Of * Jana Iyengar
> *Sent:* Friday, July 17, 2020 7:18 PM
> *To:* QUIC WG <[email-masked]>
> There's a protocol question and there's a question of endpoint behavior.
> In terms of the protocol itself, yes, there's no real need to distinguish
> Perhaps we can have a principle here: recommendations that are specific
> for Internet use are just that, and we use SHOULDs for those. IW10 makes
> sense based on this, and I would then also be fine with changing the MUST
> to a SHOULD. Perhaps we can state this principle upfront.
>
>
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Mon, 20 Jul 2020 16:30:10 -0500

Hi, Ian,

On Mon, Jul 20, 2020 at 11:49 AM Ian Swett <ianswett=
[email-masked]> wrote:

> rather than SHOULD,

Aren't RECOMMENDED and SHOULD equivalent in
https://tools.ietf.org/html/rfc2119?

Or am I misunderstanding your point?

3 <https://tools.ietf.org/html/rfc2119#section-3>. SHOULD   This word,
or the adjective "RECOMMENDED", mean that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course.

Best,

Spencer

since there are a number of SHOULDs which I believe apply both to the
> public internet and to DCs.  Then we could add a note at the top about how
> there are some values which are RECOMMENDED, including X, Y and Z, and
> those recommendations are expected to be good choices for most, but not all
> environments.
>
> It is RECOMMENDED that implementations limit bursts to the initial congestion window; see Section 7.2 <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-quic-recovery-29%23section-7.2&data=02%7C01%7Cpravb%40microsoft.com%7C9df3e5f4f0ca4e586f0908d824e0a3ad%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637299894526657906&sdata=ze%2F7SpKUbrdgwI6m%2BS6uEGSBgc2NapVtPn9vkOYFjVs%3D&reserved=0>.
>
>
>> >> Perhaps we can have a principle here: recommendations that are
>> specific for Internet use are just that, and we use SHOULDs for those. IW10
>> makes sense based on this, and I would then also be fine with changing the
>> MUST to a SHOULD. Perhaps we can state this principle upfront.
>>
>> than IW.. In datacenters we routinely send bursts that are higher without

---

From: Ian Swett <[email-masked]>
Date: Mon, 20 Jul 2020 17:33:34 -0400

Most of the items which I felt were somewhat deployment-specific
constants(ie: Initial Window and Initial RTT) are 'RECOMMENDED' in the
current recovery draft, so I thought it'd be preferable to keep that
convention.

On Mon, Jul 20, 2020 at 5:30 PM Spencer Dawkins at IETF <
[email-masked]> wrote:

>> I think it's ok to include burst limits in the set of things we expect
>> may be different in DCs, but if we do that, I'd prefer to use the
>> RECOMMENDED rather than SHOULD,
>>
>
> 3 <https://tools.ietf.org/html/rfc2119#section-3>. SHOULD   This word, or the adjective "RECOMMENDED", mean that there

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Mon, 20 Jul 2020 17:11:01 -0500

Hi, Ian,

> current recovery draft, so I thought it'd be preferable to keep that

Thanks for helping me understand - this is a stylistic change.

I've had a couple of conversations outside of QUIC recently with people who
were interpreting the two terms somewhat differently, so thought it a good
time to ask.

Best,

Spencer

>>>> There's a protocol question and there's a question of endpoint
>>>> behavior. In terms of the protocol itself, yes, there's no real need to
>>>> distinguish between Internet and DC environments; we've tried to ensure
>>>> that the protocol can be used broadly. My point was that the constants in
>>>> the spec were based on what we believe to be true for the public Internet,
>>>> and not for DC environments.
>>>>
>>>>
>>>>
>>>> That said, perhaps I was a bit too hasty. IW10 and InitialRTT values
>>>> are the others I was thinking about, but those are recommendations in the
>>>> spec, not requirements. And as Ian notes, there's no minimum timeout
>>>> anymore.
>>>>
>>>>
>>>>

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 23 Jul 2020 10:07:19 +0100

I'd like to argue a little more for something like the current text we 
arrived at:

"Implementations MUST either
    use pacing or another method to limit such bursts to the initial
    congestion window;"

I've been watching, and have comments see below:

On 20/07/2020 23:11, Spencer Dawkins at IETF wrote:
> Hi, Ian,
>
> On Mon, Jul 20, 2020, 16:33 Ian Swett <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     constants(ie: Initial Window and Initial RTT) are 'RECOMMENDED' in
>     the current recovery draft, so I thought it'd be preferable to
>     keep that convention.
>
>
> I've had a couple of conversations outside of QUIC recently with 
> people who were interpreting the two terms somewhat differently, so 
> thought it a good time to ask.
>
> Best,
>
> Spencer
>
>
>     On Mon, Jul 20, 2020 at 5:30 PM Spencer Dawkins at IETF
>     <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>         Hi, Ian,
>
>         On Mon, Jul 20, 2020 at 11:49 AM Ian Swett
>         <ianswett=[email-masked]
>         <mailto:[email-masked]>> wrote:
>
>             I think it's ok to include burst limits in the set of
>             things we expect may be different in DCs, but if we do
>             that, I'd prefer to use the RECOMMENDED rather than SHOULD,
>
>
>             3 <https://tools.ietf.org/html/rfc2119#section-3>. SHOULD
>
>         This word, or the adjective "RECOMMENDED", mean that there may
>         exist valid reasons in particular circumstances to ignore a
>         particular item, but the full implications must be understood
>         and carefully weighed before choosing a different course.
>
>         Best,
>
>         Spencer
>
>             since there are a number of SHOULDs which I believe apply
>             both to the public internet and to DCs.  Then we could add
>             a note at the top about how there are some values which
>             are RECOMMENDED, including X, Y and Z, and those
>             recommendations are expected to be good choices for most,
>             but not all environments.
>
>             In terms of text, I'd suggest taking Neal's suggestion and
>             dropping an explicit mention of the public internet:
>
>             It is RECOMMENDED that implementations limit bursts to the
>             initial congestion window; seeSection 7.2  <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-quic-recovery-29%23section-7.2&data=02%7C01%7Cpravb%40microsoft.com%7C9df3e5f4f0ca4e586f0908d824e0a3ad%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637299894526657906&sdata=ze%2F7SpKUbrdgwI6m%2BS6uEGSBgc2NapVtPn9vkOYFjVs%3D&reserved=0>.
>
>
>             On Sun, Jul 19, 2020 at 4:08 PM Praveen Balasubramanian
>             <pravb=[email-masked]
>             <mailto:[email-masked]>> wrote:
>
>                 Yes letting the NIC handle pacing will make improve
>                 CPU efficiency and improve accuracy due to fine
>                 grained hardware timers. But that’s in the future.
>                 Today’s NICs don’t pace large send offloads for TCP or
>                 UDP.
>
>                 >> Perhaps we can have a principle here:
>                 recommendations that are specific for Internet use are
>                 just that, and we use SHOULDs for those. IW10 makes
>                 sense based on this, and I would then also be fine
>                 with changing the MUST to a SHOULD. Perhaps we can
>                 state this principle upfront.
>
>                 I like the idea of stating that principle up front.
>                 SHOULD would be sufficient resolution for the burst
>                 size issue.
>
>                 *From:* QUIC <[email-masked]
>                 <mailto:[email-masked]>> *On Behalf Of * Jana
>                 Iyengar
>                 *To:* QUIC WG <[email-masked] <mailto:[email-masked]>>
>                 There's a protocol question and there's a question of
>                 endpoint behavior. In terms of the protocol itself,
>                 yes, there's no real need to distinguish between
>                 Internet and DC environments; we've tried to ensure
>                 that the protocol can be used broadly. My point was
>                 that the constants in the spec were based on what we
>                 believe to be true for the public Internet, and not
>                 for DC environments.
>
>                 That said, perhaps I was a bit too hasty. IW10 and
>                 InitialRTT values are the others I was thinking about,
>                 but those are recommendations in the spec, not
>                 requirements. And as Ian notes, there's no minimum
>                 timeout anymore.
>
>                 Perhaps we can have a principle here: recommendations
>                 that are specific for Internet use are just that, and
>                 we use SHOULDs for those.
>
I don't agree, see below.
>
>                 IW10 makes sense based on this, and I would then also
>                 be fine with changing the MUST to a SHOULD. Perhaps we
>                 can state this principle upfront.
>
>                 Ian, I share your hesitation that we don't want to
>                 make a distinction between private and public
>                 networks, but we already allow for implementations to
>                 do that with a different IW and Initial RTT. Is it
>                 On Fri, Jul 17, 2020 at 3:07 PM Jeremy Harris
>                 <[email-masked] <mailto:[email-masked]>> wrote:
>
>                     > "Implementations MUST either use pacing or
>                     another method to limit such bursts to the initial
>                     congestion window; see Section 7.2."
>                     > This seems to preclude use of segmentation
>                     offload of sizes greater than IW.. In datacenters
>                     we routinely send bursts that are higher without
>                     causing loss. The MUST here seems unnecessary. It
>                     also conflicts with the RECOMMENDED in an earlier
>                     sentence.
>

I am unsure that trying to differentiate DC from ISP-Edge from Enterprise Server
or anything else will help in the spec. Sure, guidance may differ - but I'd like
to think (???) the IETF Spec describes the "Internet case, which
is expected to be good choices for most, but not all environments... and if a system distributes
functions within your own network segment then that's your decision - and yes,
and doing this in NICs would be great ... although NICs also can generate bursts
when transports pace at a higher level...

The pacing/burst-limiting could happen in various ways by various actors, but
that's not the way transport Specs are written, I thought that's why we ended-up with
"other methods".

For me at least, it was a conscious decision to not to explicitly make this "10 packets",
because an endpoint sender that made an informed decision to use a larger IW
has at the same time decided to configure in a way that expects the network path can safely
support this burst size. I expect this matters most when the
connection sending rate is low (see below). But we missed something.

I think the suggested addition from Ian, could really help a lot:
"unless the pacing system can't provide enough sending opportunities to send at the rate specified by the congestion controller."

- This would make the language less clear, but I think in a very good way, because I agree that as the connection **sending** rate increases (and hence also the CC understanding of
the path characteristics), the effect of bursts might be expected to be less significant. At higher rates queues in devices clear more quickly, and buffering 1mS (or whatever) of data, buffers more.

Can we discuss that proposal more?

Gorry

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 24 Jul 2020 18:42:30 -0700

Gorry,

That's not the only case where you don't want to pace. If I know (and I
have a production network where this is true) that the network can absorb a
larger burst. What is the argument to make this a MUST when even the value
of the Initial Window (of 10) is a SHOULD?

- jana

On Thu, Jul 23, 2020 at 2:07 AM Gorry Fairhurst <[email-masked]>
wrote:

> I've had a couple of conversations outside of QUIC recently with people
> who were interpreting the two terms somewhat differently, so thought it a
> good time to ask.
>
> Best,
>
> Spencer
>
>>
>>>>> specific for Internet use are just that, and we use SHOULDs for those.
>>>>>
> IW10 makes sense based on this, and I would then also be fine with
>>>>> changing the MUST to a SHOULD. Perhaps we can state this principle upfront.
>>>>>
>>>>>
>>>>>

---

From: Ian Swett <[email-masked]>
Date: Mon, 27 Jul 2020 10:06:08 -0400

For me, two things make "MUST, unless" or something along those lines
preferable for me.
1) Infinite limits make me concerned.
2) QUIC doesn't specify Slow Start After Idle and to me that seems safe
only because it requires pacing and/or limiting bursts.

But I agree the difference between "MUST unless" and SHOULD is small, so I
think this is a case for adding some extra text about why it's not a strict
MUST.

PS: I'll get back to the PR today, sorry for the delay.

On Fri, Jul 24, 2020 at 9:42 PM Jana Iyengar <[email-masked]> wrote:

>>>>>> Yes letting the NIC handle pacing will make improve CPU efficiency
>>>>>> and improve accuracy due to fine grained hardware timers. But that’s in the

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 27 Jul 2020 15:28:51 +0100

On 27/07/2020 15:06, Ian Swett wrote:
> 2) QUIC doesn't specify Slow Start After Idle and to me that seems 
> safe only because it requires pacing and/or limiting bursts.
>
> But I agree the difference between "MUST unless" and SHOULD is small, 
> so I think this is a case for adding some extra text about why it's 
> not a strict MUST.
>
Great. I'll look forward to reading that new text.

> On Fri, Jul 24, 2020 at 9:42 PM Jana Iyengar <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Gorry,
>
>     That's not the only case where you don't want to pace. If I know
>     (and I have a production network where this is true) that the
>     network can absorb a larger burst. What is the argument to make
>     this a MUST when even the value of the Initial Window (of 10) is a
>     SHOULD?
>
>     - jana
>

In answer to Jana, if you know the path has specific capabilities - 
which is pretty much what I'd expect in an DC or any other "constrained 
environment", the normal Internet guidance isn't necessarily appropiate. 
Generally that is not what IETF specs target.

Gorry

>     On Thu, Jul 23, 2020 at 2:07 AM Gorry Fairhurst
>     <[email-masked] <mailto:[email-masked]>> wrote:
>
>         I'd like to argue a little more for something like the current
>         text we arrived at:
>
>>             Most of the items which I felt were somewhat
>>             deployment-specific constants(ie: Initial Window and
>>             Initial RTT) are 'RECOMMENDED' in the current recovery
>>             draft, so I thought it'd be preferable to keep that
>>             convention.
>>
>>
>>         I've had a couple of conversations outside of QUIC recently
>>         with people who were interpreting the two terms somewhat
>>         differently, so thought it a good time to ask.
>>
>>         Best,
>>
>>         Spencer
>>
>>
>>                     I think it's ok to include burst limits in the
>>                     set of things we expect may be different in DCs,
>>                     but if we do that, I'd prefer to use the
>>                     3
>>                     <https://tools.ietf.org/html/rfc2119#section-3>.
>>                     SHOULD
>>
>>                 This word, or the adjective "RECOMMENDED", mean that
>>                 there may exist valid reasons in particular
>>                 circumstances to ignore a particular item, but the
>>                 full implications must be understood and carefully
>>                 weighed before choosing a different course.
>>
>>                 Best,
>>
>>                 Spencer
>>
>>                     since there are a number of SHOULDs which I
>>                     believe apply both to the public internet and to
>>                     DCs.  Then we could add a note at the top about
>>                     how there are some values which are RECOMMENDED,
>>                     including X, Y and Z, and those recommendations
>>                     are expected to be good choices for most, but not
>>                     all environments.
>>
>>                     In terms of text, I'd suggest taking Neal's
>>                     suggestion and dropping an explicit mention of
>>                     the public internet:
>>
>>                     It is RECOMMENDED that implementations limit
>>                     bursts to the initial congestion window; seeSection 7.2  <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-quic-recovery-29%23section-7.2&data=02%7C01%7Cpravb%40microsoft.com%7C9df3e5f4f0ca4e586f0908d824e0a3ad%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637299894526657906&sdata=ze%2F7SpKUbrdgwI6m%2BS6uEGSBgc2NapVtPn9vkOYFjVs%3D&reserved=0>.
>>
>>
>>                         Yes letting the NIC handle pacing will make
>>                         improve CPU efficiency and improve accuracy
>>                         due to fine grained hardware timers. But
>>                         that’s in the future. Today’s NICs don’t pace
>>                         large send offloads for TCP or UDP.
>>
>>                         recommendations that are specific for
>>                         Internet use are just that, and we use
>>                         SHOULDs for those. IW10 makes sense based on
>>                         this, and I would then also be fine with
>>                         I like the idea of stating that principle up
>>                         front. SHOULD would be sufficient resolution
>>                         for the burst size issue.
>>
>>                         <mailto:[email-masked]>> *On Behalf Of
>>                         * Jana Iyengar
>>                         There's a protocol question and there's a
>>                         question of endpoint behavior. In terms of
>>                         the protocol itself, yes, there's no real
>>                         need to distinguish between Internet and DC
>>                         environments; we've tried to ensure that the
>>                         protocol can be used broadly. My point was
>>                         that the constants in the spec were based on
>>                         what we believe to be true for the public
>>                         Internet, and not for DC environments.
>>
>>                         That said, perhaps I was a bit too hasty.
>>                         IW10 and InitialRTT values are the others I
>>                         was thinking about, but those
>>                         are recommendations in the spec, not
>>                         requirements. And as Ian notes, there's no
>>                         minimum timeout anymore.
>>
>>                         recommendations that are specific for
>>                         Internet use are just that, and we use
>>                         SHOULDs for those.
>>
>>                         IW10 makes sense based on this, and I would
>>                         then also be fine with changing the MUST to a
>>                         Ian, I share your hesitation that we don't
>>                         want to make a distinction between private
>>                         and public networks, but we already allow for
>>                         implementations to do that with a
>>                         different IW and Initial RTT. Is it different
>>                         when talking about burst limits?
>>
>>                             > "Implementations MUST either use pacing
>>                             or another method to limit such bursts to
>>                             the initial congestion window; see
>>                             Section 7.2."

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 27 Jul 2020 10:27:44 -0700

Ian,

I've changed the text in PR to reflect your suggestion in the discussion
here: https://github.com/quicwg/base-drafts/pull/3936.

- jana

On Mon, Jul 27, 2020 at 7:29 AM Gorry Fairhurst <[email-masked]>
wrote:

> In answer to Jana, if you know the path has specific capabilities - which
> is pretty much what I'd expect in an DC or any other "constrained
>>>>>> I think it's ok to include burst limits in the set of things we
>>>>>> expect may be different in DCs, but if we do that, I'd prefer to use the
>>>>>> In terms of text, I'd suggest taking Neal's suggestion and dropping
>>>>>> an explicit mention of the public internet:
>>>>>>
>>>>>>> Ian, I share your hesitation that we don't want to make a
>>>>>>> distinction between private and public networks, but we already allow for
>>>>>>> greater than IW.. In datacenters we routinely send bursts that are higher
>>>>>>> without causing loss. The MUST here seems unnecessary. It also conflicts
>>>>>>> with the RECOMMENDED in an earlier sentence.
>>>>>>>