Subject: Re: [Potential Spoof] Re: Getting to consensus on packet number encryption

From: Roberto Peon <[email-masked]>
Date: Tue, 1 May 2018 23:22:26 +0000

I’d suggest we start working on it now.
This is only partly a joke ☺
-=R

From: QUIC <[email-masked]> on behalf of Roberto Peon <[email-masked]>
Date: Tuesday, May 1, 2018 at 4:21 PM
To: Praveen Balasubramanian <pravb=[email-masked]>, Ted Hardie <[email-masked]>
Cc: "Salz, Rich" <[email-masked]>, Benjamin Kaduk <[email-masked]>, IETF QUIC WG <[email-masked]>
Subject: [Potential Spoof] Re: Getting to consensus on packet number encryption

Yup. I know.
Prism was eye-opening partially because of the fact that the packets were not going across the internet, but rather within a secure network.

I believe that ossification alone is a good enough reason to do PNE.
The cherry for me is that it, along with other actions/mitigations, makes linkability more difficult for malicious actors.
CRIME and related was eye-opening because it exposed the power of the observable side-channel.

-=R

From: QUIC <[email-masked]> on behalf of Praveen Balasubramanian <pravb=[email-masked]>
Date: Tuesday, May 1, 2018 at 9:35 AM
To: Ted Hardie <[email-masked]>, Roberto Peon <[email-masked]>
Cc: "Salz, Rich" <[email-masked]>, Benjamin Kaduk <[email-masked]>, IETF QUIC WG <[email-masked]>
Subject: RE: Getting to consensus on packet number encryption

This is not about putting plaintext data on the network, this is about not adding PNE which is intended for preventing ossification. So I don’t understand what the trust issue being referred to is in case of the rare misconfig.

Within a DC several things are different – the deployment may need different congestion control like DCTCP, different settings for various transport parameters (like minrto and max ACK delay), larger flow control window etc. This is true for TCP today. The out of box default uses the handshake RTT to make a determination of what’s intra-DC but there is rich configuration to override the default and set it based on 4-tuple filters.

And there workloads like cloud storage and live migration that are by virtue of deployment intra-DC. These cannot be load shifted as they use private IP address space.

From: Ted Hardie [mailto:[email-masked]]
Sent: Tuesday, May 1, 2018 8:55 AM
To: Roberto Peon <[email-masked]>
Cc: Salz, Rich <[email-masked]>; Benjamin Kaduk <[email-masked]>; Praveen Balasubramanian <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

On Tue, May 1, 2018 at 8:13 AM, Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:
Also, Prism.

-=R

That's a little more succinct than I would be, but yes.  How does an application know that the flow it is initiating will traverse a topology that is deemed to be within a controlled environment?  You can say "configuration" but I fear it will turn up on a post-it  <https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fi.kinja-2Dimg.com-252Fgawker-2Dmedia-252Fimage-252Fupload-252Fs-2D-2DEdu6LPu9-2D-2D-252Fc-5Fscale-252Cf-5Fauto-252Cfl-5Fprogressive-252Cq-5F80-252Cw-5F800-252F194thbtyencs0jpg.jpg-26data-3D02-257C01-257Cpravb-2540microsoft.com-257C4da44e8ac1df4708ad8808d5af7bfca8-257C72f988bf86f141af91ab2d7cd011db47-257C1-257C1-257C636607869411258976-26sdata-3DProlSHkbF2SgMH3qdQN-252BZFokXvsEBf-252F3o9MaUnil0g8-253D-26reserved-3D0&d=DwMGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=TuVfzypo-CZLo522lgMRrLpKxcFaHvW9kdNSI57lXT0&s=By2d_to7XzLEgAL3Or9cZc2Aykye-PZEenFQDviAUlw&e=> if you do.  Loads shift, resources move, and suddenly you find that host which used to be in the same rack, being down, has load-shifted to the next instance,  in a datacenter 500 miles away across a very different network.

Many modern deployments are also in someone else's data center, so saying "Data center networks" doesn't say much about why you're trusting the network inside.
Ted

Sent via the Samsung Galaxy S7, an AT&T 4G LTE smartphone

-------- Original message --------
From: "Salz, Rich" <rsalz=[email-masked]<mailto:[email-masked]>>
Date: 5/1/18 5:25 AM (GMT-08:00)
To: Benjamin Kaduk <bkaduk=[email-masked]<mailto:[email-masked]>>, Praveen Balasubramanian <pravb=[email-masked]<mailto:[email-masked]>>
Cc: IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Getting to consensus on packet number encryption

    > I disagree that we need any more data for not doing PNE in the datacenter. Why would we add an extra encrypt-decrypt step for no obvious benefit?

I am concerned that people will mis-interpret the meaning of a datacenter, and think that a bunch of servers, or even a rack, in an open colo space is a "datacenter."  Computers keep getting faster.

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 1 May 2018 17:37:03 -0700

I'll try to summarize my position on #1079. I see two points:
1. PNE will help against ossification and also, albeit to a limited extent,
linkability.
2. PNE has a CPU cost. While we can speculate, we still don't *know* what
this cost will be in the context of deployment, and whether this cost will
be the show-stopper. PNE's CPU cost may or may not be ameliorated by NIC
implementation. We should allow PNE negotiability when we have established
that this cost is worth saving.

My sense is that we agree on (1). This is what #1079 is doing, so we should
separate the two questions and get (1) done. Whether we do (2) or not is
being discussed on this thread, and I think we can continue, but #1079
should be independently submitted.

FWIW, I believe that we want to iterate on versions, and that we will want
to get a v2 out as soon as we have things to get deployed that left pending
in v1.

Thoughts on intra-DC QUIC, related to point (2) follow:

I don't think we'll reach a meaningful end to this discussion by getting
into security for intra-DC protocols. I understand that there may be
situations where one may have fair certainty about the deployment context
and given that the security tradeoff isn't the same as disabling TLS, the
bar will be substantially lower for disabling PNE.

On QUIC in intra-DC environments:
1. As Igor noted, the value of QUIC in the intra-DC environments is
uncertain, and I'm not yet sure that compromising on PNE for an unknown
cost in an uncertain deployment context is necessarily useful.
2. Separately, I remain unconvinced, based on my experience with trying to
get QUIC used for DC transport, that this cost is going to be an important
factor in QUIC adoption within DCs. I agree that CPU cost is a factor, but
there are much bigger things here that will need attention (segmentation
offload, smaller record sizes, userspace costs) that will dominate well
before this AES operation becomes a factor. This all remains speculation
though.

Praveen: Not making PNE optional right now does not mean we won't do it. It
just means that it's an open question. I'm not yet convinced that it's a
debilitating cost or one of the proverbial "million cuts" that will keep it
from getting deployed intra-DC. You are right that you'll want to
standardize PNE negotiation if you use multiple platforms, but surely you
will be just doing this between Windows systems for a while if you start
using QUIC inside the DC. That would be a great time to use versioning to
run your own version within the DC, and compare the cost to IETF v1, and
see if it is worth removing PNE. I hope you find other costs that are worth
eliminating or changes that are worth making to the protocol too, and that
they all become part of standard QUIC as a consequence. But in the
meanwhile, I don't see why making it optional is necessary right now to add
PNE at all.

On Tue, May 1, 2018 at 4:22 PM, Roberto Peon <[email-masked]> wrote:

> *From: *QUIC <[email-masked]> on behalf of Roberto Peon <
> [email-masked]>
> *Date: *Tuesday, May 1, 2018 at 4:21 PM
> *To: *Praveen Balasubramanian <pravb=[email-masked]>, Ted
> Hardie <[email-masked]>
> *Cc: *"Salz, Rich" <[email-masked]>, Benjamin Kaduk <[email-masked]>,
> IETF QUIC WG <[email-masked]>
> *Subject: *[Potential Spoof] Re: Getting to consensus on packet number
> encryption
>
>
>
> Yup. I know.
>
> Prism was eye-opening partially because of the fact that the packets were
> not going across the internet, but rather within a secure network.
>
>
>
> The cherry for me is that it, along with other actions/mitigations, makes
> linkability more difficult for malicious actors.
> *From: *QUIC <[email-masked]> on behalf of Praveen Balasubramanian
> <pravb=[email-masked]>
> *Date: *Tuesday, May 1, 2018 at 9:35 AM
> *To: *Ted Hardie <[email-masked]>, Roberto Peon <[email-masked]>
> *Cc: *"Salz, Rich" <[email-masked]>, Benjamin Kaduk <[email-masked]>,
> IETF QUIC WG <[email-masked]>
> *Subject: *RE: Getting to consensus on packet number encryption
>
>
>
> This is not about putting plaintext data on the network, this is about not
> adding PNE which is intended for preventing ossification. So I don’t
> understand what the trust issue being referred to is in case of the rare
> misconfig.
>
>
>
> Within a DC several things are different – the deployment may need
> different congestion control like DCTCP, different settings for various
> transport parameters (like minrto and max ACK delay), larger flow control
> window etc. This is true for TCP today. The out of box default uses the
> handshake RTT to make a determination of what’s intra-DC but there is rich
> configuration to override the default and set it based on 4-tuple filters.
>
>
>
> And there workloads like cloud storage and live migration that are by
> virtue of deployment intra-DC. These cannot be load shifted as they use
> private IP address space.
>
>
>
> *From:* Ted Hardie [mailto:[email-masked]]
> *Sent:* Tuesday, May 1, 2018 8:55 AM
> *To:* Roberto Peon <[email-masked]>
> *Cc:* Salz, Rich <[email-masked]>; Benjamin Kaduk <[email-masked]>;
> Praveen Balasubramanian <[email-masked]>; IETF QUIC WG <[email-masked]
> >
> *Subject:* Re: Getting to consensus on packet number encryption
>
>
>
> On Tue, May 1, 2018 at 8:13 AM, Roberto Peon <[email-masked]> wrote:
>
> Also, Prism.
>
>
>
> -=R
>
>
>
>
>
>
>
> That's a little more succinct than I would be, but yes.  How does an
> application know that the flow it is initiating will traverse a topology
> that is deemed to be within a controlled environment?  You can say
> "configuration" but I fear it will turn up on a post-it
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fi.kinja-2Dimg.com-252Fgawker-2Dmedia-252Fimage-252Fupload-252Fs-2D-2DEdu6LPu9-2D-2D-252Fc-5Fscale-252Cf-5Fauto-252Cfl-5Fprogressive-252Cq-5F80-252Cw-5F800-252F194thbtyencs0jpg.jpg-26data-3D02-257C01-257Cpravb-2540microsoft.com-257C4da44e8ac1df4708ad8808d5af7bfca8-257C72f988bf86f141af91ab2d7cd011db47-257C1-257C1-257C636607869411258976-26sdata-3DProlSHkbF2SgMH3qdQN-252BZFokXvsEBf-252F3o9MaUnil0g8-253D-26reserved-3D0&d=DwMGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=TuVfzypo-CZLo522lgMRrLpKxcFaHvW9kdNSI57lXT0&s=By2d_to7XzLEgAL3Or9cZc2Aykye-PZEenFQDviAUlw&e=>if
> you do.  Loads shift, resources move, and suddenly you find that host which
> used to be in the same rack, being down, has load-shifted to the next
> instance,  in a datacenter 500 miles away across a very different network.
>
> Many modern deployments are also in someone else's data center, so saying
> "Data center networks" doesn't say much about why you're trusting the
> network inside.
>
> Ted
>
>
>
> From: "Salz, Rich" <rsalz=[email-masked]>
>
> To: Benjamin Kaduk <bkaduk=[email-masked]>, Praveen
> Balasubramanian <pravb=[email-masked]>
>
> Cc: IETF QUIC WG <[email-masked]>
>
>     > I disagree that we need any more data for not doing PNE in the
> datacenter. Why would we add an extra encrypt-decrypt step for no obvious
> benefit?
>
> I am concerned that people will mis-interpret the meaning of a datacenter,
> and think that a bunch of servers, or even a rack, in an open colo space is
> a "datacenter."  Computers keep getting faster.
>
>
>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 2 May 2018 02:02:37 +0000

We can get (1) done but I disagree that we can decouple (2). It doesn’t make sense to require performance overhead all the time with no benefit. For the Internet, PNE can prevent ossification so it is seen as a reasonable trade-off. For the datacenter, there is no known benefit, only cost.

We know that there is a CPU cost, this is not speculation. Doesn’t matter if the cost is 1% or 10%. Hardware implementers have also raised concerns. Every CPU cycle we spend on things that add no value to the user should be questioned. Every extra instruction we execute in the hot data path leaves less CPU for workloads, burns more power and makes for a worse comparison with HTTPS over TCP. The same argument as privacy holes holds here – just because there are some existing perf bottlenecks doesn’t mean we add more, particularly ones we cannot mitigate. I said it before that there are lots of ways we can improve UDP perf but I don’t know of any technique for reducing crypto CPU cost other than hardware offload.

The cost is certainly worth saving with something as simple as a negotiation mechanism. Am I missing something about a transport parameter being complicated to implement? Implementations are free to support only PNE and not negotiate anything else. There is actually an even simpler mechanism if a transport parameter is deemed as costly: if the peer sends PN = 0 in the CI, then its requesting no PNE and the receiver can choose to proceed or drop and force TCP fall back. This should take only an extra if check on the receiver CI processing logic to implement. I am assuming here that PNE on input of 0 will not produce 0 so it’s a safe sentinel value.

From: Jana Iyengar [mailto:[email-masked]]
Sent: Tuesday, May 1, 2018 5:37 PM
To: Roberto Peon <[email-masked]>
Cc: Praveen Balasubramanian <[email-masked]>; Ted Hardie <[email-masked]>; Salz, Rich <[email-masked]>; Benjamin Kaduk <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: [Potential Spoof] Re: Getting to consensus on packet number encryption

I'll try to summarize my position on #1079. I see two points:
1. PNE will help against ossification and also, albeit to a limited extent, linkability.
2. PNE has a CPU cost. While we can speculate, we still don't *know* what this cost will be in the context of deployment, and whether this cost will be the show-stopper. PNE's CPU cost may or may not be ameliorated by NIC implementation. We should allow PNE negotiability when we have established that this cost is worth saving.

My sense is that we agree on (1). This is what #1079 is doing, so we should separate the two questions and get (1) done. Whether we do (2) or not is being discussed on this thread, and I think we can continue, but #1079 should be independently submitted.

FWIW, I believe that we want to iterate on versions, and that we will want to get a v2 out as soon as we have things to get deployed that left pending in v1.

I don't think we'll reach a meaningful end to this discussion by getting into security for intra-DC protocols. I understand that there may be situations where one may have fair certainty about the deployment context and given that the security tradeoff isn't the same as disabling TLS, the bar will be substantially lower for disabling PNE.

On QUIC in intra-DC environments:
1. As Igor noted, the value of QUIC in the intra-DC environments is uncertain, and I'm not yet sure that compromising on PNE for an unknown cost in an uncertain deployment context is necessarily useful.
2. Separately, I remain unconvinced, based on my experience with trying to get QUIC used for DC transport, that this cost is going to be an important factor in QUIC adoption within DCs. I agree that CPU cost is a factor, but there are much bigger things here that will need attention (segmentation offload, smaller record sizes, userspace costs) that will dominate well before this AES operation becomes a factor. This all remains speculation though.

Praveen: Not making PNE optional right now does not mean we won't do it. It just means that it's an open question. I'm not yet convinced that it's a debilitating cost or one of the proverbial "million cuts" that will keep it from getting deployed intra-DC. You are right that you'll want to standardize PNE negotiation if you use multiple platforms, but surely you will be just doing this between Windows systems for a while if you start using QUIC inside the DC. That would be a great time to use versioning to run your own version within the DC, and compare the cost to IETF v1, and see if it is worth removing PNE. I hope you find other costs that are worth eliminating or changes that are worth making to the protocol too, and that they all become part of standard QUIC as a consequence. But in the meanwhile, I don't see why making it optional is necessary right now to add PNE at all.

On Tue, May 1, 2018 at 4:22 PM, Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:
I’d suggest we start working on it now.
This is only partly a joke ☺
-=R

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Roberto Peon <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, May 1, 2018 at 4:21 PM
To: Praveen Balasubramanian <pravb=[email-masked]<mailto:[email-masked]>>, Ted Hardie <[email-masked]<mailto:[email-masked]>>
Cc: "Salz, Rich" <[email-masked]<mailto:[email-masked]>>, Benjamin Kaduk <[email-masked]<mailto:[email-masked]>>, IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: [Potential Spoof] Re: Getting to consensus on packet number encryption

Yup. I know.

-=R

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Praveen Balasubramanian <pravb=[email-masked]<mailto:[email-masked]>>
Date: Tuesday, May 1, 2018 at 9:35 AM
To: Ted Hardie <[email-masked]<mailto:[email-masked]>>, Roberto Peon <[email-masked]<mailto:[email-masked]>>
Cc: "Salz, Rich" <[email-masked]<mailto:[email-masked]>>, Benjamin Kaduk <[email-masked]<mailto:[email-masked]>>, IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: RE: Getting to consensus on packet number encryption

From: Ted Hardie [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Tuesday, May 1, 2018 8:55 AM
To: Roberto Peon <[email-masked]<mailto:[email-masked]>>
Cc: Salz, Rich <[email-masked]<mailto:[email-masked]>>; Benjamin Kaduk <[email-masked]<mailto:[email-masked]>>; Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Getting to consensus on packet number encryption

-=R

That's a little more succinct than I would be, but yes.  How does an application know that the flow it is initiating will traverse a topology that is deemed to be within a controlled environment?  You can say "configuration" but I fear it will turn up on a post-it  <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fi.kinja-2Dimg.com-252Fgawker-2Dmedia-252Fimage-252Fupload-252Fs-2D-2DEdu6LPu9-2D-2D-252Fc-5Fscale-252Cf-5Fauto-252Cfl-5Fprogressive-252Cq-5F80-252Cw-5F800-252F194thbtyencs0jpg.jpg-26data-3D02-257C01-257Cpravb-2540microsoft.com-257C4da44e8ac1df4708ad8808d5af7bfca8-257C72f988bf86f141af91ab2d7cd011db47-257C1-257C1-257C636607869411258976-26sdata-3DProlSHkbF2SgMH3qdQN-252BZFokXvsEBf-252F3o9MaUnil0g8-253D-26reserved-3D0%26d%3DDwMGaQ%26c%3D5VD0RTtNlTh3ycd41b3MUw%26r%3DC0sUo-LFNBaYfyoaCsf6TA%26m%3DTuVfzypo-CZLo522lgMRrLpKxcFaHvW9kdNSI57lXT0%26s%3DBy2d_to7XzLEgAL3Or9cZc2Aykye-PZEenFQDviAUlw%26e%3D&data=02%7C01%7Cpravb%40microsoft.com%7Cc50f91bc3ec7445441de08d5afc4d79e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636608182335633282&sdata=%2F6JEbuCnlWbiUY8uaEsOGdGNWPsb2KVOiVKH0UgdU8U%3D&reserved=0> if you do.  Loads shift, resources move, and suddenly you find that host which used to be in the same rack, being down, has load-shifted to the next instance,  in a datacenter 500 miles away across a very different network.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 2 May 2018 06:24:58 +0000

A data center might use an overlay network with Kubernetes or Swarm - which might lead to ossification.

An unencrypted per CID PN with cheap multiplication permutation is another option compared to 1079.

________________________________
From: QUIC <[email-masked]> on behalf of Praveen Balasubramanian <pravb=[email-masked]>
Sent: Wednesday, May 2, 2018 4:02:37 AM
To: Jana Iyengar; Roberto Peon
Cc: Salz, Rich; Ted Hardie; IETF QUIC WG; Benjamin Kaduk
Subject: RE: [Potential Spoof] Re: Getting to consensus on packet number encryption

Yup. I know.

-=R

-=R

---

From: Benjamin Kaduk <[email-masked]>
Date: Wed, 2 May 2018 07:10:18 -0500

Hi Praveen,

> We can get (1) done but I disagree that we can decouple (2). It doesn’t make sense to require performance overhead all the time with no benefit. For the Internet, PNE can prevent ossification so it…

Most of why I am pushing back so hard on this point is that I am having a
*really* hard time getting the statement of "no benefit" (i.e., exactly zero)
to mesh with my understanding of the situation.  I don't know the best
way past this impasse -- it doesn't really seem right for me to make a
list of potential benefits and have you shoot them down, for example.  I
think maybe you are trying to make a slightly different point involving
something like ongoing operational benefits (as opposed to cost of
implementation/risk of bugs/ecosystem benefits), but I'm not entirely
sure.

> We know that there is a CPU cost, this is not speculation. Doesn’t matter if
> the cost is 1% or 10%. Hardware implementers have also raised concerns. Every

Well, I think it rather does matter for 1% vs. 10%.  If it was 0.1%
(unlikely, I suppose), then that might be "in the noise" and not worth
doing anything about, ever.  For a 1% cost, that's significant enough to
make me want to address it in my long-term plans, but it is not
necessarily a big enough impact to require immediate attention.  10%, on
the other hand, probably would require immediate attention and be a
show-stopper.

> CPU cycle we spend on things that add no value to the user should be
> questioned. Every extra instruction we execute in the hot data path leaves
> less CPU for workloads, burns more power and makes for a worse comparison
> with HTTPS over TCP. The same argument as privacy holes holds here – just

The "burns more power" argument can be applied to every single CPU
instruction; relative measures (i.e., percentage of total CPU) from
actual deployment are what I find to be most compelling.

> because there are some existing perf bottlenecks doesn’t mean we add more,
> particularly ones we cannot mitigate. I said it before that there are lots of
> ways we can improve UDP perf but I don’t know of any technique for reducing
> crypto CPU cost other than hardware offload.

And I ask again, who is "we"?  (At this point I am mostly assuming "all
QUIC implementors", but it would be nice to be sure.)

> The cost is certainly worth saving with something as simple as a negotiation
> mechanism. Am I missing something about a transport parameter being

>From the rest of the discussion, I don't think you've really sold people
on "certainly", yet.

> complicated to implement? Implementations are free to support only PNE and
> not negotiate anything else. There is actually an even simpler mechanism if a
> transport parameter is deemed as costly: if the peer sends PN = 0 in the CI,
> then its requesting no PNE and the receiver can choose to proceed or drop and
> force TCP fall back. This should take only an extra if check on the receiver
> CI processing logic to implement. I am assuming here that PNE on input of 0
> will not produce 0 so it’s a safe sentinel value.

I think some people are worried not just about the complexity/simplicity
of the mechanism implemented, but broader-scope ecosystem and
architecture issues, whether that's a general complexity reduction
initiative or more subtle issues involving middleboxes attempting to
coerce certain sorts of behavior.

-Ben

---

From: Ian Swett <[email-masked]>
Date: Wed, 02 May 2018 12:28:10 +0000

Praveen, I thought the suggested plan by Martin was to default enable PNE
and then we can separately discuss a mechanism(ie: Transport params) where
if both sides agreed, PNE would be disabled for short header packets?  That
seems fine for your use cases, correct?  Or is the concern that the WG will
reject any sort of mechanism that allows PNE to be disabled?

And as Ben said, if this adds 10% CPU, then we need to rethink this
completely, but I don't believe it will.  Hopefully I'll have some numbers
before the next interim

AND FWIW, I'd like to see a QUIC v2 within a year or two of QUIC v1.

On Wed, May 2, 2018 at 8:10 AM Benjamin Kaduk <bkaduk=
[email-masked]> wrote:

> > We can get (1) done but I disagree that we can decouple (2). It doesn’t
> make sense to require performance overhead all the time with no benefit.
> For the Internet, PNE can prevent ossification so it is seen as a
> reasonable trade-off. For the datacenter, there is no known benefit, only
> cost.
>

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 2 May 2018 14:46:51 +0200

Hi Ben, hi all,

at this point, I'm fine with moving on with a decision and I didn’t speak up recently because I did feel that just repeating my point of view over and over again (like other do) would bring the discussion forward but I have to say that I’m still skeptical about the benefits of PNE. 

I don’t think that it will help linkability as mitigation can be easy detected with simple traffic analysis (if you see both flows; if you don’t see both flows you can’t detect it no matter what) and I rate the risk of ossification on the packet number rather low (both in terms of the risk that it will happen and if even if it happens that it will hinder protocol evolution strongly) such that I simply don’t think that it is justified to spend ANY additional complexity on PNE. 

I might be wrong and I understand the fear of people about what can happen in a worst case scenario, however, I also learned during my time in the IEFT that complexity itself can be a barrier for deployment that should not be underestimated. If people don’t understand the specs we write, they will not be able to implement (and maintain) our protocols, or at least not be able to implement it correctly which might be even worse for deployment and security (than not using them at all). 

Independent of the decision on PNE, I would like people to please keep in mind that while increased security is preferable, it does not mean that this is a free ticket to add encryption wherever we can, even if there are no or maybe yet unknown benefits as it always comes with a cost.

Mirja

> Am 02.05.2018 um 14:10 schrieb Benjamin Kaduk <bkaduk=[email-masked]>:
> 
> Hi Praveen,
>

---

From: "Roni Even (A)" <[email-masked]>
Date: Wed, 2 May 2018 13:04:49 +0000

Hi,
I strongly support Mirja's view.  At least we can move forward.

I did not feel from the discussion that the benefits from PNE vs the PN were meaningful enough to make me support any direction too many variables (Ossification, linkability, complexity and processing requirements, PNE not needed for all cases so negotiate,...) with no way for evaluation.  If there was a hum I would probably hum for "not enough information" or "no opinion" but still hope that some decision will be made.

Roni Even

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Mirja Kühlewind
Sent: Wednesday, May 02, 2018 3:47 PM
To: Benjamin Kaduk
Cc: Praveen Balasubramanian; IETF QUIC WG
Subject: Re: Getting to consensus on packet number encryption

Hi Ben, hi all,

Mirja

> *really* hard time getting the statement of "no benefit" (i.e., 
> exactly zero) to mesh with my understanding of the situation.  I don't 
> know the best way past this impasse -- it doesn't really seem right 
> for me to make a list of potential benefits and have you shoot them 
> down, for example.  I think maybe you are trying to make a slightly 
> different point involving something like ongoing operational benefits 
> (as opposed to cost of implementation/risk of bugs/ecosystem 
> benefits), but I'm not entirely sure.
> 
>> We know that there is a CPU cost, this is not speculation. Doesn’t 
>> matter if the cost is 1% or 10%. Hardware implementers have also 
>> raised concerns. Every
> 
> doing anything about, ever.  For a 1% cost, that's significant enough 
> to make me want to address it in my long-term plans, but it is not 
> necessarily a big enough impact to require immediate attention.  10%, 
> on the other hand, probably would require immediate attention and be a 
> show-stopper.
> 
>> questioned. Every extra instruction we execute in the hot data path 
>> leaves less CPU for workloads, burns more power and makes for a worse 
>> comparison with HTTPS over TCP. The same argument as privacy holes 
>> holds here – just
> 
>> because there are some existing perf bottlenecks doesn’t mean we add 
>> more, particularly ones we cannot mitigate. I said it before that 
>> there are lots of ways we can improve UDP perf but I don’t know of 
>> any technique for reducing crypto CPU cost other than hardware offload.
> 
> And I ask again, who is "we"?  (At this point I am mostly assuming 
> "all QUIC implementors", but it would be nice to be sure.)
> 
>> The cost is certainly worth saving with something as simple as a 
>> negotiation mechanism. Am I missing something about a transport 
>> parameter being
> 
>> complicated to implement? Implementations are free to support only 
>> PNE and not negotiate anything else. There is actually an even 
>> simpler mechanism if a transport parameter is deemed as costly: if 
>> the peer sends PN = 0 in the CI, then its requesting no PNE and the 
>> receiver can choose to proceed or drop and force TCP fall back. This 
>> should take only an extra if check on the receiver CI processing 
>> logic to implement. I am assuming here that PNE on input of 0 will not produce 0 so it’s a safe sentinel value.
> 
> I think some people are worried not just about the 
> complexity/simplicity of the mechanism implemented, but broader-scope 
> ecosystem and architecture issues, whether that's a general complexity 
> reduction initiative or more subtle issues involving middleboxes 
> attempting to coerce certain sorts of behavior.
> 
> -Ben
>

---

From: Dmitri Tikhonov <[email-masked]>
Date: Wed, 2 May 2018 09:11:39 -0400

On Wed, May 02, 2018 at 02:46:51PM +0200, Mirja Kühlewind wrote:
> [...] I also learned during my time in the IEFT that complexity itself
> can be a barrier for deployment that should not be underestimated. If
> people don’t understand the specs we write, they will not be able
> to implement (and maintain) our protocols, or at least not be able to
> implement it correctly which might be even worse for deployment and
> security (than not using them at all).

Agreed.

  - Dmitri.

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 02 May 2018 14:46:10 +0100

As I see it, adding PNE adds complexity, and presents risks for the 
protocol.

I accept there could also be a future be risk of ossification around any 
packet header - although as said also, the incentives for a middlebox 
doing odd things with a PN in QUIC would seem to me to be quite 
different from the incentives to mangle TCP (where you can easily 
influence the receiver state machine). As Mirja stated, this also leaves 
me skeptical about the benefits of PNE in avoiding ossification, nor the 
downsides of this. The direct linakbility appears from a decision to 
continue the PN sequence in the clear part of the header across a 
migration, not because a PN was visible.

While we are summarising: the majority of IETF transports have to date 
included some form of sequence number in all transport segements. This 
has often been usefully harnessed by tools such as wireshark or similar 
for measurement, debugging, and as a basis for more in depth analysis 
and operational tools. We have decades of experience with this way of 
working - although it seems to me that this list has not proven a useful 
place to discuss that.

I didn't see strong consensus on the value of PNE, so I understand that 
this is just something that has to be decided to move on. And I do wish 
to move on.

Gorry

On 02/05/2018, 14:04, Roni Even (A) wrote:
>> Am 02.05.2018 um 14:10 schrieb Benjamin Kaduk<bkaduk=[email-masked]>:
>>
>> Hi Praveen,
>>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 2 May 2018 16:08:08 +0000

Yes that mechanism is perfectly fine and is what I have been arguing for. It should be fairly straightforward to add. I also think it is better implemented together with PNE but based on the decision on PNE, we have decoupled it. I must have missed Martin’s suggested plan about the negotiation, if that is being considered then I have no concern. Issue #1296<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fquicwg%2Fbase-drafts%2Fissues%2F1296&data=02%7C01%7Cpravb%40microsoft.com%7Cc4695c40509a4f37774208d5a496cd8c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636595889971448691&sdata=Affj5l5%2BnLCTwvLLgS9%2B3gze%2F9c9RrhFZIHwH%2Fk1Ljw%3D&reserved=0> is open and the next step would be a PR for the negotiation.

Getting before and after numbers will be great. Last time we implemented and measured PNE, it was showing to be 1%+ but we did not do more detailed tests like what happens for smaller packets and ACKs.

From: Ian Swett [mailto:[email-masked]]
Sent: Wednesday, May 2, 2018 5:28 AM
To: bkaduk=[email-masked]
Cc: Praveen Balasubramanian <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

Praveen, I thought the suggested plan by Martin was to default enable PNE and then we can separately discuss a mechanism(ie: Transport params) where if both sides agreed, PNE would be disabled for short header packets?  That seems fine for your use cases, correct?  Or is the concern that the WG will reject any sort of mechanism that allows PNE to be disabled?

And as Ben said, if this adds 10% CPU, then we need to rethink this completely, but I don't believe it will.  Hopefully I'll have some numbers before the next interim

On Wed, May 2, 2018 at 8:10 AM Benjamin Kaduk <bkaduk=[email-masked]<mailto:[email-masked]>> wrote:
Hi Praveen,

on "certainly", yet.

-Ben

---

From: Christian Huitema <[email-masked]>
Date: Wed, 2 May 2018 19:38:43 +0200

> On May 2, 2018, at 3:46 PM, Gorry Fairhurst <[email-masked]> wrote:
> 
> As I see it, adding PNE adds complexity, and presents risks for the protocol.

Personally I feel that PNE removes complexity. You don't need the synchronized offset on migration required in the current spec. You can support multipath without the added complexity of per path sequence numbers and per path acks. You could potentially do away with the numbering of connection ID. All that is significant.

-- Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 2 May 2018 13:56:06 -0400

On 2 May 2018 at 19.39.23, Christian Huitema ([email-masked]) wrote:

 You can support multipath without the added complexity of per path
sequence numbers and per path acks.

How would you do proper congestion control over multiple paths without
dedicating PN’s to a specific path?

---

From: Christian Huitema <[email-masked]>
Date: Wed, 2 May 2018 21:12:41 +0200

> On May 2, 2018, at 7:56 PM, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
> 
>>  You can support multipath without the added complexity of per path sequence numbers and per path acks. 
> 
> How would you do proper congestion control over multiple paths without dedicating PN’s to a specific path?
> 
Very simple. You know on which path a packet was sent. You can thus give congestion credits on that path when the packet is acknowledged, or congestion signals if the packet is lost. 

-- Christian Huitema