Subject: Re: Hardware acceleration and packet number encryption

From: Mark Nottingham <[email-masked]>
Date: Mon, 2 Apr 2018 16:14:51 +1000

Hi Praveen,

On 1 Apr 2018, at 4:17 am, Praveen Balasubramanian <pravb=[email-masked]> wrote:
> 
> PN size and location needs to be part of invariants. How the field is transformed and encoded in the clear is an end-to-end function and can change between versions. 

Invariants is in WGLC now, but does not include packet numbers. If you believe it needs to be in there, please raise an issue and we can discuss it (in a separate thread).

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 2 Apr 2018 16:14:51 +1000

Hi Praveen,

Cheers,

---

From: Praveen Balasubramanian <[email-masked]>
Date: Mon, 2 Apr 2018 16:17:34 +0000

Since PN is the very last field before encrypted payload, it should be ok to exclude the field from invariants. Several benefits ensue - well behaved middleboxes don't even look at an end-to-end field, leaves room for future crypto that doesn't need a cleartext nonce, datacenter QUIC could subsume PN into the encrypted portion, and cleartext QUIC in a controlled environment can just skip the encryption step. However if we add any new fields to invariants some future version onwards, would they need to be placed before PN? Since implementations will do version specific parsing this shouldn't be a big problem. 

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Mark Nottingham
Sent: Sunday, April 1, 2018 11:15 PM
To: Praveen Balasubramanian <pravb=[email-masked]>
Cc: IETF QUIC WG <[email-masked]>
Subject: Re: Hardware acceleration and packet number encryption

Hi Praveen,

Cheers,

--
Mark Nottingham   https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mnot.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7Ce135b2a8beff46bdf44208d598611381%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636582465072048656&sdata=OigvpPbNPqhLMJDYJ4xyphpFuyanfQzrY9eOwLwX2xU%3D&reserved=0

---

From: Praveen Balasubramanian <[email-masked]>
Date: Mon, 2 Apr 2018 16:17:34 +0000

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Mark Nottingham
Sent: Sunday, April 1, 2018 11:15 PM
To: Praveen Balasubramanian <pravb=[email-masked]>
Cc: IETF QUIC WG <[email-masked]>
Subject: Re: Hardware acceleration and packet number encryption

Hi Praveen,

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 4 Apr 2018 14:58:20 +1000

Hi everyone,

The editors have told your chairs that this issue is starting to block progress on other aspects of QUIC. Coming to consensus on it soon (i.e., before Stockholm, if possible) would be good.

It looks like this thread has come to a natural pause. Reading through it, I think we agree there are some unpleasant tradeoffs here, but so far we have only one concrete proposal -- PR#1079.

My AU$.05 - While we're chartered to produce a protocol that's encrypted as much as operational concerns allow, and that privacy is a natural extension of that (especially in light of RFC7258 and RFC6973), there is no *requirement* that we produce a protocol that is able to be accelerated by hardware.

That being the case, I think we need to ask ourselves if we believe that inclusion of PR#1079 will significantly inhibit deployment of the protocol. Based on the discussion so far, that doesn't seem to be the case, but I'd be interested to hear what others think.

If we can get to consensus to incorporate the PR, and folks come back later with a more hardware-friendly replacement that doesn't change the Invariants or increase linkability, I suspect the WG will be amenable to that.

What do folks think?

Cheers,

> On 29 Mar 2018, at 11:39 am, Ian Swett <ianswett=[email-masked]> wrote:
> 
> Thanks for the nice summary Jana.
> 
> As much as I'd love to have easier crypto HW acceleration, I've ended up arriving at the same conclusion.  I don't want to bite off the work to do proper multipath in QUIC v1, which I think is the only other reasonable option of those Christian outlined.
> 
> If someone comes up with a way to transform packet number to make it non-linkable, but doesn't have the downside of making hardware offload difficult, then I'm open to it.  But we've been talking about this for 2 months without any notable improvements over Martin's PR.
> 
> Given we never talk about any issue only once in QUIC, I'm sure this will come up again, but for the time being I think #1079 is the best option we have.
> 
> 
> 
> On Wed, Mar 28, 2018 at 8:03 PM Jana Iyengar <[email-masked]> wrote:
> A few quick thoughts as I catch up on this thread.
> 
> I spent some time last week working through a design using multiple PN spaces, and it is quite doable. I suspect we'll head towards multiple PN spaces as we consider multipath in the future. That said, there is complexity (as Christian notes). This complexity may be warranted when doing multipath in v2 or later, but I'm not convinced that this is necessary as a design primitive for QUICv1. 
> 
> We may want to creatively use the PN bits in v2, say to encode a path ID and a PN, for multipath. We want to retain flexibility in these bits going into v2. We've used encryption to ensure that we don't lose flexibility elsewhere in the header, and it follows that we should use PNE to retain flexibility in these bits as well. (Simplicity of design is the other value in using PNE, since handling migration linkability is non-trivial without it.)
> 
> This leaves the question of HW acceleration being at loggerheads with the design in PR #1079. First, I expect that the primary benefit of acceleration will be in DC environments. Yes, there are some gains to be had in serving the public Internet as well, but I'm unconvinced that this is the driving use case for hardware acceleration. I understand that others may disagree with me here.
> 
> AFAIK, QUIC has not been used in DC environments yet. I expect there are other things in the protocol that we'd want to change as we gain experience deploying QUIC in DCs. Spinning up a new version to try QUIC within DCs is not only appropriate, I would recommend it. This allows for rapid iterations internally, and the experience can drive subsequent changes to QUIC. It's what *I* would do if I was to deploy QUIC inside a DC.
> 
> So, in short, I think we should go ahead with PR# 1079. This ensures that future versions are guaranteed the flexibility to change the PN bits for better support of HW acceleration or multipath or what-have-you.
> 
> - jana
> 
> On Mar 26, 2018 9:41 AM, "Christian Huitema" <[email-masked]> wrote:
> 
> On 3/26/2018 8:20 AM, Swindells, Thomas (Nokia - GB/Cambridge) wrote:
>> Looking at https://en.wikipedia.org/wiki/AES_instruction_set#Intel_and_AMD_x86_architecture it seems to imply a large range of server, desktop and mobile chips all have a CPU instruction set available to do AES acceleration and other similar operations (other instruction sets are also available).
>> 
>> If we are considering the AES instructions then it looks like it is (or at least will be in the near future) a sizeable proportion of the public internet have it to be used.
>> 
> 
> Certainly, but that's not the current debate. PR #1079 is fully compatible with use of the AES instructions. The issue of the debate is that the mechanism in PR #1079 required double buffering, first encrypt the payload, then use the result of the encryption to encrypt the PN. This is not an issue in a software implementation that can readily access all bytes of the packet from memory, but it may be an issue in some hardware implementations that are designed to do just one pass over the data.
> 
> 
> -- Christian Huitema
> 
> 
>

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 4 Apr 2018 07:58:48 +0200

Hi Mark,

On Wed, Apr 04, 2018 at 02:58:20PM +1000, Mark Nottingham wrote:
> My AU$.05 - While we're chartered to produce a protocol that's encrypted as
> much as operational concerns allow, and that privacy is a natural extension
> of that (especially in light of RFC7258 and RFC6973), there is no
> *requirement* that we produce a protocol that is able to be accelerated by
> hardware.

IMHO it's important to keep in mind that what cannot be accelerated by
hardware is also subject to severe limitations in terms of software-based
processing. For example anti-ddos solutions are critical for operations
everywhere nowadays, and the more unreadable the protocol becomes on the
wire, the lower amount of protection such third-party components will be
able to provide. And it's just an example. Also, I note that 36 years after
TCP was published, many of us still find ourselves launching tcpdump or
wireshark every single day to debug protocol-level issues, or at least to
be able to say who's right and who's wrong in a conversation to try to
spot the root cause of an issue.

I already expressed my real dislike (not to say disgust) for protocol
elements encryption for these various reasons and I'm well aware that
my view is generally not shared here. I tend to consider that a properly
working protocol is more important to maintain the internet in good health
and resilient than adding some elements which supposedly will make it
harder to guess what people are doing while their devices spread their
cookies to all ads providers and "like" buttons, are infected with malware
picking their clear text conversations, and some authorities are decrypting
communications by forcing keys directly on their devices. In the end we're
just being more selective about who has access to their activity. BTW, you
know the saying, there are two types of people, those how believe their
conversations have never been intercepted, and those who think they know
where it happened ;-)

Now a last concern about performance : sure, processors do provide AES
instructions, but these ones are not that fast, perform a single round,
and need to run in loops (10 rounds for AES-128 if my memory serves me
right). I just tried here on a Xeon E5-2680 at 2.5 GHz. This machine is
able to classify 14.88 million incoming packets per second on a single
core (10G line rate). But it only performs 42 million AES-128-CBC per
second and the other modes are even slower. So having to process even a
single AES operation per packet just to know whether or not it is valid
for a session will definitely come with a significant cost on some such
machines, by representing 1/3 of the processing time budget. And to this
we must add all the low-level processing that will not be offloadable to
the hardware anymore!

At the very least if we're going to encrypt the protocol and want to be
CPU-friendly, it would be nice to perform a single AES round (4-6 cycles
approx on modern CPUs with the suitable instruction set). It will
definitely not reduce privacy but will at least make the future internet
faster than by using full AES. It will not make it work better though.

Just my two cents,
Willy

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 4 Apr 2018 12:27:00 +0200

Hi Mark,

without intending to state a technical opinion, I think I need to disagree with your assessment of the situation. Please see below.

> Am 04.04.2018 um 06:58 schrieb Mark Nottingham <[email-masked]>:
> 
> Hi everyone,
> 

The other proposal is to use a random offset that can/should be changed when the connection ID is changed. Which is mostly inline with what the draft currently says.
> 
> My AU$.05 - While we're chartered to produce a protocol that's encrypted as much as operational concerns allow,

I don’t think this is what the charter say. The charter says "Providing always-secure transport, using TLS 1.3 by default.“ plus another paragraph that talks about TLS1.3. It does not spell out the desire of some individual to „encrypt as much as possible“. 

> and that privacy is a natural extension of that (especially in light of RFC7258 and RFC6973), there is no *requirement* that we produce a protocol that is able to be accelerated by hardware.

Not there is no requirement that the protocol need to be accelerated by hardware. However, not all requirement are listed in the charter. We as a group have to define the requirement. However, there is of course an implicit requirement for all work in the IEFT that is must be deployable and serve the needs of those people who want to deploy it.

> That being the case, I think we need to ask ourselves if we believe that inclusion of PR#1079 will significantly inhibit deployment of the protocol. Based on the discussion so far, that doesn't see…

Here my assessment is also different. There are multiple people that have raised concerns that this could hinder deployment.
> If we can get to consensus to incorporate the PR, and folks come back later with a more hardware-friendly replacement that doesn't change the Invariants or increase linkability, I suspect the WG wi…

I don’t think to just go with what we have even though there are concerns, is the right approach and definitely not the approach that was chosen for other proposals. In the sake of getting version v1 out quickly, I would think that not incorporating another change that does not have consensus is the proposer approach.

Mirja

---

From: Dmitri Tikhonov <[email-masked]>
Date: Wed, 4 Apr 2018 08:54:06 -0400

> conversations have never been intercepted, and those who think they know

+1.

> right).

For hosting providers, more CPU cycles means using more hardware.  In
other words, money.  This could have an impact on the adoption of QUIC.

  - Dmitri.

---

From: Patrick McManus <[email-masked]>
Date: Wed, 4 Apr 2018 08:54:32 -0400

I think its time to move forward with 1079.

btw I think its wrong to characterize this as perf vs privacy - its cpu vs
bandwidth, the privacy is a must have.

1079 costs cpu (both in hw and sw) and in my mind it really competes with
either a multipath-pn-scheme (which we'll table for v2) or a one pass
scheme with a new nonce rather than making the packet number do double
duty. The latter is bandwidth tax and I'd rather pay the cpu cost.

The only serious alternative to me is to take the timeline hit and work out
multiple packet number spaces for v1 (which are likely part of multipath
anyhow).

-P

On Wed, Apr 4, 2018 at 12:58 AM, Mark Nottingham <[email-masked]> wrote:

> Hi everyone,
>
> The editors have told your chairs that this issue is starting to block
> progress on other aspects of QUIC. Coming to consensus on it soon (i.e.,
> before Stockholm, if possible) would be good.
>
> It looks like this thread has come to a natural pause. Reading through it,
> I think we agree there are some unpleasant tradeoffs here, but so far we
> have only one concrete proposal -- PR#1079.
>
> My AU$.05 - While we're chartered to produce a protocol that's encrypted
> as much as operational concerns allow, and that privacy is a natural
> extension of that (especially in light of RFC7258 and RFC6973), there is no
> That being the case, I think we need to ask ourselves if we believe that
> inclusion of PR#1079 will significantly inhibit deployment of the protocol.
> Based on the discussion so far, that doesn't seem to be the case, but I'd
> be interested to hear what others think.
>
> If we can get to consensus to incorporate the PR, and folks come back
> later with a more hardware-friendly replacement that doesn't change the
> Invariants or increase linkability, I suspect the WG will be amenable to
> that.
>
> What do folks think?
>
> Cheers,
>
>
> > As much as I'd love to have easier crypto HW acceleration, I've ended up
> arriving at the same conclusion.  I don't want to bite off the work to do
> proper multipath in QUIC v1, which I think is the only other reasonable
> option of those Christian outlined.
> >
> > If someone comes up with a way to transform packet number to make it
> non-linkable, but doesn't have the downside of making hardware offload
> difficult, then I'm open to it.  But we've been talking about this for 2
> months without any notable improvements over Martin's PR.
> >
> > Given we never talk about any issue only once in QUIC, I'm sure this
> will come up again, but for the time being I think #1079 is the best option
> we have.
> >
> >
> >
> > I spent some time last week working through a design using multiple PN
> spaces, and it is quite doable. I suspect we'll head towards multiple PN
> spaces as we consider multipath in the future. That said, there is
> complexity (as Christian notes). This complexity may be warranted when
> doing multipath in v2 or later, but I'm not convinced that this is
> necessary as a design primitive for QUICv1.
> >
> > We may want to creatively use the PN bits in v2, say to encode a path ID
> and a PN, for multipath. We want to retain flexibility in these bits going
> into v2. We've used encryption to ensure that we don't lose flexibility
> elsewhere in the header, and it follows that we should use PNE to retain
> flexibility in these bits as well. (Simplicity of design is the other value
> in using PNE, since handling migration linkability is non-trivial without
> it.)
> >
> > This leaves the question of HW acceleration being at loggerheads with
> the design in PR #1079. First, I expect that the primary benefit of
> acceleration will be in DC environments. Yes, there are some gains to be
> had in serving the public Internet as well, but I'm unconvinced that this
> is the driving use case for hardware acceleration. I understand that others
> may disagree with me here.
> >
> > AFAIK, QUIC has not been used in DC environments yet. I expect there are
> other things in the protocol that we'd want to change as we gain experience
> deploying QUIC in DCs. Spinning up a new version to try QUIC within DCs is
> not only appropriate, I would recommend it. This allows for rapid
> iterations internally, and the experience can drive subsequent changes to
> QUIC. It's what *I* would do if I was to deploy QUIC inside a DC.
> >
> > So, in short, I think we should go ahead with PR# 1079. This ensures
> that future versions are guaranteed the flexibility to change the PN bits
> for better support of HW acceleration or multipath or what-have-you.
> >
> > - jana
> >
> >> Looking at https://en.wikipedia.org/wiki/AES_instruction_set#Intel_and_
> AMD_x86_architecture it seems to imply a large range of server, desktop
> and mobile chips all have a CPU instruction set available to do AES
> acceleration and other similar operations (other instruction sets are also
> available).
> >>
> >> If we are considering the AES instructions then it looks like it is (or
> at least will be in the near future) a sizeable proportion of the public
> internet have it to be used.
> >>
> >
> > Certainly, but that's not the current debate. PR #1079 is fully
> compatible with use of the AES instructions. The issue of the debate is
> that the mechanism in PR #1079 required double buffering, first encrypt the
> payload, then use the result of the encryption to encrypt the PN. This is
> not an issue in a software implementation that can readily access all bytes
> of the packet from memory, but it may be an issue in some hardware
> implementations that are designed to do just one pass over the data.
> >
> >
> > -- Christian Huitema
> >
> >
> >
>
> --

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 4 Apr 2018 09:02:08 -0500

It's my intention to let the QUIC working group be the QUIC working group
most of the time between now and Montreal. Having said that ...

On Wed, Apr 4, 2018 at 7:54 AM, Patrick McManus <[email-masked]>
wrote:

> The only serious alternative to me is to take the timeline hit and work
> out multiple packet number spaces for v1 (which are likely part of
> multipath anyhow).
>

This makes sense to me (as an individual, which doesn't actually matter),
but does prod me to ask whether people are fairly confident that if
multipath needs multiple packet number spaces, that this is something that
could be added after QUICv1.

Thanks,

Spencer, as curious responsible AD

---

From: Patrick McManus <[email-masked]>
Date: Wed, 4 Apr 2018 10:18:31 -0400

> could be added after QUICv1.
imo yes, we could add them in v2 without runinng afoul of the invariants.

---

From: Brian Trammell (IETF) <[email-masked]>
Date: Wed, 4 Apr 2018 16:31:59 +0200

hi Mark, all,

IMO the decision as to whether packet number encryption is the right thing to do depends on the design we settle on for multipath, which I know is not very helpful with respect to "we need to make a decision now":

- If we end up with a multipath design with a single PN space across all subflows for a given MP connection -- a design which does have very nice properties -- then PN encryption seems to be mandatory to prevent trivial linkability across subflows.

- If we end up with a multipath design that provides multipath as layer 4.5 sitting atop multiple layer 4 connections (a la MP-TCP), then PN encryption is potentially nice to have for ossification prevention, but isn't very useful for much else AFAICT.

The fact that I believe the PN encryption question in V1 depends on a design choice in a V2 feature indicates that my opinion should logically be that PN encryption is not necessary for V1, but that we should work on improving what we have in #1079 to make it possible to easily offload to hardware, as a V2 feature.

Of course, it's more complicated than that, because isn't it always. Way more detail below (or just read draft-hardie-path-signals)...

Why encrypt packet numbers?
---------------------------

There are three broad justifications I've seen in the discussion to date for packet number encryption:

(1) "we're chartered to produce a protocol that's encrypted as much as operational concerns allow" -- like Mirja, I don't read the charter this way, though I acknowledge that might be a minority view. "Encrypt all the things" is a very nice organizing principle for work on QUIC, but without understanding (and coming to consensus to) the reasons *why* we want to encrypt the things, it is as useful a principle as "QUIC should be as blue as possible".

So we should talk about why we want to encrypt the PN:

(2) Ossification prevention. There seems to be rough consensus in the WG that exposing packet numbers will necessarily add them to the "descriptive invariants" -- i.e. parts of the wire image that we have not promised to keep the same but that will become difficult to change in future versions.

I believe this more on some days, and less on others.

On the one hand, most of the ossification that has happened to TCP's internal machinery happened because middleboxes could actually provide some perceived value to at least the person who approved the purchase, by understanding, reacting to, and modifying seq and ack numbers as well as tracking state. The QUIC packet number alone does not provide enough of a surface for middleboxes to do very much with it without breaking the protocol in ways that would become immediately apparent during testing.

On the other hand, naive attempts to apply TCP thinking to QUIC could lead to on-path breakage driven by PN inspection, and the built-in fallback to TCP could mask some of the issues this causes. (cf. Happy Eyeballs, which makes IPv6 migration less disruptive, also makes it less successful -- something like 10% of the Alexa top million sites that "support IPv6" from a DNS standpoint don't really accept connections).

Certainly, PN encryption is the conceptually simplest way to obfuscate PN behavior, if we decide that it is necessary for us to obfuscate the PN in V1 in order to preserve the ability to deploy PN encryption in V2.

(3) Linkability prevention. IMO this is only really relevant in a multipath situation, where the multipath design uses a single packet number space across all subflows (as noted above). Unintentional (on the part of the client) NAT rebinding in the presence of a server CID is already trivially linkable, and not particularly sophisticated traffic analysis in the middle can build small anonymity groups for migration based on timing alone, so PN encryption doesn't help much in the case that a rebinding might happen without client permission. (#include "brian_is_a_traffic_analysis_nihilist.md")

I actually find a fourth, more abstract reason more compelling than any of these:

(4) The path should only get to see stuff that's explicitly meant for the path. The packet number, with its present semantics, has no explicit utility for on-path devices. (It might have *implicit* utility for on-path devices, which I address below.)

Why not encrypt packet numbers?
-------------------------------

There are similarly two broad justifications for not encrypting packet numbers:

(1) Performance and scalability. Encrypting and decrypting packet numbers takes additional CPU, and two-pass encryption lends itself poorly to encryption offload.

It's difficult to make judgements based on this without a more detailed performance analysis. There are lots of things that might make QUIC slow -- an assumption in kernel UDP interfaces that UDP doesn't have to be fast, for instance -- and in the spirit of engineering we should address these in the order of their performance impact, based to the extent possible on real measurements of actual prototypes of the mechanisms proposed.

I recognize the tension between QUIC's design principle of transport-as-continuous-experiment and the desire of people who make chips to have a draft protocol they can implement offload for as early as possible, and the tension between the flexibility that userspace implementation affords and the restrictions on what is easy to do in hardware.  I also recognize that some of the people who would deploy QUIC might not if it means they have to scale their backends multiplicatively due to increased CPU requirements of a non-offloadable protocol. Hence I think we should work to make something like 1079 more offloadable.

(2) Utility to on-path devices. The path can use the packet number for various forms of classification (i.e., "is this probably QUIC?") and state tracking ("does this look like a valid packet for a flow I know about?"), as well as passive measurement and diagnostic purposes (i.e., for one-point upstream loss and reordering measurement, as well as for rejecting loss and reordered spin edges with a single spin bit).

I'm not sure any of these are really good uses of a cleartext packet number. In each of these cases a cleartext packet happens to share some properties with the ideal signals for each particular task.

We should decide for each of these cases whether they are important, then design specific signals for each of them. The spin bit + VEC discussed in London for passive latency measurement (and to be described in an individual draft we'll submit shortly for consideration for adoption) is one example of this. Passive RTT measurement can also be achieved with a cleartext PN and PN echo, but the spin+VEC has superior measurability characteristics as well as superior efficiency. Examination of other uses might lead us to add a signal back to the wire image that counts up kind of like a packet number, but need not necessarily be tied to the number the transport uses for its own internals.

(Or, as above, tl;dr, see draft-hardie-path-signals)

Cheers,

Brian

> On 4 Apr 2018, at 06:58, Mark Nottingham <[email-masked]> wrote:
> 
> Hi everyone,
>

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Wed, 4 Apr 2018 16:35:29 +0200

hi Patrick,

> On 4 Apr 2018, at 14:54, Patrick McManus <[email-masked]> wrote:
> 
> btw I think its wrong to characterize this as perf vs privacy - its cpu vs bandwidth, the privacy is a must have.

I agree with your characterization here, but I'm curious as to what aspect of privacy PN encryption guarantees. Is there something I'm missing beyond rendering linkability vastly more difficult for multipath flows with a single PN space?

Thanks, cheers,

Brian

---

From: Patrick McManus <[email-masked]>
Date: Wed, 4 Apr 2018 10:40:15 -0400

it is about linkability, but in v1 for mobility (which I wouldn't call
multipath but ymmv).

On Wed, Apr 4, 2018 at 10:35 AM, Brian Trammell (IETF) <[email-masked]>
wrote:

> > btw I think its wrong to characterize this as perf vs privacy - its cpu
> vs bandwidth, the privacy is a must have.
>
> I agree with your characterization here, but I'm curious as to what aspect
> of privacy PN encryption guarantees. Is there something I'm missing beyond
> rendering linkability vastly more difficult for multipath flows with a
> single PN space?
>
> Thanks, cheers,
>
> Brian
>
>

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Wed, 4 Apr 2018 16:43:01 +0200

Ah, ok. So intentional mobility, as opposed to unintentional rebinding... Thanks, got it. (Won't argue the philosophical question of whether that's multipath or not. :) )

Cheers,

Brian

> On 4 Apr 2018, at 16:40, Patrick McManus <[email-masked]> wrote:
> 
> it is about linkability, but in v1 for mobility (which I wouldn't call multipath but ymmv).
> 
> 
> 
> On Wed, Apr 4, 2018 at 10:35 AM, Brian Trammell (IETF) <[email-masked]> wrote:
> hi Patrick,
>

---

From: Ted Hardie <[email-masked]>
Date: Wed, 4 Apr 2018 11:06:12 -0700

On Wed, Apr 4, 2018 at 7:31 AM, Brian Trammell <[email-masked]> wrote:

>
>
> Of course, it's more complicated than that, because isn't it always. Way
> more detail below (or just read draft-hardie-path-signals)...
>
>
Given the citation above, it is probably no surprise that I generally agree
with Brian on this.  There is one place where I think we probably would
emphasize things slightly differently.  For more on his take, see
https://github.com/britram/draft-trammell-wire-image/blob/master/draft-trammell-wire-image.md
which talks a good bit about the ossification issue that he touched on in
his message here.

My emphasis is slightly different:  how much does this choice constrain the
working group's later choices?  From my take, making the choice to encrypt
the packet number in v1 has the lesser set of constraints, because it is
possible for later versions to add a path-visible signal of sequence
numbers if we see a reason to do so.  That set of sequence numbers might
not be the same as those being used for protocol state mechanics by the
endpoints, but that's okay as they wouldn't be serving the same purpose.
Looking at Brian's text on the utility of sequences to observers:

Utility to on-path devices. The path can use the packet number for various
> forms of classification (i.e., "is this probably QUIC?") and state tracking
> ("does this look like a valid packet for a flow I know about?"), as well as
> passive measurement and diagnostic purposes (i.e., for one-point upstream
> loss and reordering measurement, as well as for rejecting loss and
> reordered spin edges with a single spin bit).
>

it seems pretty clear that the passive measurement and diagnostic purposes
would work with fairly small sequence numbers, maybe even down to sizes
small enough to make the linkability problem less scary.  The other issues
("does this look like quic" and "does this look like a known flow") seem
like they don't really require packet numbers or even deliberately exposed
sequence numbers, just consistency.   In other words, if we do anything to
expose sequences, the wire image we engineer does not have to simply re-use
the packet numbers used for protocol mechanics. It can focus on the message
we want to send to the path instead.

regards,

Ted

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 4 Apr 2018 20:33:35 +0200

Hi Ted,

based on what you say here, I’d like to further pick on an important point

> Am 04.04.2018 um 20:06 schrieb Ted Hardie <[email-masked]>:
> 
> My emphasis is slightly different:  how much does this choice constrain the working group's later choices?  From my take, making the choice to encrypt the packet number in v1 has the lesser set of constraints, because it is possible for later versions to add a path-visible signal of sequence numbers if we see a reason to do so.  That set of sequence numbers might not be the same as those being used for protocol state mechanics by the endpoints, but that's okay as they wouldn't be serving the same purpose.  

Currently the PN is used for a least two different purposes: 1) congestion control/retransmissions and as 2) input for the decryption. For the first purpose, the PN would not need to be in the wire image and could just be part of the encrypted payload. I guess the main reason that we use the same bits for both purposed is to save bits...

While you say, we can still add an unencrypted PN for a third purpose (measurements), it is also possible to add any time later (in a new version or with extension frames) a separate PN for 1 to the fully encrypted part of the packet that is independent of the exposed packet number (in case that gets ossified and does not address the needs of the quic transport machinery anymore). 

So even if the exposed PN gets ossified, I think we still have ways to handle that. However, the question is, do we want to burn the bits and complexity now, or later when we might (or might not) figure out that it is needed?

Mirja

---

From: Mike Bishop <[email-masked]>
Date: Wed, 4 Apr 2018 18:38:18 +0000

At this point, I think we have multiple paths forward, pun intended.

(1) - Packet number encryption.  #1079's challenges have been amply discussed, though it's not the only possibility in this space.

(2) - Per-CID offset for obfuscation.  Rather than specifying a packet number gap, specify a number that will be added to the actual packet number for any frames sent with that CID.  This allows the path to observe, per CID, monotonically-increasing packet numbers without necessarily being able to recover the true packet number.

(3) - Status quo.

Problems I'm wrestling with:

- Specification holes.  With my editor's hat only half-on (since I'm not an editor on this particular document), I'll note that there are a number of thorny issues with (3) in the current drafts that just go away if packet numbers are encrypted.  The current obfuscation method, packet number gaps, simply does not play well with the connection migration design we chose.  Gaps assume a continuous monotonic progression through CIDs which are never used again once you've ever seen a packet with the next CID, and don't accommodate the possibility that you might be using different CIDs on different paths at the same time for probing.  This is the driving issue at the moment -- if the WG chooses (3), we then need to run these other issues to ground, which will not be quick.

- Linkability.  With (2), if an observer sees the beginning of the connection, they can figure out the offset for that CID, then track the packet number as long as that CID is in use, then make a fairly accurate guess about the offset of the next CID they see, if they're able to otherwise correlate them.  There is some correlation that can be made by the fact that one CID disappears and another one appears on (potentially) the same 4-tuple, of course.  (1) might let an implementation mask that by using more than one CID at once to blur the boundaries a little bit, but (2) would make that attempt obvious because the middlebox would be able to observe gaps that neatly fit the other CID's observed packet counts.

- "Helpfulness."  With TCP, we see boxes that try to "help" by correcting packets which appear to be out of order.  The principle we've followed so far is that we want to leave all transport-related activities in the end hosts, and we actively don't want "helpful" middle devices, because when such a device sees a gap (because some packet was sent on a different path) it interprets it either as loss or as something that needs to trigger corrective action on its part.  This only gets worse when we move from migration support to full multipath.  That's what forced MP-TCP into a per-path sequence number in addition to a global sequence number.  I think we'd prefer to avoid having both in QUIC.  (2) exposes the same increasing sequence to middleboxes, inviting them to attempt helpfulness in the same way, and likely creating the same problems that MP-TCP encountered.

- Cost.  (1) fundamentally consumes more CPU, and that's a drawback.  There have been workarounds proposed to the offload problem (e.g. pre-compute and throw away some of the encryption so that the hardware doesn't have to do two passes), but it fundamentally costs more CPU and is harder to offload.  This in turn means more CapEx for those who would deploy QUIC on the server side, with the risk of slower adoption as a result.

When you get right down to it, I'm sympathetic to the cost argument, but it doesn't outweigh the other two.  A given server can (I presume) sustain dramatically more Telnet connections than it can SSH connections, but people who advocate continuing to use Telnet for that reason are roundly ridiculed.  SSH has a different, better security profile and those differences justify the cost.  TCP versus QUIC seems similar.

As a result, I believe that the reasonable path forward must be (1).  Not necessarily in the form of #1079 -- if there's a more offload-friendly or CPU-efficient way to do it, I'm all for it.  But let's start there and move forward.  If a path signal becomes necessary, that's a separate discussion.

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Mark Nottingham
Sent: Tuesday, April 3, 2018 9:58 PM
To: IETF QUIC WG <[email-masked]>
Cc: Lars Eggert <[email-masked]>
Subject: Getting to consensus on packet number encryption

Hi everyone,

What do folks think?

Cheers,

---

From: Dmitri Tikhonov <[email-masked]>
Date: Wed, 4 Apr 2018 14:48:26 -0400

On Wed, Apr 04, 2018 at 06:38:18PM +0000, Mike Bishop wrote:
> When you get right down to it, I'm sympathetic to the cost argument,
> but it doesn't outweigh the other two.  A given server can (I
> presume) sustain dramatically more Telnet connections than it can
> SSH connections, but people who advocate continuing to use Telnet
> for that reason are roundly ridiculed..  SSH has a different, better
> security profile and those differences justify the cost.  TCP versus
> QUIC seems similar.

We are comparing HTTPS with QUIC -- thus the telnet/SSH analogy
is not applicable.

  - Dmitri.

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 4 Apr 2018 18:51:25 +0000

You are also ignoring all bulk transfer applications over WAN and in DC where the CPU costs will magnify much more compared to online streaming web video and web search. I suspect the cost will be more like 3-4x. From service deployment point of view its still HTTP with TLS, just changing a transport underneath shouldn't require throwing 3x more servers.

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Dmitri Tikhonov
Sent: Wednesday, April 4, 2018 11:48 AM
To: Mike Bishop <[email-masked]>
Cc: Lars Eggert <[email-masked]>; Mark Nottingham <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

On Wed, Apr 04, 2018 at 06:38:18PM +0000, Mike Bishop wrote:
> presume) sustain dramatically more Telnet connections than it can SSH 
> connections, but people who advocate continuing to use Telnet for that 
> reason are roundly ridiculed..  SSH has a different, better security 
> profile and those differences justify the cost.  TCP versus QUIC seems 
> similar.

We are comparing HTTPS with QUIC -- thus the telnet/SSH analogy is not applicable.

  - Dmitri.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 4 Apr 2018 21:00:44 +0200

Mike,

on your 3 options list you are missing 4) multiple PN spaces (what I call
segmented packet numbers), ie. a serial PN per CID and a unique CID per
path. We keep assuming this is very difficult and only a v2 thing.
But I think it is not very difficult to do and it would solve nearly all
the issues except ossification on a single path. And if that is a problem
(which I don’t think), it would be good to know before v2 where it really
becomes important.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 4 April 2018 at 20.51.41, Praveen Balasubramanian (
pravb=[email-masked]) wrote:

You are also ignoring all bulk transfer applications over WAN and in DC
where the CPU costs will magnify much more compared to online streaming web
video and web search. I suspect the cost will be more like 3-4x. From
service deployment point of view its still HTTP with TLS, just changing a
transport underneath shouldn't require throwing 3x more servers.

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Dmitri Tikhonov
Sent: Wednesday, April 4, 2018 11:48 AM
To: Mike Bishop <[email-masked]>
Cc: Lars Eggert <[email-masked]>; Mark Nottingham <[email-masked]>; IETF
QUIC WG <[email-masked]>

> profile and those differences justify the cost. TCP versus QUIC seems

We are comparing HTTPS with QUIC -- thus the telnet/SSH analogy is not
applicable.

- Dmitri.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 4 Apr 2018 21:00:44 +0200

Mike,

Kind Regards,
Mikkel Fahnøe Jørgensen

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Dmitri Tikhonov
Sent: Wednesday, April 4, 2018 11:48 AM
To: Mike Bishop <[email-masked]>
Cc: Lars Eggert <[email-masked]>; Mark Nottingham <[email-masked]>; IETF
QUIC WG <[email-masked]>

- Dmitri.

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 4 Apr 2018 19:14:55 +0000

+1

From: QUIC [mailto:[email-masked]] On Behalf Of Mikkel Fahnøe Jørgensen
Sent: Wednesday, April 4, 2018 12:01 PM
To: Praveen Balasubramanian <pravb=[email-masked]>; Mike Bishop <[email-masked]>; Dmitri Tikhonov <[email-masked]>
Cc: Lars Eggert <[email-masked]>; Mark Nottingham <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: RE: Getting to consensus on packet number encryption

Mike,

on your 3 options list you are missing 4) multiple PN spaces (what I call segmented packet numbers), ie. a serial PN per CID and a unique CID per path. We keep assuming this is very difficult and only a v2 thing.
But I think it is not very difficult to do and it would solve nearly all the issues except ossification on a single path. And if that is a problem (which I don’t think), it would be good to know before v2 where it really becomes important.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 4 April 2018 at 20.51.41, Praveen Balasubramanian (pravb=[email-masked]<mailto:pravb=[email-masked]>) wrote:
You are also ignoring all bulk transfer applications over WAN and in DC where the CPU costs will magnify much more compared to online streaming web video and web search. I suspect the cost will be more like 3-4x. From service deployment point of view its still HTTP with TLS, just changing a transport underneath shouldn't require throwing 3x more servers.

-----Original Message-----
From: QUIC [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Dmitri Tikhonov
Sent: Wednesday, April 4, 2018 11:48 AM
To: Mike Bishop <[email-masked]<mailto:[email-masked]>>
Cc: Lars Eggert <[email-masked]<mailto:[email-masked]>>; Mark Nottingham <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Getting to consensus on packet number encryption

- Dmitri.

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 4 Apr 2018 19:14:55 +0000

+1

Mike,

Kind Regards,
Mikkel Fahnøe Jørgensen

-----Original Message-----
From: QUIC [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Dmitri Tikhonov
Sent: Wednesday, April 4, 2018 11:48 AM
To: Mike Bishop <[email-masked]<mailto:[email-masked]>>
Cc: Lars Eggert <[email-masked]<mailto:[email-masked]>>; Mark Nottingham <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Getting to consensus on packet number encryption

- Dmitri.

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 4 Apr 2018 19:15:17 +0000

I too disagree with the assessment of the situation. +1 to everything Mirja said.

>> there is no *requirement* that we produce a protocol that is able to be accelerated by hardware.
It is however a requirement for deployment that we be able to offload QUIC. A large part of work we do in TCP is acceleration in software and hardware. If QUIC has to evolve from experiment to broad deployment performance is a key goal. This is brought up several times in TCP working group where we see proposals that wouldn’t work with widely deployed software and hardware accelerations. 

Whether we like it or not performance is a key requirement for QUIC on servers and for datacenter scenarios. Even Ian mentioned before that he'd not want to do PNE within DC and I agree. Most connections will also not be doing migration or using multi path so why pay the extra overhead in the common case for no benefit?

Since PNE doesn’t affect the invariants (its simply an extra step post packetization and pre parsing), please explain why it would block progress on other fronts? What is the list of things that get blocked?

Please let us not rush into a design that impacts per packet processing costs with no clear articulation of the problem and the benefits. 

From what I can tell there is wide disagreement in the group and nothing resembling consensus. The fact that we have poor representation from hardware developers and public cloud providers is further skewing the perception. 

My recommendations:
1. Leave the current draft as is. Make sure we have some provision for greasing the PN to prevent ossification.
2. Explore multiple PN spaces for connection migration (and eventually multipath).
3. Make PNE an optional negotiated extension for implementations that want to do connection migration in V1 with the caveat that we will revisit this for V2 when we do multipath.
4. Solicit more feedback on the protocol from hardware vendors and public and private cloud providers. I can try to loop in more folks but others should too.

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Mirja Kühlewind
Sent: Wednesday, April 4, 2018 3:27 AM
To: Mark Nottingham <[email-masked]>
Cc: Lars Eggert <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

Hi Mark,

Mirja

>>> Looking at https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FAES_instruction_set%23Intel_and_AMD_x86_architecture&data=02%7C01%7Cpravb%40microsoft.com%7C14f278ab3769469a44f908d59a16a2d6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636584344383959765&sdata=aOHc7FZoIBCE1OlqmNpvdMM0ntLerZ3FpqESVj85zfE%3D&reserved=0 it seems to imply a large range of server, desktop and mobile chips all have a CPU instruction set available to do AES acceleration and other similar operations (other instruction sets are also available).
>>> 
> Mark Nottingham   https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mnot.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7C14f278ab3769469a44f908d59a16a2d6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636584344383959765&sdata=RXba2TyvYz6otecEHoLCT4tZ2I7EodxDiyPxWrn%2FnAw%3D&reserved=0
>

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 4 Apr 2018 19:55:47 +0000

To restate the options:
(1) - Packet number encryption.
(2) - Per-CID offset.
(3) - Plain text PNs across connection migrations.
(4) - Per-CID PN spaces.

None of the alternatives mask the “connection migration” event from an on-path observer.  The “original” connection establishment is done via clearly identifiable handshake packets on a 5-tuple before short-header packets show up, while “connection migration” events are identifiable by the use of a new CID with short-header packets on a new 5-tuple w/o preceding handshake packets.

As it was pointed out many times (especially by Brian), it is not that hard to correlate cessation of packets from one 5-tuple and appearance of a new post-“connection migration” 5-tuple, especially if destination IPs for both are identical.  Moreover, since devices that wish to maintain connections during migration will likely migrate multiple connections, the on-path observer will see a group of 5-tuples from the same IP cease activity, while a similar group of post-“connection migration” 5-tuples appears from a different IP, all with matched destination IPs.

Given the above, I am left wondering if increasing the costs for content hosting, increasing per-packet power requirements for mobile and micro devices, and making the Internet harder to troubleshoot and operate is worth the mostly illusory privacy gains of (1).  Patrick’s “the privacy is a must have” is a desire I share, but does (1) provide privacy against a sophisticated on-path observer?

Yet, (2) seems simple enough and a good compromise.  As Mike points out, it allows an on-path observer who can link CIDs across “connection migrations” to infer PNs.  But if CIDs are already linked, there is no more privacy argument to be made anyway.  One thing (2) does not allow an on-path observer to do is to link CIDs via PN analysis.  (Though it still seems like the privacy gains from any of the above are mostly an illusion.)  So my vote is for (2).

- Igor

-----Original Message-----
From: Praveen Balasubramanian [mailto:pravb=[email-masked]] 
Sent: Wednesday, April 04, 2018 3:15 PM
To: Mirja Kühlewind <[email-masked]>; Mark Nottingham <[email-masked]>
Cc: Lars Eggert <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: RE: Getting to consensus on packet number encryption

My recommendations:

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Mirja Kühlewind
Sent: Wednesday, April 4, 2018 3:27 AM
To: Mark Nottingham <[email-masked]>
Cc: Lars Eggert <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

Hi Mark,

Mirja

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 4 Apr 2018 19:55:47 +0000

To restate the options:

- Igor

-----Original Message-----
From: Praveen Balasubramanian [mailto:pravb=[email-masked]] 
Sent: Wednesday, April 04, 2018 3:15 PM
To: Mirja Kühlewind <[email-masked]>; Mark Nottingham <[email-masked]>
Cc: Lars Eggert <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: RE: Getting to consensus on packet number encryption

My recommendations:

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Mirja Kühlewind
Sent: Wednesday, April 4, 2018 3:27 AM
To: Mark Nottingham <[email-masked]>
Cc: Lars Eggert <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

Hi Mark,

Mirja

---

From: Roberto Peon <[email-masked]>
Date: Wed, 4 Apr 2018 20:28:36 +0000

PN encryption plus multiple (rotating) GUIDs specified on a per-path basis, as discussed in Melbourne do seem able to provide substantively better privacy. With such a scheme nothing appears to be shared from one 5-tuple+GUID to another 5-tuple+GUID.

An implementation that cared deeply about privacy would send packets along both paths for some time. This doesn't seem like something we can reasonable require with the protocol, but it can be allowed by the protocol.

I am not particularly interested in seeing QUIC v1 become the most CPU efficient thing. I'd rather we release it, and continue iterating. Concentrating on features that prevent ossification furthers this goal and allows for an eventual (and possibly better) HW acceleration path later.
-=R

________________________________
From: QUIC <[email-masked]> on behalf of Lubashev, Igor <ilubashe=[email-masked]>
Sent: Wednesday, April 4, 2018 12:55:47 PM
To: Praveen Balasubramanian; Mirja Kühlewind; Mark Nottingham
Cc: Lars Eggert; IETF QUIC WG
Subject: RE: Getting to consensus on packet number encryption

To restate the options:

(2) - Per-CID offset.

(4) - Per-CID PN spaces.

- Igor

-----Original Message-----

>> there is no *requirement* that we produce a protocol that is able to be accelerated by hardware.

It is however a requirement for deployment that we be able to offload QUIC. A large part of work we do in TCP is acceleration in software and hardware.. If QUIC has to evolve from experiment to broad deployment performance is a key goal. This is brought up several times in TCP working group where we see proposals that wouldn’t work with widely deployed software and hardware accelerations.

My recommendations:

-----Original Message-----

Hi Mark,

Mirja

>>> Looking at https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fen.wikipedia.org-252Fwiki-252FAES-5Finstruction-5Fset-2523Intel-5Fand-5FAMD-5Fx86-5Farchitecture-26data-3D02-257C01-257Cpravb-2540microsoft.com-257C14f278ab3769469a44f908d59a16a2d6-257C72f988bf86f141af91ab2d7cd011db47-257C1-257C0-257C636584344383959765-26sdata-3DaOHc7FZoIBCE1OlqmNpvdMM0ntLerZ3FpqESVj85zfE-253D-26reserved-3D0&d=DwIGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=ym7cE5rLl46aIfj8Q7KNEpy3FDfhyfumeXz9eQXmX1g&s=WkBplmuEwMxL-oezaQ4rK1_9-0SXgkY1NJ_V3SPhYOU&e= it seems to imply a large range of server, desktop and mobile chips all have a CPU instruction set available to do AES acceleration and other similar operations (other instruction sets are also available).

> Mark Nottingham   https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fwww.mnot..net-252F-26data-3D02-257C01-257Cpravb-2540microsoft.com-257C14f278ab3769469a44f908d59a16a2d6-257C72f988bf86f141af91ab2d7cd011db47-257C1-257C0-257C636584344383959765-26sdata-3DRXba2TyvYz6otecEHoLCT4tZ2I7EodxDiyPxWrn-252FnAw-253D-26reserved-3D0&d=DwIGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=ym7cE5rLl46aIfj8Q7KNEpy3FDfhyfumeXz9eQXmX1g&s=b_XyJvAQscNJNCwHmgj11ep8Xbsf7PvhDvONVZe3PR0&e=

---

From: Roberto Peon <[email-masked]>
Date: Wed, 4 Apr 2018 20:28:36 +0000

To restate the options:

(2) - Per-CID offset.

(4) - Per-CID PN spaces.

- Igor

-----Original Message-----

My recommendations:

-----Original Message-----

Hi Mark,

Mirja

> Mark Nottingham   https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fwww.mnot.net-252F-26data-3D02-257C01-257Cpravb-2540microsoft.com-257C14f278ab3769469a44f908d59a16a2d6-257C72f988bf86f141af91ab2d7cd011db47-257C1-257C0-257C636584344383959765-26sdata-3DRXba2TyvYz6otecEHoLCT4tZ2I7EodxDiyPxWrn-252FnAw-253D-26reserved-3D0&d=DwIGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=ym7cE5rLl46aIfj8Q7KNEpy3FDfhyfumeXz9eQXmX1g&s=b_XyJvAQscNJNCwHmgj11ep8Xbsf7PvhDvONVZe3PR0&e=

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 4 Apr 2018 21:09:39 +0000

*   An implementation that cared deeply about privacy would send packets along both paths for some time.

Devices would only be able to do so under certain conditions.  But if the device is able to transit on both path in an attempt to avoid CID linkage, it could use (2) "Per-CID offset" and retransmit the same packets (except for padding and with randomized inter-packet intervals) on both paths.  The QUIC peer would discard duplicates.

The cost of privacy (sending duplicate packets) would then be paid only by those who desire privacy and only for the short duration of the migration.

  *   Igor

From: Roberto Peon [mailto:[email-masked]]
Sent: Wednesday, April 04, 2018 4:29 PM
To: Lubashev, Igor <[email-masked]>; Praveen Balasubramanian <[email-masked]>; Mirja Kühlewind <[email-masked]>; Mark Nottingham <[email-masked]>
Cc: Lars Eggert <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Lubashev, Igor <ilubashe=40akamai.com@dmarc...ietf.org<mailto:ilubashe=40akamai.com@dmarc...ietf.org>>

To restate the options:

(2) - Per-CID offset.

(4) - Per-CID PN spaces.

None of the alternatives mask the "connection migration" event from an on-path observer.  The "original" connection establishment is done via clearly identifiable handshake packets on a 5-tuple before short-header packets show up, while "connection migration" events are identifiable by the use of a new CID with short-header packets on a new 5-tuple w/o preceding handshake packets.

As it was pointed out many times (especially by Brian), it is not that hard to correlate cessation of packets from one 5-tuple and appearance of a new post-"connection migration" 5-tuple, especially if destination IPs for both are identical.  Moreover, since devices that wish to maintain connections during migration will likely migrate multiple connections, the on-path observer will see a group of 5-tuples from the same IP cease activity, while a similar group of post-"connection migration" 5-tuples appears from a different IP, all with matched destination IPs.

Given the above, I am left wondering if increasing the costs for content hosting, increasing per-packet power requirements for mobile and micro devices, and making the Internet harder to troubleshoot and operate is worth the mostly illusory privacy gains of (1).  Patrick's "the privacy is a must have" is a desire I share, but does (1) provide privacy against a sophisticated on-path observer?

Yet, (2) seems simple enough and a good compromise.  As Mike points out, it allows an on-path observer who can link CIDs across "connection migrations" to infer PNs.  But if CIDs are already linked, there is no more privacy argument to be made anyway.  One thing (2) does not allow an on-path observer to do is to link CIDs via PN analysis.  (Though it still seems like the privacy gains from any of the above are mostly an illusion.)  So my vote is for (2).

- Igor

-----Original Message-----

To: Mirja Kühlewind <[email-masked]<mailto:[email-masked]>>; Mark Nottingham <[email-masked]<mailto:[email-masked]>>

Cc: Lars Eggert <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>

>> there is no *requirement* that we produce a protocol that is able to be accelerated by hardware.

It is however a requirement for deployment that we be able to offload QUIC. A large part of work we do in TCP is acceleration in software and hardware... If QUIC has to evolve from experiment to broad deployment performance is a key goal. This is brought up several times in TCP working group where we see proposals that wouldn't work with widely deployed software and hardware accelerations.

Since PNE doesn't affect the invariants (its simply an extra step post packetization and pre parsing), please explain why it would block progress on other fronts? What is the list of things that get blocked?

My recommendations:

-----Original Message-----

To: Mark Nottingham <[email-masked]<mailto:[email-masked]>>

Cc: Lars Eggert <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>

Hi Mark,

> Am 04.04.2018 um 06:58 schrieb Mark Nottingham <[email-masked]<mailto:[email-masked]>>:

> My AU$.05 - While we're chartered to produce a protocol that's encrypted as much as operational concerns allow,

I don't think this is what the charter say. The charter says "Providing always-secure transport, using TLS 1.3 by default." plus another paragraph that talks about TLS1.3. It does not spell out the desire of some individual to "encrypt as much as possible".

> If we can get to consensus to incorporate the PR, and folks come back later with a more hardware-friendly replacement that doesn't change the Invariants or increase linkability, I suspect the WG wi…

I don't think to just go with what we have even though there are concerns, is the right approach and definitely not the approach that was chosen for other proposals. In the sake of getting version v1 out quickly, I would think that not incorporating another change that does not have consensus is the proposer approach.

Mirja

>> On 29 Mar 2018, at 11:39 am, Ian Swett <ianswett=[email-masked]<mailto:ianswett=[email-masked]>> wrote:

>> On Wed, Mar 28, 2018 at 8:03 PM Jana Iyengar <[email-masked]<mailto:[email-masked]>> wrote:

>> On Mar 26, 2018 9:41 AM, "Christian Huitema" <[email-masked]<mailto:[email-masked]>> wrote:

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 5 Apr 2018 07:28:10 +1000

Mirja,

> The other proposal is to use a random offset that can/should be changed when the connection ID is changed. Which is mostly inline with what the draft currently says.

As I understand it, the status quo has caused a significant amount of pain, complexity and potential bugs. PN encryption was proposed by Kazuho in Melbourne as a way to work around that. 

We can, of course, decide that the status quo is the right approach. However, my current reading of the WG is that it isn't acceptable.

> I don’t think this is what the charter say. The charter says "Providing always-secure transport, using TLS 1.3 by default.“ plus another paragraph that talks about TLS1.3. It does not spell out the…

I think that depends on how you read "always-secure." The charter also says we will provide "confidentiality and integrity protection of both application data and QUIC headers."

> Not there is no requirement that the protocol need to be accelerated by hardware. However, not all requirement are listed in the charter. We as a group have to define the requirement. However, ther…

Yes - a discussion that I'm trying to draw out. Thanks.

> Here my assessment is also different. There are multiple people that have raised concerns that this could hinder deployment.

As you know, we don't assess consensus just by the number of people that have raised concerns, but also the technical merit of the arguments. So far, the arguments have been all over the map, and I want them to converge. 

The data that's been provided has been very helpful, but we need to evaluate that in the context of how QUICv1 will actually be deployed and used -- something which a few threads have touched upon.

Overall, I do not think it is a goal of QUIC to be deployable on any system for any purpose; if we try to design it for all purposes, we're going to fail. The question, then, is how much deployment / implementation / adoption we need to make it succeed -- and to design for that.

> I don’t think to just go with what we have even though there are concerns, is the right approach and definitely not the approach that was chosen for other proposals. In the sake of getting version…

I wasn't proposing that we incorporate a change without consensus -- please re-read my statement. You may disagree with how the chairs eventually call consensus on this issue, and there are mechanisms for you to appeal (or even DISCUSS) that, but it's premature to start that now.

That said -- One way to prove that PNE hurts deployability is to try it; implementers have a very strong incentive to make sure their code can be deployed, so if those who have concerns are correct, it's probably the easiest way for them to be proven right.

Thanks,

---

From: Ian Swett <[email-masked]>
Date: Wed, 04 Apr 2018 21:43:46 +0000

As expressed previously, I tend to think PN encryption is the most
pragmatic way forward at the moment.

However, is anyone who would prefer a different solution(ie: packet number
jumps or separate spaces) willing to send out a PR they think solves the
link-ability issue PN encryption solves?

My mental model is there are a fair number of thorny edge cases, but maybe
if I saw them in a PR, it wouldn't be as scary as I think?

On Wed, Apr 4, 2018 at 5:28 PM Mark Nottingham <[email-masked]> wrote:

> > without intending to state a technical opinion, I think I need to
> disagree with your assessment of the situation. Please see below.
> >
> >> It looks like this thread has come to a natural pause. Reading through
> it, I think we agree there are some unpleasant tradeoffs here, but so far
> we have only one concrete proposal -- PR#1079.
> >
> > The other proposal is to use a random offset that can/should be changed
> when the connection ID is changed. Which is mostly inline with what the
> draft currently says.
>
> As I understand it, the status quo has caused a significant amount of
> pain, complexity and potential bugs. PN encryption was proposed by Kazuho
> in Melbourne as a way to work around that.
>
> We can, of course, decide that the status quo is the right approach.
> However, my current reading of the WG is that it isn't acceptable.
>
>
> >> My AU$.05 - While we're chartered to produce a protocol that's
> encrypted as much as operational concerns allow,
> >
> > I don’t think this is what the charter say. The charter says "Providing
> always-secure transport, using TLS 1.3 by default.“ plus another paragraph
> that talks about TLS1.3. It does not spell out the desire of some
> individual to „encrypt as much as possible“.
>
> I think that depends on how you read "always-secure." The charter also
> says we will provide "confidentiality and integrity protection of both
> application data and QUIC headers."
>
>
> >> and that privacy is a natural extension of that (especially in light of
> RFC7258 and RFC6973), there is no *requirement* that we produce a protocol
> that is able to be accelerated by hardware.
> >
> > Not there is no requirement that the protocol need to be accelerated by
> hardware. However, not all requirement are listed in the charter. We as a
> group have to define the requirement. However, there is of course an
> implicit requirement for all work in the IEFT that is must be deployable
> and serve the needs of those people who want to deploy it.
>
> >> That being the case, I think we need to ask ourselves if we believe
> that inclusion of PR#1079 will significantly inhibit deployment of the
> protocol. Based on the discussion so far, that doesn't seem to be the case,
> but I'd be interested to hear what others think.
> >
> > Here my assessment is also different. There are multiple people that
> have raised concerns that this could hinder deployment.
>
> As you know, we don't assess consensus just by the number of people that
> have raised concerns, but also the technical merit of the arguments. So
> far, the arguments have been all over the map, and I want them to converge.
>
> The data that's been provided has been very helpful, but we need to
> evaluate that in the context of how QUICv1 will actually be deployed and
> used -- something which a few threads have touched upon.
>
> Overall, I do not think it is a goal of QUIC to be deployable on any
> system for any purpose; if we try to design it for all purposes, we're
> going to fail. The question, then, is how much deployment / implementation
> / adoption we need to make it succeed -- and to design for that.
>
>
> > I don’t think to just go with what we have even though there are
> concerns, is the right approach and definitely not the approach that was
> chosen for other proposals. In the sake of getting version v1 out quickly,
> I would think that not incorporating another change that does not have
> consensus is the proposer approach.
>
> I wasn't proposing that we incorporate a change without consensus --
> please re-read my statement. You may disagree with how the chairs
> eventually call consensus on this issue, and there are mechanisms for you
> to appeal (or even DISCUSS) that, but it's premature to start that now.
>
> That said -- One way to prove that PNE hurts deployability is to try it;
> implementers have a very strong incentive to make sure their code can be
> deployed, so if those who have concerns are correct, it's probably the
> easiest way for them to be proven right.
>
> Thanks,
>
>
> > Mirja
> >
> >
> >
> >>
> >> What do folks think?
> >>
> >> Cheers,
> >>
> >>
> >>> As much as I'd love to have easier crypto HW acceleration, I've ended
> up arriving at the same conclusion.  I don't want to bite off the work to
> do proper multipath in QUIC v1, which I think is the only other reasonable
> >>> We may want to creatively use the PN bits in v2, say to encode a path
> ID and a PN, for multipath. We want to retain flexibility in these bits
> going into v2. We've used encryption to ensure that we don't lose
> flexibility elsewhere in the header, and it follows that we should use PNE
> to retain flexibility in these bits as well. (Simplicity of design is the
> other value in using PNE, since handling migration linkability is
> non-trivial without it.)
> >>>
> >>> AFAIK, QUIC has not been used in DC environments yet. I expect there
> are other things in the protocol that we'd want to change as we gain
> experience deploying QUIC in DCs. Spinning up a new version to try QUIC
> within DCs is not only appropriate, I would recommend it. This allows for
> rapid iterations internally, and the experience can drive subsequent
> changes to QUIC. It's what *I* would do if I was to deploy QUIC inside a DC.
> >>>
> >>>> Looking at
> https://en.wikipedia.org/wiki/AES_instruction_set#Intel_and_AMD_x86_architecture
> it seems to imply a large range of server, desktop and mobile chips all
> have a CPU instruction set available to do AES acceleration and other
> similar operations (other instruction sets are also available).
> >>>>
> >>>> If we are considering the AES instructions then it looks like it is
> (or at least will be in the near future) a sizeable proportion of the
> public internet have it to be used.
> >>>>
> >>>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 4 Apr 2018 23:22:25 +0000

PNE is one way forward for doing connection migration without linkability. I wouldn’t call it the most pragmatic since it has performance implications.

Any objections to the following?

1. Leave the current draft as is for the common case (no migration no multipath). Make sure we have some provision for greasing the PN to prevent ossification. If PN jumps are needed for this (apart from a random starting PN) we should have a PR modifying current draft.

2. For connection migration the options are:

a. Explore multiple PN spaces for connection migration (and eventually multipath), OR

b. Make PNE (and hence connection migration) an optional negotiated extension in V1 with the caveat that we will  revisit this  for V2 when we do multipath. This is required anyways because servers must be able to tell client that they cannot support migration since they are behind a legacy load balancer.

From: QUIC [mailto:[email-masked]] On Behalf Of Ian Swett
Sent: Wednesday, April 4, 2018 2:44 PM
To: Mark Nottingham <[email-masked]>
Cc: Lars Eggert <[email-masked]>; IETF QUIC WG <[email-masked]>; Mirja Kühlewind <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

As expressed previously, I tend to think PN encryption is the most pragmatic way forward at the moment.

However, is anyone who would prefer a different solution(ie: packet number jumps or separate spaces) willing to send out a PR they think solves the link-ability issue PN encryption solves?

My mental model is there are a fair number of thorny edge cases, but maybe if I saw them in a PR, it wouldn't be as scary as I think?

On Wed, Apr 4, 2018 at 5:28 PM Mark Nottingham <[email-masked]<mailto:[email-masked]>> wrote:
Mirja,

On 4 Apr 2018, at 8:27 pm, Mirja Kühlewind <[email-masked]<mailto:[email-masked]>> wrote:
>> The editors have told your chairs that this issue is starting to block progress on other aspects of QUIC. Coming to consensus on it soon (i.e.., before Stockholm, if possible) would be good.
>>

Thanks,

> Mirja
>
>
>
>>
>> What do folks think?
>>
>> Cheers,
>>
>>
>>> On 29 Mar 2018, at 11:39 am, Ian Swett <ianswett=[email-masked]<mailto:[email-masked]>> wrote:
>>>
>>>> Looking at https://en.wikipedia.org/wiki/AES_instruction_set#Intel_and_AMD_x86_architecture<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FAES_instruction_set%23Intel_and_AMD_x86_architecture&data=02%7C01%7Cpravb%40microsoft.com%7Cad26f52d9d1a4260b69308d59a753285%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636584750530033325&sdata=yGqWbBMM98tH5A22kBW4KZFhxxBgbDh44MyCKQORND0%3D&reserved=0> it seems to imply a large range of server, desktop and mobile chips all have a CPU instruction set available to do AES acceleration and other similar operations (other instruction sets are also available).
>>>>
>> Mark Nottingham   https://www.mnot.net/<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mnot.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7Cad26f52d9d1a4260b69308d59a753285%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636584750530033325&sdata=ZdSVUlRGeQe85j2DzWmG8JIMTVu0kCKXV%2BMIZo99h0Q%3D&reserved=0>
>>
>

--
Mark Nottingham   https://www.mnot.net/<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mnot.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7Cad26f52d9d1a4260b69308d59a753285%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636584750530033325&sdata=ZdSVUlRGeQe85j2DzWmG8JIMTVu0kCKXV%2BMIZo99h0Q%3D&reserved=0>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 4 Apr 2018 23:22:25 +0000

Thanks,

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 5 Apr 2018 09:27:19 +1000

Hi Praveen,

On 5 Apr 2018, at 9:22 am, Praveen Balasubramanian <[email-masked]> wrote:
> 1. Leave the current draft as is for the common case (no migration no multipath).

Just noting -- one of the things that came up in Melbourne was that migration for the "parking lot problem" -- i.e., wifi to cellular, or vice versa -- *is* in-scope for QUICv1, in many people's minds. 

> Make sure we have some provision for greasing the PN to prevent ossification. If PN jumps are needed for this (apart from a random starting PN) we should have a PR modifying current draft.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 5 Apr 2018 09:27:19 +1000

Hi Praveen,

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 4 Apr 2018 23:39:05 +0000

Yes I am not opposed to including migration in V1 if its a negotiated option. By "leave the draft as is" I meant it for connections that will not migrate. Parking lot problem/ migration  doesn’t apply to: devices connected to just one network, OS policy preventing aggressive use of metered networks, desktops and laptops with just wifi or ethernet, all datacenter servers, and Internet facing servers behind legacy load balancers. It’s not universal.

-----Original Message-----
From: Mark Nottingham [mailto:[email-masked]] 
Sent: Wednesday, April 4, 2018 4:27 PM
To: Praveen Balasubramanian <[email-masked]>
Cc: Ian Swett <ianswett=[email-masked]>; Lars Eggert <[email-masked]>; IETF QUIC WG <[email-masked]>; Mirja Kühlewind <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

Hi Praveen,

> a. Explore multiple PN spaces for connection migration (and eventually 
> multipath), OR b. Make PNE (and hence connection migration) an optional negotiated extension in V1 with the caveat that we will  revisit this  for V2 when we do multipath. This is required anyways because servers must be able to tell client that they cannot support migration since they are behind a legacy load balancer.

--
Mark Nottingham   https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mnot.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7C1ac1708c2f024d3ce96608d59a83a010%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636584812483586746&sdata=pMVsnysCkFFkvsmFjaDwuDWKshPBycW8XDYbA70sU3A%3D&reserved=0

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 4 Apr 2018 23:39:05 +0000

-----Original Message-----
From: Mark Nottingham [mailto:[email-masked]] 
Sent: Wednesday, April 4, 2018 4:27 PM
To: Praveen Balasubramanian <[email-masked]>
Cc: Ian Swett <ianswett=[email-masked]>; Lars Eggert <[email-masked]>; IETF QUIC WG <[email-masked]>; Mirja Kühlewind <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

Hi Praveen,

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 5 Apr 2018 01:40:26 +0200

Without pro / con anything:

Optional privacy does not work. In part it cannot be retrofitted when the
need arise, in part it can be incriminating to enable.

On 5 April 2018 at 01.22.38, Praveen Balasubramanian (
pravb=[email-masked]) wrote:

Make PNE (and hence connection migration) an optional negotiated extension
in V1

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 5 Apr 2018 01:40:26 +0200

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 4 Apr 2018 23:42:21 +0000

What is the privacy issue if you are not doing migration? Migration + PNE (or multiple PN spaces) should go hand in hand.

From: QUIC [mailto:[email-masked]] On Behalf Of Mikkel Fahnøe Jørgensen
Sent: Wednesday, April 4, 2018 4:40 PM
To: Praveen Balasubramanian <pravb=[email-masked]>; Ian Swett <ianswett=[email-masked]>; Mark Nottingham <[email-masked]>
Cc: Lars Eggert <[email-masked]>; IETF QUIC WG <[email-masked]>; Mirja Kühlewind <[email-masked]>
Subject: RE: Getting to consensus on packet number encryption

Optional privacy does not work. In part it cannot be retrofitted when the need arise, in part it can be incriminating to enable.

On 5 April 2018 at 01.22.38, Praveen Balasubramanian (pravb=[email-masked]<mailto:pravb=[email-masked]>) wrote:
Make PNE (and hence connection migration) an optional negotiated extension in V1

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 4 Apr 2018 23:42:21 +0000

---

From: Christian Huitema <[email-masked]>
Date: Wed, 4 Apr 2018 17:44:26 -0700

If we are in the business of repeating our positions, so be it. I am
pretty much on the same line as Patrick and others:

1) Privacy is not optional. And yes, removing "linkability through PN"
is important, because when the connection ID and the source address also
change, the only other cue for linkability is timing. Timing can be
addressed by having overlapping connections.

2) Packet number encryption is the simplest of the many options
presented so far. It is in particular much simpler to implement than the
"predictable offset" scheme presented in previous drafts.

3) I am happy to discuss alternatives that appear simpler to implement
in hardware, but so far the only plausible alternative is to insert an
extra nonce in the packet header. Given birthday paradoxes and all that,
this probably requires 8 extra bytes of overhead per packet, which is
not very attractive.

-- Christian Huitema

On 4/4/2018 4:42 PM, Praveen Balasubramanian wrote:
>
> What is the privacy issue if you are not doing migration? Migration +
> PNE (or multiple PN spaces) should go hand in hand.
>
>  
>
> *From:* QUIC [mailto:[email-masked]] *On Behalf Of *Mikkel
> Fahnøe Jørgensen
> *Sent:* Wednesday, April 4, 2018 4:40 PM
> *To:* Praveen Balasubramanian <pravb=[email-masked]>;
> Ian Swett <ianswett=[email-masked]>; Mark Nottingham
> <[email-masked]>
> *Cc:* Lars Eggert <[email-masked]>; IETF QUIC WG <[email-masked]>;
> Mirja Kühlewind <[email-masked]>
> *Subject:* RE: Getting to consensus on packet number encryption
>
>  
>
> Optional privacy does not work. In part it cannot be retrofitted when
> the need arise, in part it can be incriminating to enable.
>
>  
>
> On 5 April 2018 at 01.22.38, Praveen Balasubramanian
> (pravb=[email-masked]
> <mailto:pravb=[email-masked]>) wrote:
>

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 5 Apr 2018 09:57:38 +0900

I support adoption of #1079 for three reasons.

One reason is that privacy is a required feature as Patrick has
suggested. The other reason is simplicity as others have pointed out.

The third reason is that without PNE there is a certain risk of QUIC
failing to reach one of the chartered goals: preventing head-of-line
blocking.

In TCP, we have seen middleboxes trying to do clever things using the
way sequence number increases, including packet reordering.

Unless we encrypt PN in QUIC, there is fair chance that middleboxes
would start doing the same. A middlebox reordering (or queuing) the
QUIC packets by observing PN *is* causing head-of-line blocking since
it prevents (or delays) packets being processed in different order at
the receiver at the earliest possible moment.

Note that middlebox vendors are known to make certain assumptions
based on what they see on the wire rather than what is written down in
the specification. Therefore, stating that middlebox should not do
that is not a solution here.

It is true that #1079 has issues with hardware crypto offload. I am
happy to support switching to a better encryption scheme (if we find
one) at any moment; e.g., during the standardization of QUICv1, or as
an extension to v1, or part of vX.

But as stated, I think that PNE is a MUST for us. That makes me prefer
adopting #1079 and then, if possible, swaping it with a better
solution.

2018-04-04 21:54 GMT+09:00 Patrick McManus <[email-masked]>:
> either a multipath-pn-scheme (which we'll table for v2) or a one pass scheme
> with a new nonce rather than making the packet number do double duty. The
> latter is bandwidth tax and I'd rather pay the cpu cost.
>
>> Based on the discussion so far, that doesn't seem to be the case, but I'd be
>> interested to hear what others think.
>>
>> > On 29 Mar 2018, at 11:39 am, Ian Swett
>> > <ianswett=[email-masked]> wrote:
>> >
>> > complexity (as Christian notes). This complexity may be warranted when doing
>> > multipath in v2 or later, but I'm not convinced that this is necessary as a
>> > design primitive for QUICv1.
>> >
>> > acceleration will be in DC environments. Yes, there are some gains to be had
>> > in serving the public Internet as well, but I'm unconvinced that this is the
>> > driving use case for hardware acceleration. I understand that others may
>> > disagree with me here.
>> >
>> > not only appropriate, I would recommend it. This allows for rapid iterations
>> > internally, and the experience can drive subsequent changes to QUIC. It's
>> > what *I* would do if I was to deploy QUIC inside a DC.
>> >
>> >> it seems to imply a large range of server, desktop and mobile chips all have
>> >> a CPU instruction set available to do AES acceleration and other similar
>> >> operations (other instruction sets are also available).
>> >>
>> > compatible with use of the AES instructions. The issue of the debate is that
>> > the mechanism in PR #1079 required double buffering, first encrypt the

-- 
Kazuho Oku

---

From: Martin Thomson <[email-masked]>
Date: Thu, 5 Apr 2018 14:35:34 +1000

> an extension to v1, or part of vX.

Until this came up, I hadn't considered the use of an
extension/transport parameter to negotiate a new scheme, but it's
obvious.  This fits best with my view of things.

If that means that we get the DC-QUIC option that turns the packet
number encryption function into identity or some cheaper function,
then I think that's an OK outcome.  We should stop pretending that one
size fits all is achievable; it's not been the case for years already.

FWIW, what I got from the info that Praveen and Ian have shared is
that there are enough problems to solve with UDP that worrying about
the performance hit from encryption is premature.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 5 Apr 2018 08:00:46 +0200

When I said no pro / con, I meant: if you think that PN is always required
for privacy, then it does not work to negotiate. If everyone agrees that
that is not an issue, then negotiation is an option. It may then be tied
with additional requirements such as never supporting migration or NAT
rebinding and requiring termination if that is violated. If everyone does
not agree on this, then negotiation is not a solution the problem because
privacy is not negotiable.

Personally I believe privacy is not an issue if you isolate PN, but even if
many might agree on that the might still argue it is difficult to do
(correctly).

There is also the concern raised by some that packet reordering in transit
could happen without PN - this is always an issue - but is it worth the
cost, and is it still a real concern for a protocol that was not designed
to be strictly ordered? How does WebRTC work in this context?

On 5 April 2018 at 01.42.23, Praveen Balasubramanian ([email-masked])
wrote:

What is the privacy issue if you are not doing migration? Migration + PNE
(or multiple PN spaces) should go hand in hand.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 5 Apr 2018 08:00:46 +0200

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 5 Apr 2018 08:10:12 +0200

On Thu, Apr 05, 2018 at 07:28:10AM +1000, Mark Nottingham wrote:
> Overall, I do not think it is a goal of QUIC to be deployable on any system
> for any purpose; if we try to design it for all purposes, we're going to
> fail. The question, then, is how much deployment / implementation / adoption
> we need to make it succeed -- and to design for that.

I totally agree with you here. It's also why my -1 is a weak one (rather a
warning than a -1 in fact because I consider that robustness is more important
than prevention against linkability). we know that whatever we'll produce,
solutions will adapt if they want to adopt it, and if the the risks are too
high for adopters (such as increased sensitivity do DDoS), adopters will
simply roll back for the time it takes to find a solution.

> easiest way for them to be proven right.

I disagree on this approach. That's exactly what led us to release HPACK
with an inefficient encoding and with unsorted header fields. We all know
that deployment speed varies a lot between participants, and that those
with most concerns are also those with less ability to prove their concerns.
Specifically, single-ended implementations often face less issues than
intermediaries and are much easier to implement. And once the least concerned
ones consider that it works well enough for them, the protocol is locked and
it's forbidden to improve or even it. We need not to repeat our mistakes.

I think that the direction taken in this thread is more pragmatic : technical
and philosophical concerns are exchanged, weighted and the decision will be
taken based on this. Regarding the performance issues, several of us noted
that there are options by adopting negociable algorithms, or as I proposed,
to use a single AES round instead of all of them, which will be reasonably
cheap on modern systems.

Cheers,
Willy

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Thu, 5 Apr 2018 08:53:31 +0200

> On 5 Apr 2018, at 06:35, Martin Thomson <[email-masked]> wrote:
> 

other than "please, please don't call it DC-QUIC", this approach (packet number encoding/encryption scheme as a transport parameter) sounds like a very good approach to me. There are details here, of course...

> the performance hit from encryption is premature.

This was my intuition as well.

However, fixing kernel interface issues (as are at the root of UDP performance problems, as well as the lack of ECN signaling from userland) takes time, so those of us with hands on kernels should be working to fix those issues we *know* we will have should be working to fix them now. Fixing hardware issues -- let's face it, we'll probably never have full transport offload for QUIC, given the flexibility at the core of its design philosophy, but crypto offload is a good target -- takes longer, so anything we think we need for V2/V3, we need to start thinking about now.

IOW it's all premature, but the prematurity in this case is an externality, unless we think we know enough to decide that crypto offload isn't worth it for QUIC. (Obviously, I don't think we do)

Cheers,

Brian

---

From: Lars Eggert <[email-masked]>
Date: Thu, 5 Apr 2018 09:01:25 +0200

Hi,

On 2018-4-5, at 8:53, Brian Trammell (IETF) <[email-masked]> wrote:
> However, fixing kernel interface issues (as are at the root of UDP performance problems, as well as the lack of ECN signaling from userland) takes time

FWIW, ECN signaling is mostly only an issue on Windows at the moment. Most other stacks can do this via ancillary data on sendmsg/recvmsg. I have tested Linux, FreeBSD and Darwin and they work (modulo a bug in Darwin on TX that Apple is looking into.)

Lars

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Thu, 5 Apr 2018 09:02:58 +0200

> On 5 Apr 2018, at 09:01, Lars Eggert <[email-masked]> wrote:
> 
> Hi,
> 

Ah, I thought this was a more widely spread problem. Okay, cool, we can focus more on making UDP not suck then. :)

Thanks, cheers,

Brian

---

From: <[email-masked]>
Date: Thu, 5 Apr 2018 09:24:39 +0200

On 05/04/2018 02:44, Christian Huitema wrote:
> 3) I am happy to discuss alternatives that appear simpler to implement in
> hardware, but so far the only plausible alternative is to insert an extra nonce
> in the packet header. Given birthday paradoxes and all that, this probably
> requires 8 extra bytes of overhead per packet, which is not very attractive.

Can you quantify "not very attractive" ? Given the complex ramifications of the 
alternatives, is it wise to shy away from an 8-byte overhead ? TCP options like 
timestamps weigh the same and survived...

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 5 Apr 2018 00:42:41 -0700

On Wed, Apr 4, 2018 at 5:44 PM, Christian Huitema <[email-masked]>
wrote:

> 1) Privacy is not optional. And yes, removing "linkability through PN" is
> important, because when the connection ID and the source address also
> 2) Packet number encryption is the simplest of the many options presented
> so far. It is in particular much simpler to implement than the "predictable
> offset" scheme presented in previous drafts.
>
> hardware, but so far the only plausible alternative is to insert an extra
> nonce in the packet header. Given birthday paradoxes and all that, this
> probably requires 8 extra bytes of overhead per packet, which is not very
> attractive.
>
8 seems pretty close to the line unless we are going to rekey *very*
frequently. Probably more like 12 bytes.

In any case, I concur that we should do PN encryption and IMO PR#1079 is
the best -- or at least close to the best -- design we have. I would prefer
to land it, and then if we find something better later, we can improve it
then.

-Ekr

-- Christian Huitema
> *From:* QUIC [mailto:[email-masked] <[email-masked]>] *On
> Behalf Of *Mikkel Fahnøe Jørgensen
> *To:* Praveen Balasubramanian <pravb=[email-masked]>
> <pravb=[email-masked]>; Ian Swett
> <ianswett=[email-masked]>
> <ianswett=[email-masked]>; Mark Nottingham <[email-masked]>
> <[email-masked]>
> *Cc:* Lars Eggert <[email-masked]> <[email-masked]>; IETF QUIC WG
> <[email-masked]> <[email-masked]>; Mirja Kühlewind
> <[email-masked]> <[email-masked]>

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 5 Apr 2018 16:56:36 +0900

2018/04/05 15:00、Mikkel Fahnøe Jørgensen <[email-masked]>のメール:

> There is also the concern raised by some that packet reordering in transit could happen without PN - this is always an issue - but is it worth the cost, and is it still a real concern for a protocol that was not designed to be strictly ordered?

I agree that reordering by middleboxes *might not* happen. But I would also argue that we should encrypt the PN if we think reordering *might* happen.

If we encrypt PN now, we will have the chance to revisit the design in the future versions of QUIC. But if we ship without PNE and then find out middleboxes start using the PN under the assumption that they are sent in-order, we would forever be forced to stick to them sent in-order, in cleartext. 

Ossification closes the door to future modifications.

> How does WebRTC work in this context?
> 
> 
>> On 5 April 2018 at 01.42.23, Praveen Balasubramanian ([email-masked]) wrote:
>>

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 5 Apr 2018 11:26:53 +0200

Let me repeat another position then as well:

> Am 05.04.2018 um 02:44 schrieb Christian Huitema <[email-masked]>:
> 
> If we are in the business of repeating our positions, so be it. I am pretty much on the same line as Patrick and others:
> 
> 1) Privacy is not optional. And yes, removing "linkability through PN" is important, because when the connection ID and the source address also change, the only other cue for linkability is timing. Timing can be addressed by having overlapping connections.
> 
Timing doesn’t help here either because you still have the same destination IP, both port numbers and the fact that a migrated connection does not have a handshake. If we want to address the linkability problem, we would need to do much more, probably baring more hits on performance. 

I thought we discussed this already endlessly and concluded that PNE does not help linkability.

Then an argument for ossification was broad up. However, we really cannot compare the situation here with TCP. With TCP everything is in the clear incl. also the retransmissions. In case of MPTCP this was rather the problem the the seq# because if you’d use the same seq# number space on both connections it would like loss but without a retransmission. Further boxes that reorder, usually do that in situation where packets were transmitted for a known reason out-of order previously, e.g. when multiple paths are used in the network. The recommendation we still give in the IETF transport area is that in such a situation the scheduling should not be done on a per-packet level but per-flow. However, those scheme that do per-packet scheduling usually also apply mechanism to re-order afterwards. They can just use the TCP seq# for that but for non-TCP they would simply add their own encapsulation with a seq#…

Please keep in mind that even if an (unencrypted) PN gets ossified by the network,
in case of QUIC that does not mean that we can’t change the transport machinery anymore. Currently the exposed PN is also used for congestion control and retransmission, however, we can still add another PN in the encrypted part and resolved the fact that the current PN is overloaded for different functions.

Currently, I really still don’t see the benefit of PNE but only additional complexity and known performance degradation.

Mirja

---

From: "Philipp S. Tiesel" <[email-masked]>
Date: Thu, 5 Apr 2018 11:57:27 +0200

> On 5. Apr 2018, at 09:56, Kazuho Oku <[email-masked]> wrote:
> 
> 2018/04/05 15:00、Mikkel Fahnøe Jørgensen <[email-masked] <mailto:[email-masked]>>のメール:
> 

For the case of the PN, I also see this problem, but (inspired by Brian), 
I started thinking of this from a different perspective:

What if we think of the "clear PN" as a path signal, which is coincidentally used as PN in the transport in v1.
For me, the consequences seem the following:
 - It instantly becomes somewhat of an invariant, for which we can clearly define
   behaviour (monotonically increasing, jumps infrequent, …) 
 - Need to design the "clear PN” for space efficiency – use a variable length
   integer that can reduce to one byte if we have a "transport PN" in the encrypted
   part in v2

Maybe a “faster” way forward without needing to feel ossified.

>> On 5 April 2018 at 01.42.23, Praveen Balasubramanian ([email-masked] <mailto:[email-masked]>) wrote:
>> 

AVE!
  Philipp S. Tiesel / phils…
-- 
   {phils}--->---([email-masked])--->---(http://phils.in-panik.de)----,
      wenn w eine   aube ist dn      man au dran dre en                   |
           o     Schr        an muss     hc         h   (Kurt Schwitters) |
:wq!  <----(phone: +49-179-6737439)---<---(jabber: [email-masked])----'

---

From: Christian Huitema <[email-masked]>
Date: Thu, 5 Apr 2018 08:34:14 -0700

> On Apr 5, 2018, at 2:26 AM, Mirja Kühlewind <[email-masked]> wrote:
> ...

Mirja,

You rightly point out that the connection ID and the Packet Number are not the only elements that provide linkability. There is also of course the UDP source port. That one is not much of an issue for servers, but it is an issue for clients. We are not spelling that out in the draft. We should, because clients can trivially close that hole when doing migration.

I am not sure that the absence of negotiation divulges much data. It marks the path as originating from a migration, but it does not tell from where. But there might be an ossification issue. We will get that ossification if we train middleboxes to believe that connection always start with an observable negotiation. So maybe we should explore ways to grease that.

Any other privacy hole that we should fix?

-- Christian Huitema

---

From: Frederick Kautz <[email-masked]>
Date: Thu, 05 Apr 2018 15:58:58 +0000

Are you concerned that middleware boxes may be trained to reject
migrations, thereby forcing a new connection with a visible negotiation?

On Thu, Apr 5, 2018, 8:34 AM Christian Huitema <[email-masked]> wrote:

> > Timing doesn’t help here either because you still have the same
> destination IP, both port numbers and the fact that a migrated connection
> does not have a handshake. If we want to address the linkability problem,
> we would need to do much more, probably baring more hits on performance.
>
> Mirja,
>
> You rightly point out that the connection ID and the Packet Number are not
> the only elements that provide linkability. There is also of course the UDP
> source port. That one is not much of an issue for servers, but it is an
> issue for clients. We are not spelling that out in the draft. We should,
> because clients can trivially close that hole when doing migration.
>
> I am not sure that the absence of negotiation divulges much data. It marks
> the path as originating from a migration, but it does not tell from where.
> But there might be an ossification issue. We will get that ossification if
> we train middleboxes to believe that connection always start with an
> observable negotiation. So maybe we should explore ways to grease that.
>

---

From: Christian Huitema <[email-masked]>
Date: Thu, 5 Apr 2018 09:08:36 -0700

> On Apr 5, 2018, at 8:58 AM, Frederick Kautz <[email-masked]> wrote:
> 
> Are you concerned that middleware boxes may be trained to reject migrations, thereby forcing a new connection with a visible negotiation?

Yes. Hence the need to grease. For example, have clients do some gratuitous migration to a new source port number rather frequently.

-- Christian Huitema

---

From: Praveen Balasubramanian <[email-masked]>
Date: Thu, 5 Apr 2018 16:41:54 +0000

Love this idea. Expanding it a bit further I don’t think we need to make PN transform a transport parameter but rather connection migration itself. We could also simply use QUIC versioning for this. 

1. Ship QUIC v1 with no support for connection migration. Servers behind load balancers that are not QUIC aware will only advertise this version. I expect this to be a large fraction of servers especially those hosted in AWS, Azure, GCP. *This is important - I want the group to acknowledge that servers needs to be able to advertise support for connection migration*.
2. Ship QUIC v2 with support for connection migration. If a client and server use this version they MUST do PNE as per #1079. If we find a more efficient scheme before November then we use that instead. 

Eventually we will ship QUIV vN which has support for multi-path and we can revisit if PNE is needed or multiple PN spaces will work. Until then I can live with higher CPU cost for the fraction of connections that will migrate. 

Now there is the ossification problem in QUIC v1 as referred to above. We already require a random starting PN. The next problem is one that Kazuho brought up where middleboxes will start seeing strict increasing PN and try to reorder. I was thinking about short random jumps to fix this problem but the monotonic increase will remain. While I think sane middleboxes will not do any reordering, there might be a few crazy outliers like with TCP. I think we can solve this by making the in clear PN jump both forward and back within a "reordering degree". I am thinking about some ideas here and will reply back if I find a simple scheme.

>> FWIW, what I got from the info that Praveen and Ian have shared is that there are enough problems to solve with UDP that worrying about the performance hit from encryption is premature
Martin, we have plans to make UDP screaming fast in software. But the real advantage for TCP comes from the batch offloads (see email with subject " Impact of hardware offloads on network stack performance"). Batch offloads for QUIC are impossible to do without crypto offload. I would be happy to explain more on this front. I think we will exhaust software optimizations soon and then get blocked on crypto offload.

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Martin Thomson
Sent: Wednesday, April 4, 2018 9:36 PM
To: Kazuho Oku <[email-masked]>
Cc: Lars Eggert <[email-masked]>; Mark Nottingham <[email-masked]>; IETF QUIC WG <[email-masked]>; Patrick McManus <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

> an extension to v1, or part of vX.

Until this came up, I hadn't considered the use of an extension/transport parameter to negotiate a new scheme, but it's obvious.  This fits best with my view of things.

If that means that we get the DC-QUIC option that turns the packet number encryption function into identity or some cheaper function, then I think that's an OK outcome.  We should stop pretending that one size fits all is achievable; it's not been the case for years already.

FWIW, what I got from the info that Praveen and Ian have shared is that there are enough problems to solve with UDP that worrying about the performance hit from encryption is premature.

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 5 Apr 2018 17:57:20 +0100

> Yes. Hence the need to grease. For example, have clients do some gratuitous migration to a new source port number rather frequently.
This concern about "migration", seems an odd concern to me - and that 
probably stems from a very different perspective. My desire to see 
numbers here is to allow for toolchains to plot, analyse, connection 
progress and a variety of other traffic characteristics - complimenting 
the SPIN bit(s); I see it can also provide opportunities to leverage 
what we know about using hardware offload fro transport processing (and 
I expect potentially crypto), etc.

Yes potentially such numbers could be used to help build connection 
state in CGNs or whatever middlebox that wants to see connections 
continue to progress (but then they can also count packets right now t 
do that). To me such use is totally independent of how QUIC chooses to 
use these numbers internally in future, and in no way stops the QUIC 
protocol evolving.

My point is that I keep hearing issues: e.g. the potential opportunities 
for discovering linkability - yet the uses I had in mind do not need to 
be per-path, for these topics there could just as well be one sequence 
space per "path" being used.

Gorry

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 6 Apr 2018 02:18:54 +0900

> Yes. Hence the need to grease. For example, have clients do some gratuitous migration to a new source port number rather frequently.

I agree that it might be a good idea to grease.

OTOH I do not think that gratuitous migration would be a good solution
due to performance issues. We would be required to start from INITCWND
every time the client switches to a new address. The fact that we
cannot run two paths simultaneously in QUIC v1 makes the performance
drawback big.

Rather, I'd prefer making handshake packets indistinguishable from
short header packets. In essence, you move the flags of the long
header inside the AEAD-protected area. An endpoint that receives a
packet carrying a CID that does not belong to any known connection
trial-decrypts the packet as a initial packet and handles it as a
connection initiation, or sends a stateful reset if the
trial-decryption fails.

-- 
Kazuho Oku

---

From: Mike Bishop <[email-masked]>
Date: Thu, 5 Apr 2018 17:27:10 +0000

Spec says:

An endpoint MUST NOT return to the send rate used for the previous path unless it is reasonably sure that the previous send rate is valid for the new path. For instance, a change in the client’s port number is likely indicative of a rebinding in a middlebox and not a complete change in path. This determination likely depends on heuristics, which could be imperfect; if the new path capacity is significantly reduced, ultimately this relies on the congestion controller responding to congestion signals and reducing send rates appropriately.

You're not required to reset the CWND if you have reason to believe the underlying path is the same, so doing this shouldn’t cause any transport information to get reset.

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Kazuho Oku
Sent: Thursday, April 5, 2018 10:19 AM
To: Christian Huitema <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>; Mirja Kühlewind <[email-masked]>; Frederick Kautz <[email-masked]>
Subject: Re: Privacy holes (was: Re: Getting to consensus on packet number encryption)

2018-04-06 1:08 GMT+09:00 Christian Huitema <[email-masked]<mailto:[email-masked]>>:

>> On Apr 5, 2018, at 8:58 AM, Frederick Kautz <[email-masked]<mailto:[email-masked]>> wrote:

OTOH I do not think that gratuitous migration would be a good solution due to performance issues. We would be required to start from INITCWND every time the client switches to a new address. The fact that we cannot run two paths simultaneously in QUIC v1 makes the performance drawback big.

Rather, I'd prefer making handshake packets indistinguishable from short header packets. In essence, you move the flags of the long header inside the AEAD-protected area. An endpoint that receives a packet carrying a CID that does not belong to any known connection trial-decrypts the packet as a initial packet and handles it as a connection initiation, or sends a stateful reset if the trial-decryption fails.

--

Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 5 Apr 2018 10:30:55 -0700

Hi

On 5 April 2018 at 19.27.24, Mike Bishop ([email-masked]) wrote:

 An endpoint that receives a packet carrying a CID that does not belong to
any known connection trial-decrypts the packet as a initial packet and
handles it as a connection initiation, or sends a stateful reset if the
trial-decryption fails.

I can see hiding things may be a good thing, but having to trial decrypt
all unknown packets as potentially creating a huge load, unless you combine
it with my proposal to add a checksum to the encrypted packet number so you
only need one crypto block on average to reject random packet content.

It won’t solve all DDoS, but it will help. Forcing trial decryption fo the
full packet makes me a bit anxious.

Mikkel

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 6 Apr 2018 02:43:23 +0900

2018-04-06 2:27 GMT+09:00 Mike Bishop <[email-masked]>:
> Spec says:
>
> An endpoint MUST NOT return to the send rate used for the previous path
> unless it is reasonably sure that the previous send rate is valid for the
> new path. For instance, a change in the client’s port number is likely
> indicative of a rebinding in a middlebox and not a complete change in path.
> This determination likely depends on heuristics, which could be imperfect;
> if the new path capacity is significantly reduced, ultimately this relies on
> the congestion controller responding to congestion signals and reducing send
> rates appropriately.
>
>
>
> You're not required to reset the CWND if you have reason to believe the
> underlying path is the same, so doing this shouldn’t cause any transport
> information to get reset.

Thank you for pointing that out. I missed that.

OTOH, I still wonder if frequent gratuitous migration is a good solution.

I'd assume that in case of users behind a large-scale NAT connecting
to a CDN, there would be many connections sharing the same client-IP,
server-IP, server-port. The only difference in the 5-tuple is the
clients' port.

If we require frequent gratuitous migration, wouldn't either the NAT
will run out of port addresses belonging to the same client-IP or
required to switch to a different IP address?

> Cc: IETF QUIC WG <[email-masked]>; Mirja Kühlewind
> <[email-masked]>; Frederick Kautz <[email-masked]>
>> Yes. Hence the need to grease. For example, have clients do some
>> gratuitous migration to a new source port number rather frequently.
>
>
>
> OTOH I do not think that gratuitous migration would be a good solution due
> to performance issues. We would be required to start from INITCWND every
> time the client switches to a new address. The fact that we cannot run two
> paths simultaneously in QUIC v1 makes the performance drawback big.
>
>
>
> Rather, I'd prefer making handshake packets indistinguishable from short
> header packets. In essence, you move the flags of the long header inside the
> AEAD-protected area. An endpoint that receives a packet carrying a CID that
> does not belong to any known connection trial-decrypts the packet as a
> initial packet and handles it as a connection initiation, or sends a
> stateful reset if the trial-decryption fails.
>
>
>
>> -- Christian Huitema
>
>
>
>
>
>
>
> --
>
> Kazuho Oku
>
>

-- 
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 6 Apr 2018 02:50:58 +0900

2018-04-06 2:30 GMT+09:00 Mikkel Fahnøe Jørgensen <[email-masked]>:
> I can see hiding things may be a good thing, but having to trial decrypt all
> unknown packets as potentially creating a huge load, unless you combine it
> with my proposal to add a checksum to the encrypted packet number so you

While you might be correct in pointing out that requiring
trial-decryption is not a good thing in general, I think it is an
orthogonal issue to whether if we can move the packet type bits into
AEAD-encrypted area.

In the current form of QUIC, a server is required to AES-decrypt the
Initial packet. So it is possible to mount a brute-force attack
targeting the symmetric crypto by sending some junk together with a
packet header that looks like a Initial packet.

The approach I proposed merely moves when the packet type is being
checked, from prior-to-decryption to post-decryption. Maybe I should
have explained it this way. Sorry if that has caused the confusion.

-- 
Kazuho Oku

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 5 Apr 2018 17:59:04 +0000

*   I'd prefer making handshake packets indistinguishable from short header packets. In essence, you move the flags of the long header inside the AEAD-protected area. An endpoint that receives a packet carrying a CID that does not belong to any known connection trial-decrypts the packet as a initial packet and handles it as a connection initiation, or sends a stateful reset if the trial-decryption fails.
This change would wreck stateless load balancers that rely on being able to distinguish CI packets (and route them based on IPs) from other packets (and route them based on CID).

- Igor

From: Mikkel Fahnøe Jørgensen [mailto:[email-masked]]
Sent: Thursday, April 05, 2018 1:31 PM
To: Mike Bishop <[email-masked]>; Christian Huitema <[email-masked]>; Kazuho Oku <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>; Mirja Kühlewind <[email-masked]>; Frederick Kautz <[email-masked]>
Subject: RE: Privacy holes (was: Re: Getting to consensus on packet number encryption)

Hi

On 5 April 2018 at 19.27.24, Mike Bishop ([email-masked]<mailto:[email-masked]>) wrote:

 An endpoint that receives a packet carrying a CID that does not belong to any known connection trial-decrypts the packet as a initial packet and handles it as a connection initiation, or sends a stateful reset if the trial-decryption fails.

I can see hiding things may be a good thing, but having to trial decrypt all unknown packets as potentially creating a huge load, unless you combine it with my proposal to add a checksum to the encrypted packet number so you only need one crypto block on average to reject random packet content.

It won’t solve all DDoS, but it will help. Forcing trial decryption fo the full packet makes me a bit anxious.

Mikkel

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 6 Apr 2018 03:09:13 +0900

2018-04-06 2:59 GMT+09:00 Lubashev, Igor <[email-masked]>:

>
>    - I'd prefer making handshake packets indistinguishable from short
>    header packets. In essence, you move the flags of the long header inside
>    the AEAD-protected area. An endpoint that receives a packet carrying a CID
>    that does not belong to any known connection trial-decrypts the packet as a
> This change would wreck stateless load balancers that rely on being able
> to distinguish CI packets (and route them based on IPs) from other packets
> (and route them based on CID).
>

It would not, *if* the stateless load balancer can AES-decrypt the packet
(using the key derived from the packet header) to see if the packet is in
fact a initial packet. Or another way of solving the issue will be to
always route the packet based on CID, then forward the packet between the
endpoints behind the load balancer.

I agree that it would be a no-deal for stateless load balancers that are
incapable of doing such things.

>
>
> - Igor
>
>
>
> *From:* Mikkel Fahnøe Jørgensen [mailto:[email-masked]]
> *Sent:* Thursday, April 05, 2018 1:31 PM
> *To:* Mike Bishop <[email-masked]>; Christian Huitema <
> [email-masked]>; Kazuho Oku <[email-masked]>
> *Cc:* IETF QUIC WG <[email-masked]>; Mirja Kühlewind <
> [email-masked]>; Frederick Kautz <[email-masked]>
> *Subject:* RE: Privacy holes (was: Re: Getting to consensus on packet
> number encryption)
>
>
>
> Hi
>
>
>
>  An endpoint that receives a packet carrying a CID that does not belong
> to any known connection trial-decrypts the packet as a initial packet and

-- 
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 6 Apr 2018 03:14:00 +0900

> incapable of doing such things.

PS. Note that the purpose of AES-encrypting the handshake packets is purely
for obfuscation. It might be worth considering changing it to some
snake-oil scrambling scheme if it helps stateless load balancer adopt the
proposed scheme.

-- 
Kazuho Oku

---

From: Praveen Balasubramanian <[email-masked]>
Date: Thu, 5 Apr 2018 18:19:22 +0000

IMO we should not have a design that forces load balancers to decrypt.

From: QUIC [mailto:[email-masked]] On Behalf Of Kazuho Oku
Sent: Thursday, April 5, 2018 11:09 AM
To: Lubashev, Igor <[email-masked]>
Cc: Mirja Kühlewind <[email-masked]>; huitema <[email-masked]>; Mike Bishop <[email-masked]>; Mikkel Fahnøe Jørgensen <[email-masked]>; IETF QUIC WG <[email-masked]>; Frederick Kautz <[email-masked]>
Subject: Re: Privacy holes (was: Re: Getting to consensus on packet number encryption)

2018-04-06 2:59 GMT+09:00 Lubashev, Igor <[email-masked]<mailto:[email-masked]>>:

It would not, *if* the stateless load balancer can AES-decrypt the packet (using the key derived from the packet header) to see if the packet is in fact a initial packet. Or another way of solving the issue will be to always route the packet based on CID, then forward the packet between the endpoints behind the load balancer.

I agree that it would be a no-deal for stateless load balancers that are incapable of doing such things.

- Igor

From: Mikkel Fahnøe Jørgensen [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Thursday, April 05, 2018 1:31 PM
To: Mike Bishop <[email-masked]<mailto:[email-masked]>>; Christian Huitema <[email-masked]<mailto:[email-masked]>>; Kazuho Oku <[email-masked]<mailto:[email-masked]>>
Cc: IETF QUIC WG <[email-masked]<mailto:[email-masked]>>; Mirja Kühlewind <[email-masked]<mailto:[email-masked]>>; Frederick Kautz <[email-masked]<mailto:[email-masked]>>
Subject: RE: Privacy holes (was: Re: Getting to consensus on packet number encryption)

Hi

Mikkel

--
Kazuho Oku

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 5 Apr 2018 18:21:00 +0000

*   It would not, *if* the stateless load balancer can AES-decrypt the packet (using the key derived from the packet header) to see if the packet is in fact a initial packet.

Would not it need to first decide that the CID in the packet is unknown (which means stateful)?  It would need to do some integrity check on CID, which would require a large nonce to avoid accidentally classifying a random CID as valid, because the cost of this mistake can be very large for the client (see below).

  *   Or another way of solving the issue will be to always route the packet based on CID, then forward the packet between the endpoints behind the load balancer.

This would not really work for a global load balancer – the one capable of forwarding across a continent to a different anycast pop (due to client’s change in attachment point or BGP reconvergence).

From: Kazuho Oku [mailto:[email-masked]]
Sent: Thursday, April 05, 2018 2:09 PM
To: Lubashev, Igor <[email-masked]>
Cc: Mikkel Fahnøe Jørgensen <[email-masked]>; Mike Bishop <[email-masked]>; Christian Huitema <[email-masked]>; IETF QUIC WG <[email-masked]>; Mirja Kühlewind <[email-masked]>; Frederick Kautz <[email-masked]>
Subject: Re: Privacy holes (was: Re: Getting to consensus on packet number encryption)

- Igor

Cc: IETF QUIC WG <[email-masked]<mailto:[email-masked]>>; Mirja Kühlewind <[email-masked]<mailto:[email-masked]>>; Frederick Kautz <[email-masked]<mailto:[email-masked]>>

Hi

Mikkel

--
Kazuho Oku

---

From: Praveen Balasubramanian <[email-masked]>
Date: Thu, 5 Apr 2018 18:27:08 +0000

I’d like to clarify that when I said v1 and v2 that we will sim ship them in the same RFC. I did not intend to suggest moving migration out of scope. And then clients that want to migrate and servers that support migration will negotiate v2 and rest of the cases will  be v1.

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Praveen Balasubramanian
Sent: Thursday, April 5, 2018 9:42 AM
To: Martin Thomson <[email-masked]>; Kazuho Oku <[email-masked]>
Cc: Lars Eggert <[email-masked]>; Mark Nottingham <[email-masked]>; IETF QUIC WG <[email-masked]>; Patrick McManus <[email-masked]>
Subject: RE: Getting to consensus on packet number encryption

>> the performance hit from encryption is premature

---

From: Praveen Balasubramanian <[email-masked]>
Date: Thu, 5 Apr 2018 18:27:08 +0000

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 5 Apr 2018 18:27:23 +0000

Ø  Or another way of solving the issue will be to always route the packet based on CID, then forward the packet between the endpoints behind the load balancer.

Moreover, the routing info in a random CID may look “syntactically valid” but be “semantically invalid” (pointing at a dead path or a set of servers not capable of serving QUIC traffic).

From: Lubashev, Igor [mailto:ilubashe=[email-masked]]
Sent: Thursday, April 05, 2018 2:21 PM
To: Kazuho Oku <[email-masked]>
Cc: Mirja Kühlewind <[email-masked]>; Christian Huitema <[email-masked]>; Mike Bishop <[email-masked]>; Mikkel Fahnøe Jørgensen <[email-masked]>; IETF QUIC WG <[email-masked]>; Frederick Kautz <[email-masked]>
Subject: RE: Privacy holes (was: Re: Getting to consensus on packet number encryption)

From: Kazuho Oku [mailto:[email-masked]]
Sent: Thursday, April 05, 2018 2:09 PM
To: Lubashev, Igor <[email-masked]<mailto:[email-masked]>>
Cc: Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>>; Mike Bishop <[email-masked]<mailto:[email-masked]>>; Christian Huitema <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>; Mirja Kühlewind <[email-masked]<mailto:[email-masked]>>; Frederick Kautz <[email-masked]<mailto:[email-masked]>>
Subject: Re: Privacy holes (was: Re: Getting to consensus on packet number encryption)

- Igor

Hi

Mikkel

--
Kazuho Oku

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 5 Apr 2018 18:27:23 +0000

- Igor

Hi

Mikkel

--
Kazuho Oku

---

From: Praveen Balasubramanian <[email-masked]>
Date: Thu, 5 Apr 2018 18:33:41 +0000

Windows supports querying of the ECN bits (recvmsg) on recent versions but does not yet support setting them (sendmsg). Adding support for set is on our TODO list. Please see https://github.com/quicwg/base-drafts/wiki/ECN-in-QUIC which has per OS details.

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Brian Trammell (IETF)
Sent: Thursday, April 5, 2018 12:03 AM
To: Lars Eggert <[email-masked]>
Cc: Mark Nottingham <[email-masked]>; IETF QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>; Patrick McManus <[email-masked]>; Kazuho Oku <[email-masked]>
Subject: Re: ECN signaling from userland Re: Getting to consensus on packet number encryption

Thanks, cheers,

Brian

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 6 Apr 2018 03:35:37 +0900

2018-04-06 3:21 GMT+09:00 Lubashev, Igor <[email-masked]>:

>
>    - It would not, *if* the stateless load balancer can AES-decrypt the
>    packet (using the key derived from the packet header) to see if the packet
>    is in fact a initial packet.
>
>
>
> Would not it need to first decide that the CID in the packet is unknown
> (which means stateful)?  It would need to do some integrity check on CID,
> which would require a large nonce to avoid accidentally classifying a
> random CID as valid, because the cost of this mistake can be very large for
> the client (see below).
>

AEAD tag that is used for checking if a packet is legitimate as a initial
packet is currently 16 octets. That would not change. I believe that it
would be enough to avoid 1-RTT packets being misclassified as handshake
packets.

>
>
>    - Or another way of solving the issue will be to always route the
>    packet based on CID, then forward the packet between the endpoints behind
>    the load balancer.
>
>
>
> This would not really work for a global load balancer – the one capable of
> forwarding across a continent to a different anycast pop (due to client’s
> change in attachment point or BGP reconvergence).
>

Agreed.

>
>
>
>
>
>
> *From:* Kazuho Oku [mailto:[email-masked]]
> *Sent:* Thursday, April 05, 2018 2:09 PM
> *To:* Lubashev, Igor <[email-masked]>
> *Cc:* Mikkel Fahnøe Jørgensen <[email-masked]>; Mike Bishop <
> [email-masked]>; Christian Huitema <[email-masked]>; IETF QUIC
> WG <[email-masked]>; Mirja Kühlewind <[email-masked]>;
> Frederick Kautz <[email-masked]>
> *Subject:* Re: Privacy holes (was: Re: Getting to consensus on packet
> number encryption)
>
>
>
>
>
>
>

-- 
Kazuho Oku

---

From: Frederick Kautz <[email-masked]>
Date: Thu, 5 Apr 2018 12:23:14 -0700

Maybe there could be some form of unencrypted header which load balancers
could use to land it to the correct location? This id could be set by the
server after a successful negotiation, and would be shared across all
connections to that same server. It can be used to ensure that the same
server internally always handles the given connection while attempting to
keep linkability low.

On Thu, Apr 5, 2018 at 11:35 AM, Kazuho Oku <[email-masked]> wrote:

>> This would not really work for a global load balancer – the one capable
>> of forwarding across a continent to a different anycast pop (due to
>> client’s change in attachment point or BGP reconvergence).
>>
>
> Agreed.
>
>
>>
>>
>>
>>
>>
>>
>> [email-masked]>; Christian Huitema <[email-masked]>; IETF
>> QUIC WG <[email-masked]>; Mirja Kühlewind <[email-masked]>;

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 5 Apr 2018 22:36:51 +0200

> required to switch to a different IP address?

Or much more commonly simply drop the packets by lack of new source ports.
I've witnessed this problem a lot already at places having to access HTTP
services not using keep-alive.

Willy

---

From: Martin Thomson <[email-masked]>
Date: Fri, 6 Apr 2018 10:38:38 +1000

> Yes. Hence the need to grease. For example, have clients do some gratuitous migration to a new source port number rather frequently.

This seems like a simple fix, particularly since Mike recently added
text that suggests occasional use of new connection IDs.  In the
spirit of keeping the wheels greased:

https://github.com/quicwg/base-drafts/pull/1269

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 6 Apr 2018 10:10:50 +0900

2018-04-05 18:57 GMT+09:00 Philipp S. Tiesel <[email-masked]>:

> to be strictly ordered?
> I agree that reordering by middleboxes *might not* happen. But I would
> also argue that we should encrypt the PN if we think reordering *might*
> happen.
>
> If we encrypt PN now, we will have the chance to revisit the design in the
> future versions of QUIC. But if we ship without PNE and then find out
> middleboxes start using the PN under the assumption that they are sent
> in-order, we would forever be forced to stick to them sent in-order, in
> cleartext.
>
> What if we think of the "clear PN" as a path signal, which is
> coincidentally used as PN in the transport in v1.
>

I am not sure if that would solve the middlebox reordering issue or any
ossification caused by the assumption that packet numbers increase in the
order the packets are being sent.

It's confusing, but the discussion about reordering / ossification is about
what happens on each path, while the discussion about privacy is about what
happens when switching to a new 5-tuple.

-- 
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 6 Apr 2018 11:06:00 +0900

> https://github.com/quicwg/base-drafts/pull/1269
Would you mind explaining how you would solve the issue?

If I understand correctly, Christian's concern is that some middleboxes
might only create a hole if the first packet sent by a client (using a new
port) looks like a Initial packet.

Consider the case where you want to build a middlebox that allows QUIC, but
no other UDP-based protocols.

By observing the QUIC packets, you think that you can implement such a
middlebox by creating a hole only when the UDP flow starts with a packet
starting with 0xff (long header + Initial). But such a design will block
migration.

The question is if we need to address the issue proactively, and how.

FWIW, I can see at least three possible approaches here.

1. frequent gratuitous migration by clients, so that middlebox vendors can
notice that there design is wrong prior to shipping it
2. make long header packets not easily indistinguishable from short header
packets (by encrypting the type field)
3. mimic 0-RTT handshake when gratuitously switching to a new UDP flow
(depends on #1262)

1 and 2 have been discussed on this thread. I haven't yet seen 3 being
discussed.

The concerns so far being raised for 1 is that NATs might run out of
unassigned ports and that switching to a new path might trigger the restart
of congestion control on large-scale NATs that use multiple public
addresses.

The concern so far being raised for 2 is that stateless load balancers
would be required to decrypt the type field (or the entire packet,
depending on how we encrypt) to determine if a packet is trying to initiate
a new connection.

-- 
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 6 Apr 2018 11:15:39 +0900

> (and route them based on CID).

Reconsidering this, thanks to symmetric CID, I think we now have a way to
statelessly determine whether or not a packet is initial without decrypting
it, even if the type field is encrypted.

Initial packets always carry a DCID of 8 octets.

So if you could configure your servers to use DCID of any other size, you
can determine if a packet is initial or not just by checking the size of
DCID.

And assuming that there would be non-neglible amount of servers using DCID
of 8 octets, the size of the field would be unusable as a way to detect if
a packet is initial from the middlebox's point of view.

Igor, Praveen, does the approach alleviate your concern?

-- 
Kazuho Oku

---

From: Martin Thomson <[email-masked]>
Date: Fri, 6 Apr 2018 12:34:17 +1000

> port) looks like a Initial packet.

Oh, I didn't get the same impression, but it does seem like a valid
concern.  The idea I had was to have clients migrate often enough that
those middleboxes will see those migrations and not incorrectly assume
that flows have to start with Initial packets.

Of course, migrating often like this might not provide enough
incentive to avoid that practice because clients will likely make new
connections when the old one fails.  But given that this sort of
failure probably involves a timeout before the repair, it's probably
still noticeable enough to generate support tickets (and incentive to
allow the migration to pass).

Existing middleboxes that pass UDP will be fine, so we at least have
that in our favour.

> notice that there design is wrong prior to shipping it

That is what this was angling for.  The question being how frequent.

> 1 and 2 have been discussed on this thread. I haven't yet seen 3 being

It has been suggested a few times elsewhere.  The question is how far
you go: do you send Initial and Handshake packets that use the
handshake packet protection and carry apparently real ClientHello and
ServerHello messages?  Do you restrict the flow of packets to match
0-RTT?

> of congestion control on large-scale NATs that use multiple public

That is possible, but it depends on how often you attempt this.  CGNAT
does try to keep IP addresses for the same device stable.  If not,
then a congestion controller reset for relatively infrequent
migrations isn't so bad if heuristics fail.  It's only when you have
very frequent migration that the resets becomes burdensome.

> The concern so far being raised for 2 is that stateless load balancers would
> be required to decrypt the type field (or the entire packet, depending on
> how we encrypt) to determine if a packet is trying to initiate a new
> connection.

Yeah, that's hard.  Having the load balancer hit the slow path for
every packet seems pretty fatal.

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 6 Apr 2018 11:56:41 +0900

Hi Martin,

Thank you for the response. My comments inline.

> very frequent migration that the resets becomes burdensome.

How frequent is the issue here.

If it's too frequent, some NATs could run out of unassigned ports, and if
depending on their eviction algorithm, shutdown UDP-based flows running
other protocols that do not do gratuitous switching. Other NATs could start
assigning new public addresses as they run out of unassigned ports.

If it's too infrequent, then the chance of seeing misimplemented middlebox
becomes higher.

> would

That was my understanding until I started to consider the impact of
symmetric CIDs.

The length of DCID can be a signal to the stateless load balancers for
distinguishing an initial packet. If people interested in using that type
of load balancer can agree that they will be using CID other than 8 octets,
the length can be used as the signal by them.

More detail about using DCID can be found on
https://www.ietf.org/mail-archive/web/quic/current/msg03809.html.

-- 
Kazuho Oku

---

From: "Lubashev, Igor" <[email-masked]>
Date: Fri, 6 Apr 2018 03:48:39 +0000

> The length of DCID can be a signal to the stateless load balancers.

Short headers do not have an explicit CID length, so we are back to validating the integrity of CID, no? What am I missing?

-----Original Message-----
From: Kazuho Oku [[email-masked]]
Received: Thursday, 05 Apr 2018, 10:56PM
To: Martin Thomson [[email-masked]]
CC: Mirja Kühlewind [[email-masked]]; QUIC WG [[email-masked]]; Christian Huitema [[email-masked]]; Frederick Kautz [[email-masked]]
Subject: Re: Privacy holes (was: Re: Getting to consensus on packet number encryption)

Hi Martin,

Thank you for the response.. My comments inline.

2018-04-06 11:34 GMT+09:00 Martin Thomson <[email-masked]<mailto:[email-masked]>>:

If it's too frequent, some NATs could run out of unassigned ports, and if depending on their eviction algorithm, shutdown UDP-based flows running other protocols that do not do gratuitous switching. Other NATs could start assigning new public addresses as they run out of unassigned ports.

If it's too infrequent, then the chance of seeing misimplemented middlebox becomes higher.

That was my understanding until I started to consider the impact of symmetric CIDs.

The length of DCID can be a signal to the stateless load balancers for distinguishing an initial packet. If people interested in using that type of load balancer can agree that they will be using CID other than 8 octets, the length can be used as the signal by them.

More detail about using DCID can be found on https://www.ietf.org/mail-archive/web/quic/current/msg03809.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail-2Darchive_web_quic_current_msg03809.html&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=WsJaFd1M4_j4sdAutwoZzcMiZ8Ut8mmyHO9Jn0FAQOI&s=gTP2rfuzpXp0_GdcmG8aA_N4hlvaYnhj6H_CVyXBK10&e=>.

--
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 6 Apr 2018 13:02:23 +0900

2018-04-06 12:48 GMT+09:00 Lubashev, Igor <[email-masked]>:

> Short headers do not have an explicit CID length, so we are back to
> validating the integrity of CID, no? What am I missing?
>

Yeah I missed that, apparently. Thank you for pointing that out.

So I would propose to encrypt the type bits _and_ have DCID length in the
header in cleartext.

By doing that, it will still be possible for stateless load balancers to
determine if a packet is initial (by checking if DCIL is 8) without
decrypting the packet, at the same time being impossible for a middlebox to
reliably determine the same thing (assuming that some servers will
advertise DCID of 8 octets).

> *From:* Kazuho Oku [[email-masked]]
> *Received:* Thursday, 05 Apr 2018, 10:56PM
> *To:* Martin Thomson [[email-masked]]
> *CC:* Mirja Kühlewind [[email-masked]]; QUIC WG [
> [email-masked]]; Christian Huitema [[email-masked]]; Frederick Kautz [
> [email-masked]]
> More detail about using DCID can be found on https://www.ietf.org/mail-
> archive/web/quic/current/msg03809.html
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail-2Darchive_web_quic_current_msg03809.html&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=WsJaFd1M4_j4sdAutwoZzcMiZ8Ut8mmyHO9Jn0FAQOI&s=gTP2rfuzpXp0_GdcmG8aA_N4hlvaYnhj6H_CVyXBK10&e=>
> .
>
> --
> Kazuho Oku
>

-- 
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 6 Apr 2018 04:33:51 +0000

So eventually CID's will ossify into certain lenghts by convention to things working smoothly in the general case?

________________________________
From: QUIC <[email-masked]> on behalf of Kazuho Oku <[email-masked]>
Sent: Friday, April 6, 2018 6:02:23 AM
To: Lubashev, Igor
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: Privacy holes (was: Re: Getting to consensus on packet number encryption)

So I would propose to encrypt the type bits _and_ have DCID length in the header in cleartext.

By doing that, it will still be possible for stateless load balancers to determine if a packet is initial (by checking if DCIL is 8) without decrypting the packet, at the same time being impossible for a middlebox to reliably determine the same thing (assuming that some servers will advertise DCID of 8 octets).

-----Original Message-----
From: Kazuho Oku [[email-masked]<mailto:[email-masked]>]
Received: Thursday, 05 Apr 2018, 10:56PM
To: Martin Thomson [[email-masked]<mailto:[email-masked]>]
CC: Mirja Kühlewind [[email-masked]<mailto:[email-masked]>]; QUIC WG [[email-masked]<mailto:[email-masked]>]; Christian Huitema [[email-masked]<mailto:[email-masked]>]; Frederick Kautz [[email-masked]<mailto:fkautz@alumni..cmu.edu>]
Subject: Re: Privacy holes (was: Re: Getting to consensus on packet number encryption)

Hi Martin,

--
Kazuho Oku

--
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 6 Apr 2018 14:08:35 +0900

2018-04-06 13:33 GMT+09:00 Mikkel Fahnøe Jørgensen <[email-masked]>:

> So eventually CID’s will ossify into certain lenghts by convention to
> things working smoothly in the general case?
>

That is a good point, but I think that the answer is no.

In the long header format, the lengths of CID's are sent in clear in the
latest working version of the draft. And if I recall correctly, people are
interested in greasing the CID lengths by advertising CIDs longer than
necessary against a minor portion of the clients. We assume that the
shortest length of CID is zero, and an endpoint that uses a zero-length CID
can grease it to any length.

To summarize, we will be greasing CID length regardless of if we include it
as part of the short header.

> ------------------------------
> *From:* QUIC <[email-masked]> on behalf of Kazuho Oku <
> [email-masked]>
> *Sent:* Friday, April 6, 2018 6:02:23 AM
> *To:* Lubashev, Igor
> *Cc:* [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]
>
>> [email-masked]]; Christian Huitema [[email-masked]]; Frederick Kautz
>> [[email-masked] <fkautz@alumni..cmu.edu>]
>> https://www.ietf.org/mail-archive/web/quic/current/msg03809.html

-- 
Kazuho Oku

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Fri, 6 Apr 2018 09:16:32 +0200

(Repeating and expanding my comment on #1269 to the list)

+1 to CID rotation being pointless if we don't do this. However, this seems dangerous if not done carefully, and it links CID rotation rate to the maximum usable UDP port consumption rate, which is (most probably) widely variable across access networks and (AFAICT) largely unknown in the literature. We have anecdata here, and we need data.

Note that QUIC presently doesn't have an advisory on-path state clearance signal (we hsd discussed this as a signal bound to public reset in #20, but it morphed into Stateless Reset, which has completely different semantics), which means that there is neither a present nor yet a potential future method to reduce the NAT state load this will generate.

I'd feel a lot less apprehensive about this if there were an advisory, integrity-protected signal meant for path consumption to clear state for a path. Of course it wouldn't be used at all on day one, but the tradeoff it presents is "hey, this is QUIC, it will burn through your UDP port state faster than any other UDP protocol and multiplicatively faster than TCP; but hey look, you can clear state when you see this path-verifiable singal to reduce the impact this has."

Cheers,

Brian

> On 6 Apr 2018, at 02:38, Martin Thomson <[email-masked]> wrote:
>

---

From: Martin Thomson <[email-masked]>
Date: Fri, 6 Apr 2018 17:45:05 +1000

We do have data about rate of binding creation, courtesy of tests done
for WebRTC.  In essence, bindings can be created at rates that exceed
most needs (I think that they went down to single digit milliseconds,
far lower than I think is relevant here).

Justin Uberti has the numbers, but I struggled to find the email or
slides; we can ask if you care. This result differed from the data
that was available when STUN was first developed, where it was found
that there were non-trivial numbers of NAT devices that lost bindings
below 20ms (see https://tools.ietf.org/html/rfc5245#appendix-B.1).

The question about exhaustion is relevant, I agree.  There, the value
doesn't need measurement as much as asking the right person what their
configuration is.  Take the number of ports open for bindings per IP,
the duration of a binding, and how many clients are expected to be
behind each IP address or how many bindings each client is limited to.
We'd probably need to just ask a few CGNAT operators; anything smaller
is unlikely to be as sensitive as something that runs at a decent
scale.  But getting data in the usual fashion - making bindings until
something breaks - might be a little antisocial.

If we're going to saturate NAT bindings with this, then a signal might
be a good solution.  Or it might just be that we can recommend a rate
limit on these little migrations.  Based on typical timeouts on UDP
(30s is very common, see also experience with ICE), we could say that
a little migration every 10s is OK and you only have each connection
taking 3 bindings.  That doesn't seem that bad.

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Fri, 6 Apr 2018 10:36:38 +0200

> On 6 Apr 2018, at 09:45, Martin Thomson <[email-masked]> wrote:
> 

There are two relevant metrics here: binding creation rate and maximum concurrent state. The latter is where I expect there to be more variability, and potential unfortunate side effects of an engineering decision made with a single consideration. "People stuck behind marginal NATs outside their own control don't deserve to use QUIC" is not a great statement to make, even accidentally.

> below 20ms (see https://tools.ietf.org/html/rfc5245#appendix-B.1).

This is useful, thanks!

> something breaks - might be a little antisocial.

Agreed that CGNAT is where the highest pain magnitude will be, but there is a very long tail of CPE out there, where configuration information will be harder to come by. But that's why we have a TCP fallback, I guess.

> taking 3 bindings.  That doesn't seem that bad.

It's multiplicative by a factor of three, which means that the max flow concurrency will reduce by a factor of three. I agree it's probably tolerable, but we do need to understand what we're actually getting for that cost. I remain dubious about the actual privacy benefits WRT third party passive surveillance here -- they don't accrue at all unless there's a CGNAT in the way that doesn't bind an address/port range to a customer at a given point in time. In the common case, ISTM that just burn more in-network state than they additionally require at the surveillance point to track the change. Defenses with such scaling curves are not a good idea.

Another issue this raises: this will wreak varying levels havoc on anything that uses IPFIX, NetFlow, sFlow, or similar technologies for accounting or forensics. For accounting, whether two flow records happen to represent the same flow is irrelevant, since they'll generally be dumped into larger aggregates anyway. Replacing a single long-lived flow (a common max active time is 30min) with its equivalent split into 10s windows will lead to a 180x increase in export bandwidth for these flows in the first stage of the collection infrastructure, but the flow counts handled by these infrastructures (when connected to the open Internet) are usually dominated by single-packet UDP and ICMP garbage anyway, so this probably won't have too terrible an impact.

Most such infrastructures are sampled (sFlow explicitly so), and there are well-known issues with differential visibility of packet-sampled flows based on flow duration. QUIC flows rotating ports every 10s would be systematically undercounted with respect to long-duration TCP flows, which will provide a strong incentive for correction if it materially impacts settlement.

Especially forensics for operational security will remain very interested in understanding whether a group of 5-tuple flows are related to the same activity. In most enterprise environments, though, any CGNAT mappings are reversable, so there will not necessarily be a strong incentive

I want to like this proposal, but I'm increasingly left with the impression that this is neither a particularly useful nor a completely safe thing to do at scale.

Cheers,

Brian

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 6 Apr 2018 18:14:41 +0900

2018-04-06 17:36 GMT+09:00 Brian Trammell (IETF) <[email-masked]>:

> There are two relevant metrics here: binding creation rate and maximum
> concurrent state. The latter is where I expect there to be more
> variability, and potential unfortunate side effects of an engineering
> decision made with a single consideration. "People stuck behind marginal
> NATs outside their own control don't deserve to use QUIC" is not a great
> statement to make, even accidentally.
>
> Agreed that CGNAT is where the highest pain magnitude will be, but there
> is a very long tail of CPE out there, where configuration information will
> be harder to come by. But that's why we have a TCP fallback, I guess.
>
> It's multiplicative by a factor of three, which means that the max flow
> concurrency will reduce by a factor of three. I agree it's probably
> tolerable, but we do need to understand what we're actually getting for
> that cost. I remain dubious about the actual privacy benefits WRT third
> party passive surveillance here -- they don't accrue at all unless there's
> a CGNAT in the way that doesn't bind an address/port range to a customer at
> a given point in time. In the common case, ISTM that just burn more
> in-network state than they additionally require at the surveillance point
> to track the change. Defenses with such scaling curves are not a good idea.
>

I think I share the same concern.

One way to minimize the impact could be to suggest certain amount of
connections (e.g., 5%) to gratuitously migrate to a new tuple just once,
right after the connection is being established.

If we specify that way, we can assume that the impact on the NAT will be
having 5% more flows. Middlebox vendor that mistakenly tries to use the
cleartext bits in the packet header as the signal of connection initiation
will also see failures at the same rate.

> Another issue this raises: this will wreak varying levels havoc on
> anything that uses IPFIX, NetFlow, sFlow, or similar technologies for
> accounting or forensics. For accounting, whether two flow records happen to
> represent the same flow is irrelevant, since they'll generally be dumped
> into larger aggregates anyway. Replacing a single long-lived flow (a common
> max active time is 30min) with its equivalent split into 10s windows will
> lead to a 180x increase in export bandwidth for these flows in the first
> stage of the collection infrastructure, but the flow counts handled by
> these infrastructures (when connected to the open Internet) are usually
> dominated by single-packet UDP and ICMP garbage anyway, so this probably
> won't have too terrible an impact.
>
> Most such infrastructures are sampled (sFlow explicitly so), and there are
> well-known issues with differential visibility of packet-sampled flows
> based on flow duration. QUIC flows rotating ports every 10s would be
> systematically undercounted with respect to long-duration TCP flows, which
> will provide a strong incentive for correction if it materially impacts
> settlement.
>
> Especially forensics for operational security will remain very interested
> in understanding whether a group of 5-tuple flows are related to the same
> activity. In most enterprise environments, though, any CGNAT mappings are
> reversable, so there will not necessarily be a strong incentive
>
> I want to like this proposal, but I'm increasingly left with the
> impression that this is neither a particularly useful nor a completely safe
> thing to do at scale.
>
> Cheers,
>
> Brian
>
> >
> >> +1 to CID rotation being pointless if we don't do this. However, this
> seems dangerous if not done carefully, and it links CID rotation rate to
> the maximum usable UDP port consumption rate, which is (most probably)
> widely variable across access networks and (AFAICT) largely unknown in the
> literature. We have anecdata here, and we need data.
> >>
> >> Note that QUIC presently doesn't have an advisory on-path state
> clearance signal (we hsd discussed this as a signal bound to public reset
> in #20, but it morphed into Stateless Reset, which has completely different
> semantics), which means that there is neither a present nor yet a potential
> future method to reduce the NAT state load this will generate.
> >>
> >> I'd feel a lot less apprehensive about this if there were an advisory,
> integrity-protected signal meant for path consumption to clear state for a
> path. Of course it wouldn't be used at all on day one, but the tradeoff it
> presents is "hey, this is QUIC, it will burn through your UDP port state
> faster than any other UDP protocol and multiplicatively faster than TCP;
> but hey look, you can clear state when you see this path-verifiable singal
> to reduce the impact this has."
> >>
> >> Cheers,
> >>
> >> Brian
> >>

-- 
Kazuho Oku

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Fri, 6 Apr 2018 11:38:55 +0200

Hi, Kazuho,

> On 6 Apr 2018, at 11:14, Kazuho Oku <[email-masked]> wrote:
> 
> One way to minimize the impact could be to suggest certain amount of connections (e.g., 5%) to gratuitously migrate to a new tuple just once, right after the connection is being established.
> 
> If we specify that way, we can assume that the impact on the NAT will be having 5% more flows. Middlebox vendor that mistakenly tries to use the cleartext bits in the packet header as the signal of connection initiation will also see failures at the same rate.

If the goal is only ossification agility, ensuring that migration can actually work, then this seems like a much better approach.

However, I would submit that if we think we are in a position where:

- there is a strong incentive for firewalls/NATs to want to see start- (and end-) of flow (which seems to me to be the threat model we're trying to mitigate with this); and

- we do not want arbitrary information in the QUIC wire image to be used to detect these conditions, since that might cause bits of the wire image we don't want to end up in the accidental invariants;

then we really should consider adding explicit advisory flow state signals, both for establishment and teardown. The theory here is that if you give in-network functions a "grip" in the wire image of the protocol they can use for state management, they won't try to grab on to the slippery bits. Together with probabilistic migration, these signals would divorce "on-path state" from "transport session state", reducing the privacy impact with respect to SYN/FIN in TCP.

(The alternative is to make the wire image so inscrutable that on-path state tracking becomes completely futile and goes away. I'm not sure we're that good at inscrutable wire images.)

Cheers,

Brian

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 6 Apr 2018 02:44:13 -0700

Adding a “grib” sounds reasonable.

I previously mentioned my concern with CID ossification because length or
content might be used in lack of anything better.

The defense suggested - to grease the CID, will not work when a load
balancer says “I only support QUIC connections that use CID length x, and
sets the upper two bits like this.”, and this load balancer then gets a big
market share. Then greasing is out the window.

So it is better to provide an explicit signal for the cases that are
immediately obvious.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 6 April 2018 at 11.39.07, Brian Trammell (IETF) ([email-masked]) wrote:

Hi, Kazuho,

If the goal is only ossification agility, ensuring that migration can
actually work, then this seems like a much better approach.

- there is a strong incentive for firewalls/NATs to want to see start- (and
end-) of flow (which seems to me to be the threat model we're trying to
mitigate with this); and

- we do not want arbitrary information in the QUIC wire image to be used to
detect these conditions, since that might cause bits of the wire image we
don't want to end up in the accidental invariants;

then we really should consider adding explicit advisory flow state signals,
both for establishment and teardown. The theory here is that if you give
in-network functions a "grip" in the wire image of the protocol they can
use for state management, they won't try to grab on to the slippery bits.
Together with probabilistic migration, these signals would divorce "on-path
state" from "transport session state", reducing the privacy impact with
respect to SYN/FIN in TCP.

(The alternative is to make the wire image so inscrutable that on-path
state tracking becomes completely futile and goes away. I'm not sure we're
that good at inscrutable wire images.)

Cheers,

Brian

---

From: Roland Zink <[email-masked]>
Date: Fri, 6 Apr 2018 12:01:25 +0200

> taking 3 bindings.  That doesn't seem that bad.
I have found this :

RFC 4787 Network Address Translation (NAT) Behavioral Requirements for 
Unicast UDP (https://tools.ietf.org/html/rfc4787#page-12):
    REQ-5:  A NAT UDP mapping timer MUST NOT expire in less than two
       minutes, unless REQ-5a applies.

       a) For specific destination ports in the well-known port range
          (ports 0-1023), a NAT MAY have shorter UDP mapping timers that
          are specific to the IANA-registered application running over
          that specific destination port.

RFC6888 Common Requirements for Carrier-Grade NATs (CGNs):

   REQ-8:  Once an external port is deallocated, it SHOULD NOT be
       reallocated to a new mapping until at least 120 seconds have
       passed, with the exceptions being:

       a.  If the CGN tracks TCP sessions (e.g., with a state machine, as
           inSection 3.5.2.2 of [RFC6146] 
<https://tools.ietf.org/html/rfc6146#section-3.5.2.2>), TCP ports MAY be reused
           immediately.

       b.  If external ports are statically assigned to internal
           addresses (e.g., address X with port range 1000-1999 is
           assigned to subscriber A, 2000-2999 to subscriber B, etc.),
           and the assignment remains constant across state loss, then
           ports MAY be reused immediately.

       c.  If the allocated external ports used address-dependent or
           address-and-port-dependent filtering before state loss, they
           MAY be reused immediately.

       The length of time and the maximum number of ports in this state
       MUST be configurable by the CGN administrator.

So assuming 30 seconds seems a bit short. When there would be signals 
for session start and end then NATs could track how long mappings are 
necessary. Avoiding that and creating many bindings may be unsocial.

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 6 Apr 2018 19:10:51 +0900

Hi Brian,

My comments inline.

> then we really should consider adding explicit advisory flow state signals, both for establishment and teardown. The theory here is that if you give in-network functions a "grip" in the wire image…

I think that this is a very interesting question asking what the
requirements of QUIC as a protocol is. Thank you for bringing the
topic to table.

My argument against having an explicit signal for flow creation is
that it is impossible to have such signal in a reliable way, and that
trying to have no signal is better than having an unreliable signal.

Let me explain why I think we cannot have a reliable signal.

Consider the case where two NAT's exist. Such deployments exist, for
example when you connect through a Wifi router connected to a cellular
network.

There is a possibility that the NAT running next to the client (e.g.,
Wifi router) rebinding an existing QUIC connection to a new source
port without client intentionally requesting it. In such case, the
second NAT will start seeing a new flow that starts with a packet that
the client thinks it is part of an existing flow.

I think that in this example it is clear that a signal of flow
creation cannot be delivered to the second NAT, since the client does
not even know that a flow has been created.

The same argument goes for running an observer outside any number of NAT.

For flow closure, I think saving the battery by not sending a FIN
packet after quiescence is more important than sending good signal to
middleboxes.

> (The alternative is to make the wire image so inscrutable that on-path state tracking becomes completely futile and goes away. I'm not sure we're that good at inscrutable wire images.)

I would rather argue that the goal of the alternative approach is to
make the intentional flow creation not easily distinguishable to the
extent that the middlebox vendors will refer to the RFC. Then, they
can decide either to not limit QUIC flows starting from a 1-RTT
packet, or to drop QUIC completely. And the issue is solved.

-- 
Kazuho Oku

---

From: Martin Thomson <[email-masked]>
Date: Fri, 6 Apr 2018 20:40:06 +1000

> right after the connection is being established.

I like this idea.  Do it rarely, but with a high enough probability
that it is noticed.

I don't think that we need to specify it, rather permit it (maybe with
observations about the costs of doing it too often), then establish a
convention.  We can reach enough of the client population in this
group that we can probably coordinate effectively such that we reach a
common enough proportion.  Add migrate-after-idle as the current PR
suggests, which has a much lower chance of expanding state and I'm not
seeing much need for "grips".

> having 5% more flows.

5% more only during the first timeout interval (whatever that is) and
thereafter no more with this design.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 6 Apr 2018 20:55:12 +1000

I don't know how to game this one out.

If we're looking for mutual benefit, the best I can come up with is
that the middlebox knows about flow up/down, so it can extend timeouts
for most flows and endpoints thereby enjoy better chance of flow
liveness after longer periods of quiescence.  That would be something
like with TCP where you can expect timeouts to be in the order of
minutes or tens of minutes (30 minutes if you pick the right port
number) and not merely tens of seconds.

The problem here is that you need enough of the middlebox population
to extend timeouts in the common case.  Tuning to specific networks is
difficult and time-consuming.  So endpoints would deploy this "grip"
without any expectation of reciprocation for potentially a very long
time.  That's in the order of the end-of-life cycle for home routers,
or an entire Internet era.

In the meantime, endpoints have to assume flow liveness based on the
Internet of today.  30s is the benchmark now, despite the wishful
"requirement" for 2 minutes in RFC 4787.

Thus we end up with the tension between the documented wire image that
says that there is a "done" signal, and the actual wire image, which
has activity every 30 seconds.  Add genuine cases of migration,
crashes, stateless resets, and the other reasons that the "done"
signal doesn't appear and the timer has to come down.

There might be benefits if everything comes together, but it's a long
game to play.

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Fri, 6 Apr 2018 14:39:58 +0200

hi Kazuho,

Thanks for your message, comments inline, omitting some history for clarity...

> On 6 Apr 2018, at 12:10, Kazuho Oku <[email-masked]> wrote:
> 
> Hi Brian,
> 
> My comments inline.
> 

I should say I'm not all that convinced that an advisory "flow start" signal is a good idea(*) -- I'm just saying that we seem to be trying very hard to keep a de facto advisory flow start signal from forming (which I would agree would be extremely undesirable indeed)... and we should step back and think about what properties flow start and end should have in QUIC, explicitly, as a function of requirements as opposed to philosophy.

> Let me explain why I think we cannot have a reliable signal.

<snip> All this makes very good sense and I agree with it: the signal cannot possibly be reliable in the general case.

> middleboxes

This is a hard problem, and it cuts both ways; if a server wants to maintain the ability to connect a client after more than the standard UDP NAT timeout (30s), it needs to send keepalives more often than once every 30s (thereby waking the client if it's on battery). Which side of this coin wastes more power is difficult to determine, since it's entirely dependent on the prevalence of each kind of activity in a given network. We can make plausible guesses both in favor of "get rid of FIN, it wastes endpoint power" or "we really need FIN to keep from wasting endpoint power".

So I'm more convinced of the value of advisory-end than advisory-start.

> packet, or to drop QUIC completely. And the issue is solved.

Let me try to rephrase this in wildly different terminology: "we can't come up with a grip that we think will wotk, so we should make flow start as slippery as possible"; or rather, a middlebox that thinks it's seeing QUIC should make its drop decision based on "this is QUIC" and not anything based on the state of the connection. That definitely has desirable properties with respect to future flexibility -- the start and end signaling would end up in the invariants rather quickly whether intentional or not. And ISTM that the probabilistic migration behavior would help to achieve it.

Cheers,

Brian

(*): "Advisory flow start" is somewhat different than "path-visible proof of return routability" and "path-visible indication of consent to continue communicating". The latter, IMO, has more value as part of a design to support in-network rejection of UDP garbage. But it's a more complex signal, since it involves revealing things that are hard to fake at flow establishment time, and is of course subject to the philosophical question of whether that's a good idea.

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Fri, 6 Apr 2018 14:54:30 +0200

hi Martin,

> On 6 Apr 2018, at 12:55, Martin Thomson <[email-masked]> wrote:
> 

Flow down in this case is way more useful than flow up.

> or an entire Internet era.

We, as a community, are currently managing transitions that have taken that long. Insert your favorite IPv6 joke here, of course, but ECN comes to mind as well.

> "requirement" for 2 minutes in RFC 4787.

"Flow liveness" is a path property, not a global one, a function of the minimum timeout of all connection-breaking state along the path. I suspect that endpoint pairs in a world ruled by UDP state timeout will end up widely deploy simple probing algorithms to discover it whether or not there's a simple way to signal that state can be disposed of early.

This is really a separate issue, though -- the question about state disposal is disjoiny from how to keep flow start signaling from ossifying into an unintentional magic packet sequence you have to show the network. It makes sense to think about the "grippiness" of the protocol in terms of the eventual endpoint as well as management reqt's though.

> There might be benefits if everything comes together, but it's a long

I've been having some variant of this particular conversation for (checks watch) about four years now. ;)

Cheers,

Brian

---

From: Ian Swett <[email-masked]>
Date: Fri, 06 Apr 2018 12:55:14 +0000

There's some slightly older data from QUIC on slide 10 in this IETF
presenation:
https://www.ietf.org/proceedings/88/slides/slides-88-tsvarea-10.pdf

TLDR: 90% of networks have at least a 1 minute timeout, but only 50% have
at least a 2 minute timeout.

On Fri, Apr 6, 2018 at 6:01 AM Roland Zink <[email-masked]> wrote:

>           in Section 3.5.2.2 of [RFC6146] <https://tools.ietf.org/html/rfc6146#section-3.5.2.2>), TCP ports MAY be reused
>           immediately.
>
> So assuming 30 seconds seems a bit short. When there would be signals for
> session start and end then NATs could track how long mappings are
> On Fri, Apr 6, 2018 at 5:16 PM, Brian Trammell (IETF) <[email-masked]> <[email-masked]> wrote:
>
> On 6 Apr 2018, at 02:38, Martin Thomson <[email-masked]> <[email-masked]> wrote:
>
> On Fri, Apr 6, 2018 at 2:08 AM, Christian Huitema <[email-masked]> <[email-masked]> wrote:
>
> On Apr 5, 2018, at 8:58 AM, Frederick Kautz <[email-masked]> <[email-masked]> wrote:
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 6 Apr 2018 09:02:04 -0500

Hi, Brian,

On Fri, Apr 6, 2018 at 7:54 AM, Brian Trammell (IETF) <[email-masked]>
wrote:

> We, as a community, are currently managing transitions that have taken
> that long. Insert your favorite IPv6 joke here, of course, but ECN comes to
> mind as well.
>
> "Flow liveness" is a path property, not a global one, a function of the
> minimum timeout of all connection-breaking state along the path. I suspect
> that endpoint pairs in a world ruled by UDP state timeout will end up
> widely deploy simple probing algorithms to discover it whether or not
> there's a simple way to signal that state can be disposed of early.
>
> This is really a separate issue, though -- the question about state
> disposal is disjoiny from how to keep flow start signaling from ossifying
> into an unintentional magic packet sequence you have to show the network.
> It makes sense to think about the "grippiness" of the protocol in terms of
> the eventual endpoint as well as management reqt's though.
>

Honestly, we should be checking a sundial.

If I'm understanding that the problem is what happens when signals are
unreliable and devices have to include workarounds that accommodate missed
signals, I THINK that's a conversation I overheard sometime around 2000, in
L2Triggers. I wasn't a participant in that conversation, but IIRC, the
reason that TRIGTRAN's scope was limited to first-hop routers in 2002 was
because of the problems L2Triggers had identified with signals coming from
routers that weren't locally attached, and we couldn't turn TRIGTRAN into
something useful even with that limitation.

The signals being discussed aren't the same signals, but the problems sound
very, very familiar.

Keeping in mind that I started Section 4 of
https://tools.ietf.org/html/draft-dawkins-panrg-what-not-to-do-00 with
TRIGTRAN, that's probably not an good sign ...

Do the right thing, of course.

Spencer

---

From: Kazuho Oku <[email-masked]>
Date: Sat, 7 Apr 2018 01:10:58 +0900

> I should say I'm not all that convinced that an advisory "flow start" signal is a good idea(*) -- I'm just saying that we seem to be trying very hard to keep a de facto advisory flow start signal f…

Thank you for the clarification.

> This is a hard problem, and it cuts both ways; if a server wants to maintain the ability to connect a client after more than the standard UDP NAT timeout (30s), it needs to send keepalives more oft…

The case of server reviving the connection is interesting.

If we are to have some signal, I think that should be "path-visible
indication of consent to continue communicating" as you describe,
specifically, as a signal that can be set on every packet for which,
at the moment of sending, the endpoint is sure that the server will
not send any more packets unless it receives a new packet from the
client.

A NAT aware of the signal can drop an idle UDP flow on which the last
packet had that flag set, after couple of round-trips, thereby giving
other flows a longer idle timeout.

The benefit of defining the signal this way is that there would be no
need to send FIN after quiescence to shut down the hole.

That said, I would prefer not discussing this as a feature of QUICv1
for several reasons: i) this can be implemented in v2 ii) having such
signal is not going give us benefit in the short-term, so no need to
hurry iii) many of us are interested in shipping v1 as early as
possible iv) HTTP (which is the application protocol in focus for v1)
is not going to suffer a lot from the server-revives-connection
problem since HTTP is a request-response protocol.

-- 
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Sat, 7 Apr 2018 01:15:35 +0900

> seeing much need for "grips".

+1 to all the comments. The approach has very small (not if the least)
impact on existing QUIC implementations, middleboxes, and the
specification.

> thereafter no more with this design.

Thank you for the correction. Nice catch :-)

-- 
Kazuho Oku

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 6 Apr 2018 16:31:51 +0000

Transport parameters can also be used for negotiation. Opened issue #1271 https://github.com/quicwg/base-drafts/issues/1271 for negotiation of connection migration.

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 6 Apr 2018 16:31:51 +0000

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 6 Apr 2018 16:54:36 +0000

I am a supporter of the flow closure signal. I understand that it may not be reliable on the network or not all endpoints may generate it (Kazuho's point about bringing up battery to send the "FIN/RST"). But we have a genuine use case for this in the cloud. The software switches in public cloud hosts typically do stateful processing (encap/NAT/other packet transforms) to implement SDN functionality like network virtualization and  load balancing. Since flow state tracking is expensive, today these software middleboxes use TCP termination packets to keep a longer timeout for TCP and a much shorter timeout for UDP. If there was a explicit closure signal for QUIC, then we could increase the timeouts and thereby reduce the keep-alive traffic needed by a large fraction. 

Not every middlebox is on the Internet and not every scenario for QUIC is the web browser/app loading a web page. When building a general purpose transport we should aim to define mechanisms like flow closure signal and leave the policy of whether to send one to end devices. 

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Kazuho Oku
Sent: Friday, April 6, 2018 9:11 AM
To: Brian Trammell (IETF) <[email-masked]>
Cc: huitema <[email-masked]>; QUIC WG <[email-masked]>; Mirja Kühlewind <[email-masked]>; Martin Thomson <[email-masked]>; Frederick Kautz <[email-masked]>
Subject: Re: Grips in the Wire Image for In-Network State (was Re: Privacy holes (was Re: Getting to consensus on packet number encryption))

2018-04-06 21:39 GMT+09:00 Brian Trammell (IETF) <[email-masked]>:
>>> - there is a strong incentive for firewalls/NATs to want to see 
>>> start- (and end-) of flow (which seems to me to be the threat model 
>>> we're trying to mitigate with this); and
>>>
>>> - we do not want arbitrary information in the QUIC wire image to be 
>>> used to detect these conditions, since that might cause bits of the 
>>> wire image we don't want to end up in the accidental invariants;
>>>

If we are to have some signal, I think that should be "path-visible indication of consent to continue communicating" as you describe, specifically, as a signal that can be set on every packet for which, at the moment of sending, the endpoint is sure that the server will not send any more packets unless it receives a new packet from the client.

A NAT aware of the signal can drop an idle UDP flow on which the last packet had that flag set, after couple of round-trips, thereby giving other flows a longer idle timeout.

The benefit of defining the signal this way is that there would be no need to send FIN after quiescence to shut down the hole.

That said, I would prefer not discussing this as a feature of QUICv1 for several reasons: i) this can be implemented in v2 ii) having such signal is not going give us benefit in the short-term, so no need to hurry iii) many of us are interested in shipping v1 as early as possible iv) HTTP (which is the application protocol in focus for v1) is not going to suffer a lot from the server-revives-connection problem since HTTP is a request-response protocol.

>>> (The alternative is to make the wire image so inscrutable that 
>>> on-path state tracking becomes completely futile and goes away. I'm 
>>> not sure we're that good at inscrutable wire images.)
>>

--
Kazuho Oku

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 9 Apr 2018 12:54:43 +1000

Praveen,

In previous discussions, there's been strong interest in including connection migration -- in a limited form (for the "parking lot" use case) -- in QUICv1 (discussed in Melbourne in some detail; see <https://github.com/quicwg/wg-materials/blob/master/interim-18-01/minutes.md#connection-migration>). 

It looks like the discussion in <https://github.com/quicwg/base-drafts/issues/1271> about making it optional and how that gets indicated / negotiated is going nicely. Thanks for opening an issue.

AFAICT, (at least) two assumptions need to be proven to have that help the PNE issue:

a) There isn't a linkability issue during NAT rebinding

b) Having two packet number schemes is acceptable to implementers (given the probable popularity of migration)

I haven't yet seen much discussion of either of these specific aspects.

As an aside -- the other threads about modifications to PNE to make them single-pass seem extremely promising; I'd encourage folks there to continue discussions and make a PR.

Cheers,

> On 6 Apr 2018, at 2:41 am, Praveen Balasubramanian <pravb=[email-masked]> wrote:
>

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 9 Apr 2018 12:54:43 +1000

Praveen,

Cheers,

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Mon, 9 Apr 2018 10:36:21 +0200

hi Mark, all,

> On 9 Apr 2018, at 04:54, Mark Nottingham <[email-masked]> wrote:
> 
> Praveen,
> 

Isn't the question we should be asking here "does QUIC with connection migration disabled (and no PNE) have similar linkability characteristics to QUIC with connection migration enabled (and PNE)?"?

I'm still confused as to how PNE is supposed to mitigate linkability across a NAT rebinding that occurs without client knowledge in the presence of a server-supplied CID, which seems to me to be the most common NAT rebinding event type. If it indeed does not, then the linkability characteristics seem identical to me.

> As an aside -- the other threads about modifications to PNE to make them single-pass seem extremely promising; I'd encourage folks there to continue discussions and make a PR.

+1. Making the design of PNE less problematic for offload and other future optimizations won't solve the underlying requirements problem (i.e. we are both treating linkability among 5-tuple flows by a specific on-path device, the LB, as a requirement to support via CID, as well as a requirement to mitigate linkability among 5-tuple flows by a all generic on-path devices via PNE, which is inherently contradictory  without a very precise definition of under which circumstances which requirement takes precedence). But it will reduce the cost from "probably mostly harmless" to "almost definitely mostly harmless", which should make the decision easier.

Cheers,

Brian

---

From: Ian Swett <[email-masked]>
Date: Mon, 09 Apr 2018 20:20:09 +0000

On Mon, Apr 9, 2018 at 4:36 AM Brian Trammell (IETF) <[email-masked]>
wrote:

> > In previous discussions, there's been strong interest in including
> connection migration -- in a limited form (for the "parking lot" use case)
> -- in QUICv1 (discussed in Melbourne in some detail; see <
> https://github.com/quicwg/wg-materials/blob/master/interim-18-01/minutes.md#connection-migration
> >).
> >
> > It looks like the discussion in <
> https://github.com/quicwg/base-drafts/issues/1271> about making it
> optional and how that gets indicated / negotiated is going nicely. Thanks
> for opening an issue.
> >
> Isn't the question we should be asking here "does QUIC with connection
> migration disabled (and no PNE) have similar linkability characteristics to
> QUIC with connection migration enabled (and PNE)?"?
>
> I'm still confused as to how PNE is supposed to mitigate linkability
> across a NAT rebinding that occurs without client knowledge in the presence
> of a server-supplied CID, which seems to me to be the most common NAT
> rebinding event type. If it indeed does not, then the linkability
> characteristics seem identical to me.
>

Agreed.  It would only be non-identical in the case of 0 byte connection
IDs, which I don't believe anyone expects to be common from client to
server.

> > As an aside -- the other threads about modifications to PNE to make them
> single-pass seem extremely promising; I'd encourage folks there to continue
> discussions and make a PR.
>
> +1. Making the design of PNE less problematic for offload and other future
> optimizations won't solve the underlying requirements problem (i.e. we are
> both treating linkability among 5-tuple flows by a specific on-path device,
> the LB, as a requirement to support via CID, as well as a requirement to
> mitigate linkability among 5-tuple flows by a all generic on-path devices
> via PNE, which is inherently contradictory  without a very precise
> definition of under which circumstances which requirement takes
> precedence). But it will reduce the cost from "probably mostly harmless" to
> "almost definitely mostly harmless", which should make the decision easier.
>
> Cheers,
>
> Brian
>
>
> >
> > Cheers,
> >
> >
> >
> >> Love this idea. Expanding it a bit further I don’t think we need to
> make PN transform a transport parameter but rather connection migration
> itself. We could also simply use QUIC versioning for this.
> >>
> >> 1. Ship QUIC v1 with no support for connection migration. Servers
> behind load balancers that are not QUIC aware will only advertise this
> version. I expect this to be a large fraction of servers especially those
> hosted in AWS, Azure, GCP. *This is important - I want the group to
> acknowledge that servers needs to be able to advertise support for
> connection migration*.
> >> 2. Ship QUIC v2 with support for connection migration. If a client and
> server use this version they MUST do PNE as per #1079. If we find a more
> efficient scheme before November then we use that instead.
> >>
> >> Eventually we will ship QUIV vN which has support for multi-path and we
> can revisit if PNE is needed or multiple PN spaces will work. Until then I
> can live with higher CPU cost for the fraction of connections that will
> migrate.
> >>
> >> Now there is the ossification problem in QUIC v1 as referred to above.
> We already require a random starting PN. The next problem is one that
> Kazuho brought up where middleboxes will start seeing strict increasing PN
> and try to reorder. I was thinking about short random jumps to fix this
> problem but the monotonic increase will remain. While I think sane
> middleboxes will not do any reordering, there might be a few crazy outliers
> like with TCP. I think we can solve this by making the in clear PN jump
> both forward and back within a "reordering degree". I am thinking about
> some ideas here and will reply back if I find a simple scheme.
> >>
> that there are enough problems to solve with UDP that worrying about the
> performance hit from encryption is premature
> >> Martin, we have plans to make UDP screaming fast in software. But the
> real advantage for TCP comes from the batch offloads (see email with
> subject " Impact of hardware offloads on network stack performance"). Batch
> offloads for QUIC are impossible to do without crypto offload. I would be
> happy to explain more on this front. I think we will exhaust software
> optimizations soon and then get blocked on crypto offload.
> >>
> >>
> >> Cc: Lars Eggert <[email-masked]>; Mark Nottingham <[email-masked]>;
> IETF QUIC WG <[email-masked]>; Patrick McManus <[email-masked]>
> extension/transport parameter to negotiate a new scheme, but it's obvious.
> This fits best with my view of things.
> >>
> number encryption function into identity or some cheaper function, then I
> think that's an OK outcome.  We should stop pretending that one size fits
> all is achievable; it's not been the case for years already.
> >>
> >> FWIW, what I got from the info that Praveen and Ian have shared is that
> there are enough problems to solve with UDP that worrying about the
> performance hit from encryption is premature.
> >>
> >
> > --

---

From: <[email-masked]>
Date: Mon, 9 Apr 2018 23:37:08 +0200

On 04/04/2018 20:06, Ted Hardie wrote:
> it seems pretty clear that the passive measurement and diagnostic purposes would
> work with fairly small sequence numbers, maybe even down to sizes small enough
> to make the linkability problem less scary.  The other issues ("does this look
> like quic" and "does this look like a known flow") seem like they don't really
> require packet numbers or even deliberately exposed sequence numbers, just
> consistency.   In other words, if we do anything to expose sequences, the wire
> image we engineer does not have to simply re-use the packet numbers used for
> protocol mechanics. It can focus on the message we want to send to the path instead.

If we're to secure a few bits as a "message we want to send to the path", e.g. 
for loss measurement, please note that subsequence numbers are just one option 
among many, with known shortcomings (only losses between sender and observer; 
lack of robustness to strong loss and reordering).

Instead, building on top of the Spin Bit, we propose a 7-bit extension named 
"Spin Counters":

     http://snaggletooth.akam.ai/IETF-101-HotRFC/10-Ferrieux.pdf

they give both up- and downstream losses, are robust to any level of loss and 
reordering, and leverage the Spin Bit's autoscaling to one congestion window.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 10 Apr 2018 01:04:52 +0000

Adding another PN would probably not be sufficient-- one would need to ensure that the ossified PN was presented with the right sequencing as well, which might be difficult, or expensive.
Ossification is surprisingly difficult to work around once it sets in, as even figuring out how it is impacting you can be substantial work.

-=R

On 4/5/18, 2:27 AM, "QUIC on behalf of Mirja Kühlewind" <[email-masked] on behalf of [email-masked]> wrote:

    

    
    
    Mirja

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 11 Apr 2018 19:18:21 +0000

Going back to Christian's original question on privacy holes, there is an attack for linking IP addresses in the voluntary migration case.

Let's consider the parking lot problem or in general case losing one network and roaming to another network. This is the primary use case for connection migration. In this case all active QUIC connections that can migrate, will do so. When these connections migrate they can change the CID, the local port number and packet numbers. However do notice that only the local 2-tuple changes for each connection, the server's 2-tuple remains the same. 

Let's assume a powerful adversary can collect a network sniff of all this traffic and builds a massive dataset. The adversary can then run a machine learning algorithm to identify time instants where a bunch of connections change their local 2-tuple at near the same time instant but continue going to the same server side 2-tuples. This will allow the adversary to link two IP addresses. The more times the user roams between networks back and forth the richer the correlation. The more open connections to different servers, the richer the correlation. Geolocation information can minimize the number of addresses the adversary needs to correlate. 

The fundamental problem seems to be that the substrate is UDP/IP and it is in the clear and allows linkability. 

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Christian Huitema
Sent: Thursday, April 5, 2018 8:34 AM
To: Mirja Kühlewind <[email-masked]>
Cc: [email-masked]
Subject: Privacy holes (was: Re: Getting to consensus on packet number encryption)

Mirja,

-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Wed, 11 Apr 2018 17:17:04 -0700

> The fundamental problem seems to be that the substrate is UDP/IP and it is in the clear and allows linkability. 

The server 2-tuple does provide some information, but we should not
overstate it. One element of the 2-tuple, the port number, is likely to
be set to 443 for pretty much all servers, so I don't think it is
terribly useful. The IP address is useful, but in many cases these
address will be shared by a large number of connections, either because
it is a big service like Microsoft.com, or because it ends in some kind
of CDN.

I don't like the general line of reasoning that "because there is
another hole here, we should not fix this hole there." If you accept
that, you end up with so many unpatched holes that the situation is
indeed hopeless. But if we fix now the holes that we can fix, and if
other holes prove important, we can fix them later. For example, if
tracking by IP is an important problem, we can start using multiple
addresses per server, or maybe some form of onion routing, or some CDN
trick. But if we stick a clear text ID in the headers, we lose.

-- Christian Huitema

---

From: Praveen Balasubramanian <[email-masked]>
Date: Thu, 12 Apr 2018 01:46:41 +0000

It's not just the server information. Its timing information that is far more valuable for the adversary. Not all clients will failover from wifi to cellular at the same time and hence the correlation will be very strong. You just need a few failover events in a day for a very strong correlation. What also makes this easy is that In a capture, a migration looks very different than a new connection. The attack seems very practical for a powerful adversary.

>> I don't like the general line of reasoning that "because there is another hole here, we should not fix this hole there."
I didn't state this and incremental progress is useful. I was simply trying to list possible privacy holes since that was the question on this thread.

>> we can start using multiple addresses per server, or maybe some form of onion routing, or some CDN trick
In an ideal world yes. But most deployments will be incremental on current infra.

>> But if we stick a clear text ID in the headers, we lose
This is too broad - sticking the *same* cleartext ID is a problem. Different cleartext IDs is no different than using encryption i.e. make clear text fields look like a new connection on the wire. 

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Christian Huitema
Sent: Wednesday, April 11, 2018 5:17 PM
To: [email-masked]
Subject: Re: Privacy holes

> The fundamental problem seems to be that the substrate is UDP/IP and it is in the clear and allows linkability. 

The server 2-tuple does provide some information, but we should not overstate it. One element of the 2-tuple, the port number, is likely to be set to 443 for pretty much all servers, so I don't think it is terribly useful. The IP address is useful, but in many cases these address will be shared by a large number of connections, either because it is a big service like Microsoft.com, or because it ends in some kind of CDN.

I don't like the general line of reasoning that "because there is another hole here, we should not fix this hole there." If you accept that, you end up with so many unpatched holes that the situation is indeed hopeless. But if we fix now the holes that we can fix, and if other holes prove important, we can fix them later. For example, if tracking by IP is an important problem, we can start using multiple addresses per server, or maybe some form of onion routing, or some CDN trick. But if we stick a clear text ID in the headers, we lose.

-- Christian Huitema

---

From: Ian Swett <[email-masked]>
Date: Thu, 12 Apr 2018 23:36:15 +0000

Yes, for a powerful adversary, this seems fairly tractable.

Making connection migration look like a new connection would likely help a
lot.

On Wed, Apr 11, 2018 at 3:18 PM Praveen Balasubramanian <pravb=
[email-masked]> wrote:

> Going back to Christian's original question on privacy holes, there is an
> attack for linking IP addresses in the voluntary migration case.
>
> Let's consider the parking lot problem or in general case losing one
> network and roaming to another network. This is the primary use case for
> connection migration. In this case all active QUIC connections that can
> migrate, will do so. When these connections migrate they can change the
> CID, the local port number and packet numbers. However do notice that only
> the local 2-tuple changes for each connection, the server's 2-tuple remains
> the same.
>
> Let's assume a powerful adversary can collect a network sniff of all this
> traffic and builds a massive dataset. The adversary can then run a machine
> learning algorithm to identify time instants where a bunch of connections
> change their local 2-tuple at near the same time instant but continue going
> to the same server side 2-tuples. This will allow the adversary to link two
> IP addresses. The more times the user roams between networks back and forth
> the richer the correlation. The more open connections to different servers,
> the richer the correlation. Geolocation information can minimize the number
> of addresses the adversary needs to correlate.
>

---

From: Ian Swett <[email-masked]>
Date: Thu, 12 Apr 2018 23:36:15 +0000

On Wed, Apr 11, 2018 at 3:18 PM Praveen Balasubramanian <pravb=
[email-masked]> wrote:

> the path as originating from a migration, but it does not tell from where..

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 13 Apr 2018 09:32:29 +0900

2018-04-13 8:36 GMT+09:00 Ian Swett <ianswett=[email-masked]>:
> Yes, for a powerful adversary, this seems fairly tractable..
>

I am not sure if that is the right path.

We have three ways in which a new set of 5-tuple starts getting used.

* new connection
* voluntary migration
* involuntary migration (i.e. NAT rebinding)

For the involuntary migration case, the first packet will always be a
short header packet.

If we make voluntary migration look like a new connection, the chance
of involuntary migration getting blocked by a middlebox becomes higher
(as we have seen gateways blocking the TLS 1.3 records, or the '07'
case).

Therefore, my preference goes to making all the three cases
indistinguishable (which is ideal but hard), or keeping voluntary and
non-voluntary migration look the same (as we do now) at the same time
making sure that the middlebox developers understand migration happens
[1].

[1] https://www.ietf.org/mail-archive/web/quic/current/msg03820.html

> On Wed, Apr 11, 2018 at 3:18 PM Praveen Balasubramanian
> <pravb=[email-masked]> wrote:
>>
>> migrate, will do so. When these connections migrate they can change the CID,
>> the local port number and packet numbers. However do notice that only the
>> local 2-tuple changes for each connection, the server's 2-tuple remains the
>> same.
>>
>> > On Apr 5, 2018, at 2:26 AM, Mirja Kühlewind
>> > <[email-masked]..ethz.ch> wrote:
>> > ...
>> > does not have a handshake. If we want to address the linkability problem, we
>> > would need to do much more, probably baring more hits on performance.
>>
>> Mirja,
>>

-- 
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 13 Apr 2018 09:32:29 +0900

* new connection
* voluntary migration

-- 
Kazuho Oku

---

From: Martin Thomson <[email-masked]>
Date: Fri, 13 Apr 2018 11:09:00 +1000

This is also my preference, in full realization of the downsides of
that.  Trying to make migration look like a handshake is also really
hard to get right.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 13 Apr 2018 11:09:00 +1000

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 13 Apr 2018 13:06:40 +0200

Hi Roberto,

see below.

> Am 10.04.2018 um 03:04 schrieb Roberto Peon <[email-masked]>:
> 

Yes, that is easy. Why do you think that is difficult or expensive?

> Ossification is surprisingly difficult to work around once it sets in, as even figuring out how it is impacting you can be substantial work.

That is true for a protocol like TCP very the invariant is large and non of the bits are encrypted. For QUIC we have a completely different situation.

Mirja

---

From: Mike Bishop <[email-masked]>
Date: Fri, 13 Apr 2018 16:31:44 +0000

+1.  There are too many sharp corners on trying to make migration look like 0-RTT, while still enabling the recipient to know which Initial/0-RTT packets are really Initial/0-RTT with consistency.

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Martin Thomson
Sent: Thursday, April 12, 2018 6:09 PM
To: Kazuho Oku <[email-masked]>
Cc: Praveen Balasubramanian <pravb=[email-masked]>; IETF QUIC WG <[email-masked]>; Ian Swett <ianswett=[email-masked]>
Subject: Re: Privacy holes (was: Re: Getting to consensus on packet number encryption)

This is also my preference, in full realization of the downsides of that.  Trying to make migration look like a handshake is also really hard to get right.

On Fri, Apr 13, 2018 at 10:32 AM, Kazuho Oku <[email-masked]> wrote:
>> Making connection migration look like a new connection would likely 
>> help a lot.
>
>>> Going back to Christian's original question on privacy holes, there 
>>> is an attack for linking IP addresses in the voluntary migration case.
>>>
>>> network and roaming to another network. This is the primary use case 
>>> for connection migration. In this case all active QUIC connections 
>>> that can migrate, will do so. When these connections migrate they 
>>> can change the CID, the local port number and packet numbers. 
>>> However do notice that only the local 2-tuple changes for each 
>>> connection, the server's 2-tuple remains the same.
>>>
>>> Let's assume a powerful adversary can collect a network sniff of all 
>>> this traffic and builds a massive dataset. The adversary can then 
>>> run a machine learning algorithm to identify time instants where a 
>>> bunch of connections change their local 2-tuple at near the same 
>>> time instant but continue going to the same server side 2-tuples. 
>>> This will allow the adversary to link two IP addresses. The more 
>>> times the user roams between networks back and forth the richer the 
>>> correlation. The more open connections to different servers, the 
>>> richer the correlation. Geolocation information can minimize the number of addresses the adversary needs to correlate.
>>>
>>> The fundamental problem seems to be that the substrate is UDP/IP and 
>>> it is in the clear and allows linkability.
>>>
>>> Subject: Privacy holes (was: Re: Getting to consensus on packet 
>>> number
>>> encryption)
>>>
>>>
>>>
>>> > destination IP, both port numbers and the fact that a migrated 
>>> > connection does not have a handshake. If we want to address the 
>>> > linkability problem, we would need to do much more, probably baring more hits on performance.
>>>
>>> Mirja,
>>>
>>> You rightly point out that the connection ID and the Packet Number 
>>> are not the only elements that provide linkability. There is also of 
>>> course the UDP source port. That one is not much of an issue for 
>>> servers, but it is an issue for clients. We are not spelling that 
>>> out in the draft. We should, because clients can trivially close that hole when doing migration.
>>>
>>> I am not sure that the absence of negotiation divulges much data. It 
>>> marks the path as originating from a migration, but it does not tell from where.
>>> But there might be an ossification issue. We will get that 
>>> ossification if we train middleboxes to believe that connection 
>>> always start with an observable negotiation. So maybe we should explore ways to grease that.
>>>

---

From: Mike Bishop <[email-masked]>
Date: Fri, 13 Apr 2018 16:31:44 +0000

---

From: Roberto Peon <[email-masked]>
Date: Mon, 16 Apr 2018 19:56:19 +0000

Paraphrasing:

> PN insufficient to fix ossification, why is that difficult/expensive?

As described in the other subthread, you'd have to emulate the same ordering of the previously-ossified PN while also doing whatever newness you want with the new PN.
That sounds fabulously complex. You'd end up running two different implementations simultaneously.

> Ossification/invariant without encryption:
Encryption solves it, sounds like we agree on that 😊

-=R

________________________________
From: QUIC <[email-masked]> on behalf of Mirja Kühlewind <[email-masked]>
Sent: Friday, April 13, 2018 4:06:40 AM
To: Roberto Peon
Cc: [email-masked]; Christian Huitema
Subject: Re: Getting to consensus on packet number encryption

Hi Roberto,

see below.

Mirja

---

From: Christian Huitema <[email-masked]>
Date: Mon, 16 Apr 2018 22:05:04 -0700

Since you all want to read more text, I wrote a summary of the various
proposals in the QUIC Wiki:
https://github.com/quicwg/base-drafts/wiki/Summary-of-the-PN-encryption-issues-and-alternatives.

Of course, that summary reflects my own modest understanding. But then
it's a wiki, so you can easily correct me.

-- Christian Huitema

On 4/16/2018 12:56 PM, Roberto Peon wrote:
> As described in the other subthread, you'd have to emulate the same
> ordering of the previously-ossified PN while also doing whatever
> newness you want with the new PN.
> ------------------------------------------------------------------------
> *From:* QUIC <[email-masked]> on behalf of Mirja Kühlewind
> <[email-masked]>
> *Sent:* Friday, April 13, 2018 4:06:40 AM
> *To:* Roberto Peon
> *Cc:* [email-masked]; Christian Huitema
> *Subject:* Re: Getting to consensus on packet number encryption
>  
> Hi Roberto,
>
> see below.
>
> > Adding another PN would probably not be sufficient-- one would need
> to ensure that the ossified PN was presented with the right sequencing
> as well, which might be difficult, or expensive.
>
> > Ossification is surprisingly difficult to work around once it sets
> in, as even figuring out how it is impacting you can be substantial work.
>
> That is true for a protocol like TCP very the invariant is large and
> non of the bits are encrypted. For QUIC we have a completely different
> situation.
>
> Mirja
>
>
> >
> > -=R
> >
> > On 4/5/18, 2:27 AM, "QUIC on behalf of Mirja Kühlewind"
> <[email-masked] on behalf of [email-masked]>
> wrote:
> >
> >> If we are in the business of repeating our positions, so be it. I
> am pretty much on the same line as Patrick and others:
> >>
> >> 1) Privacy is not optional. And yes, removing "linkability through
> PN" is important, because when the connection ID and the source
> address also change, the only other cue for linkability is timing.
> Timing can be addressed by having overlapping connections.
> >>
> >    Then an argument for ossification was broad up. However, we
> really cannot compare the situation here with TCP. With TCP everything
> is in the clear incl. also the retransmissions. In case of MPTCP this
> was rather the problem the the seq# because if you’d use the same seq#
> number space on both connections it would like loss but without a
> retransmission. Further boxes that reorder, usually do that in
> situation where packets were transmitted for a known reason out-of
> order previously, e.g. when multiple paths are used in the network.
> The recommendation we still give in the IETF transport area is that in
> such a situation the scheduling should not be done on a per-packet
> level but per-flow. However, those scheme that do per-packet
> scheduling usually also apply mechanism to re-order afterwards. They
> can just use the TCP seq# for that but for non-TCP they would simply
> add their own encapsulation with a seq#…
> >
> >    in case of QUIC that does not mean that we can’t change the
> transport machinery anymore. Currently the exposed PN is also used for
> congestion control and retransmission, however, we can still add
> another PN in the encrypted part and resolved the fact that the
> current PN is overloaded for different functions.
> >
> >    Currently, I really still don’t see the benefit of PNE but only
> additional complexity and known performance degradation.
> >
> >    Mirja
> >
> >
> >
> >
> >
>

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 17 Apr 2018 15:15:09 +0900

Hi Christian,

Thank you for writing the wonderful summary.

Reading it, I think that I now see two issues with the alternative
approaches that use a weaker cipher (i.e. alternative PN encryption, 64bit
encrypted PN).

> We can argue that the algorithm does not have to be as robust as AES. The
main requirement is to prevent real time analysis of sequence numbers by
middle-boxes. This is achieved as long as the encryption key cannot be
retrieved in a short time by the middle-boxes. If they could do that, we
would be once again on the path to ossification. XOR and simple obfuscation
probably don't meet that goal, but IPCrypt probably does, and a more robust
algorithm certainly would.

As you state, it is true that preventing real time analysis is important.
But are we sure that we do not need to consider offline attacks? I would be
interested in arguing about the other requirement than ossification:
privacy.

A pervasive monitor might collect CIDs and encrypted PNs that were being
used, and then use the encrypted PNs as a source to correlate CIDs to
reconstruct a QUIC connection. Are we sure that a weak cipher does not leak
such trace?

Considering that, I think that the alternative approaches should have
per-CID PN encryption keys, even though I am not sure if there are no other
privacy-related issues.

The other issue is about crypto agility. #1179 has agility in sense that
the corresponding CTR mode is used for encrypting the PN. OTOH, the two
alternatives using a simple cipher are not agile. So we might need to
define two types of simpler ciphers for PN encryption, if consider that
privacy issues (like the one above) might arise in the future.

2018-04-17 14:05 GMT+09:00 Christian Huitema <[email-masked]>:

> proposals in the QUIC Wiki: https://github.com/quicwg/base
> -drafts/wiki/Summary-of-the-PN-encryption-issues-and-alternatives.
>
>
-- Christian Huitema
> ordering of the previously-ossified PN while also doing whatever newness
> you want with the new PN.
> *From:* QUIC <[email-masked]> <[email-masked]> on behalf of
> > Am 10.04.2018 um 03:04 schrieb Roberto Peon <[email-masked]>
> <[email-masked]>:
> >
> > Adding another PN would probably not be sufficient-- one would need to
> ensure that the ossified PN was presented with the right sequencing as
> well, which might be difficult, or expensive.
>
> > Ossification is surprisingly difficult to work around once it sets in,
> as even figuring out how it is impacting you can be substantial work.
>
> That is true for a protocol like TCP very the invariant is large and non
> of the bits are encrypted. For QUIC we have a completely different
> situation.
>
> Mirja
>
>
> >
> > -=R
> >
> > On 4/5/18, 2:27 AM, "QUIC on behalf of Mirja Kühlewind" <[email-masked]
> on behalf of [email-masked]>
> <[email-masked]@tik.ee.ethz.ch> wrote:
> >
> >> Am 05.04.2018 um 02:44 schrieb Christian Huitema <[email-masked]>
> <[email-masked]>:
> >>
> >    Then an argument for ossification was broad up. However, we really
> cannot compare the situation here with TCP. With TCP everything is in the
> clear incl. also the retransmissions. In case of MPTCP this was rather the
> problem the the seq# because if you’d use the same seq# number space on
> both connections it would like loss but without a retransmission. Further
> boxes that reorder, usually do that in situation where packets were
> transmitted for a known reason out-of order previously, e.g. when multiple
> paths are used in the network. The recommendation we still give in the IETF
> transport area is that in such a situation the scheduling should not be
> done on a per-packet level but per-flow. However, those scheme that do
> per-packet scheduling usually also apply mechanism to re-order afterwards.
> They can just use the TCP seq# for that but for non-TCP they would simply
> >    in case of QUIC that does not mean that we can’t change the transport
> machinery anymore. Currently the exposed PN is also used for congestion
> control and retransmission, however, we can still add another PN in the
> encrypted part and resolved the fact that the current PN is overloaded for
> different functions.
> >

-- 
Kazuho Oku

---

From: Christian Huitema <[email-masked]>
Date: Tue, 17 Apr 2018 11:58:58 -0700

On 4/16/2018 11:15 PM, Kazuho Oku wrote:
> approaches that use a weaker cipher (i..e. alternative PN encryption,
> 64bit encrypted PN).
>
> > We can argue that the algorithm does not have to be as robust as
> AES.. The main requirement is to prevent real time analysis of
> sequence numbers by middle-boxes. This is achieved as long as the
> encryption key cannot be retrieved in a short time by the
> middle-boxes. If they could do that, we would be once again on the
> path to ossification. XOR and simple obfuscation probably don't meet
> that goal, but IPCrypt probably does, and a more robust algorithm
> certainly would.
>
> As you state, it is true that preventing real time analysis is
> important. But are we sure that we do not need to consider offline
> attacks? I would be interested in arguing about the other requirement
> than ossification: privacy.
>
> A pervasive monitor might collect CIDs and encrypted PNs that were
> being used, and then use the encrypted PNs as a source to correlate
> CIDs to reconstruct a QUIC connection. Are we sure that a weak cipher
> does not leak such trace?

Yes, you are right. The argument about tolerating weak algorithms only
applies if we have another solution to prevent linkability, such as
different sequence numbers and encryption keys for each Connection ID.
So it comes with a high cost in term of software complexity.

> per-CID PN encryption keys, even though I am not sure if there are no
> other privacy-related issues.
Yes.
>
> The other issue is about crypto agility. #1179 has agility in sense
> that the corresponding CTR mode is used for encrypting the PN. OTOH,
> the two alternatives using a simple cipher are not agile. So we might
> need to define two types of simpler ciphers for PN encryption, if
> consider that privacy issues (like the one above) might arise in the
> future.
Yes.  I updated the Wiki page to reflect your comments.

-- Christian Huitema

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 18 Apr 2018 13:48:19 +0900

Hi Christian,

Thank you for updating the wiki page. And the comparison table is awesome.

Looking at that, (it might be unsurprising but) my two cents go to
merging #1079 and continue to consider a hardware-friendly variant as
an extension (until it turns out to be unnecessary).

The reasons I prefer #1079 are:
 * it is easy to implement in software
 * efficient on wire
 * computational overhead is negligible
 * crypto is secure and agile to changes
 * majority of the implementors seem to be happy with the approach

On the PR I see a proposal of a hardware friendly variant (that
modifies AES-GCM). All of us might end up happy with a single scheme.

If some of us (especially those interested in hardware crypto offload)
are unhappy with the solution, we can consider having a negotiated
extension of QUIC that uses a different scheme.

For that variant, my preference goes to the one using nonce due to the
following reasons:
 * it requires the least change from #1079 in terms of key scheduling
 * it would have the broadest range of hardware support

The latter point is from my assumption that most if not all of devices
that have AES-GCM support are capable of accepting an explicit nonce,
while some if not many of them might be incapable of doing some 32-bit
or 64-bit crypto for decrypting PN.

-- 
Kazuho Oku

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 18 Apr 2018 18:09:23 +1000

Thanks Christian; that's extremely helpful (and considering that I was about to embark on the same task, but do a much worse job, I'm especially appreciative).

Everyone please have a read and direct your comments back here.

One aspect that still hasn't been made clear AFAICT -- you say:

"""The process requires two passes, fetching some output of the encryption to seed the PN encryption. This is problematic for hardware implementations that typically don't buffer the output for further access."""

... and then later:

"""Single pass, thus mostly implementable in hardware if we assume that the PN encryption/decryption can be done in software."""

Having a bit more context about the hardware implementation limitations here would be extremely helpful. Specifically, is "problematic" one of:

* "More difficult for existing hardware (e.g., without firmware updates, etc.)"
* "Impossible for existing hardware, but possible for bespoke hardware"
* "Impossible in any hardware"
* Prohibitively expensive for current/future hardware
* Something else?

Cheers,

> On 17 Apr 2018, at 3:05 pm, Christian Huitema <[email-masked]> wrote:
> 
> Since you all want to read more text, I wrote a summary of the various proposals in the QUIC Wiki: https://github.com/quicwg/base-drafts/wiki/Summary-of-the-PN-encryption-issues-and-alternatives.
> 
> Of course, that summary reflects my own modest understanding. But then it's a wiki, so you can easily correct me.
> 
> -- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Wed, 18 Apr 2018 09:38:01 -0700

On 4/18/2018 1:09 AM, Mark Nottingham wrote:
> .... and then later:

I tried to express the issue in my own words, based on discussions in
this list. But as I said, this is a Wiki. People with more hands-on
experience on hardware acceleration would be welcome to add a small
section describing the issue with hardware implementations.

-- Christian Huitema

---

From: "Deval, Manasi" <[email-masked]>
Date: Mon, 23 Apr 2018 17:55:06 +0000

Hi All,

I had brought up the issue with PNE several weeks ago as a barrier to hardware offload. After further review, it looks like a hardware offload can implement the PNE at a small cost. 

The implementation can modify current HW crypto accelerators to support encrypting a buffer in the first pass and then encrypting packet number in the 2nd pass as already discussed on this thread. The exact requirement (header checksum, packet length encoding) is still in flux so there may be some small variations depending on the accelerator and final algorithm chosen for PNE. Future offload designs can do more to further reduce the overhead.

This re-opens the issue of crypto offload in the context of other offloads. There is another thread talking about the offload in context of segmentation. I'll respond to that thread separately.

Thanks,
Manasi

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Christian Huitema
Sent: Wednesday, April 18, 2018 9:38 AM
To: Mark Nottingham <[email-masked]>
Cc: [email-masked]
Subject: Re: Getting to consensus on packet number encryption

> * Prohibitively expensive for current/future hardware

I tried to express the issue in my own words, based on discussions in this list. But as I said, this is a Wiki. People with more hands-on experience on hardware acceleration would be welcome to add a small section describing the issue with hardware implementations.

-- Christian Huitema

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 23 Apr 2018 15:49:11 -0700

Thanks for the update, Manasi. In practice, can you state what this small
cost is? Is it the second crypto pass, or are there other deployment costs
that you are thinking about?

On Mon, Apr 23, 2018 at 10:55 AM, Deval, Manasi <[email-masked]>
wrote:

> Hi All,
>
> I had brought up the issue with PNE several weeks ago as a barrier to
> hardware offload. After further review, it looks like a hardware offload
> can implement the PNE at a small cost.
>
> The implementation can modify current HW crypto accelerators to support
> encrypting a buffer in the first pass and then encrypting packet number in
> the 2nd pass as already discussed on this thread. The exact requirement
> (header checksum, packet length encoding) is still in flux so there may be
> some small variations depending on the accelerator and final algorithm
> chosen for PNE. Future offload designs can do more to further reduce the
> overhead.
>
> This re-opens the issue of crypto offload in the context of other
> offloads. There is another thread talking about the offload in context of
> segmentation. I'll respond to that thread separately.
>
> Thanks,
> Manasi
>
>
> > Thanks Christian; that's extremely helpful (and considering that I was
> about to embark on the same task, but do a much worse job, I'm especially
> appreciative).
> >
> > """The process requires two passes, fetching some output of the
> encryption to seed the PN encryption. This is problematic for hardware
> implementations that typically don't buffer the output for further
> access."""
> >
> > .... and then later:
> >
> > """Single pass, thus mostly implementable in hardware if we assume that
> the PN encryption/decryption can be done in software."""
> >
> > Having a bit more context about the hardware implementation limitations
> here would be extremely helpful. Specifically, is "problematic" one of:
> >
> I tried to express the issue in my own words, based on discussions in this
> list. But as I said, this is a Wiki. People with more hands-on experience
> on hardware acceleration would be welcome to add a small section describing
> the issue with hardware implementations.
>
> -- Christian Huitema
>
>

---

From: Ian Swett <[email-masked]>
Date: Mon, 23 Apr 2018 23:22:06 +0000

Yes, thanks for the update.

Can you estimate the portion of existing hardware(that supports TCP crypto
offload) which could support crypto offload for QUIC with PNE vs without
PNE?

On Mon, Apr 23, 2018 at 6:49 PM Jana Iyengar <[email-masked]> wrote:

>> experience on hardware acceleration would be welcome to add a small section
>> describing the issue with hardware implementations.
>>
>> -- Christian Huitema
>>
>>
>

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 24 Apr 2018 11:53:39 +0900

Manasi, thank you for the update.

In addition to the points that Jana and Ian made, it would be great if
you could provide us some estimation on if using different keys for PN
encryption and payload encryption affects HW offload performance /
portion of existing hardware that can support this form of encryption.

On https://github.com/quicwg/base-drafts/pull/1079#issuecomment-382007477,
Antoine has recently suggested that the PNE scheme proposed in #1079
can be tweaked to use the same key for encrypting the PN and the
payload (they will use same key but different nonces).

This is minor to the points made by Jana and Ian, but I think it would
be something good to know to make a decision.

2018-04-24 8:22 GMT+09:00 Ian Swett <ianswett=[email-masked]>:
>>> hardware offload. After further review, it looks like a hardware offload can
>>> implement the PNE at a small cost.
>>>

-- 
Kazuho Oku

---

From: Christian Huitema <[email-masked]>
Date: Wed, 25 Apr 2018 11:13:34 +0100

> The implementation can modify current HW crypto accelerators to support encrypting a buffer in the first pass and then encrypting packet number in the 2nd pass as already discussed on this thread.…

Thanks for the information, Manasi. I have modified the wiki page
describing the PNE issues and alternatives to reflect this new data:
https://github.com/quicwg/base-drafts/wiki/Summary-of-the-PN-encryption-issues-and-alternatives.
With that new information, it appears that PR #1079 is superior to every
other alternative.

-- Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 25 Apr 2018 07:53:29 -0400

Another observation point:

If there are so many concurrent paths in the host system that PNE matters,
then the cost of an additional keyed lookup is likely higher than PNE. You
can’t do much better than 40ns per lookup under ideal circumstances where a
minimum number of cache lines are hit, unless your working set is small.
Red-Black tree easily gets up to 2-400ns with current hardware. PNE is
below 30ns estimated - iff the decryption context is readily available.

The question is then, would avoiding PNE lead to additional lookups that
would not otherwise be required. Or, on the contrary would PNE lead to
additional lookups due to a separate encryption context. Lookup broadly
means fetching cache lines not otherwise required.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 25 April 2018 at 12.14.35, Christian Huitema ([email-masked]) wrote:

-- Christian Huitema

---

From: Mike Bishop <[email-masked]>
Date: Wed, 25 Apr 2018 19:54:03 +0000

Yes -- it seems that the biggest objection to #1079 was the difficulty in hardware implementation.  If we're hearing that hardware implementation is feasible at a reasonable cost, then I think we might have a winner.

The CPU cost for a software implementation is still worth considering, and an option to not encrypt is probably reasonable to limit that burden for implementations / use cases that don't care.

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Christian Huitema
Sent: Wednesday, April 25, 2018 3:14 AM
To: Deval, Manasi <[email-masked]>; Mark Nottingham <[email-masked]>
Cc: [email-masked]
Subject: Re: Getting to consensus on packet number encryption

> The implementation can modify current HW crypto accelerators to support encrypting a buffer in the first pass and then encrypting packet number in the 2nd pass as already discussed on this thread.…

Thanks for the information, Manasi. I have modified the wiki page describing the PNE issues and alternatives to reflect this new data:
https://github.com/quicwg/base-drafts/wiki/Summary-of-the-PN-encryption-issues-and-alternatives.
With that new information, it appears that PR #1079 is superior to every other alternative.

-- Christian Huitema

---

From: Ian Swett <[email-masked]>
Date: Wed, 25 Apr 2018 19:56:57 +0000

Hi Mike,

To clarify, are you suggesting adding a way to disable packet number
encryption via negotiation in the v1 spec as well as adopting #1079?  Or
would the choice of whether PNE is to be used unilateral, such as a
transport param?

On Wed, Apr 25, 2018 at 3:54 PM Mike Bishop <[email-masked]> wrote:

> Yes -- it seems that the biggest objection to #1079 was the difficulty in
> hardware implementation.  If we're hearing that hardware implementation is
> feasible at a reasonable cost, then I think we might have a winner.
>
> The CPU cost for a software implementation is still worth considering, and
> an option to not encrypt is probably reasonable to limit that burden for
> implementations / use cases that don't care.
>
> To: Deval, Manasi <[email-masked]>; Mark Nottingham <[email-masked]
> >
> Cc: [email-masked]

---

From: Mike Bishop <[email-masked]>
Date: Wed, 25 Apr 2018 22:01:32 +0000

I think that’s been suggested before, though we’d need to sort out the details of what that looks like.  I don’t have a particular design in mind.

From: Ian Swett [mailto:[email-masked]]
Sent: Wednesday, April 25, 2018 12:57 PM
To: Mike Bishop <[email-masked]>
Cc: Christian Huitema <[email-masked]>; Deval, Manasi <[email-masked]>; Mark Nottingham <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

Hi Mike,

To clarify, are you suggesting adding a way to disable packet number encryption via negotiation in the v1 spec as well as adopting #1079?  Or would the choice of whether PNE is to be used unilateral, such as a transport param?

On Wed, Apr 25, 2018 at 3:54 PM Mike Bishop <[email-masked]<mailto:[email-masked]>> wrote:
Yes -- it seems that the biggest objection to #1079 was the difficulty in hardware implementation.  If we're hearing that hardware implementation is feasible at a reasonable cost, then I think we might have a winner.

-----Original Message-----
From: QUIC <[email-masked]<mailto:[email-masked]>> On Behalf Of Christian Huitema
Sent: Wednesday, April 25, 2018 3:14 AM
To: Deval, Manasi <[email-masked]<mailto:[email-masked]>>; Mark Nottingham <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

-- Christian Huitema

---

From: Ian Swett <[email-masked]>
Date: Thu, 26 Apr 2018 00:21:05 +0000

It has been, and I'm personally supportive of it, because I believe it'll
be useful for datacenter QUIC use cases.

But the WG has to decide:
 1) Is it allowable to disable PNE?
 2) What the mechanism is?  ie: Unilateral via TransportParams or
negotiated via an extension or ?

On Wed, Apr 25, 2018 at 6:01 PM Mike Bishop <[email-masked]> wrote:

> I think that’s been suggested before, though we’d need to sort out the
> details of what that looks like.  I don’t have a particular design in mind.
>
>
>
> *From:* Ian Swett [mailto:[email-masked]]
> *Sent:* Wednesday, April 25, 2018 12:57 PM
> *To:* Mike Bishop <[email-masked]>
> *Cc:* Christian Huitema <[email-masked]>; Deval, Manasi <
> [email-masked]>; Mark Nottingham <[email-masked]>; IETF QUIC WG <
> [email-masked]>

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 26 Apr 2018 10:22:37 +0900

2018-04-26 9:21 GMT+09:00 Ian Swett <ianswett=[email-masked]>:
> It has been, and I'm personally supportive of it, because I believe it'll be
> useful for datacenter QUIC use cases.
>

There are two concerns that PNE is trying to address: ossification and privacy.

I won't mind if some portion of QUIC-like traffic has no mechanism to
prevent ossification. This is because we can train the middleboxes to
not ossify the PN field by having PNE in the majority of the traffic
that they will see.

OTOH, I would be worried of privacy implications. Once you define a
protocol, it is hard to control how it is going to be used.

But with that said, I think we might better wait to see if somebody
still thinks he/she *needs* QUIC without PNE.

>  2) What the mechanism is?  ie: Unilateral via TransportParams or negotiated
> via an extension or ?
>
>> Cc: Christian Huitema <[email-masked]>; Deval, Manasi
>> <[email-masked]>; Mark Nottingham <[email-masked]>; IETF QUIC WG
>> <[email-masked]>
>> https://github.com/quicwg/base-drafts/wiki/Summary-of-the-PN-encryption-issues-and-alternatives..

-- 
Kazuho Oku

---

From: Erik Kline <[email-masked]>
Date: Thu, 26 Apr 2018 01:46:08 +0000

Couldn't a middlebox have a policy where it permits QUIC sessions w/o
PNE but blocks sessions with PNE?  Then implementations would be
forced to choose how adapt: break altogether, maybe try TCP, or
disable PNE and carry on.

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 26 Apr 2018 11:25:10 +0900

> disable PNE and carry on.

There are two ways of negotiating the use (or non-use) of PNE. One is
use Transport Parameters and the other is to use a different QUIC
version.

Use of Transport Parameters is secure because it is part of the TLS
handshake. Version negotiation has it's own protection against
downgrade (or upgrade) attacks (see
https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#rfc.section.6.4.4).

So I would assume that a middlebox trying to intervene in the
negotiation of PNE use can be detected.

That said, I do see your point that if some middleboxes decide to pass
QUIC without PNE but block QUIC with PNE, we might be incentivized to
use QUIC without PNE. In such case, we might end up having privacy
leaks & ossification.

I think that is a good argument for always having PNE turned on.

-- 
Kazuho Oku

---

From: Patrick McManus <[email-masked]>
Date: Thu, 26 Apr 2018 08:37:39 -0400

> be useful for datacenter QUIC use cases.

can you layout the datacenter and PNE mismatch? It is just "more work" or
is it an interaction with other hardware offloads?

---

From: Ian Swett <[email-masked]>
Date: Thu, 26 Apr 2018 16:15:39 +0000

>From my perspective, it's just "more work" and certain workloads are really
CPU cost sensitive in datacenters in a way that isn't true on the public
internet.

That being said, datacenters may start moving to large MTUs earlier than
the public internet, which would greatly decrease the practical cost of PNE.

Given where we're at, even though I believe it's likely someone will want
to disable PNE for some use case at some point, I think we should spec PNE
as the only option for now and when someone needs to disable it, it can be
up to them to create a mechanism, use it in their own implementation, and
if they care to standardize it, they can?

---

From: Patrick McManus <[email-masked]>
Date: Thu, 26 Apr 2018 12:35:57 -0400

sgtm

On Thu, Apr 26, 2018 at 12:15 PM, Ian Swett <[email-masked]> wrote:

> From my perspective, it's just "more work" and certain workloads are
> really CPU cost sensitive in datacenters in a way that isn't true on the
> public internet.
>
>> On Wed, Apr 25, 2018 at 8:21 PM, Ian Swett <ianswett=40google.com@dmarc.
>> ietf.org> wrote:
>>
>>> It has been, and I'm personally supportive of it, because I believe
>>> it'll be useful for datacenter QUIC use cases.
>>>
>>

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 26 Apr 2018 16:51:46 +0000

> if some middleboxes decide to pass QUIC without PNE but block QUIC with PNE, we might be incentivized to use QUIC without PNE.

If a middlebox really dislikes PNE, it can just block QUIC altogether, forcing a TCP fallback.  Is TCP better than QUIC-without-PNE from any POV in that situation?  (And if so, this should be the recommendation to QUIC stacks with negotiable PNE -- fallback to TCP, if path is blocking PNE.)

- Igor

-----Original Message-----
From: Kazuho Oku [mailto:[email-masked]] 
Sent: Wednesday, April 25, 2018 10:25 PM
To: Erik Kline <[email-masked]>
Cc: Mark Nottingham <[email-masked]>; Mike Bishop <[email-masked]>; Ian Swett <ianswett=[email-masked]>; Christian Huitema <[email-masked]>; IETF QUIC WG <[email-masked]>; Deval, Manasi <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

> disable PNE and carry on.

There are two ways of negotiating the use (or non-use) of PNE. One is use Transport Parameters and the other is to use a different QUIC version.

Use of Transport Parameters is secure because it is part of the TLS handshake. Version negotiation has it's own protection against downgrade (or upgrade) attacks (see https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#rfc.section.6.4.4).

So I would assume that a middlebox trying to intervene in the negotiation of PNE use can be detected.

That said, I do see your point that if some middleboxes decide to pass QUIC without PNE but block QUIC with PNE, we might be incentivized to use QUIC without PNE. In such case, we might end up having privacy leaks & ossification.

>>>> I think that’s been suggested before, though we’d need to sort out 
>>>> the details of what that looks like.  I don’t have a particular design in mind.
>>>>
>>>>
>>>>
>>>> <[email-masked]>; Mark Nottingham <[email-masked]>; IETF 
>>>> QUIC WG <[email-masked]>
>>>> To clarify, are you suggesting adding a way to disable packet 
>>>> number encryption via negotiation in the v1 spec as well as 
>>>> adopting #1079?  Or would the choice of whether PNE is to be used 
>>>> unilateral, such as a transport param?
>>>>
>>>>
>>>>
>>>> Yes -- it seems that the biggest objection to #1079 was the 
>>>> difficulty in hardware implementation.  If we're hearing that 
>>>> hardware implementation is feasible at a reasonable cost, then I think we might have a winner.
>>>>
>>>> The CPU cost for a software implementation is still worth 
>>>> considering, and an option to not encrypt is probably reasonable to 
>>>> limit that burden for implementations / use cases that don't care.
>>>>
>>>> > I had brought up the issue with PNE several weeks ago as a 
>>>> > barrier to hardware offload. After further review, it looks like 
>>>> > a hardware offload can implement the PNE at a small cost.
>>>> >
>>>> > The implementation can modify current HW crypto accelerators to 
>>>> > support encrypting a buffer in the first pass and then encrypting 
>>>> > packet number in the 2nd pass as already discussed on this 
>>>> > thread. The exact requirement (header checksum, packet length 
>>>> > encoding) is still in flux so there may be some small variations 
>>>> > depending on the accelerator and final algorithm chosen for PNE. 
>>>> > Future offload designs can do more to further reduce the overhead.
>>>>

--
Kazuho Oku

---

From: Praveen Balasubramanian <[email-masked]>
Date: Thu, 26 Apr 2018 18:41:52 +0000

> it's likely someone will want to disable PNE for some use case at some point
Yes there is a very good use case for this in datacenter and private networks because ossification or privacy concerns do not apply and the CPU cost is higher. And the added CPU cost will be a much higher percentage for smaller packets like ACKs. Hardware offloads will take some time to ship, stabilize and deploy.

Intra-datacenter does not imply only a single implementation doing something custom. We have a lot of TCP traffic between Windows and Linux systems for example. The mechanism must be standardized otherwise it breaks interop. Transport parameter is a better negotiation mechanism than versions because otherwise each such customization will cut the version space in half. Issue #1296<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fquicwg%2Fbase-drafts%2Fissues%2F1296&data=02%7C01%7Cpravb%40microsoft.com%7Cc4695c40509a4f37774208d5a496cd8c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636595889971448691&sdata=Affj5l5%2BnLCTwvLLgS9%2B3gze%2F9c9RrhFZIHwH%2Fk1Ljw%3D&reserved=0> tracks PN transform negotiation.

From: QUIC [mailto:[email-masked]] On Behalf Of Patrick McManus
Sent: Thursday, April 26, 2018 9:36 AM
To: Ian Swett <[email-masked]>
Cc: Mark Nottingham <[email-masked]>; huitema <[email-masked]>; Patrick McManus <[email-masked]>; Mike Bishop <[email-masked]>; IETF QUIC WG <[email-masked]>; Deval, Manasi <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

sgtm

On Thu, Apr 26, 2018 at 12:15 PM, Ian Swett <[email-masked]<mailto:[email-masked]>> wrote:
From my perspective, it's just "more work" and certain workloads are really CPU cost sensitive in datacenters in a way that isn't true on the public internet.

That being said, datacenters may start moving to large MTUs earlier than the public internet, which would greatly decrease the practical cost of PNE.

Given where we're at, even though I believe it's likely someone will want to disable PNE for some use case at some point, I think we should spec PNE as the only option for now and when someone needs to disable it, it can be up to them to create a mechanism, use it in their own implementation, and if they care to standardize it, they can?

On Thu, Apr 26, 2018 at 8:38 AM Patrick McManus <[email-masked]<mailto:[email-masked]>> wrote:

On Wed, Apr 25, 2018 at 8:21 PM, Ian Swett <ianswett=[email-masked]<mailto:ianswett=[email-masked]>> wrote:
It has been, and I'm personally supportive of it, because I believe it'll be useful for datacenter QUIC use cases.

can you layout the datacenter and PNE mismatch? It is just "more work" or is it an interaction with other hardware offloads?

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 27 Apr 2018 15:38:38 +0900

> If a middlebox really dislikes PNE, it can just block QUIC altogether, forcing a TCP fallback.  Is TCP better than QUIC-without-PNE from any POV in that situation?  (And if so, this should be the r…

The argument here is that if we define PNE as an option turned on by
default and see some middleboxes blocking PNE, we might be
incentivized to turn off PNE for *all* connections, because detecting
blockade and then falling back to a less secure protocol is hard and
time consuming.

-- 
Kazuho Oku

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Fri, 27 Apr 2018 19:20:53 +0200

Hi, all,

I am deeply skeptical of the possible existence of a box that would permit QUIC with unencrypted PN, but not with encrypted PN. The PN is simply not valuable enough to the path on its own to do the design work to detect and interfere with that case. Better in that case just to block QUIC and force TCP fallback.

Again, I would like to encourage everyone to consider that a middlebox must serve some perceived function to see deployment. 

That said, it seems like we have a way forward for not-terrible-for-offload PNE, with negotiation to shut it off in controlled environments where one wants to shave energy expenditure. +1 to Praveen’s suggestion to add transport parameters to the standard for this for purposes of interoperability.

Cheers, Brian

Sent from my iPhone

> On 27 Apr 2018, at 08:38, Kazuho Oku <[email-masked]> wrote:
> 
>>>> 2018-04-26 9:21 GMT+09:00 Ian Swett <ianswett=[email-masked]..org>:

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 27 Apr 2018 17:08:11 -0700

I agree that negotiating PNE runs the risk of having two types of
connections on the Internet. I agree that the risk of blockage doesn't seem
high, but honestly, having seen how middlebox features get deployed, I'm
not convinced that it won't happen (though I suspect it's unlikely to
happen at a large scale). I understand the desire to have it off within DCs
though, so I'm sensitive to that need.

I'd like to hear more about why making PNE optional isn't a decision that
we can punt to later though. I'd like to move along with PNE now, and come
back to negotiating this as an option once we have some deployment
work/experience in intra DC environments. I'm not yet convinced that this
is going to be your biggest cost in an intra-DC environment, and if it is,
then we can surely revisit this decision and add a param later to make PNE
optional, either later in v1 or in v2. We are talking about a very narrow
view into total cost at this point -- this one AES op is unlikely to be the
bottleneck. But I have no data, and I don't think anyone has any full
deployment experience yet.

TL;DR: I'd like to suggest that we move ahead with PR #1079, and continue
discussion on adding optional PNE in Issue #1296.

Brian: You're right that a middlebox must serve some perceived function to
see deployment. The problem is that the value and cost of any deployed
function however is also perceived, and everyone has different perceptions.

On Fri, Apr 27, 2018 at 10:20 AM, Brian Trammell (IETF) <[email-masked]>
wrote:

> Hi, all,
>
> I am deeply skeptical of the possible existence of a box that would permit
> QUIC with unencrypted PN, but not with encrypted PN. The PN is simply not
> valuable enough to the path on its own to do the design work to detect and
> interfere with that case. Better in that case just to block QUIC and force
> TCP fallback.
>
> Again, I would like to encourage everyone to consider that a middlebox
> must serve some perceived function to see deployment.
>
> That said, it seems like we have a way forward for
> not-terrible-for-offload PNE, with negotiation to shut it off in controlled
> environments where one wants to shave energy expenditure. +1 to Praveen’s
> suggestion to add transport parameters to the standard for this for
> purposes of interoperability..
>
> Cheers, Brian
>
>
>
> Sent from my iPhone
>
> >>> if some middleboxes decide to pass QUIC without PNE but block QUIC
> with PNE, we might be incentivized to use QUIC without PNE.
> >>
> >> If a middlebox really dislikes PNE, it can just block QUIC altogether,
> forcing a TCP fallback.  Is TCP better than QUIC-without-PNE from any POV
> in that situation?  (And if so, this should be the recommendation to QUIC
> stacks with negotiable PNE -- fallback to TCP, if path is blocking PNE.)
> >
> >> Cc: Mark Nottingham <[email-masked]>; Mike Bishop <[email-masked]>;
> Ian Swett <ianswett=[email-masked]>; Christian Huitema <
> [email-masked]>; IETF QUIC WG <[email-masked]>; Deval, Manasi <
> [email-masked]>
> use Transport Parameters and the other is to use a different QUIC version.
> >>
> handshake. Version negotiation has it's own protection against downgrade
> (or upgrade) attacks (see https://quicwg.github.io/base-
> drafts/draft-ietf-quic-transport.html#rfc.section.6.4.4).
> >>
> QUIC without PNE but block QUIC with PNE, we might be incentivized to use
> QUIC without PNE. In such case, we might end up having privacy leaks &
> ossification.
> >>
> >>>> 2018-04-26 9:21 GMT+09:00 Ian Swett <ianswett=40google.com@dmarc.
> ietf...org>:
> >>>>>> https://github.com/quicwg/base-drafts/wiki/Summary-of-
> the-PN-encryption-issues-and-alternatives..

---

From: Martin Thomson <[email-masked]>
Date: Mon, 30 Apr 2018 13:46:30 +1000

OK, going back to the OP to close the loop on a few things.

I have updated PR #1079 to rebase it on top of the addition of a
packet length field.  As far as I'm concerned, this is now ready.  We
might later consider a few things, like varint packet numbers and
removal of the sequence number from NEW_CONNECTION_ID, but those are
separate refinements.

I'm glad to see that there is a plausible implementation of this in
hardware that isn't inordinately expensive.  I appreciate that this
complicates offload in some cases, but I still believe this to be the
best design we have right now in terms of all the competing factors
we've considered (and we've probably covered every angle now; more
than once).

My assessment is that we should merge this.  Chairs, is this your assessment?

---

From: Boris Pismenny <[email-masked]>
Date: Mon, 30 Apr 2018 09:58:14 +0300

Hi all,

I'm responsible for the architecture of crypto offload in Mellanox NICs. 
Here are my 2 cents on PNE.

It is possible to implement any PNE scheme in HW, but it is a question 
of cost and ROI.
The current PNE proposal requires an additional AES stage in the NIC 
pipeline - silicon real estate or complex scheduling of a single AES 
engine and two passes over the data. No other network crypto standard 
I've encountered has such a feature. Thus, ROI is likely to be low.

Best,
Boris.

On 4/30/2018 6:46 AM, Martin Thomson wrote:
>>>> Looking at https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FAES_instruction_set%23Intel_and_AMD_x86_architecture&data=02%7C01%7Cborisp%40mellanox.com%7C426e09d1c4d44186499a08d5ae4cfd51%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C636606568059138841&sdata=xJlYVkw4qoyUyxxdXGjVQ%2B1DBpfcoo2AbkQpj8XZnlg%3D&reserved=0 it seems to imply a large range of server, desktop and mobile chips all have a CPU instruction set available to do AES acceleration and other similar operations (other instruction sets are also available)..
>>>>
>> Mark Nottingham   https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mnot.net%2F&data=02%7C01%7Cborisp%40mellanox.com%7C426e09d1c4d44186499a08d5ae4cfd51%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C636606568059138841&sdata=Gz248vu518Gu2ZxYXDLO8vldtycSv6dok%2FuPrwnQFD4%3D&reserved=0
>>
>

---

From: Roberto Peon <[email-masked]>
Date: Mon, 30 Apr 2018 20:23:49 +0000

This kind of 'obvious' "breakage" is less easy to game (it is unsubtle), and thus preferable.
-=R

    

    - Igor

    -----Original Message-----

    

    

    

    

    Use of Transport Parameters is secure because it is part of the TLS handshake. Version negotiation has it's own protection against downgrade (or upgrade) attacks (see https://urldefense.proofpoint.com/v2/url?u=https-3A__quicwg.github.io_base-2Ddrafts_draft-2Dietf-2Dquic-2Dtransport.html-23rfc.section.6.4.4&d=DwIGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=zC1VGmf3GQmWVe467ufLNulxuNhUTlKEQwOA80nBUUY&s=fvtchwAG-YxJcR8wUAezTwCSN6X8EH8E-7MlSo1p1nM&e=).

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    
    >>>> To: Deval, Manasi <[email-masked]>; Mark Nottingham 

    

    

    

    

    

    

    

    >>>> With that new information, it appears that PR #1079 is superior to 
    
    >>>> every other alternative.

    

    

    

    
    --
    
    Kazuho Oku

---

From: Christian Huitema <[email-masked]>
Date: Mon, 30 Apr 2018 21:34:40 +0100

On 4/28/2018 1:08 AM, Jana Iyengar wrote:
> connections on the Internet. I agree that the risk of blockage doesn't
> seem high, but honestly, having seen how middlebox features get
> deployed, I'm not convinced that it won't happen (though I suspect
> it's unlikely to happen at a large scale). I understand the desire to
> have it off within DCs though, so I'm sensitive to that need.
>
> I'd like to hear more about why making PNE optional isn't a decision
> that we can punt to later though. I'd like to move along with PNE now,
> and come back to negotiating this as an option once we have some
> deployment work/experience in intra DC environments. I'm not yet
> convinced that this is going to be your biggest cost in an intra-DC
> environment, and if it is, then we can surely revisit this decision
> and add a param later to make PNE optional, either later in v1 or in
> v2. We are talking about a very narrow view into total cost at this
> point -- this one AES op is unlikely to be the bottleneck. But I have
> no data, and I don't think anyone has any full deployment experience yet.
>
> TL;DR: I'd like to suggest that we move ahead with PR #1079, and
> continue discussion on adding optional PNE in Issue #1296.

+1

-- Christian Huitema

---

From: Ian Swett <[email-masked]>
Date: Mon, 30 Apr 2018 20:38:28 +0000

SGTM

---

From: Praveen Balasubramanian <[email-masked]>
Date: Mon, 30 Apr 2018 21:55:31 +0000

I disagree that we need any more data for not doing PNE in the datacenter. Why would we add an extra encrypt-decrypt step for no obvious benefit? The cost of this operation will keep magnifying as UDP performance improves. We have several optimizations planned for UDP, but short of hardware offload, none for crypto – this is an important trend to account for performance wise. For short packets like ACKs an extra encrypt decrypt step can become a large fraction of CPU cost over time.

If QUIC is not intended to be a general purpose transport but only for the Internet, then we should say so in the charter. Then we can invent custom protocols over UDP for datacenter scenarios and give up on interop.

Also, what is the urgency behind getting PNE adopted – is it blocking progress on interop or other issues? If it is deemed urgent then I am fine with adopting PNE as long as we add a negotiation mechanism in V1.

From: QUIC [mailto:[email-masked]] On Behalf Of Ian Swett
Sent: Monday, April 30, 2018 1:38 PM
To: huitema <[email-masked]>
Cc: Jana Iyengar <[email-masked]>; Lubashev, Igor <[email-masked]>; Mark Nottingham <[email-masked]>; Mike Bishop <[email-masked]>; Erik Kline <[email-masked]>; Brian Trammell <[email-masked]>; IETF QUIC WG <[email-masked]>; Deval, Manasi <[email-masked]>; Kazuho Oku <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

SGTM

On Mon, Apr 30, 2018 at 4:34 PM Christian Huitema <[email-masked]<mailto:[email-masked]>> wrote:

+1

-- Christian Huitema

---

From: "Lubashev, Igor" <[email-masked]>
Date: Mon, 30 Apr 2018 22:20:00 +0000

It does not seem like there is any difference in the subtlety of "blocking QUIC" and "blocking QUIC that negotiated PNE".  In both cases the connection ends up blocked, and it is up to the QUIC stack how to fail over (and it would failover to a non-PNE transport).

As long as there is a TCP fallback, a network that is willing to block traffic with PNE block traffic with PNE.  Since PNE is very likely to be the default for web browsers, no middlebox can "accidentally" block PNE.

- Igor

-----Original Message-----
From: Roberto Peon [mailto:[email-masked]] 
Sent: Monday, April 30, 2018 4:24 PM
To: Lubashev, Igor <[email-masked]>; Kazuho Oku <[email-masked]>; Erik Kline <[email-masked]>
Cc: Mark Nottingham <[email-masked]>; Christian Huitema <[email-masked]>; Ian Swett <[email-masked]>; Mike Bishop <[email-masked]>; IETF QUIC WG <[email-masked]>; Deval, Manasi <[email-masked]>
Subject: Re: Getting to consensus on packet number encryption

    

    - Igor

    -----Original Message-----

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    
    --
    
    Kazuho Oku

---

From: Martin Thomson <[email-masked]>
Date: Tue, 1 May 2018 09:49:03 +1000

On Tue, May 1, 2018 at 7:55 AM, Praveen Balasubramanian
<pravb=[email-masked]> wrote:
> Also, what is the urgency behind getting PNE adopted – is it blocking
> progress on interop or other issues?

Yes.  There are a considerable number of issues backing up behind this now.

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 1 May 2018 00:32:18 +0000

What are the examples of issues that have a dependency on PNE? This is a transform that is applied post constructing the packet and then before decryption on receive, so I was assuming its very decoupled from the core protocol and could be added at the end. If any part of the protocol adds a dependency on PNE, then it’s a cause for concern because it precludes the no-op transform for datacenter option. So stating any such dependencies is important IMO.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 1 May 2018 10:34:18 +1000

Copying from the PR:

Closes #1174
Closes #1043
Closes #1048
Closes #1034
Closes #850
Closes #990
Closes #734

This isn't about other dependencies, but this is holding a bunch of issues open.