Subject: Alternate header proposal

From: Jana Iyengar <[email-masked]>
Date: Tue, 14 Feb 2017 23:05:15 -0800

After discussion with various folks, I have put together an alternate
proposal for packet header organization, described below (also on github
here <https://gist.github.com/janaiyengar/b05acb3af17d6e938d25befc69c11eae>.)
This borrows the long/short header idea from Martin's formulation, but
turns the flags field into a type byte, with only 18 codepoints defined.
This proposal eliminates the need for an explicit magic field, and uses
packet types for directionality of cleartext packets.

Hope this makes sense, and happy to discuss modifications to this format.
Details can of course be changed, but I'm quite liking the use of a 1-byte
codepoint instead of bits for various fields.

-------------------

QUIC packet headers can be separated into long form and short form headers.
Long form headers are used for version negotiation, connection
establishment,
and public reset packets. Short form headers are version-specific, and are
used
for 1-RTT packets, minimizing their header size.

While all packets are identified using the same Type octet, headers are
separated into the two categories for two reasons: (i) short form headers
are
only used after version negotiation and establishment of 1-RTT keys, and
(ii) editorial convenience.

This formulation eschews specific bits indicating the presence of
various fields in favor of an octet indicating one of 256 packet types.
Of these 256 types, this version defines and uses only 18: 6 with long-form
headers for the initial phase of the connection, and 12 with short-form
headers
for after the version negotiation and 1-RTT keys are established.

# Long-Form Headers

```
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+
|     Type      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                        Connection ID                          +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           Version                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  Packet Number / Proof                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Payload                          ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
```

These long-form headers are used for anything that doesn't have 1-RTT packet
protection and prior to the completion of version negotiation. Once both
conditions are met, a sender may switch to sending short-form headers.
While inefficient, long headers may also be used for 1-RTT packets. The
long
form allows for special packets, such as the version negotiation and the
public
reset packets to be represented in this uniform fixed-length packet format.

* Octet 0: Packet Type
  * 39: Version Negotiation packet
  * 3a: Public Reset packet
  * 3d: 0-RTT packet
  * 3e: Server cleartext packet
  * 3f: Client cleartext packet
  * 1c: 1-RTT packet with version field, connection ID,
        and 2-byte packet number

The remainder of the packet layout is the same regardless of type, the
difference being what rules for how to fill the values out and their
semantics.

A client cleartext packet (type = 0x3c) then contains:
* Octets 1-8: connection ID (initially randomly chosen)
* Octets 9-12: version
* Octets 13-16: packet number (low 4 octets, starts at a random 31-bit
value)
* Octets 17+: payload

The client MUST choose a random value and use it as the Connection ID until
the
server replies with a server-selected connection ID. The client's
connection ID
would have no semantic value, though they might serve to provide proof that
the server received the packet via echoing, see below.

A server cleartext packet (type = 0x3e) contains:
* Octets 1-8: connection ID (server-selected value)
* Octets 9-12: version (echoed)
* Octets 13-16: packet number (low 4 octets, random 31-bit initial value)
* Octets 17+: payload

Both 0-RTT and 1-RTT packets with long-form headers contain:
* Octets 1-8: connection ID (server-selected value)
* Octets 9-12: version
* Octets 13-16: packet number (low 4 octets)
* Octets 17+: payload

A version negotiation packet contains:
* Octets 1-8: connection ID (server-selected value, may be used in a
subsequent
  connection to reach the same server)
* Octets 9-12: version (echoed)
* Octets 13-16: proof (first 4 octets of client-selected connection ID)
* Octets 17+: payload = version list

A public reset packet is sent when the server has no state for a received
packet. A server may therefore have to respond to either a long-form or a
short-form packet. A public reset packet contains:
* Octets 1-8: connection ID (server-selected value)
* Octets 9-12: version
* Octets 13-16: proof (octets 1-5 of received packet)

Echoing details from the packet in both version negotiation and public
reset
provides return routeability (#244), while maintaining a consistent header
shape for all packets.

# Short Form Header

```
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+
|      Type     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                   Connection ID (optional)                    +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Packet Number (1/2/4)                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Payload                          ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
```

The short form header is used after the version and 1-RTT keys are
negotiated.
The short form header is defined to be specific to a version. In this
version,
bit 0 of the first octet (i.e., 0x80) is used as the key phase bit,
resulting
in the following packet types.
* Octet 0: Packet Type
  * 04 and 84: 1-RTT packet (packet number size = 1)
  * 14 and 94: 1-RTT packet (packet number size = 2)
  * 34 and b4: 1-RTT packet (packet number size = 4)
  * 0c and 8c: 1-RTT packet with Connection ID (packet number size = 1)
  * 1c and 9c: 1-RTT packet with Connection ID (packet number size = 2)
  * 3c and bc: 1-RTT packet with Connection ID (packet number size = 4)

---

From: Martin Thomson <[email-masked]>
Date: Wed, 15 Feb 2017 19:43:45 +1100

I like this general approach.

I would prefer if the packet number were in the same place in both
forms (that seems easy enough to do by moving the version).  Then we
get a contiguous proof block (connection ID + packet number) for a
public reset.

I note that you can use 0x08 to check if the packet is long or short.
We should try to keep that property until we can't any more.

I'm OK with the server proposing a new connection ID on a version
negotiation packet (though not yet 100% confident that it's the right
answer), but it doesn't make any sense to propose a new connection ID
on a public reset.  Echoing the client value is a chance to provide
stronger proof that it saw the packet it is reacting to.

>   * 34 and b4: 1-RTT packet (packet number size = 4)

I don't think that we need these.  I haven't seen any reason why
connection ID can't be negotiated off during the handshake (if that
even makes sense), reducing your 18 types to 12.

I'm not sure why you have 1c mentioned in two places.  Was your intent
to define a long-form packet that can carry 1-RTT?  Because you need
to explicitly mark it as long-form in that case or we'll have some
confusion about whether version is present or not.  (Note that you
need two 1-RTT long-form packets to deal with key changes.)

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 15 Feb 2017 10:14:35 +0100

Hi Jana, hi all,

in general I like the idea of short and long packets because optimizing 
header bits really only make a lot of sense if this space can be used for 
payload bits instead.

However, couple of comments:

1) I'd say that having undefined codepoints enables actually the kind of 
extensibility (beside versioning) that you've been arguing against 
previously. I don't think that this a bad thing because it's under endpoint 
control, however, having codepoints instead of bits with a special meaning 
actually makes it harder to deploy. If the endpoint sees an unknown 
codepoint, it can only drop the packet completely (it doesn't even know where 
the payload starts), while with an dedicated bit that declares a certain 
space to be used for additional information that can always be ignored. In 
summary, I think your proposal needs a header length field, but still I 
prefer the flags field because it's less complex.

2) Further effectively as Martin basically pointed out in his follow-up mail, 
you still have one bit that indicates the short or long header. I think 
that's good and we should maintain this but in that case we should also call 
this bit out like this because that's easier to understand for humans (who 
need to implement this).

3) I still don't really understand the need for different sizes of the packet 
number field, especially I don't understand why we have 3 different values. I 
assume you mainly try to save bits in a data center scenario, right? But in 
this case you could even run your own version of quic where the only 
difference is that you always have a shorter packet number.

4) And finally, I think it would be safer to have the connection ID on all 
packets. But maybe we can just use a shorter connection ID? What's the reason 
for a 64 bits connection ID again? Also if we allow for packets without the 
connection ID, that's probably something the server should tell the client 
because the server might know if that's needed for some kind of load 
balancing setup at server side or not.

Mirja

On 15.02.2017 08:05, Jana Iyengar wrote:
> proposal for packet header organization, described below (also on github here
> <https://gist.github.com/janaiyengar/b05acb3af17d6e938d25befc69c11eae>.) This
> borrows the long/short header idea from Martin's formulation, but turns the
> flags field into a type byte, with only 18 codepoints defined. This proposal
> eliminates the need for an explicit magic field, and uses packet types for
> directionality of cleartext packets.
>
> Long form headers are used for version negotiation, connection establishment,
> and public reset packets. Short form headers are version-specific, and are used
> separated into the two categories for two reasons: (i) short form headers are
> headers for the initial phase of the connection, and 12 with short-form headers
> While inefficient, long headers may also be used for 1-RTT packets. The long
> form allows for special packets, such as the version negotiation and the public
> difference being what rules for how to fill the values out and their semantics.
>
> * Octets 13-16: packet number (low 4 octets, starts at a random 31-bit value)
> * Octets 17+: payload
>
> The client MUST choose a random value and use it as the Connection ID until the
> server replies with a server-selected connection ID. The client's connection ID
> * Octets 1-8: connection ID (server-selected value, may be used in a subsequent
> Echoing details from the packet in both version negotiation and public reset
> The short form header is used after the version and 1-RTT keys are negotiated.
> The short form header is defined to be specific to a version. In this version,
> bit 0 of the first octet (i.e., 0x80) is used as the key phase bit, resulting

---

From: Martin Thomson <[email-masked]>
Date: Wed, 15 Feb 2017 22:17:26 +1100

On 15 February 2017 at 20:14, Mirja Kühlewind
<[email-masked]> wrote:
> packets. But maybe we can just use a shorter connection ID? What's the
> reason for a 64 bits connection ID again?

If anything I think that 64 bits is too short :)  Server people keep
asking for more anyway.  Managing routing, authentication and whatever
else this field might be used for, all in a mere 64 bits, is pretty
challenging.

I think that the notion of removing connection ID needs a lot more
discussion.  Reasons I can see for removing it are that the server
doesn't need it, or the client doesn't want it (hello linkability).
Removing it does jeopardize some connection migration cases though, so
we probably want to work through the use cases in more detail.

---

From: Ian Swett <[email-masked]>
Date: Wed, 15 Feb 2017 07:48:56 -0500

I like this general approach a lot, because it defines a consistent set of
easy to parse packets for prior to version negotiation, and allows plenty
of extensibility both pre and post version negotiation if we need it.
Having connection ID come after the flags byte feels right, because it's
routing info.  I also think it'll be relatively easy to implement.

Mirja, to address your point about extensibility.  I think we want the
ability to evolve the design and add new features, in this case via packet
types.  One can imagine adding an 8 byte packet number format in the
future, for example, or even a format with Martin's authenticated repeated
options if we decide it's compelling?

Mirja, re #3 about packet number sizes.  This is very much about
user-facing traffic, not datacenters.  2 bytes is enough for all the user
facing traffic we have today.  But that allows a max CWND that's less than
what TCP allows, so 4 bytes was added because it seemed fairly futureproof.
 1 byte isn't really that necessary, though it's commonly used on short
connections and it's really easy to implement and we use it today, so Jana
defined it.

Martin and Mirja, re: When to include connection ID: Yes, it's clear this
needs more discussion.  I think it makes sense to have both sets of
codepoints in there for now, and we can remove them later if we really
think connection ID is never necessary.  Currently, it's necessary in cases
when NATs rapidly rebind.

I do think we need a long-form packet that can carry 1-RTT, and I think
it's critical if we want to drop connection ID from short form packets,
since it could allow us to re-establish a connection with the same server
using a new different ID.  It's also useful for out of band key
negotiation, which I believe is why Jana added it.

If we're only going to include connection ID in packets which also contain
version, we could bump the length to 16 bytes if there's a compelling use
case.

This proposal also allows us to grease every unencrypted byte in the
packet, which makes me excited, but I'll write that up on the grease issue.

---

From: Martin Duke <[email-masked]>
Date: Wed, 15 Feb 2017 11:08:23 -0800

The packet formats look great. I also like the idea of a version-agnostic
long form header with version complexity only in the short form.

But I have issues with the first byte:

Perhaps I'm missing something simple here, but how does 0x8 specify
short/long form? The first six short headers Jana presented differ in this
bit from the second six. In fact there is no bit that is consistently set
in the short form while cleared in the long form, or vice versa.

Also, we have a number of outstanding manageability issues that could use
2-3 bits in the public flags, and I'd like to compress the uses here to
make space for those.

Furthermore, I think having an actual long/short bit would be quite useful
and I'd like to arrange the codepoints to allow that.

So I think processing would be somewhat easier with the following
arrangement:
0x80: Long Form header
If a long form header:
  0x80-0xd0 - covers the six types of long-form packet
  The last 4 are reserved, and greased if we don't find uses for them
If a short form header
  0x40 - key phase
  0x20 - conn Id included
  0x10 and 0x08 - pkt # length
  The last three bits are reserved/greased.

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 15 Feb 2017 12:56:20 -0800

> I like this general approach.

I do too. It comes from my discussions with you, so thank you for those.

> get a contiguous proof block (connection ID + packet number) for a

Packet number after version was deliberate, to allow packet number to
change in future versions. An incompatible receiver only has to read up
through the version field, and ignores the packet number field. If we move
the packet number to before the version, we burn the 4-byte packet number
in the long-form header for future versions too. It's not the craziest
thing to burn in, so I'm not opposed to this idea. You do get the nice
contiguous block... if we have packet number before version in the long
header, here's a reformulation of the public reset packet.

A public reset packet is sent when the server has no state for a received
packet. A server may therefore have to respond to either a long-form or a
short-form packet. A public reset packet contains:
* Octets 1-12: proof (octets 1-12 of received packet)
* Octets 13-16: version

For a client that sends a connection ID on every packet, this packet can be
simply interpreted as:
* Octets 1-8: connection ID (echoed)
* Octets 9-12: 1/2/4-byte packet number (echoed)
* Octets 13-16: version

I think I like this. Can anyone think of reasons to not burn packet number
in the long header?

> We should try to keep that property until we can't any more.

I'd argue against that. This means that all code points including 0x80 are
lost, and I don't see what you gain. Checking if the packet is long or
short is easy: if the type is in {39, 3a, 3d, 3e, 3f, 1c} then it's long,
and if it's in {04, 84, 14, 94, 34, b4, 0c, 8c, 1c, 9c, 3c, bc} then it's
short. This isn't hard to do.

> stronger proof that it saw the packet it is reacting to.

Agreed. The contiguous block idea above resolves this.

- jana

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 15 Feb 2017 14:05:13 -0800

Hi Mirja,

> header bits really only make a lot of sense if this space can be used for

Cool -- agreed.

> previously. I don't think that this a bad thing because it's under

My disagreement with the earlier proposal was an arbitrary extensibility
mechanism that I didn't want baked into the protocol... but I think Martin
may have been proposing using it only up until we get to RFC, in which case
I misunderstood his proposal. This proposal generalizes that idea for sure,
but it also combines that with requirement to identify QUIC packets, which
was done with the magic field in the previous proposal.

> endpoint control, however, having codepoints instead of bits with a
> special meaning actually makes it harder to deploy. If the endpoint sees an
> unknown codepoint, it can only drop the packet completely (it doesn't even
> know where the payload starts), while with an dedicated bit that declares a
> certain space to be used for additional information that can always be
> ignored. In summary, I think your proposal needs a header length field, but
> still I prefer the flags field because it's less complex.
>

This is deliberate. An endpoint that sees an unknown codepoint should drop
it since it can't tell it as a QUIC packet from random garbage.
Interpreting a piece of an unknown header as header-length is quite
problematic and I don't think you gain anything from doing this. If you
can't interpret the header, what is the endpoint to do with the payload?

Middleboxes are trickier:  you may have a box that understands the early
codepoints but doesn't understand later codepoints. Ian has an idea for
greasing the codepoint space for short-form headers, which should help with
this problem.

2) Further effectively as Martin basically pointed out in his follow-up
> mail, you still have one bit that indicates the short or long header. I
> think that's good and we should maintain this but in that case we should
> also call this bit out like this because that's easier to understand for
> humans (who need to implement this).
>

As I said earlier in my response to Martin, I don't think that's right, and
I don't think we want a bit set aside.

> 3) I still don't really understand the need for different sizes of the
> packet number field, especially I don't understand why we have 3 different
> values. I assume you mainly try to save bits in a data center scenario,
> right? But in this case you could even run your own version of quic where
> the only difference is that you always have a shorter packet number.
>

I agree with Ian's response -- we use the different packet number sizes for
the bandwidth-constrained direction (server -> client) on the public
Internet. I'll note that in the common case in Google's Internet-side
deployment, QUIC packets in the bandwidth-constrained direction currently
have a 2 or 3 byte header (flags, 1-2 bytes of packet number.)

> reason for a 64 bits connection ID again? Also if we allow for packets
> without the connection ID, that's probably something the server should tell
> the client because the server might know if that's needed for some kind of
> load balancing setup at server side or not.
>

This is currently negotiated during the handshake as the TCID (truncated
connection ID) param, which is described in the draft. Sending the entire
connection ID on all packets in both directions is quite wasteful.
Connection ID need only be sent if it is required for routing. If the
5-tuple is adequate, then sending a connection ID on each packet is a waste
of bandwidth.

- jana

>> here

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 15 Feb 2017 14:27:02 -0800

> in the short form while cleared in the long form, or vice versa.

You're right, there's no single bit that can be used in this proposal.

> 2-3 bits in the public flags, and I'd like to compress the uses here to

... or we could add more codepoints. We can make the codepoints contiguous
if that helps. Let's discuss the issues independent of representation
first, and then figure out representation.

>   The last three bits are reserved/greased.

This looks close to an intermediate proposal I had come up with :-) I'm not
opposed to this, but it makes the split between long and short forms more
explicit than necessary. I don't think processing is harder with the
current proposal, though sure, you could make the packet types contiguous
if that helps.

Importantly, the two forms are not really that different from a processing
point-of-view if you look closely; the separation is really more useful for
reasoning and understanding. There is the distinction of the short form
being version-specific, but that's really only to say that a receiver that
negotiates version X should only process the codepoints defined for that
version.  Except for the version negotiation and public reset packets, the
rest of the types are data bearing packets (with or without some fields and
key phase signaling) and should be treated as such. As to going from
codepoint to bits that indicate presence of fields, it's a simple matter of
mapping codepoints to local per-packet state, and you continue processing
the packet with the local state.

I don't think you need explicit bits; mapping from a codepoint to explicit
bits at the endpoints should work fine IMO.

- jana

>> I like this general approach a lot, because it defines a consistent set
>> of easy to parse packets for prior to version negotiation, and allows
>> plenty of extensibility both pre and post version negotiation if we need
>> it.  Having connection ID come after the flags byte feels right, because
>> it's routing info.  I also think it'll be relatively easy to implement.
>>
>> If we're only going to include connection ID in packets which also
>> contain version, we could bump the length to 16 bytes if there's a
>> compelling use case.
>>
>> On Wed, Feb 15, 2017 at 6:17 AM, Martin Thomson <[email-masked]
>> > wrote:
>>

---

From: Martin Duke <[email-masked]>
Date: Wed, 15 Feb 2017 14:41:34 -0800

> first, and then figure out representation.

Without trying to legislate these issues here, the manageability proposals
basically involve setting a few bits in the flags to give basic loss and
flow control information to middleboxes. They are orthogonal to header
length and (largely) to each otehr, so you're multiplying the number of
codepoints by up to 8 (32 if ECE/CWR in the flags has a constituency) in a
way that makes it much harder for the middlebox to read if those codepoints
don't in practice correspond to a particular bit.

These flags would apply to both long and short form packets, so there are
not even any economies there.

> bits at the endpoints should work fine IMO.

I agree 100% for long form, where there are a series of oddball packet
types. But for short form, there are many permutations that are largely
independent of each other, so little is lost by formally organizing them as
bits corresponding to fields.

---

From: Mike Bishop <[email-masked]>
Date: Wed, 15 Feb 2017 23:01:13 +0000

It seems like there’s an emerging idea (which has been present all along) that there are really two subdivisions of QUIC:

  *   QUIC-for-all-time:  Forever-fixed packet layouts carrying version negotiation, public reset, etc.; can carry version-specific headers/payload as well if you’ve agreed on the version
     *   If you change these, you’re defining a new protocol that has to be distinguishable from QUIC in some way
  *   QUIC-for-now:  Version-specific packet layouts carrying version-specific payload

It sounds like what you’re saying here is that all long-form packet codepoints are part of QUIC-for-all-time (deliberately not coining the term “QFAT”), and a receiver has to assume that any other codepoint might be part of QUIC-for-now in a version it doesn’t know (which gets dropped, either because it knows the purported version and doesn’t recognize the codepoint or doesn’t know the purported version and it shouldn’t be seeing those without negotiation having completed).

Currently, we describe the whole thing together, and have a follow-up section that says “BTW, these things won’t change in future versions.”  Without necessarily splitting QUIC-for-all-time into a separate document, it might be useful to tease that line apart further and make the immutable stuff at least a fully separate section.

From: QUIC [mailto:[email-masked]] On Behalf Of Jana Iyengar
Sent: Wednesday, February 15, 2017 2:27 PM
To: Martin Duke <[email-masked]>
Cc: Mirja Kühlewind <[email-masked]>; Ian Swett <[email-masked]>; IETF QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: Re: Alternate header proposal

On Wed, Feb 15, 2017 at 11:08 AM, Martin Duke <[email-masked]<mailto:[email-masked]>> wrote:
The packet formats look great. I also like the idea of a version-agnostic long form header with version complexity only in the short form.

Perhaps I'm missing something simple here, but how does 0x8 specify short/long form? The first six short headers Jana presented differ in this bit from the second six. In fact there is no bit that is consistently set in the short form while cleared in the long form, or vice versa.

Also, we have a number of outstanding manageability issues that could use 2-3 bits in the public flags, and I'd like to compress the uses here to make space for those.

... or we could add more codepoints. We can make the codepoints contiguous if that helps. Let's discuss the issues independent of representation first, and then figure out representation.

Furthermore, I think having an actual long/short bit would be quite useful and I'd like to arrange the codepoints to allow that.

So I think processing would be somewhat easier with the following arrangement:
0x80: Long Form header
If a long form header:

This looks close to an intermediate proposal I had come up with :-) I'm not opposed to this, but it makes the split between long and short forms more explicit than necessary. I don't think processing is harder with the current proposal, though sure, you could make the packet types contiguous if that helps.

Importantly, the two forms are not really that different from a processing point-of-view if you look closely; the separation is really more useful for reasoning and understanding. There is the distinction of the short form being version-specific, but that's really only to say that a receiver that negotiates version X should only process the codepoints defined for that version.  Except for the version negotiation and public reset packets, the rest of the types are data bearing packets (with or without some fields and key phase signaling) and should be treated as such. As to going from codepoint to bits that indicate presence of fields, it's a simple matter of mapping codepoints to local per-packet state, and you continue processing the packet with the local state.

I don't think you need explicit bits; mapping from a codepoint to explicit bits at the endpoints should work fine IMO.

- jana

On Wed, Feb 15, 2017 at 4:48 AM, Ian Swett <[email-masked]<mailto:[email-masked]>> wrote:
I like this general approach a lot, because it defines a consistent set of easy to parse packets for prior to version negotiation, and allows plenty of extensibility both pre and post version negotiation if we need it.  Having connection ID come after the flags byte feels right, because it's routing info.  I also think it'll be relatively easy to implement.

Mirja, to address your point about extensibility.  I think we want the ability to evolve the design and add new features, in this case via packet types.  One can imagine adding an 8 byte packet number format in the future, for example, or even a format with Martin's authenticated repeated options if we decide it's compelling?

Mirja, re #3 about packet number sizes.  This is very much about user-facing traffic, not datacenters.  2 bytes is enough for all the user facing traffic we have today.  But that allows a max CWND that's less than what TCP allows, so 4 bytes was added because it seemed fairly futureproof.  1 byte isn't really that necessary, though it's commonly used on short connections and it's really easy to implement and we use it today, so Jana defined it.

Martin and Mirja, re: When to include connection ID: Yes, it's clear this needs more discussion.  I think it makes sense to have both sets of codepoints in there for now, and we can remove them later if we really think connection ID is never necessary.  Currently, it's necessary in cases when NATs rapidly rebind.

I do think we need a long-form packet that can carry 1-RTT, and I think it's critical if we want to drop connection ID from short form packets, since it could allow us to re-establish a connection with the same server using a new different ID.  It's also useful for out of band key negotiation, which I believe is why Jana added it.

If we're only going to include connection ID in packets which also contain version, we could bump the length to 16 bytes if there's a compelling use case.

This proposal also allows us to grease every unencrypted byte in the packet, which makes me excited, but I'll write that up on the grease issue.

On Wed, Feb 15, 2017 at 6:17 AM, Martin Thomson <[email-masked]<mailto:[email-masked]>> wrote:
On 15 February 2017 at 20:14, Mirja Kühlewind

---

From: Martin Duke <[email-masked]>
Date: Wed, 15 Feb 2017 15:05:17 -0800

We actually have a "what cannot change in future versions" section, but
having it be a whole set of header types is much cleaner than the current
text.

On Wed, Feb 15, 2017 at 3:01 PM, Mike Bishop <[email-masked]>
wrote:

> It seems like there’s an emerging idea (which has been present all along)
> that there are really two subdivisions of QUIC:
>
>    - QUIC-for-all-time:  Forever-fixed packet layouts carrying version
>    negotiation, public reset, etc.; can carry version-specific headers/payload
>    as well if you’ve agreed on the version
>       - If you change these, you’re defining a new protocol that has to
>       be distinguishable from QUIC in some way
>    - QUIC-for-now:  Version-specific packet layouts carrying
>    version-specific payload
>
>
>
> It sounds like what you’re saying here is that all long-form packet
> codepoints are part of QUIC-for-all-time (deliberately not coining the term
> “QFAT”), and a receiver has to assume that any other codepoint *might* be
> part of QUIC-for-now in a version it doesn’t know (which gets dropped,
> either because it knows the purported version and doesn’t recognize the
> codepoint or doesn’t know the purported version and it shouldn’t be seeing
> those without negotiation having completed).
>
>
>
> Currently, we describe the whole thing together, and have a follow-up
> section that says “BTW, these things won’t change in future versions.”
> Without necessarily splitting QUIC-for-all-time into a separate document,
> it might be useful to tease that line apart further and make the immutable
> stuff at least a fully separate section.
>
>
>
> *From:* QUIC [mailto:[email-masked]] *On Behalf Of *Jana Iyengar
> *Sent:* Wednesday, February 15, 2017 2:27 PM
> *To:* Martin Duke <[email-masked]>
> *Cc:* Mirja Kühlewind <[email-masked]>; Ian Swett <
> [email-masked]>; IETF QUIC WG <[email-masked]>; Martin Thomson <
> [email-masked]>
> *Subject:* Re: Alternate header proposal
>
>
>
> This looks close to an intermediate proposal I had come up with :-) I'm
> not opposed to this, but it makes the split between long and short forms
> more explicit than necessary. I don't think processing is harder with the

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 15 Feb 2017 15:17:38 -0800

On Wed, Feb 15, 2017 at 2:41 PM, Martin Duke <[email-masked]>
wrote:

>>> Also, we have a number of outstanding manageability issues that could
>>> use 2-3 bits in the public flags, and I'd like to compress the uses here to
>>> make space for those.
>>>
>>
>> ... or we could add more codepoints. We can make the codepoints
>> contiguous if that helps. Let's discuss the issues independent of
>> representation first, and then figure out representation.
>>
>>
>

I don't think they all do -- for instance, these bits would not need to be
expressed on the public reset and version negtiation packets, would they?
Depending on the signal in question, we may be able to contain the
combinatorial explosion. If unavoidable, we can surely give up a bit.

Why do you need ECE/CWR bits in the header? I haven't carefully considered
yet how to do this in QUIC (and I haven't carefully read Ingemar's draft
yet), but those would/should be in explicit frames I would imagine.

> bits corresponding to fields.

I'm not sure that's true... more accurately, this isn't a demonstrated
problem yet. The short forms currently constitute 12 packet types, which is
not too many. If we end up having combinatorial explosion as we move
forward, then sure, we can re-jigger this type byte to be flags+type.

---

From: Martin Duke <[email-masked]>
Date: Wed, 15 Feb 2017 15:26:31 -0800

> yet), but those would/should be in explicit frames I would imagine.

The draft puts them in the ACK frame. There is a case, I think, to have
them in the public header, though I'm not convinced of it at all. The point
I am trying to make is that there are uses for these bits that are not
related to the header layout, and in the extreme case this would be as many
as 5 (I am personally convinced we need only 2).

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 15 Feb 2017 15:36:18 -0800

On Wed, Feb 15, 2017 at 3:01 PM, Mike Bishop <[email-masked]>
wrote:

> Yes. Working through Martin's proposal and trying to fix the things in
there led me to a couple of other proposals along the way, but chatting
with Martin about the codepoint idea led me to this proposal, which I think
is a nice and elegant simplification of those proposals. The core of this
proposal is in making existing uses of the Flags byte slightly more rigid,
and making the rest of the combinatorial space available for extensibility.

> “QFAT”), and a receiver has to assume that any other codepoint
>

... and you just coined it :-)

> *might* be part of QUIC-for-now in a version it doesn’t know (which gets
> dropped, either because it knows the purported version and doesn’t
> recognize the codepoint or doesn’t know the purported version and it
> shouldn’t be seeing those without negotiation having completed).
>

Yes that is correct. As Martin (Duke; E_TOO_MANY_MARTINS) points out,
there's also the version-agnostic headers fields now, which, in the long
forms, are the parts preceding the version field. Note that the entire
packet header for the long form does not need to be version agnostic.
There's no reason fields following the version field cannot change across
versions.

> stuff at least a fully separate section.

Yes, agreed. The long/short split is roughly along those lines, but we need
to carefully and explicitly draw the line in the transport document.

- jana

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 15 Feb 2017 15:47:10 -0800

On Wed, Feb 15, 2017 at 3:26 PM, Martin Duke <[email-masked]>
wrote:

>
>> Why do you need ECE/CWR bits in the header? I haven't carefully
>> considered yet how to do this in QUIC (and I haven't carefully read
>> Ingemar's draft yet), but those would/should be in explicit frames I would
>> imagine.
>>
>

I understand and agree with the point that we may have things that need to
be put in the packet header (FWIW, and we can discuss this separately, but
I'm hard-pressed to understand why ECN bits are needed in the packet header
instead of in the ack frame.) Either ways I think we'll have to discuss
this on a case-by-case basis, but in the meanwhile, this proposal gives you
another degree of freedom -- codepoint space.

The current proposal only addresses things we have reasonable agreement on
in the wg, and not on things that are yet under discussion. No matter what
header format we use, we're going to have to revisit it if there are issues
that touch the header in the future, and to that extent, this header format
will have to remain malleable. I wouldn't say we've done away with explicit
bits, but I'd sure like to try and avoid them in favor of codepoints if we
can, since it seems like a big enough space.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 16 Feb 2017 11:16:34 +1100

> I think I like this. Can anyone think of reasons to not burn packet number

If I understand what you mean by "burn" sure.  Note that the semantics
don't need to be understood, it's just that the octets need to be
present.

I'd argue for a fixed 4 octets in the long form though (I'm not sure
what you intended with your mention of 1/2/4).

> lost, and I don't see what you gain. Checking if the packet is long or short
> is easy: if the type is in {39, 3a, 3d, 3e, 3f, 1c} then it's long, and if
> it's in {04, 84, 14, 94, 34, b4, 0c, 8c, 1c, 9c, 3c, bc} then it's short.
> This isn't hard to do.

That's fine with me.  Note that I retain my desire to trim this list
further.  I will continue to push on the optional connection ID thing
(as the lowest hanging fruit), but am happy to see us make forward
progress on the other aspects in the meantime.

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 15 Feb 2017 16:36:17 -0800

> don't need to be understood, it's just that the octets need to be

Sorry, I meant to say "bake in" these 4 octets for all time, and agreed on
your point about semantics.

> what you intended with your mention of 1/2/4).

My mention was in the public reset packet, which may be sent in response to
a long packet with 4-byte packet number or a short packet with a 1/2/4-byte
packet number.

> short

SGTM. The reason for leaving in connection ID is to keep the packet as
self-descriptive as possible, allowing network devices to know whether to
look for a connection ID in the packet or not, since connection ID is used
for routing. Additionally, network tooling (say, wireshark) to be able to
tell whether to look for a connection ID in this packet or not.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 16 Feb 2017 12:05:03 +1100

> a long packet with 4-byte packet number or a short packet with a 1/2/4-byte

Ahh, that's fine.  In the case that packet number is short, I guess
the public reset will be sending the first 2 or 3 octets of the
payload.  That's something for implementations to remember.  We'll
need special text on that.  I think that it's relatively easy to
implement that way, but it's not something that will be obvious.

> tell whether to look for a connection ID in this packet or not.

Well, if you retain the ability to break stateless inspection by
removing connection ID, I'm not sure what value there is in explicitly
telling them that they are screwed given that they have no recourse.

Unless you are saying that the consequence is that 5-tuple identifies
the flow when the connection ID is absent, but I'd just use the same
argument.  Connection ID is only really useful if you are multiplexing
on the same 5-tuple - don't do this please - or you have changes in
5-tuple.  If you remove it, then you can't change 5-tuple.

(If you haven't guessed already, connection ID is the single feature
of this protocol that I am most conflicted about.  I can see so many
advantages and so many disadvantages.  They are all potentially huge
and that means that I don't know how to make the trade-offs sensibly.)

---

From: Ryan Hamilton <[email-masked]>
Date: Wed, 15 Feb 2017 17:27:58 -0800

> and that means that I don't know how to make the trade-offs sensibly.)

​As you're thinking through the pros and cons of this issue, keep in mind
that short NAT rebinding timeouts for UDP flows are a terribly unfortunate
fact of life on the internet today. If a QUIC connection is unable to
change the 5-tuple, I suspect that will be a big challenge for QUIC
adoption since these NAT rebindings are essentially invisible to the client.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 16 Feb 2017 12:32:43 +1100

On 16 February 2017 at 12:27, Ryan Hamilton <[email-masked]> wrote:
> As you're thinking through the pros and cons of this issue, keep in mind
> fact of life on the internet today. If a QUIC connection is unable to change
> the 5-tuple, I suspect that will be a big challenge for QUIC adoption since
> these NAT rebindings are essentially invisible to the client.

That would be one of the big/huge advantages, of which I'm acutely
aware.  And also why I'm quite skeptical about having the ability to
trim connection ID, but I guess your counter will be data-center use
cases.

---

From: Ryan Hamilton <[email-masked]>
Date: Wed, 15 Feb 2017 17:37:37 -0800

​​On Wed, Feb 15, 2017 at 5:32 PM, Martin Thomson <[email-masked]>
wrote:

> change
> aware.

​Fair enough! Sorry for harping on it.​

> And also why I'm quite skeptical about having the ability to

Actually, I think it's super useful for the web use case. ​In Google QUIC,
we have the server always omit the connection ID in packets it sends to the
client (as the result of negotiation) because the client's socket is only
associated with a single connection. Since this is the direction that the
vast majority of the data flows, saving the bytes is valuable. (But in the
other direction, from the client to the server, we always send the full
connection ID, because the ​server has to multiplex connections from client
for which their underlying 5-tuple may change.) I hope that we are able to
do this with IETF QUIC as well.

(Man, I hate saying "Google QUIC" and IETF QUIC. We should retroactively
rename Google QUIC to GUIC or something :)

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 15 Feb 2017 17:39:18 -0800

> payload.  That's something for implementations to remember.  We'll

Right, payload in this version, but really, whatever follows the packet
number. Of course, if the connection ID is absent in the received packet
then the structure of what's echoed back is different. But the client
should know how to interpret the 12 bytes since it sent them.

> 5-tuple.  If you remove it, then you can't change 5-tuple.

You are assuming that connection ID is gone in both directions, which isn't
true. In the common case, which is what Google's deployment does, you allow
NAT rebinding at the client by having the client send connection ID on
packets. If they're absent in the server->client packets, that disallows
IP/port change at the server, which is entirely fine, since the server's
not behind a NAT.

> and that means that I don't know how to make the trade-offs sensibly.)

Do you have more disadvantages besides the potential of increased
link-ability?

---

From: Martin Thomson <[email-masked]>
Date: Thu, 16 Feb 2017 12:41:39 +1100

> Do you have more disadvantages besides the potential of increased

Bloat.  We just took great paints to trim a couple of octets from the
packet number, saving a whole 2 octets in the best case.  But
connection ID is 8.

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 15 Feb 2017 17:45:21 -0800

> rename Google QUIC to GUIC or something :)

HistoriQ? Actually, let's not change the name and just get used to calling
it Google QUIC and IETF QUIC :-)

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 15 Feb 2017 17:55:24 -0800

> packet number, saving a whole 2 octets in the best case.  But

It's optional and commonly not needed in the server->client direction,
which makes it avoidable it in the commonly bandwidth-constrained
direction. It's what Chrome negotiates currently: it tells the server that
it doesn't need connection ID on incoming packets, but it sends them on
outgoing packets.

---

From: Christian Huitema <[email-masked]>
Date: Wed, 15 Feb 2017 16:29:10 -1000

On 2/15/2017 1:36 PM, Jana Iyengar wrote:
> On Wed, Feb 15, 2017 at 3:01 PM, Mike Bishop
> <[email-masked] <mailto:[email-masked]>>
> wrote:
>
>     It seems like there’s an emerging idea (which has been present all
>     along) that there are really two subdivisions of QUIC:
>
>       * QUIC-for-all-time:  Forever-fixed packet layouts carrying
>         version negotiation, public reset, etc.; can carry
>         version-specific headers/payload as well if you’ve agreed on
>         the version
>           o If you change these, you’re defining a new protocol that
>             has to be distinguishable from QUIC in some way
> there led me to a couple of other proposals along the way, but
> chatting with Martin about the codepoint idea led me to this proposal,
> which I think is a nice and elegant simplification of those proposals.
> The core of this proposal is in making existing uses of the Flags byte
> slightly more rigid, and making the rest of the combinatorial space
> available for extensibility.
>
I am a little bit worried about the proposal to set the packet number
size to 32 bits in "QUIC for now". I know that "4 billion messages in
100 milliseconds" appears ridiculously large right now, but do you
really believe that we are not going to see 500 Tbps links in the
future? Or maybe 1Tbps link to the Moon, or to Mars? Maybe not in my
lifetime, but in yours? At that point, how do we get the "short form" to
use 64 bit numbers?

-- Christian Huitema

---

From: Martin Thomson <[email-masked]>
Date: Thu, 16 Feb 2017 13:58:50 +1100

On 16 February 2017 at 13:29, Christian Huitema <[email-masked]> wrote:
> I am a little bit worried about the proposal to set the packet number size
> to 32 bits in "QUIC for now". I know that "4 billion messages in 100
> milliseconds" appears ridiculously large right now, but do you really
> believe that we are not going to see 500 Tbps links in the future? Or maybe
> 1Tbps link to the Moon, or to Mars? Maybe not in my lifetime, but in yours?
> At that point, how do we get the "short form" to use 64 bit numbers?

You can stick the extra high bits after the version in the long form.
The short form can just use a new type field to indicate that the
packet number is gigantic.

Of course, this all assumes that you a) have 500 Tbps or higher, and
b) that you get reordering over ~4 times the BDP on that link.  That
latter part is where I think this breaks down some.

---

From: marcelo bagnulo braun <[email-masked]>
Date: Thu, 16 Feb 2017 08:32:47 +0100

El 16/02/17 a las 02:39, Jana Iyengar escribió:
>
>
> You are assuming that connection ID is gone in both directions, which 
> isn't true. In the common case, which is what Google's deployment 
> does, you allow NAT rebinding at the client by having the client send 
> connection ID on packets. If they're absent in the server->client 
> packets, that disallows IP/port change at the server, which is 
> entirely fine, since the server's not behind a NAT.
>

I guess one case when the connection ID could be useful is the case 
where the server is located in a multihomed network and it is served by 
two or more NATs

I understand that as of today, when there is a failure, proprietary 
protocols are used to synch the state between the NATs in order to allow 
the other NAT to act as backup when the initial NAT (or the path through 
that NAT) fails.

Having the connection ID would allow a much cleaner/cheaper solution, 
since the IP of the server can change and the connection survive.

I guess you make a similar argument about VM migration. Having the 
connection ID in all packets would allow to migrate the VM without 
keeping the address (making unnecessary all the mechanisms to keep the 
address unchanged in VM migration).

And there is the case of multipath, that some form of connection ID will 
be needed, at least in the first packet of the new subflow.

However, I am unconvinced any of these are strong enough to make the 
connection ID mandatory. IMHO it is enough to make it optional (with as 
recommendation for clients to use it in all packets and leave it open 
for servers).

Regards, marcelo

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 16 Feb 2017 07:59:27 +0000

Hi

I believe that it should be sufficient to have the ECN feedback in the ACK frames. There are three alternatives in the draft, however I don’t see the actual feedback as a burning issue.
The more tricky part is how to safely enable ECN given that there are possible issues both as regards to access to the ECN bits in the IP header and in the support in the network.
Marcelo suggested that a dedicated ECN frame is exchanged by the peers after the connection setup. I hope to update the ECN draft soon based on the comments that I received.

Question: Is CWR necessary with QUIC, I always thought that it is something that is a special thing for TCP with its limited header space.
Or is CWR used for other means than to just clear the ECE flag for classic ECN ?

/Ingemar

From: Jana Iyengar [mailto:[email-masked]]
Sent: den 16 februari 2017 00:47
To: Martin Duke <[email-masked]>
Cc: Mirja Kühlewind <[email-masked]>; Ian Swett <[email-masked]>; IETF QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: Re: Alternate header proposal

On Wed, Feb 15, 2017 at 3:26 PM, Martin Duke <[email-masked]<mailto:[email-masked]>> wrote:

Why do you need ECE/CWR bits in the header? I haven't carefully considered yet how to do this in QUIC (and I haven't carefully read Ingemar's draft yet), but those would/should be in explicit frames I would imagine.

The draft puts them in the ACK frame. There is a case, I think, to have them in the public header, though I'm not convinced of it at all. The point I am trying to make is that there are uses for these bits that are not related to the header layout, and in the extreme case this would be as many as 5 (I am personally convinced we need only 2).

I understand and agree with the point that we may have things that need to be put in the packet header (FWIW, and we can discuss this separately, but I'm hard-pressed to understand why ECN bits are needed in the packet header instead of in the ack frame.) Either ways I think we'll have to discuss this on a case-by-case basis, but in the meanwhile, this proposal gives you another degree of freedom -- codepoint space.

The current proposal only addresses things we have reasonable agreement on in the wg, and not on things that are yet under discussion. No matter what header format we use, we're going to have to revisit it if there are issues that touch the header in the future, and to that extent, this header format will have to remain malleable. I wouldn't say we've done away with explicit bits, but I'd sure like to try and avoid them in favor of codepoints if we can, since it seems like a big enough space.

---

From: marcelo bagnulo braun <[email-masked]>
Date: Thu, 16 Feb 2017 09:13:50 +0100

El 16/02/17 a las 08:59, Ingemar Johansson S escribió:
>
> Hi
>
> I believe that it should be sufficient to have the ECN feedback in the 
> ACK frames. There are three alternatives in the draft, however I don’t 
> see the actual feedback as a burning issue.
>
> The more tricky part is how to safely enable ECN given that there are 
> possible issues both as regards to access to the ECN bits in the IP 
> header and in the support in the network.
>
> Marcelo suggested that a dedicated ECN frame is exchanged by the peers 
> after the connection setup. I hope to update the ECN draft soon based 
> on the comments that I received.
>
> Question: Is CWR necessary with QUIC, I always thought that it is 
> something that is a special thing for TCP with its limited header space.
>

I guess if the ECE information is sent reliably in a ECN quic frame, we 
dont need CWR, since the ACK for that frame would fullfill the same purpose?

Regards, marcelo

> /Ingemar
>
> *From:*Jana Iyengar [mailto:[email-masked]]
> *Sent:* den 16 februari 2017 00:47
> *Cc:* Mirja Kühlewind <[email-masked]>; Ian Swett 
> <[email-masked]>; IETF QUIC WG <[email-masked]>; Martin Thomson 
> <[email-masked]>
>         Why do you need ECE/CWR bits in the header? I haven't
>         carefully considered yet how to do this in QUIC (and I haven't
>         carefully read Ingemar's draft yet), but those would/should be
>         in explicit frames I would imagine.
>
>     The draft puts them in the ACK frame. There is a case, I think, to
>     have them in the public header, though I'm not convinced of it at
>     all. The point I am trying to make is that there are uses for
>     these bits that are not related to the header layout, and in the
>     extreme case this would be as many as 5 (I am personally convinced
>     we need only 2).
>
> I understand and agree with the point that we may have things that 
> need to be put in the packet header (FWIW, and we can discuss this 
> separately, but I'm hard-pressed to understand why ECN bits are needed 
> in the packet header instead of in the ack frame.) Either ways I think 
> we'll have to discuss this on a case-by-case basis, but in the 
> meanwhile, this proposal gives you another degree of freedom -- 
> codepoint space.
>
> The current proposal only addresses things we have reasonable 
> agreement on in the wg, and not on things that are yet under 
> discussion. No matter what header format we use, we're going to have 
> to revisit it if there are issues that touch the header in the future, 
> and to that extent, this header format will have to remain malleable. 
> I wouldn't say we've done away with explicit bits, but I'd sure like 
> to try and avoid them in favor of codepoints if we can, since it seems 
> like a big enough space.
>

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Thu, 16 Feb 2017 13:58:19 +0100

hi Martin, all..

(and hi Jana, I like the general approach)

trying to catch up on this thread, falling back to jump-in-randomly:

> On 16 Feb 2017, at 02:41, Martin Thomson <[email-masked]> wrote:
> 

I remain unconvinced of the complexity/space tradeoff for the 8- and 16-bit packet number cases. Your earlier point on handling public resets --

> implement that way, but it's not something that will be obvious.

-- is an excellent illustration of the point here. If we're clawing 16 to 24 bits back only to add 64 bits on Connection ID... the value prop goes from "questionable" to "uncompelling"...

> and that means that I don't know how to make the trade-offs sensibly.)

This might be slightly cowardly, but I share your conflict. Perhaps the presence/absence/utility of connection ID shouldn't be a property of QUIC's design, then, but rather a property of the applications using QUIC. Indeed, this isn't even a thing that should always be left up to the app developer -- there are certainly networks I would be far less interested in increasing my trackability on than others, independent of application.

Recommend two or three points in the space ("always" for load balancing and rebinding, "sometimes" for load balancing and limited rebinding, "never" for reducing traceability) and make supporting them all MTI.

Cheers,

Brian

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 16 Feb 2017 14:31:46 +0100

> Am 16.02.2017 um 08:59 schrieb Ingemar Johansson S <[email-masked]>:
> 

ECE and CWR are both TCP-specifc. For quic you could also provide feedback in the form of counters for ECT1/ECT1/CE (similar to RTCP)...

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 16 Feb 2017 14:33:50 +0100

Hi,

> Am 16.02.2017 um 02:55 schrieb Jana Iyengar <[email-masked]>:
> 
> On Wed, Feb 15, 2017 at 5:41 PM, Martin Thomson <[email-masked]> wrote:
> It's optional and commonly not needed in the server->client direction, which makes it avoidable it in the commonly bandwidth-constrained direction. It's what Chrome negotiates currently: it tells the server that it doesn't need connection ID on incoming packets, but it sends them on outgoing packets.
> 

How do you know if it’s needed or not? Often a client might not know if it is behind a NAT or not. Also there might be other stateful middleboxes that could utilize the connection id (if the 5-tuple changes) or otherwise might break the connection.

Mirja

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Thu, 16 Feb 2017 15:14:01 +0100

hi Martin, all..

Cheers,

Brian

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Thu, 16 Feb 2017 15:14:10 +0100

> On 16 Feb 2017, at 03:58, Martin Thomson <[email-masked]> wrote:
> 

I don't know; I think it'd be fun to design a networking experiment that involves accelerating the moon to relativistic speeds...

---

From: Ian Swett <[email-masked]>
Date: Thu, 16 Feb 2017 10:31:28 -0500

I was a bit concerned about that as well, but I don't think we need more
than 4 bytes for the handshake(long form) and it's easy to add new version
specific code points for 8 byte packet numbers in the future if we wanted
them.  Or we can add the codepoints now if we know we're going to want
them.  Martin's point is valid, that this really only becomes a concern
with large scale reordering or loss.

On Wed, Feb 15, 2017 at 9:58 PM, Martin Thomson <[email-masked]>
wrote:

> size

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 16 Feb 2017 08:11:14 -0800

> packet number is gigantic.

+1. This is why the packet number was after the version earlier, but we can
pretty much do whatever we want in the long form after the version field.

> latter part is where I think this breaks down some.

I'm not arguing for having larger packet numbers now, but just on the point
of reordering: it's difficult to tell if there will be reordering or not.
If large bandwidths are gotten by combining multiple lambdas in one fiber
for instance, then reordering could even be the norm.

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 16 Feb 2017 08:13:10 -0800

On Thu, Feb 16, 2017 at 5:31 AM, Mirja Kühlewind <
[email-masked]> wrote:

> ECE and CWR are both TCP-specifc. For quic you could also provide feedback
> in the form of counters for ECT1/ECT1/CE (similar to RTCP)...

+1 to we can use entirely different feedback mechanisms. We definitely
don't need CWR, since as Marcelo points out we can use ACKs of packets
containing an ECN feedback frame instead.

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 16 Feb 2017 08:20:10 -0800

On Thu, Feb 16, 2017 at 6:14 AM, Brian Trammell (IETF) <[email-masked]>
wrote:

> I remain unconvinced of the complexity/space tradeoff for the 8- and
> 16-bit packet number cases. Your earlier point on handling public resets --
>

The complexity isn't high, and we save bits.

> -- is an excellent illustration of the point here. If we're clawing 16 to
> 24 bits back only to add 64 bits on Connection ID... the value prop goes
> from "questionable" to "uncompelling"...
>

Again, connection ID is optional in this proposal. We're not simply adding
64 bits for everyone. I'll note again that the common packet header size in
this format will be 2-3 bytes: 1 type byte (short form) + 1-2 bytes of
packet number.

- jana

And jumping back to another message:
> This might be slightly cowardly, but I share your conflict. Perhaps the
> presence/absence/utility of connection ID shouldn't be a property of QUIC's
> design, then, but rather a property of the applications using QUIC. Indeed,
> this isn't even a thing that should always be left up to the app developer
> -- there are certainly networks I would be far less interested in
> increasing my trackability on than others, independent of application.
>

Recommend two or three points in the space ("always" for load balancing and
> rebinding, "sometimes" for load balancing and limited rebinding, "never"
> for reducing traceability) and make supporting them all MTI.
>

---

From: Matt Holdrege <[email-masked]>
Date: Thu, 16 Feb 2017 17:15:22 +0100

FWIW I completely agree with the idea of letting the application deal with
connection ID and all that goes with that. QUIC should be quick, not just
in terms of routing/switching packets, but by avoiding upper layer policy
issues.

-Matt

On Thu, Feb 16, 2017 at 1:58 PM, Brian Trammell (IETF) <[email-masked]>
wrote:

> Recommend two or three points in the space ("always" for load balancing
> and rebinding, "sometimes" for load balancing and limited rebinding,
> "never" for reducing traceability) and make supporting them all MTI.
>
> Cheers,
>
> Brian
>

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Fri, 17 Feb 2017 16:20:19 +0100

> On 16 Feb 2017, at 17:20, Jana Iyengar <[email-masked]> wrote:
> 
> On Thu, Feb 16, 2017 at 6:14 AM, Brian Trammell (IETF) <[email-masked]> wrote:
> hi Martin, all..
> 

In the general case, I agree, this is not complex; moving to codepoints for header formats removes the pressure on bits for encoding packet lengths. (Adding a high-order 32 bits for u64 packet lengths after version negotiation is a bit more complex, but it's difficult to reason about the cost of complexity in an environment where you might get two billion packets of reordering...)

As long as the internal packet number is a u64 and the receiver wraps also on u8 and u16 packet numbers -- and conservativeness about switching down to u8 and u16 packet numbers (i.e. only when there are many fewer than 127 or 32767 in flight, respectively) is clear and mandatory in the draft, I guess my only remaining problems with it are purely aesthetic. :)

Cheers,

Brian

> Again, connection ID is optional in this proposal. We're not simply adding 64 bits for everyone. I'll note again that the common packet header size in this format will be 2-3 bytes: 1 type byte (short form) + 1-2 bytes of packet number.
> 
> - jana
>

---

From: Martin Thomson <[email-masked]>
Date: Sat, 18 Feb 2017 09:03:23 +1100

On 18 February 2017 at 02:20, Brian Trammell (IETF) <[email-masked]> wrote:
> conservativeness about switching down to u8 and u16 packet numbers (i.e. only when there are many fewer than 127 or 32767 in flight, respectively)

It's more conservative even than that, it's 64 and 16384 respectively
right now.  That could be overly conservative.

The multiplier/divisor is currently 4, where I think that you could
easily make a case for a lower value.  It's also MUST, which I think
is wrong (I thought I raised this already, but forgot apparently:
https://github.com/quicwg/base-drafts/issues/323).

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 22 Feb 2017 08:58:04 -0800

I'm generally positive on this proposal and I think it's an improvement on
the current
header, so I'd be fine with merging it on the principle of getting closer
to what we want.

With that said, there are some things I'm less happy about:

> * Octets 13-16: proof (octets 1-5 of received packet)

This "proof" stuff doesn't work properly for a hash-based public reset
design,
because the proof has to have high entropy. In general, I don't see a lot of
value in overloading this field. Just put any required proof-type elements
in a data payload. That's probably easier for an implementation anyway.

 0                   1                   2                   3

I'm still unpersuaded that it's that important to shave bits on the packet
number here,
especially on a per-packet basis.

Also, maybe I'm missing something, but where's the key change bit? That
seems to have
gone missing and AFAICT you are still going to need it unless you have some
thought
of swapping to long headers during the key change period until you see acks
or something
else equally gross.

-Ekr

---

From: Martin Thomson <[email-masked]>
Date: Fri, 24 Feb 2017 14:10:52 +1100

> of swapping to long headers during the key change period until you see acks

Wasn't there another set of types for that?  They must have gone
missing, which I'm sure is an accident.

I'm of the opinion that we probably just want to use a bit for this
and the long/short marker as well.  Even more so for the latter.

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 23 Feb 2017 22:55:34 -0800

> and the long/short marker as well.  Even more so for the latter.

Yeah, it's there... I just didn't write it out in the figure to keep the
Type byte the same for both.
I'll write it out separately in my PR. I'm leaning towards long/short
marker as well. I'll add that in the PR too.

---

From: Kazuho Oku <[email-masked]>
Date: Sat, 25 Feb 2017 11:05:11 +0900

Hi,

I generally like the proposed scheme, it looks like much simpler and
would be easier to route.

That said, I have one question.

Under the proposed new scheme, would it be possible to distinguish
between a client-generated and a server-generated connection ID? If
that is possible, could you please let me know how that should be
done?

One of the reasons why server-driven connection ID generation is
preferred (note that in Tokyo we agreed to abandon the use of
client-driven ID generation in favor of server-driven generation) is
because server deployments using anycast IPs would like to make the
POP identifiers part of the connection ID, so that they would be able
to continue serving the connection when packets start arriving at a
different POP when the client address changes due to roaming.

In such deployment, a POP that receives a QUIC packet will forward the
packet to a POP that deems responsible for handling the packet based
on the value of the connection ID, unless the packet (by looking at
the connection ID) is known to be operable within the POP.

The fact means that unless the use of client-generated connection ID
is abandoned or unless there is an easy way to distinguish between a
client-gerenated and a server-generated connection ID, such
deployments would be forced into making contacts between POPs to
determine if there is already an established state for an incoming
connection ID.

This would be especially problematic during connection establishment.
One of the motives under the development of QUIC is to reduce time
spent in connection establishment. We would not like to see
communications between POPs of CDNs (that are located at distinct
places around the globe) happening during connection establishment.

2017-02-15 16:05 GMT+09:00 Jana Iyengar <[email-masked]>:
> here.) This borrows the long/short header idea from Martin's formulation,
> but turns the flags field into a type byte, with only 18 codepoints defined.

-- 
Kazuho Oku

---

From: Ian Swett <[email-masked]>
Date: Sat, 25 Feb 2017 11:19:09 -0500

I'm not sure this fully solves your issue, but I don't believe QUIC will
allow port or IP migration during the handshake, so landing at a different
POP is only an issue to deal with once 1RTT keys are available.

Possibly you can do conventional IP/port load balancing on all long form
unprotected client packets, and then switch to using connection ID on all
short form packets?

Alternately, if only the SHLO could change the connection ID, then
all(except Finished?) unprotected client to server packets would use the
client's connection ID and the change of connection ID would follow the
change to 1RTT keys.  I don't think the rules on when a server can change
the connection ID are fully fleshed out yet.

On Fri, Feb 24, 2017 at 9:05 PM, Kazuho Oku <[email-masked]> wrote:

> connection
> > ID

---

From: "Lubashev, Igor" <[email-masked]>
Date: Sat, 25 Feb 2017 17:55:27 +0000

That's how I read the proposal, too. The load balancing is stateless (*) only after 1rtt keys are available. Before that, there must be state, so the availability of connections depends on the packets arriving on the same load balancer before 1 rtt keys are available. We can try hard to ensure that pre-1rtt packets arriving within a pop are routed to the same load balancer (if it is still available).

If a pre-1rtt packet arrives to a different pop (only a problem for anycast-routed CDN service, not a problem for DNS-based CDN service), the connection will be reset, and the client needs to try again.

- Igor

(*) The load balancing will become truly stateless if you encode not just the identity of a pop but the identity of the backend server in the connection id.

-----Original Message-----
From: Ian Swett [[email-masked]]
Received: Saturday, 25 Feb 2017, 11:19AM
To: Kazuho Oku [[email-masked]]
CC: Jana Iyengar [[email-masked]]; IETF QUIC WG [[email-masked]]
Subject: Re: Alternate header proposal

I'm not sure this fully solves your issue, but I don't believe QUIC will allow port or IP migration during the handshake, so landing at a different POP is only an issue to deal with once 1RTT keys are available.

Possibly you can do conventional IP/port load balancing on all long form unprotected client packets, and then switch to using connection ID on all short form packets?

Alternately, if only the SHLO could change the connection ID, then all(except Finished?) unprotected client to server packets would use the client's connection ID and the change of connection ID would follow the change to 1RTT keys.  I don't think the rules on when a server can change the connection ID are fully fleshed out yet.

On Fri, Feb 24, 2017 at 9:05 PM, Kazuho Oku <[email-masked]<mailto:[email-masked]>> wrote:
Hi,

--
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 27 Feb 2017 14:35:58 +0900

Ian, Igor,

Thank you for your answers.

> the connection ID are fully fleshed out yet.

Yes. I think it would be possible to implement either of the two approaches.

I was worrying if we would need to add to the spec. a bit within the
connection ID to distinguish between client-generated and
server-generated ones.

But I am happy to know that, by reading your advice, it seems like
that it would be possible to distinguish between the two using the
type of the packet.

Once this change gets merged, I will try to confirm that such
distinction can be made by using the types.

Thank you in advance.

-- 
Kazuho Oku