Subject: Ossification

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 30 Jan 2017 18:08:25 +0100

Hi all,

I kept thinking about the problem of ossification based on some discussion we had last Thursday. Even if we try our best to make quic headers as flexible as possible and try to maintain this by greasing, we will still see ossification that we don’t want. I’m mostly thinking about the connection ID in data packets here. I pretty sure that we will see middblebox that will only check for the ‚special packet‘ bit and assume to find the connection ID at a certain position in the header if that bit is not set. 

In general I see three approaches to face this problem:

1) Try to not expose any data at all (as soon as crypto context is established). This would mean that we should not provide a connection ID (meaning we will not support use cases where in network devices could can make use of it) and we should encrypt the packet number.

2) Provide the version number in every packet. This would at least increase the change that a middlebox would check the version number before making further implicit assumptions. However, this would probably ossify the version number we choose for the initial version as middleboxes would probably just treat this as a magic number to detect quic and are slow in updating. If we want to prevent that we would probably also need something like an online registry, where a certain number is mapped to a certain version and maybe even periodically change that mapping…? Not sure how brilliant that idea is…

3) Decide for a common wire image for all quic packets now and fix that for all version of quic. I know the intention of quic versioning would be to be as flexible as possible but I don’t think this option is as bad as it sounds. First of all, I believe we already have a good idea of the limited set of information we want to provide to middbleboxes (see below). And second, I'm also rather positive that if we agree on this now, we anyway want to maintain this information for all future versions of quic, given this should make deployment of future versions in a middleboxed network easier/possible. 

What I think we need is the following:

- a magic number: similar as already discussed for the version negotiation packet, this makes it super easy to recognize quic and therefore takes away any incentives to misuse anything else for this purpose. Note that one could still standardize another UDP-based protocol (which we might then not be called quic anymore) with a different magic number but deployment might be harder as soon as the quic magic number is widely deployed. So this basically introduces another kind of versioning with higher costs, while hopefully the actual quic versioning will not be misused and can therefore be used to change quic at low cost.

- connection ID (as routing information; might be zero in the first packet…)

- packet number (can be used for loss detection of losses that occurred before the observation point if increased linearly without gaps), and

- packet number echo (can be used for RTT measurements and proof of return routability)

And that’s it. 

My 2c…
Mirja

---

From: Martin Thomson <[email-masked]>
Date: Tue, 31 Jan 2017 11:35:37 +1100

> - packet number (can be used for loss detection of losses that occurred before the observation point if increased linearly without gaps), and

On this point, existing implementations skip packet numbers
periodically.  That looks like loss to the other side (or the path),
but even spurious losses don't have a bit impact on things like
congestion control if they are rare enough.

One thing that we've discussed is the use of gaps to verify packet
receipt after a connection migration.  An endpoint that detects a move
can start skipping more aggressively to protect against optimistic ack
attacks on that new path.  I guess that you could argue that echoing a
packet number achieves the same effect more efficiently, but we would
need to assess that in the context of the bytes it would expend
overall.

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 31 Jan 2017 09:16:05 +0000

I see real merit in moving beyond (1) for another reason if this exposes 
at least connection ID, packet number, packet number echo (asuming one 
can also derive (begin/end of flow, etc for a traffic flow.

Over the years I've seen the IETF produce quite a few "transport" 
protocols, and while each of them has had different requirements and 
ambitions, I'd like to observe that successful ones often include an 
ability for network operators and independent researchers to both gather 
per-flow traffic measurements and to understand pathologies of how 
traffic flows share capacity and understand where problems exist within 
networks. Doing this within the network relies on visibility of at least 
fields with visibility of basic flow information in the payload.

It's OK fro me tohave rules about whether packet numbers have to 
monotomically increase to be useful at a receiver, whether receivers can 
usefully use re-resequenced packets, or data with gaps, etc. Even with 
these constraints, a set of visible fields could still open-up data on 
whether my current network link is seeing more "gaps", "resequencing", 
"whatever" compared to a measurement at another time of day, or another 
type of network link - all of which can be helpful in understanding how 
to operate my network, and whether this traffic is co-existing with 
other transport services. As we look ahead to networks with greater 
variability (5G?), more reordering, or wider mixes of traffic and 
forwarding rules, this becomes more important, not less.

Gorry

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 31 Jan 2017 10:55:04 -0800

I'm loathe to increase the QUIC packet header size to avoid increasingly
additional overhead. I agree that (1) seems fundamentally hard to do
without substantial computational cost and other overhead, so I'd be fine
with exposing some things. I think this is expected to be discussed in
Chicago anyways, FWIW.

That said, my expectation is that there are three types of middleboxes that
we want to address:
(i) Those that want to separate QUIC connections from unwarranted traffic;
(ii) Those that want to track QUIC connection state; and
(iii) Those that want to firewall certain services (eg., WebRTC).
Middleboxes are interested in blocking services, not the transport.

(iii) is basically solved using the same signals that middleboxes use
today: server port number. Blocking UDP to port 443 effectively blocks use
of QUIC for HTTPS.

For (i) and (ii), if middleboxes are being modified to detect QUIC, I don't
think they'd have trouble going through the state machine and remembering
where the QUIC connection ID falls. I appreciate your concern about
ossification of specific versions, but if there are ossified middleboxes
that need to track connection state (type (iii) above), then we're stuck
with the position of the connection ID anyways.

Specifying the version or sending a magic number in each packet has
quantifiable overhead cost but the benefits are unknown, and I'd be
inclined against them. For in-network measurements, as we've discussed
previously, there may be value in sending the "largest observed" in that
header.

On Tue, Jan 31, 2017 at 1:16 AM, Gorry Fairhurst <[email-masked]>
wrote:

> at least connection ID, packet number, packet number echo (asuming one can
> also derive (begin/end of flow, etc for a traffic flow.
>
> per-flow traffic measurements and to understand pathologies of how traffic
> flows share capacity and understand where problems exist within networks.
> Doing this within the network relies on visibility of at least fields with
> visibility of basic flow information in the payload.
>
> type of network link - all of which can be helpful in understanding how to
> operate my network, and whether this traffic is co-existing with other
> transport services. As we look ahead to networks with greater variability
> (5G?), more reordering, or wider mixes of traffic and forwarding rules,
> this becomes more important, not less.
>
> Gorry
>
>
>>> - packet number (can be used for loss detection of losses that occurred
>>> before the observation point if increased linearly without gaps), and
>>>

---

From: Mirja Kühlewind <[email-masked]>
Date: Tue, 31 Jan 2017 20:24:29 +0100

Hi Jana,

a few comments below.

> Am 31.01.2017 um 19:55 schrieb Jana Iyengar <[email-masked]>:
> 
> I'm loathe to increase the QUIC packet header size to avoid increasingly additional overhead. I agree that (1) seems fundamentally hard to do without substantial computational cost and other overhead, so I'd be fine with exposing some things. I think this is expected to be discussed in Chicago anyways, FWIW.
> 
> That said, my expectation is that there are three types of middleboxes that we want to address:
> (iii) Those that want to firewall certain services (eg., WebRTC). Middleboxes are interested in blocking services, not the transport.
> 
> (iii) is basically solved using the same signals that middleboxes use today: server port number. Blocking UDP to port 443 effectively blocks use of QUIC for HTTPS.
> 
> For (i) and (ii), if middleboxes are being modified to detect QUIC, I don't think they'd have trouble going through the state machine and remembering where the QUIC connection ID falls. I appreciate your concern about ossification of specific versions, but if there are ossified middleboxes that need to track connection state (type (iii) above), then we're stuck with the position of the connection ID anyways.

That’s my point. Knowing that the position of the connection ID will ossify, we should maybe consider to actively decide that we want to support that and design the protocol respectively.

> 
> Specifying the version or sending a magic number in each packet has quantifiable overhead cost but the benefits are unknown, and I'd be inclined against them. For in-network measurements, as we've discussed previously, there may be value in sending the "largest observed" in that header.

We already agreed that having some kind of a magic number in the version negotiation packets makes sense to avoid conflicts with other UDP traffic. If you have seen version negotiation, you can use the connection ID in subsequent packets to assign it to a known flow and distinguish it from other UDP-based (cross) traffic. However, if you have not seen the handshake, there is no way to distinguish UDP-based quic traffic. This makes the quic handshake a special signal for the network which I would rather like to avoid because it’s not. Yes, putting a magic number in all packets is overhead. But this is the trade-off I’d like to discuss.

Mirja

> 
> On Tue, Jan 31, 2017 at 1:16 AM, Gorry Fairhurst <[email-masked]> wrote:
> I see real merit in moving beyond (1) for another reason if this exposes at least connection ID, packet number, packet number echo (asuming one can also derive (begin/end of flow, etc for a traffic flow.
> 
> Over the years I've seen the IETF produce quite a few "transport" protocols, and while each of them has had different requirements and ambitions, I'd like to observe that successful ones often include an ability for network operators and independent researchers to both gather per-flow traffic measurements and to understand pathologies of how traffic flows share capacity and understand where problems exist within networks. Doing this within the network relies on visibility of at least fields with visibility of basic flow information in the payload.
> 
> It's OK fro me tohave rules about whether packet numbers have to monotomically increase to be useful at a receiver, whether receivers can usefully use re-resequenced packets, or data with gaps, etc. Even with these constraints, a set of visible fields could still open-up data on whether my current network link is seeing more "gaps", "resequencing", "whatever" compared to a measurement at another time of day, or another type of network link - all of which can be helpful in understanding how to operate my network, and whether this traffic is co-existing with other transport services. As we look ahead to networks with greater variability (5G?), more reordering, or wider mixes of traffic and forwarding rules, this becomes more important, not less.
> 
> Gorry
> 
>

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 31 Jan 2017 11:37:41 -0800

On Tue, Jan 31, 2017 at 11:24 AM, Mirja Kühlewind <
[email-masked]> wrote:

> > For (i) and (ii), if middleboxes are being modified to detect QUIC, I
> don't think they'd have trouble going through the state machine and
> remembering where the QUIC connection ID falls. I appreciate your concern
> about ossification of specific versions, but if there are ossified
> middleboxes that need to track connection state (type (iii) above), then
> we're stuck with the position of the connection ID anyways.
>
> That’s my point. Knowing that the position of the connection ID will
> ossify, we should maybe consider to actively decide that we want to support
> that and design the protocol respectively.

I agree, and I think this should be discussed.

> We already agreed that having some kind of a magic number in the version
> negotiation packets makes sense to avoid conflicts with other UDP traffic.
> If you have seen version negotiation, you can use the connection ID in
> subsequent packets to assign it to a known flow and distinguish it from
> other UDP-based (cross) traffic. However, if you have not seen the
> handshake, there is no way to distinguish UDP-based quic traffic. This
> makes the quic handshake a special signal for the network which I would
> rather like to avoid because it’s not. Yes, putting a magic number in all
> packets is overhead. But this is the trade-off I’d like to discuss.

It is a special signal -- it is the beginning of a QUIC connection. I'm not
sure why you would avoid that.

If you're looking for a magic signal in every packet, I don't think it's
going to be effective in the long term. It's simple enough for folks
generating bogus traffic to send it with the magic signal. If you're trying
to thwart redirected traffic, then you look for a handshake sequence, which
gives you a pivot to hang the rest of the connection from.

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 31 Jan 2017 19:58:58 +0000

Yes at least the connection ID must be in the clear for load balancing purposes - both in the network for ECMP and in the end host for spreading traffic amongst cores. Essentially I don’t think we can assume that all middleboxes will be stateful or that there will be no need for stateless offloads to the NICs. And for such use cases being able to distinguish QUIC traffic is very useful to avoid for example spreading other UDP traffic causing packet reordering and reassembly etc. So I think a magic number is useful not only to distinguish QUIC and QUIC versions but also to distinguish other UDP traffic. But such a generic mechanism for UDP is presumably outside the scope of this working group?

Thanks

From: QUIC [mailto:[email-masked]] On Behalf Of Jana Iyengar
Sent: Tuesday, January 31, 2017 11:38 AM
To: Mirja Kühlewind <[email-masked]>
Cc: Gorry (erg) <[email-masked]>; IETF QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: Re: Ossification

On Tue, Jan 31, 2017 at 11:24 AM, Mirja Kühlewind <[email-masked]<mailto:[email-masked]>> wrote:
Hi Jana,

a few comments below.

> Am 31.01.2017 um 19:55 schrieb Jana Iyengar <[email-masked]<mailto:[email-masked]>>:
>

It is a special signal -- it is the beginning of a QUIC connection. I'm not sure why you would avoid that.

If you're looking for a magic signal in every packet, I don't think it's going to be effective in the long term. It's simple enough for folks generating bogus traffic to send it with the magic signal. If you're trying to thwart redirected traffic, then you look for a handshake sequence, which gives you a pivot to hang the rest of the connection from.

Mirja

>
> On Tue, Jan 31, 2017 at 1:16 AM, Gorry Fairhurst <[email-masked]<mailto:[email-masked]>> wrote:
> <[email-masked]<mailto:[email-masked]>>  wrote:

---

From: Christian Huitema <[email-masked]>
Date: Tue, 31 Jan 2017 10:02:26 -1000

On 1/31/2017 9:24 AM, Mirja Kühlewind wrote:
...
> ossify, we should maybe consider to actively decide that we want to 
> support that and design the protocol respectively.

Maybe we should start with an analysis of what the connection ID is used 
for, and determine which of these uses requires bytes in the header, 
versus for example an option in a connect message. And maybe how many 
bytes. So far the candidates are:

1) Demultiplexing at receiver. Arguably, this can be also be done by 
just looking at port numbers.

2) Reconnecting after a NAT mapping change. Without a connection ID, 
this would have to be treated like any other mobility event.

3) Reconnecting after a mobility event. The connection ID provides a 
simple mechanism. Without connection ID, we would have to do an explicit 
connection resume, similar to TLS resume.

4) Load balancing over several paths, as in TCP multipath. Connection ID 
enables a simple solution, perhaps too simple. TCP multipath manages 
separate sequence numbers and congestion controls for each path, and to 
do that we would need different connection ID for each path.

For mobility and multipath, having the connection ID in all packets has 
a substantial privacy downside, so arguably we will need to develop 
"something like TLS resume" in any case. So maybe we need to think 
really hard about these issues!

-- Christian Huitema

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 31 Jan 2017 20:06:04 +0000

I would like to add that today RSS and ECMP for UDP likely don’t do flow level load balancing and stick to using the two-tuple (src IP, dest IP) because UDP traffic can be fragmented so packets of the same flow could get reordered. The question is would we want this to be better for QUIC given that its likely to emulate TCP and not fragment packets. I see that Christian has brought up privacy concerns over connection ID being in the clear which is a valid concern so there’s also a security vs. performance tradeoff.

From: QUIC [mailto:[email-masked]] On Behalf Of Praveen Balasubramanian
Sent: Tuesday, January 31, 2017 11:59 AM
To: Jana Iyengar <[email-masked]>; Mirja Kühlewind <[email-masked]>
Cc: Gorry (erg) <[email-masked]>; IETF QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: RE: Ossification

Thanks

From: QUIC [mailto:[email-masked]] On Behalf Of Jana Iyengar
Sent: Tuesday, January 31, 2017 11:38 AM
To: Mirja Kühlewind <[email-masked]<mailto:[email-masked]>>
Cc: Gorry (erg) <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>; Martin Thomson <[email-masked]<mailto:[email-masked]>>
Subject: Re: Ossification

a few comments below.

Mirja

---

From: Mirja Kühlewind <[email-masked]>
Date: Tue, 31 Jan 2017 21:19:01 +0100

Hi Christian,

we discussed most of these cases at the interim and also discussed a possible solution where the client could ask the server for a next connection ID that it would use after a mobility event (case 3) and the server could also provide this new ID to the load balancer (case 4). However, if the client is not aware of the network change (case 2), it will just send the old connection ID over the new path which would make it possible to link these two connections. This part is hard to avoid. Also note that even if we would not have a connection ID at all, the (random) packet number would provide similar opportunities for linkage. I believe the conclusion (so far) was that the charter requirement is to be not worth than (MP)TCP and as such it’s probably worth to have a connection ID to support use case 2-4 (agree that 1 is no use case).
However, this will need further discussion in Chicago!

Mirja

> Am 31.01.2017 um 21:02 schrieb Christian Huitema <[email-masked]>:
> 
> Maybe we should start with an analysis of what the connection ID is used for, and determine which of these uses requires bytes in the header, versus for example an option in a connect message. And maybe how many bytes. So far the candidates are:
> 
> 1) Demultiplexing at receiver. Arguably, this can be also be done by just looking at port numbers.
> 
> 2) Reconnecting after a NAT mapping change. Without a connection ID, this would have to be treated like any other mobility event.
> 
> 3) Reconnecting after a mobility event. The connection ID provides a simple mechanism. Without connection ID, we would have to do an explicit connection resume, similar to TLS resume.
> 
> 4) Load balancing over several paths, as in TCP multipath. Connection ID enables a simple solution, perhaps too simple. TCP multipath manages separate sequence numbers and congestion controls for each path, and to do that we would need different connection ID for each path.
> 
> For mobility and multipath, having the connection ID in all packets has a substantial privacy downside, so arguably we will need to develop "something like TLS resume" in any case. So maybe we need to think really hard about these issues!
> 
> -- Christian Huitema

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 31 Jan 2017 21:22:12 +0100

On 31/01/2017 19:55, Jana Iyengar wrote:
> I'm loathe to increase the QUIC packet header size to avoid 
> increasingly additional overhead.
That wasn't specifically what I asked.
> I agree that (1) seems fundamentally hard to do without substantial 
> computational cost and other overhead, so I'd be fine with exposing 
> some things.
OK.
> I think this is expected to be discussed in Chicago anyways, FWIW.
>
My point was related to operating a network and allowing independent 
research on traffic. Maybe that's a fourth case? - tracking state seems 
rather more than passive monitoring.
> today: server port number. Blocking UDP to port 443 effectively blocks 
> use of QUIC for HTTPS.
>
> remembering where the QUIC connection ID falls. I appreciate your 
> concern about ossification of specific versions, but if there are 
> ossified middleboxes that need to track connection state (type (iii) 
> above), then we're stuck with the position of the connection ID anyways.
>
> previously, there may be value in sending the "largest observed" in 
> that header.
>
I didn't ask for a magic number either, assuming that packet numbers 
were a feature that would be in every version, largest observed would I 
expect be helpful here.

Gorry
>     I see real merit in moving beyond (1) for another reason if this
>     exposes at least connection ID, packet number, packet number echo
>     (asuming one can also derive (begin/end of flow, etc for a traffic
>     flow.
>
>     protocols, and while each of them has had different requirements
>     and ambitions, I'd like to observe that successful ones often
>     include an ability for network operators and independent
>     researchers to both gather per-flow traffic measurements and to
>     understand pathologies of how traffic flows share capacity and
>     understand where problems exist within networks. Doing this within
>     the network relies on visibility of at least fields with
>     It's OK for me to have rules about whether packet numbers have to
>     monotomically increase to be useful at a receiver, whether
>     receivers can usefully use re-resequenced packets, or data with
>     gaps, etc. Even with these constraints, a set of visible fields
>     could still open-up data on whether my current network link is
>     seeing more "gaps", "resequencing", "whatever" compared to a
>     measurement at another time of day, or another type of network
>     link - all of which can be helpful in understanding how to operate
>     my network, and whether this traffic is co-existing with other
>     transport services. As we look ahead to networks with greater
>         <[email-masked]
>         <mailto:[email-masked]>>  wrote:
>
>             - packet number (can be used for loss detection of losses
>             that occurred before the observation point if increased
>             linearly without gaps), and
>

---

From: Mike Bishop <[email-masked]>
Date: Tue, 31 Jan 2017 20:54:32 +0000

Regarding (iii), I don’t think we’re that ossified on UDP yet.  HTTP/QUIC doesn’t reserve a UDP port number.  You only find out about the host/port through the Alt-Svc advertisement, and that can advertise any port it likes.  There may be a convention that servers use :443….  But then again, it might be worth avoiding that convention if it helps minimize ossification.

From: QUIC [mailto:[email-masked]] On Behalf Of Jana Iyengar
Sent: Tuesday, January 31, 2017 10:55 AM
To: Gorry (erg) <[email-masked]>
Cc: Mirja Kühlewind <[email-masked]>; IETF QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: Re: Ossification

Gorry