Subject: obsrvtaion on packet numbers

From: Roni Even <[email-masked]>
Date: Wed, 6 Sep 2017 08:10:01 +0000

Hi,
I have some observations on the new text on packet numbers

"A QUIC endpoint MUST NOT reuse a packet number within the same connection (that is, under the same cryptographic keys). If the packet number for sending reaches 2^64 - 1, the sender MUST close the connection without sending a CONNECTION_CLOSE frame or any further packets; the sender MAY send a Stateless Reset packet in response to further packets that it receives."

I post them here and not in github since I am not sure where to discuss it.

1.       "the sender MAY send a Stateless Reset" - I believe that only a sender which is a server can do it.

2.       "If the packet number for sending reaches 2^64 - 1,..." Here I am confused. I understand that since wrap around is not allowed than the sender cannot send a CONNECTION_CLOSE or use the Immediate Close termination procedure. I am not sure about application close termination procedure since I assume it will also require some QUIC messages to close connection.  On the other hand the sender may wish to continue sending using a new connection, so is this a connection migration ? I assume that the state of the open streams should be kept.  I think that some text is missing here and for the connection migration procedure.

Roni Even

---

From: Martin Thomson <[email-masked]>
Date: Wed, 6 Sep 2017 20:20:48 +1000

Hi Roni,

The text about the sender clearly needs to be qualified.  s/the
sender/a server/ in this case (I've made that change already).

Let's just examine what this entails.  2^64 packets is an awful lot of
packets, even accounting for the randomized start and frequent new
connection IDs[1].  So many that we might not need to worry about this
happening for a while (that's 0.5 trillion packets per second for a
solid year if I'm reading the number correctly or 4 Pbps, that's the
one with 15 zeros).

So it's not obvious that special attention is needed here other than
what is written (i.e., stop sending packets).  We could advise the use
of CONNECTION_CLOSE, but then we have to say something about when to
send it.  A graceful shutdown is even better, but that's even more
amorphous and difficult to provide concrete advice about.  Better to
leave this corner unpainted in my opinion.

What's more, it's clear that both sides know what is going on.  Once
this magic value is hit, then no more packets will be sent and both
peers know it.  No real need for any signaling.

Note that this isn't a connection migration scenario.  Any new
connection would - at best - be able to retain cryptographic keys.
Stream state would have to be lost.

[1] On average new connection IDs would also get us here quicker
maybe, though it would still take 272 per second for the same year.

On Wed, Sep 6, 2017 at 6:10 PM, Roni Even <[email-masked]> wrote:
> “A QUIC endpoint MUST NOT reuse a packet number within the same connection
> (that is, under the same cryptographic keys). If the packet number for
> sending reaches 2^64 - 1, the sender MUST close the connection without
> sending a CONNECTION_CLOSE frame or any further packets; the sender MAY send
> a Stateless Reset packet in response to further packets that it receives.”
>
>
>
> 1.       “the sender MAY send a Stateless Reset” – I believe that only a
> sender which is a server can do it.
>
> 2.       “If the packet number for sending reaches 2^64 - 1,…” Here I am
> confused. I understand that since wrap around is not allowed than the sender
> cannot send a CONNECTION_CLOSE or use the Immediate Close termination
> procedure. I am not sure about application close termination procedure since
> I assume it will also require some QUIC messages to close connection.  On
> the other hand the sender may wish to continue sending using a new
> connection, so is this a connection migration ? I assume that the state of
> the open streams should be kept.  I think that some text is missing here and
> for the connection migration procedure.
>
>
>
> Roni Even
>
>

---

From: Roni Even <[email-masked]>
Date: Thu, 7 Sep 2017 05:36:47 +0000

Hi Martin,
I am OK with leaving it as is but here are some other numbers. 
VR or live feed of video events  using 16 cameras (360 degree view) at 4K resolution 30 fps 24 bit per pixel 300:1 compression will require about 300Mb/sec if my calculation is correct.  if a packet is about 1200 bytes of media (this is 25k packets per sec) it will be a over 24 hours to reach the end if starting packet number at 2^32. 
Going to 8k at 60fps will take less time but I assume it will run into network bandwidth issues.
Roni

> -----Original Message-----
> From: Martin Thomson [mailto:[email-masked]]
> Sent: יום ד 06 ספטמבר 2017 13:21
> To: Roni Even
> Cc: QUIC WG
> Subject: Re: obsrvtaion on packet numbers
> 
> Hi Roni,
> 
> The text about the sender clearly needs to be qualified.  s/the sender/a
> server/ in this case (I've made that change already).
> 
> Let's just examine what this entails.  2^64 packets is an awful lot of packets,
> even accounting for the randomized start and frequent new connection
> IDs[1].  So many that we might not need to worry about this happening for a
> while (that's 0.5 trillion packets per second for a solid year if I'm reading the
> number correctly or 4 Pbps, that's the one with 15 zeros).
> 
> So it's not obvious that special attention is needed here other than what is
> written (i.e., stop sending packets).  We could advise the use of
> CONNECTION_CLOSE, but then we have to say something about when to
> amorphous and difficult to provide concrete advice about.  Better to leave
> this corner unpainted in my opinion.
> 
> What's more, it's clear that both sides know what is going on.  Once this
> magic value is hit, then no more packets will be sent and both peers know it.
> No real need for any signaling.
> 
> Note that this isn't a connection migration scenario.  Any new connection
> would - at best - be able to retain cryptographic keys.
> [1] On average new connection IDs would also get us here quicker maybe,
> though it would still take 272 per second for the same year.
> 
> > “A QUIC endpoint MUST NOT reuse a packet number within the same
> > connection (that is, under the same cryptographic keys). If the packet
> > number for sending reaches 2^64 - 1, the sender MUST close the
> > connection without sending a CONNECTION_CLOSE frame or any further
> > packets; the sender MAY send a Stateless Reset packet in response to
> further packets that it receives.”
> >
> >
> >
> > confused. I understand that since wrap around is not allowed than the
> > sender cannot send a CONNECTION_CLOSE or use the Immediate Close
> > termination procedure. I am not sure about application close
> > termination procedure since I assume it will also require some QUIC
> > messages to close connection.  On the other hand the sender may wish
> > to continue sending using a new connection, so is this a connection
> > migration ? I assume that the state of the open streams should be
> > kept.  I think that some text is missing here and for the connection
> migration procedure.
> >
> >
> >
> > Roni Even
> >
> >

---

From: joel jaeggli <[email-masked]>
Date: Wed, 6 Sep 2017 23:11:50 -0700

> VR or live feed of video events  using 16 cameras (360 degree view) at 4K resolution 30 fps 24 bit per pixel 300:1 compression will require about 300Mb/sec if my calculation is correct.  if a packe…

2^32 isn't half of 2^64, it's half of 2^33. 25kpps is 2 billion odd
packets a day whereas 2^64 is 1.84 * 10^19.

> Going to 8k at 60fps will take less time but I assume it will run into network bandwidth issues.

There are much larger flows.

---

From: Roni Even <[email-masked]>
Date: Thu, 7 Sep 2017 06:17:41 +0000

> From: joel jaeggli [mailto:[email-masked]]
> Sent: יום ה 07 ספטמבר 2017 09:12
> To: Roni Even; Martin Thomson
> Cc: QUIC WG
> > VR or live feed of video events  using 16 cameras (360 degree view) at 4K
> resolution 30 fps 24 bit per pixel 300:1 compression will require about
> 300Mb/sec if my calculation is correct.  if a packet is about 1200 bytes of
> media (this is 25k packets per sec) it will be a over 24 hours to reach the end if
> starting packet number at 2^32.
> 
> 2^32 isn't half of 2^64, it's half of 2^33. 25kpps is 2 billion odd packets a day
> whereas 2^64 is 1.84 * 10^19.
[Roni Even]I was basing it on the text assuming that packet number does not start at 0  " The initial value for packet number MUST be selected from an uniform
   random distribution between 0 and 2^31-1 "
> >> Let's just examine what this entails.  2^64 packets is an awful lot
> >> of packets, even accounting for the randomized start and frequent new
> >> connection IDs[1].  So many that we might not need to worry about
> >> this happening for a while (that's 0.5 trillion packets per second
> >> for a solid year if I'm reading the number correctly or 4 Pbps, that's the
> one with 15 zeros).
> >>
> >> what is written (i.e., stop sending packets).  We could advise the
> >> use of CONNECTION_CLOSE, but then we have to say something about
> when
> >> to send it.  A graceful shutdown is even better, but that's even more
> >> this magic value is hit, then no more packets will be sent and both peers
> know it.
> >>> connection (that is, under the same cryptographic keys). If the
> >>> packet number for sending reaches 2^64 - 1, the sender MUST close
> >>> the connection without sending a CONNECTION_CLOSE frame or any
> >>> further packets; the sender MAY send a Stateless Reset packet in
> >>> response to
> >>> confused. I understand that since wrap around is not allowed than
> >>> the sender cannot send a CONNECTION_CLOSE or use the Immediate
> Close

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 7 Sep 2017 14:36:18 -0700

Roni,

> [Roni Even]I was basing it on the text assuming that packet number does
> not start at 0  " The initial value for packet number MUST be selected from
> an uniform

Even if your initial packet number is the largest possible, that is, 2^32,
you still have > 2^63 numbers left.
2^64 = 2^63 + 2^63, which is the total number of possible packet numbers,
and 2^63 + 2^32 is the number of packets you can send with this worst case
initial packet number.

- jana

---

From: Roberto Peon <[email-masked]>
Date: Thu, 7 Sep 2017 21:45:24 +0000

Concretely: 2^63 represents 11,690,840.9 years of packets at 25kpps, or 292 years at 1 Gpps.

Even if it was only 2^32, though, a new connection per day would be reasonable—many sites won’t want to maintain connection state for substantially more than a day (people update binaries/change config, need to shuffle load around to deal with load asymmetries, etc.).

-=R

From: QUIC <[email-masked]> on behalf of Jana Iyengar <[email-masked]>
Date: Thursday, September 7, 2017 at 2:36 PM
To: Roni Even <[email-masked]>
Cc: joel jaeggli <[email-masked]>, QUIC WG <[email-masked]>, Martin Thomson <[email-masked]>
Subject: Re: obsrvtaion on packet numbers

Roni,

Even if your initial packet number is the largest possible, that is, 2^32, you still have > 2^63 numbers left.
2^64 = 2^63 + 2^63, which is the total number of possible packet numbers, and 2^63 + 2^32 is the number of packets you can send with this worst case initial packet number.

- jana

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 8 Sep 2017 05:47:04 +0000

A connection a day is not sufficient, but the real problem is the stream ID when sending multiple new streams per package - which happens in partially reliable transmissions with uni-directional streams - you can exhaust ID space within hours.

Mikkel

________________________________
From: QUIC <[email-masked]> on behalf of Roberto Peon <[email-masked]>
Sent: Thursday, September 7, 2017 11:45:24 PM
To: Jana Iyengar; Roni Even
Cc: joel jaeggli; QUIC WG; Martin Thomson
Subject: Re: obsrvtaion on packet numbers

-=R

Roni,

- jana

---

From: Roni Even <[email-masked]>
Date: Sun, 10 Sep 2017 08:24:46 +0000

Hi,
I started the thread since I tried to understand how to close a connection when packet number reach the end. The text does not specify this case and Martin Thomson claimed that it is not important currently since this case has low probability.
It is much simpler to close a connection when stream IDs are exhausted and start a new connection, so less worry.
Roni

From: Mikkel Fahnøe Jørgensen [mailto:[email-masked]]
Sent: יום ו 08 ספטמבר 2017 08:47
To: Roberto Peon; Jana Iyengar; Roni Even
Cc: joel jaeggli; QUIC WG; Martin Thomson
Subject: Re: obsrvtaion on packet numbers

Mikkel

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Roberto Peon <[email-masked]<mailto:[email-masked]>>

-=R

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Jana Iyengar <[email-masked]<mailto:[email-masked]>>
Date: Thursday, September 7, 2017 at 2:36 PM
To: Roni Even <[email-masked]<mailto:[email-masked]>>
Cc: joel jaeggli <[email-masked]<mailto:[email-masked]>>, QUIC WG <[email-masked]<mailto:[email-masked]>>, Martin Thomson <[email-masked]<mailto:[email-masked]>>
Subject: Re: obsrvtaion on packet numbers

Roni,

- jana

---

From: Roberto Peon <[email-masked]>
Date: Mon, 11 Sep 2017 17:45:47 +0000

Mikkel—

Not sure what you mean, since it’d take ~11 million years at 65000 packets/second to exhaust the ID space. If you used up 4 IDs per stream, it is still > 2 million years.

Am I misunderstanding?

-=R

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thursday, September 7, 2017 at 10:47 PM
To: Roberto Peon <[email-masked]>, Jana Iyengar <[email-masked]>, Roni Even <[email-masked]>
Cc: joel jaeggli <[email-masked]>, QUIC WG <[email-masked]>, Martin Thomson <[email-masked]>
Subject: Re: obsrvtaion on packet numbers

Mikkel

-=R

Roni,

- jana

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 11 Sep 2017 11:15:20 -0700

Hi Robert,

there is another thread on unreliable streams where this is covered, but
I’ll try to summarise:

With uni-directional streams (otherwise overhead is unfeasible) you can
manage streams as paired bi-directional streams, or as uni-directional
streams where you continuously add more frames over time, or (which is
relevant here) you can create a virtual stream where each new frame gets a
new stream id but the application stores a virtual stream id at the start
of each frame. There is no virtual stream id concept in QUIC, but
applications can meaningfully implement it and it makes just as much sense
as adding message headers within a single stream.

With a virtual stream id you can afford to loose some frames while
maintaining ordering because each stream id will be larger than the
previous. This could be used for video frames where you don’t care about
older frames that do not get through in time and you don’t want
retransmission to block more recent frames. Video frames are large, but you
can have other frame types that are small, such as stock tickers.

In the above, you can churn a lot of frames. Another scenario is a lot of
independent but small messages, such as neutron beam detection.

A packet has about 1200 bytes payload and may have 100 stream ids per
packet.
You can move about 1000 packets per second and thus 100K messages per
second. With a bit more bandwidth you easily get 1 million messages per
second and thus 1 million frames per second.

If you don’t have partial reliability as per virtual stream id example
above, you might cause a massive traffic jam on a simple packet
retransmission, so you do not want to send all the messages in the same
stream - then you might just as well have used TCP instead.

With 1 million messages per second you run out of IDs in roughly 4000
seconds or little more than 1 hour because the stream ID space is unsigned
32 bit.

If you only pass 65000 messages per second you have 2^32 / 65000 / 3600
hours = 18 hours which is still short enough to having to care about
implementing reconnection logic.

Because that many messages need to be short, it makes sense to use a stream
id format that is compressed but which can express significantly more than
32 bits.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 11 September 2017 at 19.46.19, Roberto Peon ([email-masked]) wrote:

Mikkel—

Not sure what you mean, since it’d take ~11 million years at 65000
packets/second to exhaust the ID space. If you used up 4 IDs per stream, it
is still > 2 million years.

Am I misunderstanding?

-=R

*From: *Mikkel Fahnøe Jørgensen <[email-masked]>
*Date: *Thursday, September 7, 2017 at 10:47 PM
*To: *Roberto Peon <[email-masked]>, Jana Iyengar <[email-masked]>, Roni Even
<[email-masked]>
*Cc: *joel jaeggli <[email-masked]>, QUIC WG <[email-masked]>, Martin
Thomson <[email-masked]>
*Subject: *Re: obsrvtaion on packet numbers

A connection a day is not sufficient, but the real problem is the stream ID
when sending multiple new streams per package - which happens in partially
reliable transmissions with uni-directional streams - you can exhaust ID
space within hours.

Mikkel

------------------------------

*From:* QUIC <[email-masked]> on behalf of Roberto Peon <[email-masked]
*Sent:* Thursday, September 7, 2017 11:45:24 PM
*To:* Jana Iyengar; Roni Even
*Cc:* joel jaeggli; QUIC WG; Martin Thomson
*Subject:* Re: obsrvtaion on packet numbers

Concretely: 2^63 represents 11,690,840.9 years of packets at 25kpps, or 292
years at 1 Gpps.

Even if it was only 2^32, though, a new connection per day would be
reasonable—many sites won’t want to maintain connection state for
substantially more than a day (people update binaries/change config, need
to shuffle load around to deal with load asymmetries, etc.).

-=R

*From: *QUIC <[email-masked]> on behalf of Jana Iyengar <
[email-masked]>
*Date: *Thursday, September 7, 2017 at 2:36 PM
*To: *Roni Even <[email-masked]>
*Cc: *joel jaeggli <[email-masked]>, QUIC WG <[email-masked]>, Martin
Thomson <[email-masked]>
*Subject: *Re: obsrvtaion on packet numbers

Roni,

> 2^32 isn't half of 2^64, it's half of 2^33. 25kpps is 2 billion odd
packets a day
> whereas 2^64 is 1.84 * 10^19.
[Roni Even]I was basing it on the text assuming that packet number does not
start at 0  " The initial value for packet number MUST be selected from an
uniform
   random distribution between 0 and 2^31-1 "

- jana

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 13 Sep 2017 15:42:30 +0200

Hi Martin, hi all,

two quick comments below.

> peers know it.  No real need for any signaling.

I'm uncertain if we need to say more but I don't think having the discussion 
how likely or unlikely this case is over and over again helps (also people 
seems to have fun going through the numbers). However, even if the case is 
unlikely, it needs to be fully specified.

> Stream state would have to be lost.

But it could. If there is a way to say in a new connection that you would 
like to resume state if the other end still has the full state (including 
stream and retransmission state) for this other connection, than that's 
probably also all you need for making multipath work :-)

Mirja

---

From: Dmitri Tikhonov <[email-masked]>
Date: Wed, 13 Sep 2017 09:56:15 -0400

> unlikely, it needs to be fully specified.

I agree.  Furthermore, I think that the rationale behind some of
the decisions needs to be explained, preferably in the draft.

I posed these questions in a previous thread [1], but since they
went unanswered, I feel justified restating them here:

    1. Why are there packet number and byte offset limits?  In
       other words, why are there limits on the duration of a
       connection?

    2. How are particular limits chosen?

  - Dmitri.

1. https://www.ietf.org/mail-archive/web/quic/current/msg02144.html

---

From: Dmitri Tikhonov <[email-masked]>
Date: Wed, 13 Sep 2017 16:58:17 -0400

>     2. How are particular limits chosen?

In 2012, Jim Roskind wrote in "QUIC: Design Document and Specification
Rationale" [1]:

  " Header: Packet Sequence Number
  " 
  " In addition to sequencing packets, watching for duplicates,
  " and communicating what packets are missing, this number is
  " a critical part of the encryption.  This number forms the
  " basis of the IV used to decrypt each packet.  As a result,
  " it must conceptually be large, as it must never repeat during
  " the lifetime of the connection.  That need forced this sequence
  " number to be conceptually large, around 2^64 (more packets than
  " any connection would dream of sending), but we usually don’t
  " need to provide all 8 bytes in each packet!

And

  " even though we expect to support very large file transfers, it
  " will be rare that a stream will convey data that uses many of
  " the current 64 bits of maximum offset!

It seems that the author of QUIC took the [pragmatic] position that
these limits will never be reached.  On the other hand, his goals
at the time were also more modest:

  " The two primary support motivations for this transport are to
  " better support SPDY, and to further coalesce traffic. SPDY
  " currently runs over TCP, but has encountered a few performance
  " limitations.
                  (Ibid.)

This document seems to be the origin of the current limits in the
IETF QUIC Draft.  Now that QUIC is not just a transport for SPDY
but a generic transport protocol, perhaps it is time for a
reassessment.

  - Dmitri.

1. https://docs.google.com/document/d/1RNHkx_VvKWyWg6Lr8SZ-saqsQx7rFV-ev2jRFUoVD34/preview