Subject: draft-lubashev-quic-partial-reliability

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 19 Dec 2017 03:39:58 +0000

I've just posted a draft on adding partial reliability to QUIC streams.

Datatracker link: https://tools.ietf.org/html/draft-lubashev-quic-partial-reliability
Github link: https://github.com/igorlord/draft-lubashev-quic-partial-reliability

This draft introduces a generic way to add partial reliability to QUIC streams using a single new frame: MIN_STREAM_FRAME (and keeping one new per-stream variable: Min Stream Offset).

The high-level idea is that the sender keeps track of messages within a stream and can "expire" old messages whenever it wants.  The "expiration" is a signal to the transport to not retransmit expired data (and a signal to the other endpoint that this data will not be retransmitted).  All details are in the draft.

I am very interested in feedback on this proposal.

  *   Igor

---

From: Phil Sorber <[email-masked]>
Date: Tue, 19 Dec 2017 03:55:52 +0000

This is interesting and addresses some things that I have had some
conversations with others about as well. I'll try to read and give feedback
in the next couple of weeks.

Thanks.

On Mon, Dec 18, 2017 at 8:40 PM Lubashev, Igor <[email-masked]> wrote:

> I’ve just posted a draft on adding partial reliability to QUIC streams.
>
>
>
> Datatracker link:
> https://tools.ietf.org/html/draft-lubashev-quic-partial-reliability
>
> Github link:
> https://github.com/igorlord/draft-lubashev-quic-partial-reliability
>
>
>
> This draft introduces a generic way to add partial reliability to QUIC
> streams using a single new frame: MIN_STREAM_FRAME (and keeping one new
> per-stream variable: Min Stream Offset).
>
>
>
> The high-level idea is that the sender keeps track of messages within a
> stream and can “expire” old messages whenever it wants.  The “expiration”
> is a signal to the transport to not retransmit expired data (and a signal
> to the other endpoint that this data will not be retransmitted).  All
> details are in the draft.
>
>
>
>    - Igor

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 19 Dec 2017 09:38:02 +0000

Hi Igor,

Thanks for wrting this up, it is an interesting proposal. It seems flexible enough to apply to a few different use cases, although I do wonder how expressive or accommodating APIs could be.

Some basic questions:

Is this proposal the "alternative single-stream method"? I think it is but may be assuming incorrectly.

With this proposal, partial reliability seems to unilaterally apply to all streams, based on the potential reception of the MIN_STREAM_DATA frame at any point during the connection lifetime. This seems a different approach than others where the reliability type is indicated on stream creation. Could this feature be negotiated for the connection (or path ;) )? I can see a case for simplistic client in constrained devices that may not want the overhead of supporting this feature.

Similar to above, is this capability symmetric? Could it be restricted to one direction of traffic?

What stream are MIN_STREAM_DATA frames sent on? Could the stream ID field be omitted if the frame is only sent on the stream it affects?
________________________________________
From: QUIC [[email-masked]] on behalf of Lubashev, Igor [[email-masked]]
Sent: 19 December 2017 03:39
To: QUIC WG
Subject: draft-lubashev-quic-partial-reliability

The high-level idea is that the sender keeps track of messages within a stream and can “expire” old messages whenever it wants.  The “expiration” is a signal to the transport to not retransmit expired data (and a signal to the other endpoint that this data will not be retransmitted).  All details are in the draft.

  *   Igor

-----------------------------
http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and
may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in
error, please delete it from your system.
Do not use, copy or disclose the
information in any way nor act in reliance on it and notify the sender
immediately.
Please note that the BBC monitors e-mails
sent or received.
Further communication will signify your consent to
this.
-----------------------------

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 19 Dec 2017 13:13:04 +0000

Thanks Lucas!

Yes, I have seen other proposals, and this is my alternative.

The idea is that all streams, other than stream 0, can be partially reliable, in both directions, without a need to negotiate (other than version negotiation, in case the initial QUIC version does not support this).

The bookkeeping is simple enough and the runtime resources required is just one uint64_t per steam, so I do not see a problem for constrained devices.

Ultimately, partial reliability is a feature that an application either needs or does not. If an application needs it, it most likely requires it. In any case, an application can do its own negotiation if it desires to do so. For example, it can use stream 1 for that.

I did not fully understand the last question. Are you asking whether it is possible for the connection to negotiate just a single stream (in each direction) that would be partially reliable and, therefore, not send a stream id with MIN_STREAM_DATA? I could actually think of a version of MIN_STREAM_DATA that has an implied stream id -- the stream id of a prior/following STREAM frame in the same packet. That would be an optimization that I am happy to add, if there is enough support for it.

- Igor

-----Original Message-----
From: Lucas Pardue [[email-masked]]
Received: Tuesday, 19 Dec 2017, 4:38AM
To: Lubashev, Igor [[email-masked]]; QUIC WG [[email-masked]]
Subject: RE: draft-lubashev-quic-partial-reliability

Hi Igor,

Some basic questions:

  *   Igor

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 19 Dec 2017 15:46:56 +0000

Igor wrote:

The bookkeeping is simple enough and the runtime resources required is just one uint64_t per steam, so I do not see a problem for constrained devices.
Ultimately, partial reliability is a feature that an application either needs or does not. If an application needs it, it most likely requires it. In any case, an application can do its own negotiation if it desires to do so. For example, it can use stream 1 for that.
Now I understand this is connection-wide, I generally agree. However, for something like conventional HTTP/QUIC, the reliable guarantee assurances are required by the mapping. If partial reliability is default enabled transport feature, applications like HTTP/QUIC must restrict or disable it - I'm not sure if the editors/drafts have broached that subject yet (I've certainly had some feedback on my draft about such restrictions).

Apologies, I hadn't drunk any coffee when first replying and got myself confused. All QUIC frames of this nature should normally include a Stream ID, which identifies the stream it is sent on. I think your suggestion relies on serial ordering/processing (even inside a packet), which might be a hard sell.

Separately, in terms of asymmetry, I was thinking that you might always want (as a policy) clients to transmit with full reliability but provide data to them with partial reliability. I think this is again an application mapping concern.

Regards,
Lucas

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 19 Dec 2017 09:15:19 -0800

I think this proposal largely makes good sense.

Be default no MIN_STREAM_DATA frame is sent which is then the current
stream behaviour.
Any application protocol can define its own rules about which streams can
behave partially reliable.

A counter argument is that QUIC explicitly distinguishes between uni- and
bi-directional streams, so why not between partial- and fully reliable
streams? The answer might be that uni- streams can close state early, while
there is no similar concern for partial reliability.

Another concern is time-based partial reliability. If data is old, don’t
retransmit regardless. But an application protocol can handle this.

My main concern is that sending MIN_FRAME_DATA can generate A LOT of
traffic with lots of tiny messages where you only care about, say, the last
three 1 byte messages sent (hoping at least one gets through). The
MIN_FRAME_DATA could be much larger than the messages forcing less frequent
traffic. Of course, locally you can have you own rules, and just
periodically update MIN_FRAME_DATA, but this is a bit odd.

A variation of the concept could be sending MIN_FRAME_DATA as a delta to
most recent: never expect more than X bytes older than most recently seen,
but that also has it issues when messages are variable length.

As to client vs server asymmetri: It strongly suggest not to make a
distinction here because there are many symmetric peer to peer use cases
and also opposite cases: server to client with partially reliable video,
client to server with partially reliable game state, server to server with
partially reliable gossip protocol to statistically advance a common state
(e.g. "I have processed so many bytes now, how about you?”).

Kind Regards,
Mikkel Fahnøe Jørgensen

On 19 December 2017 at 16.47.08, Lucas Pardue ([email-masked])
wrote:

Igor wrote:

The bookkeeping is simple enough and the runtime resources required is just
one uint64_t per steam, so I do not see a problem for constrained devices.

Ultimately, partial reliability is a feature that an application either
needs or does not. If an application needs it, it most likely requires it.
In any case, an application can do its own negotiation if it desires to do
so. For example, it can use stream 1 for that.

Now I understand this is connection-wide, I generally agree. However, for
something like conventional HTTP/QUIC, the reliable guarantee assurances
are required by the mapping. If partial reliability is default enabled
transport feature, applications like HTTP/QUIC must restrict or disable it
– I’m not sure if the editors/drafts have broached that subject yet (I’ve
certainly had some feedback on my draft about such restrictions).

I did not fully understand the last question. Are you asking whether it is
possible for the connection to negotiate just a single stream (in each
direction) that would be partially reliable and, therefore, not send a
stream id with MIN_STREAM_DATA? I could actually think of a version of
MIN_STREAM_DATA that has an implied stream id -- the stream id of a
prior/following STREAM frame in the same packet. That would be an
optimization that I am happy to add, if there is enough support for it.

Apologies, I hadn’t drunk any coffee when first replying and got myself
confused. All QUIC frames of this nature should normally include a Stream
ID, which identifies the stream it is sent on. I think your suggestion
relies on serial ordering/processing (even inside a packet), which might be
a hard sell.

Separately, in terms of asymmetry, I was thinking that you might always
want (as a policy) clients to transmit with full reliability but provide
data to them with partial reliability. I think this is again an application
mapping concern.

Regards,

Lucas

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 19 Dec 2017 17:22:12 +0000

Mikkel wrote:

> As to client vs server asymmetri: It strongly suggest not to make a distinction here because there are many symmetric peer to peer use cases and also opposite cases: server to client with partially reliable video, client to server with partially reliable game state, server to server with partially reliable gossip protocol to statistically advance a common state (e.g. "I have processed so many bytes now, how about you?”).

Yes, early specialisation is a route to disappointment. There are other discussions around unnecessary asymmetry at this level.

Regards
Lucas

----------------------------

http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.

---------------------

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 19 Dec 2017 09:40:46 -0800

Another thought:

The partial reliability problem is really like a reverse ACK problem.
You can send a frame similar to ACK, be referring to you own streams rather
than the peers packets. For any stream where the conceptual MIN_STREAM_DATA
is 0, not transmission is made. Otherwise all unclosed streams have an
entry in a frame carrying many streams at once. This might be delivered
unreliable similar to ACK, since old reverse ACK are not that interesting.

You loose some details about sent vs unsent, but peer can deduce that
because a stream does not have gaps. (A stream could pretend to have lost
some data by never sending it if it gets behind, it it is the same thing).

This would use much less space achieving largely the same concept as with
MIN_STREAM_DATA frames for one stream at time.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 19 December 2017 at 18.22.16, Lucas Pardue ([email-masked])
wrote:

Yes, early specialisation is a route to disappointment. There are other
discussions around unnecessary asymmetry at this level.

Regards

Lucas

http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and may contain personal
views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance
on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.

---------------------

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 19 Dec 2017 19:02:34 +0000

*   Another concern is time-based partial reliability. If data is old, don’t retransmit regardless. But an application protocol can handle this

Yes, this is an application concern, since only the application can know what “old” means.  Also, since MIN_FRAME_DATA  frame is emitted ONLY if some expired stream data is lost (not ACKed), you would not expect to see many such frames on the wire, even if messages keep expiring all the time.

  *   My main concern is that sending MIN_FRAME_DATA [sic: MIN_STREAM_DATA] can generate A LOT of traffic with lots of tiny messages where you only care about, say, the last three 1 byte messages sent (hoping at least one gets through). The MIN_FRAME_DATA could be much larger than the messages

This is the most expensive case for MIN_STREAM_DATA – many partially-reliable streams, each carrying 1-byte messages, with each new message expiring all earlier messages, over a high-loss connection.  In this case, the cost of MIN_STREAM_DATA could approach 50% of the cost of the other frames (9 bytes for STREAM + some ACKs vs 5 bytes for MIN_STREAM_DATA frames).

You remove any of the conditions above, and MIN_STREAM_DATA cost drop off rapidly:

  *   Few partially-reliable streams: either multiple messages get consolidated into a single STREAM frame (requiring a single M_S_D frame to expire) or the overhead of the QUIC packet itself (MAC+IP+UDP+QUIC headers) starts to dominate dramatically.
  *   Multi-byte messages: STREAM frames are larger, so M_S_D frame’s proportion is smaller
  *   New messages expiring NOT all earlier messages: more chance for the network to retransmit lost messages, reducing how often M_S_D is sent.
  *   Low-loss connection: few packets are lost, so few M_S_D frames need sending.

“Partially reliable” is not “unreliable”.  The transport still provides message atomicity and ordering guarantees, in addition to retransmission of “still current” data.

If you want completely unreliable (like UDP), you may want something slightly different.  For example, we may decide that is the first STREAM has OFF=1 and Offset=0, this introduces a “completely unreliable” stream.  No retransmissions are expected at all.  However, you still want to do something for flow control (you need to periodically synchronize your stream and connection flow control).  Maybe, if your unreliable stream went idle with the last transmission unacknowledged, you could send a STREAM frame with LEN=1, Length=0, and OFF=1, Offset=current_stream_offset.

If you like this, I can write this in a draft.

  *   Igor

From: Mikkel Fahnøe Jørgensen [mailto:[email-masked]]
Sent: Tuesday, December 19, 2017 12:15 PM
To: Lucas Pardue <[email-masked]>; Lubashev, Igor <[email-masked]>; [email-masked]
Subject: RE: draft-lubashev-quic-partial-reliability

Be default no MIN_STREAM_DATA frame is sent which is then the current stream behaviour.
Any application protocol can define its own rules about which streams can behave partially reliable.

A counter argument is that QUIC explicitly distinguishes between uni- and bi-directional streams, so why not between partial- and fully reliable streams? The answer might be that uni- streams can close state early, while there is no similar concern for partial reliability.

Another concern is time-based partial reliability. If data is old, don’t retransmit regardless. But an application protocol can handle this.

My main concern is that sending MIN_FRAME_DATA can generate A LOT of traffic with lots of tiny messages where you only care about, say, the last three 1 byte messages sent (hoping at least one gets through). The MIN_FRAME_DATA could be much larger than the messages forcing less frequent traffic. Of course, locally you can have you own rules, and just periodically update MIN_FRAME_DATA, but this is a bit odd.

A variation of the concept could be sending MIN_FRAME_DATA as a delta to most recent: never expect more than X bytes older than most recently seen, but that also has it issues when messages are variable length.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 19 December 2017 at 16.47.08, Lucas Pardue ([email-masked]<mailto:[email-masked]>) wrote:
Igor wrote:

The bookkeeping is simple enough and the runtime resources required is just one uint64_t per steam, so I do not see a problem for constrained devices.
Ultimately, partial reliability is a feature that an application either needs or does not. If an application needs it, it most likely requires it. In any case, an application can do its own negotiation if it desires to do so. For example, it can use stream 1 for that.
Now I understand this is connection-wide, I generally agree. However, for something like conventional HTTP/QUIC, the reliable guarantee assurances are required by the mapping. If partial reliability is default enabled transport feature, applications like HTTP/QUIC must restrict or disable it – I’m not sure if the editors/drafts have broached that subject yet (I’ve certainly had some feedback on my draft about such restrictions).

Apologies, I hadn’t drunk any coffee when first replying and got myself confused. All QUIC frames of this nature should normally include a Stream ID, which identifies the stream it is sent on. I think your suggestion relies on serial ordering/processing (even inside a packet), which might be a hard sell.

Regards,
Lucas

---

From: Roberto Peon <[email-masked]>
Date: Tue, 19 Dec 2017 19:23:25 +0000

There is the API, implementation, and protocol.
API:
I’d be nice to have an API which would allow applications to express:

1)       Drop-dead-time: The time beyond which the data is not useful to the recipient

2)       Expansion-factor: The maximum ‘expansion factor’ allowed for the data (do I send the bytes 1.2 (~FEC) times, 2 times (duplicate sends)?)

3)       Priority: duh

Implementation:
If the implementation took all of the above into account, it might decide to never attempt to send some data, as, given the BW estimate flowing back from the congestion-avoidance algorithm, it would not be received within the drop-dead-time. This would better use the available bandwidth.

Protocol:
Protocol-wise, we could  delegate the transmission to QUIC or some other higher-level layer.
However, if we delegate away from QUIC, the proxies will be quite unlikely to participate (other than processing stream cancels). This would be sad given the wide deployment of proxies (i.e. reverse proxies) in CDNs, etc.

++ on partially reliable != unreliable.

TCP, after all, is “just” a partially reliable protocol with long enough timeouts that we deign to call it reliable ☺

-=R

From: QUIC <[email-masked]> on behalf of "Lubashev, Igor" <[email-masked]>
Date: Tuesday, December 19, 2017 at 11:03 AM
To: Mikkel Fahnøe Jørgensen <[email-masked]>, Lucas Pardue <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: RE: draft-lubashev-quic-partial-reliability

  *   Igor

Kind Regards,
Mikkel Fahnøe Jørgensen

Regards,
Lucas

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 19 Dec 2017 11:25:57 -0800

Yes, this is an application concern, since only the application can know
what “old” means.  Also, since MIN_FRAME_DATA  frame is emitted ONLY if
some expired stream data is lost (not ACKed), you would not expect to see
many such frames on the wire, even if messages keep expiring all the time.

I could be that the approach is not aggressive enough. If the connection
has low loss-rate, it might start retransmitting a lot of data that the
application already knows is no longer relevant, before the NACK (ACK gap)
arrives cost excessive bandwidth.

In a recent TCP vs QUIC paper posted here, that was an unanswered question
about why QUIC took double the bandwidth of TCP when both were competing
using CUBIC. This might be related to QUIC being able to make faster
retransmission decisions on more packets which do not count against its
flow control window (right?).

---

From: Christian Huitema <[email-masked]>
Date: Tue, 19 Dec 2017 11:31:01 -0800

On 12/18/2017 7:39 PM, Lubashev, Igor wrote:

> The high-level idea is that the sender keeps track of messages within
> a stream and can “expire” old messages whenever it wants.  The
> “expiration” is a signal to the transport to not retransmit expired
> data (and a signal to the other endpoint that this data will not be
> retransmitted).  All details are in the draft.

I read your draft, and it does indeed provide "partial reliability." My
problem is that for RTP style applications, you don't want just partial
reliability. You also want application frame delimitation, such as
delivering isolated VOIP frames, doing  loss compensation, or
implementing application level FEC for large video frames. The streams
in QUIC don't provide that. They implement a "byte stream" abstraction,
not a "packet stream". So I really wonder whether FTP style applications
are better served by adding partial reliability to the current concept
of streams, or whether we should just add a different service, maybe
creating "RTP_STREAM" frames.

-- Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 19 Dec 2017 11:40:20 -0800

I did come up with a proposal with segmented stream data for partial
reliability.
It’s not too coherent as it evolved, but here it is:

https://github.com/quicwg/base-drafts/issues/433

Applications could also do their internal framing and forward the min for
each internal frame and let transport decide if retransmission is
worthwhile.

If the application needs to make business decisions it might be out of the
loop for too long. QUIC might be scheduled a lot more often than
applications talking to QUIC.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 19 December 2017 at 20.31.17, Christian Huitema ([email-masked])
wrote:

-- Christian Huitema

---

From: Ian Swett <[email-masked]>
Date: Tue, 19 Dec 2017 15:19:47 -0500

Thanks for writing this up Igor, it's an interesting idea, and I like how
it allows any stream besides 0 to be partially reliable.  I'm a bit
concerned(or possibly confused) about the effort to change how flow control
works, see comments below.  I believe this proposal implicitly assumes that
it's easier and more efficient to use streams for ordering and make the
application deal with interpreting out of order data than the current model
of making applications reorder a large number of small streams?  That may
or may not be true in a given application, but I can imagine cases when it
is true.

At the beginning you say:
"The key to partial reliablity is notifying the transport and the peer
 when data previously enqueued for transmission no longer needs to be
 transmitted."

I believe there is another challenge, which is indicating to the receiver
when they should stop expecting to receive data.  Conveniently, this is
exactly what MIN_STREAM_DATA does, but I think it's worth noting in the
introduction.

In terms of the downsides of using streams as is, you say:
"Hence, a message-per-stream
 approach requires each message to contain an extra header portion to
 associate the message with a logical application stream.  In case of
 short messages, this approach introduces a significant overhead due
 to STREAM frames and message headers.  It also places the burden on
 the application to reorder data arriving on multiple QUIC streams."

In practice, I believe a lot of applications using partial reliability
today have such headers, since they're commonly designed with UDP in mind.

In section 4, you discuss the separation of Unsent bytes from Sent data,
but I'm still confused on the motivation there, since separating them seems
to make flow control more complex.  Currently the connection flow control
used is the sum of the largest byte offsets of all streams on a connection,
and now that would be dramatically different, correct?  Can you add more
background on why these are separated and how they are applied to various
limits?  If I read correctly, MIN_STREAM_DATA frame is effectively able to
increase a peer's flow control limits by declaring previously used credits
up for grabs?

This proposal requires out of order delivery be supported by QUIC streams,
which is worth explicitly noting, maybe in section 5, since it's currently
not required.  Also, the receiver needs to be able to determine what the
message boundaries are if there are gaps in the stream.  How easy that is
would depend upon what is being sent on the stream.  It might deserve some
discussion about how one might do this and what restrictions should be
imposed on how the sender bundles data, if that ends up being necessary?
ie: Can a STREAM frame contain the end of one message and the beginning of
another in a single frame?

On Tue, Dec 19, 2017 at 10:46 AM, Lucas Pardue <[email-masked]>
wrote:

> Igor wrote:
>
>
>
> The bookkeeping is simple enough and the runtime resources required is
> just one uint64_t per steam, so I do not see a problem for constrained
> devices.
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 19 Dec 2017 20:31:24 +0000

> From: Christian Huitema [mailto:[email-masked]]
> Sent: den 19 december 2017 20:31
> To: Lubashev, Igor <[email-masked]>; QUIC WG <[email-masked]>
> Subject: Re: draft-lubashev-quic-partial-reliability
> 
> > a stream and can "expire" old messages whenever it wants.  The
> > "expiration" is a signal to the transport to not retransmit expired
> I read your draft, and it does indeed provide "partial reliability." My problem
> is that for RTP style applications, you don't want just partial reliability. You
> also want application frame delimitation, such as delivering isolated VOIP
> frames, doing  loss compensation, or implementing application level FEC for
> large video frames. The streams in QUIC don't provide that. They implement
> a "byte stream" abstraction, not a "packet stream". So I really wonder
> whether FTP style applications are better served by adding partial reliability
> to the current concept of streams, or whether we should just add a different
> service, maybe creating "RTP_STREAM" frames.

Thanks Christian, I now understand what I did not understand with the partial-reliability draft. The reason is that I look at this from an RTP point of view. To make QUIC {partial,un}reliable streams useful for real time media then I agree that one need something like RTP_STREAM. With that said I am not sure that RTP is the best common denominator around, perhaps MEDIA_STREAM is a better alternative?, where one use the abstraction media frames (video frames, audio frames...) instead ?, that would allow support for other not RTP type packetization like for instance MPEG-TS .

---

From: Ian Swett <[email-masked]>
Date: Tue, 19 Dec 2017 15:48:25 -0500

For message style payloads commonly carried over UDP, like RTP, I added an
issue to create a MESSAGE frame for QUIC.  Does that fit what you have in
mind?
https://github.com/quicwg/base-drafts/issues/814

On Tue, Dec 19, 2017 at 3:31 PM, Ingemar Johansson S <
[email-masked]> wrote:

> problem
> Thanks Christian, I now understand what I did not understand with the
> partial-reliability draft. The reason is that I look at this from an RTP
> point of view. To make QUIC {partial,un}reliable streams useful for real
> time media then I agree that one need something like RTP_STREAM. With that
> said I am not sure that RTP is the best common denominator around, perhaps
> MEDIA_STREAM is a better alternative?, where one use the abstraction media
> frames (video frames, audio frames...) instead ?, that would allow support
> for other not RTP type packetization like for instance MPEG-TS .
>
> >
> > -- Christian Huitema
> >
> >
>
>

---

From: Roberto Peon <[email-masked]>
Date: Tue, 19 Dec 2017 20:56:20 +0000

It would be best to have a grouping of individually named atoms (whether streams or messages, doesn’t matter) so that we don’t jettison the ability to cache.
-=R

From: QUIC <[email-masked]> on behalf of Ian Swett <[email-masked]>
Date: Tuesday, December 19, 2017 at 12:49 PM
To: Ingemar Johansson S <[email-masked]>
Cc: "Lubashev, Igor" <[email-masked]>, Christian Huitema <[email-masked]>, QUIC WG <[email-masked]>
Subject: Re: draft-lubashev-quic-partial-reliability

For message style payloads commonly carried over UDP, like RTP, I added an issue to create a MESSAGE frame for QUIC.  Does that fit what you have in mind?
https://github.com/quicwg/base-drafts/issues/814

On Tue, Dec 19, 2017 at 3:31 PM, Ingemar Johansson S <[email-masked]<mailto:[email-masked]>> wrote:
> From: Christian Huitema [mailto:[email-masked]<mailto:[email-masked]>]
> To: Lubashev, Igor <[email-masked]<mailto:[email-masked]>>; QUIC WG <[email-masked]<mailto:[email-masked]>>

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 19 Dec 2017 21:45:42 +0000

> My problem is that for RTP style applications, you don't want just partial reliability. You also want application frame delimitation [...]

This is what this draft is trying to give you.

>From Section 5 "Sender Interface and Behavior"

"A typical sender would call an API function providing this functionality
   whenever any data previously enqueued for transmission expires, per
   application semantics.  The sender would keep track of the message
   boundaries of such data."

Hence, since the sender knows message boundaries (you call them "application frame delimitation") within the data stream, the sender can choose the offset to expire data.  The one thing this draft will NOT let you do is expire selective ranges of data within a data stream -- it only supports expiring "all data up to offset X".

- Igor

-----Original Message-----
From: Christian Huitema [mailto:[email-masked]] 
Sent: Tuesday, December 19, 2017 2:31 PM
To: Lubashev, Igor <[email-masked]>; QUIC WG <[email-masked]>
Subject: Re: draft-lubashev-quic-partial-reliability

> retransmitted).  All details are in the draft.

I read your draft, and it does indeed provide "partial reliability." My problem is that for RTP style applications, you don't want just partial reliability. You also want application frame delimitation, such as delivering isolated VOIP frames, doing  loss compensation, or implementing application level FEC for large video frames. The streams in QUIC don't provide that. They implement a "byte stream" abstraction, not a "packet stream". So I really wonder whether FTP style applications are better served by adding partial reliability to the current concept of streams, or whether we should just add a different service, maybe creating "RTP_STREAM" frames.

-- Christian Huitema

---

From: Colin Perkins <[email-masked]>
Date: Tue, 19 Dec 2017 21:47:58 +0000

> On 19 Dec 2017, at 20:31, Ingemar Johansson S <[email-masked]> wrote:
> 

RTP has many other features that will confuse if we’re not careful… Framed messages, rather than unframed byte streams, might be the key abstraction? Once we have that, partial reliability, dependancies, deadlines, and unordered delivery become natural to add.

-- 
Colin Perkins
https://csperkins.org/

---

From: Christian Huitema <[email-masked]>
Date: Tue, 19 Dec 2017 13:58:52 -0800

On 12/19/2017 12:48 PM, Ian Swett wrote:

> For message style payloads commonly carried over UDP, like RTP, I
> added an issue to create a MESSAGE frame for QUIC.  Does that fit what
> you have in mind?

Not quite. RTP has the concept of streams, with applications negotiating
what stream carries what kind of media payload. They also have a concept
of "sequence of messages", so they can determine which packets in a
stream are missing and apply whatever corrective process is appropriate.
And then there time stamps, and source identifiers...

Applications could indeed use a MESSAGE frame and put their own RTP
style headers inside it. But that feels like punting to the app. I am
kind of hoping that we can do a bit better.

-- Christian Huitema

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 20 Dec 2017 01:21:31 +0000

*   This proposal requires out of order delivery be supported by QUIC streams

It does not.  All delivery, whether of data or gaps-in-data, is strictly in-order.  I can see a receiver API look something like:

class Stream {
      …
size_t receive(void *buf, size_t buf_size) throw QuickException;
size_t receive(size_t &expired_bytes, void *buf, size_t buf_size) throw QuickException;
…
};

The version of receive with expired_bytes will return how many bytes have expired followed by the data received in buf.  Clients that do not expect missing messages will just call the version of receive() w/o expired_bytes and will get an exception if any data is expired.   (APIs that are event callback based will just declare a new stream event type “StreamExpiredData”.)

  *   Also, the receiver needs to be able to determine what the message boundaries are if there are gaps in the stream.

It will not.  Since sender keeps track of where message boundaries are and controls minimum retransmittable offset, the receiver is guaranteed that any data after a gap corresponds to the start of a new message.
(MAY in Section 6 “Receiver Interface and Behavior” is not a MUST to allow for an optimization of keeping data that has arrived after MIN_STREAM_DATA due to reordering, if the application has not yet been notified about the data gap.)

  *   Can a STREAM frame contain the end of one message and the beginning of another in a single frame?

Yes, it can, since nothing is changed in the behavior of STREAM frames.  The assumption is that the application can figure out itself how long each fully-delivered message is.

  *   In section 4, you discuss the separation of Unsent bytes from Sent data, but I'm still confused on the motivation there, since separating them seems to make flow control more complex.

I guess I failed to be clear here, as Unsent Bytes was designed to NOT complicate flow control accounting (in fact, it has no effect on flow control accounting at all).

Imagine an HD video conferencing app has sent 10MB of data to a stream (maybe in multiple messages).  Now, say, it was only able to transmit 1MB out of those 10MB when a new key frame is received.

If it has to use normal stream and connection flow control credits to advance this stream by the additional 9MB, it may might not be able to do so w/o violating MAX_STREAM_DATA or MAX_DATA.  The sender would need to “skip” those 9MB over several RTTs, if the receiver is only willing to commit, say, 2MB of buffers to this stream/connection (it is not aware that the sender has 9MB of octets to skip).  Moreover, this would be blocking all other streams from making progress by closing down the connection window.

Unsent Bytes was designed to solve this program, as it *does not count toward stream or connection flow control* (so it does not violate them and does not reduce connection window), but it is a hint to open the stream and connection windows to accommodate skipping byte octets (since they do not require any buffer space).  The stream would be able to resume in at most 1 rtt, and it would not block other streams meanwhile.

However, I now see a problem – while opening the stream window by 9MB+ is ok (since any octets at offset lower than Min Stream Offset are to be discarded), there is no requirement that the extra connection window is used for that stream only.  A malicious client can render connection window meaningless by sending MIN_STREAM_DATA with a huge Unsent Bytes and then never sending on that stream again.  Oops!  I will fix this!

  *   I believe there is another challenge, which is indicating to the receiver when they should stop expecting to receive data.  Conveniently, this is exactly what MIN_STREAM_DATA does, but I think it's worth noting in the introduction.

I meant exactly this by “notifying the transport and the peer when data previously enqueued for transmission no longer needs to be transmitted“ (that’s what “and the peer” was for) but said it very awkwardly.  Will fix by focusing only on the peer (interaction between the app and local transport does not concern wire protocol).

  *   Igor

From: Ian Swett [mailto:[email-masked]]
Sent: Tuesday, December 19, 2017 3:20 PM
To: Lucas Pardue <[email-masked]>
Cc: Lubashev, Igor <[email-masked]>; [email-masked]
Subject: Re: draft-lubashev-quic-partial-reliability

Thanks for writing this up Igor, it's an interesting idea, and I like how it allows any stream besides 0 to be partially reliable.  I'm a bit concerned(or possibly confused) about the effort to change how flow control works, see comments below.  I believe this proposal implicitly assumes that it's easier and more efficient to use streams for ordering and make the application deal with interpreting out of order data than the current model of making applications reorder a large number of small streams?  That may or may not be true in a given application, but I can imagine cases when it is true.

I believe there is another challenge, which is indicating to the receiver when they should stop expecting to receive data.  Conveniently, this is exactly what MIN_STREAM_DATA does, but I think it's worth noting in the introduction.

In practice, I believe a lot of applications using partial reliability today have such headers, since they're commonly designed with UDP in mind.

In section 4, you discuss the separation of Unsent bytes from Sent data, but I'm still confused on the motivation there, since separating them seems to make flow control more complex.  Currently the connection flow control used is the sum of the largest byte offsets of all streams on a connection, and now that would be dramatically different, correct?  Can you add more background on why these are separated and how they are applied to various limits?  If I read correctly, MIN_STREAM_DATA frame is effectively able to increase a peer's flow control limits by declaring previously used credits up for grabs?

This proposal requires out of order delivery be supported by QUIC streams, which is worth explicitly noting, maybe in section 5, since it's currently not required.  Also, the receiver needs to be able to determine what the message boundaries are if there are gaps in the stream.  How easy that is would depend upon what is being sent on the stream.  It might deserve some discussion about how one might do this and what restrictions should be imposed on how the sender bundles data, if that ends up being necessary?  ie: Can a STREAM frame contain the end of one message and the beginning of another in a single frame?

On Tue, Dec 19, 2017 at 10:46 AM, Lucas Pardue <[email-masked]<mailto:[email-masked]>> wrote:
Igor wrote:

Regards,
Lucas

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 20 Dec 2017 01:34:11 +0000

The two proposals are similar in a way that both effectively contain a “zero-fill” frame, though MIN_STREAM_DATA proposal is explicit about notifying the receiver about a gap in data and does not just fill with 0x00.

Since QUIC is likely to be a library linked into an application, they are scheduled together.  But in any case, I find it important to allow applications full control over when to expire data, although some QUIC implementations could provide higher-level abstractions for message expiration.

From: Mikkel Fahnøe Jørgensen [mailto:[email-masked]]
Sent: Tuesday, December 19, 2017 2:40 PM
To: Lubashev, Igor <[email-masked]>; Christian Huitema <[email-masked]>; QUIC WG <[email-masked]>
Subject: Re: draft-lubashev-quic-partial-reliability

I did come up with a proposal with segmented stream data for partial reliability.
It’s not too coherent as it evolved, but here it is:

https://github.com/quicwg/base-drafts/issues/433<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_quicwg_base-2Ddrafts_issues_433&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=4OKe8eZRrhTX1wql3X2hD9bN3t056-dLmp5rte7hsrM&s=hJuopoxDzIyFbhIYR3bZdHCSotkqVTsGCSjUcpwOfRg&e=>

Applications could also do their internal framing and forward the min for each internal frame and let transport decide if retransmission is worthwhile.

If the application needs to make business decisions it might be out of the loop for too long. QUIC might be scheduled a lot more often than applications talking to QUIC.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 19 December 2017 at 20.31.17, Christian Huitema ([email-masked]<mailto:[email-masked]>) wrote:

-- Christian Huitema

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 20 Dec 2017 03:03:19 +0000

*   If the connection has low loss-rate, it might start retransmitting a lot of data that the application already knows is no longer relevant […]

Are you worried about a case of message expiration being shorter than an rtt, and hence the application would expire messages at a rate faster than the ACKs can be received?   That would not cause any extra retransmissions, but it would cause unnecessary MIN_STREAM_DATA frames, indeed.

  *   QUIC bytes “exempt” from Flow Control

I think, yes, QUIC packets have many bytes exempt from flow control (STREAM frame headers and all non-STREAM frames).  One-stream-per-message is going to have the same overhead in “exempt bytes” as emitting one MIN_STREAM_DATA per message.

From: Mikkel Fahnøe Jørgensen [mailto:[email-masked]]
Sent: Tuesday, December 19, 2017 2:26 PM
To: Lubashev, Igor <[email-masked]>; Lucas Pardue <[email-masked]>; [email-masked]
Subject: RE: draft-lubashev-quic-partial-reliability

I could be that the approach is not aggressive enough. If the connection has low loss-rate, it might start retransmitting a lot of data that the application already knows is no longer relevant, before the NACK (ACK gap) arrives cost excessive bandwidth.

In a recent TCP vs QUIC paper posted here, that was an unanswered question about why QUIC took double the bandwidth of TCP when both were competing using CUBIC. This might be related to QUIC being able to make faster retransmission decisions on more packets which do not count against its flow control window (right?).

---

From: "Eggert, Lars" <[email-masked]>
Date: Wed, 20 Dec 2017 08:40:15 +0000

Hi,

> I’ve just posted a draft on adding partial reliability to QUIC streams.

with my chair hat on - and without having discussed this with Mark - I would like to remind people that partial reliability is called out in the charter as something that is explicitly *not* currently in scope.

Give that we're further trying to limit QUICv1 by reducing the scope even compared to what is in the charter, I think we should really put this proposal on the back burner for a while.

This is not a comment about the quality of the proposal itself - I haven't read it. It's a reminder that we already had to push our milestones back, and starting to discuss things that are not QUICv1 doesn't seem like the most effective use of our cycles.

Lars