Subject: Partially Reliable Message Stream

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 30 May 2018 16:44:36 +0000

I've just uploaded a new draft for partially reliable QUIC streams.  Note: this feature is likely not in scope for V1, but it can be an extension for V1 and/or a part of V2.

The new version 03 (https://tools.ietf.org/html/draft-lubashev-quic-partial-reliability-03) no longer needs complex flow control changes and removes the need to transmit multiple frames in the same packet.

  *   Igor

P.S.
  There is also a new version 02, which includes a more complex algorithm with more features and different trade-offs.  But I think version 03 is a better match for the needs so far.

---

From: Mike Bishop <[email-masked]>
Date: Wed, 30 May 2018 22:46:51 +0000

I find the one octet gap a bit confusing, but as I think about it, I see why you need it.  If all the stream data arrived successfully (but hadn't been acknowledged yet due to loss of the ACK, delay, etc.) and the EXPIRED_STREAM_DATA gets lost, the receiver can only retroactively realize there was a jump.  Having an octet that is never transmitted ensures the receiver actually sees a gap, which means that an in-order API will not proceed until it has received either the missing octet or an EXPIRED_STREAM_DATA informing it the octet will never arrive.

This simplification (versus -02) comes at a price:  If an API were exposing stream data out-of-order, then in your example, the receiver knows that a message always begins on a ten-byte boundary.  A receiver can no longer find ten-byte boundaries, because the offset on the read side doesn't match the offset on the send side.  I agree with you that this seems like a reasonable trade-off for the simpler flow control.

One conflict I see in the doc:

*         Section 3 says:  Receipt of an EXPIRED_STREAM_DATA does not advance the largest received offset for the stream.

  *   Section 5 says:  A receiver SHOULD discard any stream data received for an offset smaller than the new smallest receive offset, possibly advancing the largest received offset for the stream.

Other minor nits are better done via PR.
From: QUIC <[email-masked]> On Behalf Of Lubashev, Igor
Sent: Wednesday, May 30, 2018 9:45 AM
To: QUIC WG <[email-masked]>
Subject: Partially Reliable Message Stream

  *   Igor

P.S.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 31 May 2018 10:50:59 +1000

Like Mike, I find this gap thing confusing.  And Mike's message
helped, but not a lot.

It took a while to realize that forcing a gap creates a new starting
point for the next message that forces the recipient to recognize and
deal with.  However, the logic for identifying that is just hard and
it's an unnecessary imposition in my view.

Say I send a message with a length of 10 and only two octets make it
onto the wire (as with your example), then it is immediately
abandoned.  Your solution would have the EXPIRED_STREAM_DATA force a
gap after the second octet, so that the next message could start at
offset 3 rather than offset 10.  That saves the remaining 7 octets of
flow control credit.  (Use bigger numbers and perhaps it seems more
motivating).

If the octets that cover the gap were sent, then you have a problem
because now you have sent two versions of the same octets, so you can
only do this if the octets were never sent.  It starts to get closer
and closer to the point where a new stream is just easier.

Worst, I think is that the receiver has to have complicated logic for
identifying that a gap exists and dealing with it.  It has to read up
to what it has, observe that the data has expired, then learn where
the gap ends, then resynchronize based on 1+gap_end for the next
message.

Better to eat the flow control cost in my view.  Or send RST_STREAM
and make another stream.
On Thu, May 31, 2018 at 8:48 AM Mike Bishop <[email-masked]> wrote:
>
> I find the one octet gap a bit confusing, but as I think about it, I see why you need it.  If all the stream data arrived successfully (but hadn’t been acknowledged yet due to loss of the ACK, delay, etc.) and the EXPIRED_STREAM_DATA gets lost, the receiver can only retroactively realize there was a jump.  Having an octet that is never transmitted ensures the receiver actually sees a gap, which means that an in-order API will not proceed until it has received either the missing octet or an EXPIRED_STREAM_DATA informing it the octet will never arrive.
>
>
>
> This simplification (versus -02) comes at a price:  If an API were exposing stream data out-of-order, then in your example, the receiver knows that a message always begins on a ten-byte boundary.  A receiver can no longer find ten-byte boundaries, because the offset on the read side doesn’t match the offset on the send side.  I agree with you that this seems like a reasonable trade-off for the simpler flow control.
>
>
>
> ·         Section 3 says:  Receipt of an EXPIRED_STREAM_DATA does not advance the largest received offset for the stream.
>
> Section 5 says:  A receiver SHOULD discard any stream data received for an offset smaller than the new smallest receive offset, possibly advancing the largest received offset for the stream.
>
>
>
> I’ve just uploaded a new draft for partially reliable QUIC streams.  Note: this feature is likely not in scope for V1, but it can be an extension for V1 and/or a part of V2.
>
>
>
> Igor

---

From: Roberto Peon <[email-masked]>
Date: Thu, 31 May 2018 01:03:14 +0000

I find it confusing as well.  I far prefer keeping the offsets monotonically increasing as it allows for real-world use of a partially reliable stream (where you may not recieve that data but you can at least figure out what you did get and interpret it).

I dont know how one wohud resolve the race even within the sending application -- one thread sends, another thread sends while the first says to forget about stuff. I dont know what offset stuff ends up at in such cases, and so I cannot frame data in a way that is non-synchronous. That defeats some of the main reasons to do partial reliability (i.e. to reduce jitter).

Draft 02 seemed to be going in a better direction w.r.t the usecases i can conceive of. I think rewinding to that and then potentially figuring out how to simplify it would be fun!

I'm excited to have this being discussed, since there is pent up demand to use it already.
-=R

Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: QUIC <[email-masked]> on behalf of Martin Thomson <[email-masked]>
Sent: Wednesday, May 30, 2018 5:50:59 PM
To: Mike Bishop
Cc: QUIC WG; Lubashev, Igor
Subject: Re: Partially Reliable Message Stream

Better to eat the flow control cost in my view.  Or send RST_STREAM
and make another stream.
On Thu, May 31, 2018 at 8:48 AM Mike Bishop <[email-masked]> wrote:
> The new version 03 (https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dlubashev-2Dquic-2Dpartial-2Dreliability-2D03&d=DwIFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=yyRpbe14kTYgY413gjVkOalWE_UC3xYerKpJqS8HiQo&s=XZ295xiZsN11yP8GLcdMs1jf7z0b2_WhiIxMLLorZXo&e=) no longer needs complex flow control changes and removes the need to transmit multiple frames in the same packet.
>
>
>
> Igor
>
>
>
> P.S.
>

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 31 May 2018 01:20:28 +0000

Thanks for the review, Mike!

Yes, you got the purpose of the gap exactly right.

> Conflict in the doc.

Actually, I do not see a conflict, but it is true that the accounting is tricky, and the wording could be a bit better. The receiver maintains two distinct variables:
* smallest receive offset -- the smallest offset it is waiting for
* largest received offset -- just what it says, probably used for nothing else by the receiver but verifying compliance with flow control (important, since "largest received offset" - "smallest receive offset" = "size of in-use receive buffer" for the stream).

The quotation from section 5 says that when a new STREAM frame is received, data at offsets smaller than "smallest receive offset" should be discarded (since the receiver should not care for it anymore -- it is expired), but "largest received offset" should still advance if needed (since data has been received -- compliance with flow control is still required for STREAM frames).

-----Original Message-----
From: Mike Bishop [[email-masked]]
Received: Wednesday, 30 May 2018, 6:47PM
To: Lubashev, Igor [[email-masked]]; QUIC WG [[email-masked]]
Subject: RE: Partially Reliable Message Stream

The new version 03 (https://tools.ietf.org/html/draft-lubashev-quic-partial-reliability-03<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dlubashev-2Dquic-2Dpartial-2Dreliability-2D03&d=DwMFAg&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=_kwXz2JdtqD8uvvd8h8cZ-GQ3_FgDf7dRoorJ-DYVhg&s=t0au9Hcc2L02NXlpLb99Y4e-72lBv2kC2hHZuGifUWg&e=>) no longer needs complex flow control changes and removes the need to transmit multiple frames in the same packet.

  *   Igor

P.S.

---

From: Mike Bishop <[email-masked]>
Date: Thu, 31 May 2018 03:31:40 +0000

Ah, I see - the second is talking about actions taken on receipt of a STREAM frame, not an ESD frame; I missed that.

From: Lubashev, Igor <[email-masked]>
Sent: Wednesday, May 30, 2018 6:20 PM
To: [email-masked]; Mike Bishop <[email-masked]>
Subject: RE: Partially Reliable Message Stream

-----Original Message-----
From: Mike Bishop [[email-masked]]
Received: Wednesday, 30 May 2018, 6:47PM
To: Lubashev, Igor [[email-masked]]; QUIC WG [[email-masked]]
Subject: RE: Partially Reliable Message Stream

Other minor nits are better done via PR.
From: QUIC <[email-masked]<mailto:[email-masked]>> On Behalf Of Lubashev, Igor
Sent: Wednesday, May 30, 2018 9:45 AM
To: QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: Partially Reliable Message Stream

  *   Igor

P.S.

---

From: Mike Bishop <[email-masked]>
Date: Thu, 31 May 2018 03:32:53 +0000

Just to be clear....  You're suggesting multiple threads that would be writing to a single stream in parallel, rather than a single thread "owning" the stream?

From: Roberto Peon <[email-masked]>
Sent: Wednesday, May 30, 2018 6:03 PM
To: Mike Bishop <[email-masked]>; Martin Thomson <[email-masked]>
Cc: QUIC WG <[email-masked]>; Lubashev, Igor <ilubashe=[email-masked]>
Subject: Re: Partially Reliable Message Stream

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Martin Thomson <[email-masked]<mailto:[email-masked]>>

---

From: Roberto Peon <[email-masked]>
Date: Thu, 31 May 2018 15:21:04 +0000

Yes. You may want a single resource for an output which you may be generating in parallel. Even if a single thread 'owns' the stream, you've lost the ability to coordinate the data it recieves by monotonically increasing offset.

Renumbering the offsets breaks the file-like abstraction we've effectively used for http.

-=R

Sent via the Samsung Galaxy S7, an AT&T 4G LTE smartphone

-------- Original message --------
From: Mike Bishop <[email-masked]>
Date: 5/30/18 8:33 PM (GMT-08:00)
To: Roberto Peon <[email-masked]>, Martin Thomson <[email-masked]>
Cc: QUIC WG <[email-masked]>, "Lubashev, Igor" <ilubashe=[email-masked]>
Subject: RE: Partially Reliable Message Stream

Just to be clear….  You’re suggesting multiple threads that would be writing to a single stream in parallel, rather than a single thread “owning” the stream?

Get Outlook for Android<https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_ghei36&d=DwMFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=VOUj_MnwWf_PgdHvcGV1Se1o3k5jC8m8noKDixq8HqA&s=-xJb_JWOdL_DpDM43E06S54LPE-xkpYkSen4El-wkS4&e=>

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 31 May 2018 13:37:26 -0700

I'm not keen to get into the weeds of this discussion at this time (as you
say, it's for v2), but I understand the overall design. A few higher-order
points with this design direction:

1. There's clear demarcation of app messages, but they aren't really made
first class citizens. If you did that, I imagine that you might have a
simpler design. For instance, define a MSG_STREAM frame, which supports
message streams (instead of the current byte streams), and then you can
cancel messages easily without having to do stream offset shenanigans.

2. Your design has exactly one ordering capability, which is limiting.
Generally, apps that can drop messages and read subsequent ones are also
capable of handling messages out of sequence. You'll want to support
partial ordering as well and not constrain messages to be only delivered in
strictly linear order.

3. I'm keen to see what we can do with using streams as messages. We don't
need to punt everything to the app (as your text suggests), but we can
start with a grouping of streams (or messages) and building flow control
mechanisms for these groups. I suspect this might be cleaner, and it gives
you flexibility to consider various orderings of messages. When we're
closer to v2, I'm happy to spin up a draft exploring this direction.

On Thu, May 31, 2018 at 8:21 AM, Roberto Peon <[email-masked]> wrote:

> Yes. You may want a single resource for an output which you may be
> generating in parallel. Even if a single thread 'owns' the stream, you've
> lost the ability to coordinate the data it recieves by monotonically
> increasing offset.
>
> Just to be clear….  You’re suggesting multiple threads that would be
> writing to a single stream in parallel, rather than a single thread
> “owning” the stream?
>
>
>
> *From:* Roberto Peon <[email-masked]>
> *Sent:* Wednesday, May 30, 2018 6:03 PM
> *To:* Mike Bishop <[email-masked]>; Martin Thomson <
> [email-masked]>
> *Cc:* QUIC WG <[email-masked]>; Lubashev, Igor <ilubashe=
> [email-masked]>
> *Subject:* Re: Partially Reliable Message Stream
>
>
>
> I find it confusing as well.  I far prefer keeping the offsets
> monotonically increasing as it allows for real-world use of a partially
> reliable stream (where you may not recieve that data but you can at least
> figure out what you did get and interpret it).
>
> I dont know how one wohud resolve the race even within the sending
> application -- one thread sends, another thread sends while the first says
> to forget about stuff. I dont know what offset stuff ends up at in such
> cases, and so I cannot frame data in a way that is non-synchronous. That
> defeats some of the main reasons to do partial reliability (i.e. to reduce
> jitter).
>
> Draft 02 seemed to be going in a better direction w.r.t the usecases i can
> conceive of. I think rewinding to that and then potentially figuring out
> how to simplify it would be fun!
>
> Get Outlook for Android
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_ghei36&d=DwMFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=VOUj_MnwWf_PgdHvcGV1Se1o3k5jC8m8noKDixq8HqA&s=-xJb_JWOdL_DpDM43E06S54LPE-xkpYkSen4El-wkS4&e=>
>
>
> ------------------------------
>
> *From:* QUIC <[email-masked]> on behalf of Martin Thomson <
> [email-masked]>
> *Sent:* Wednesday, May 30, 2018 5:50:59 PM
> *To:* Mike Bishop
> *Cc:* QUIC WG; Lubashev, Igor
> *Subject:* Re: Partially Reliable Message Stream
>
>
>
> > I find the one octet gap a bit confusing, but as I think about it, I see
> why you need it.  If all the stream data arrived successfully (but hadn’t
> been acknowledged yet due to loss of the ACK, delay, etc.) and the
> EXPIRED_STREAM_DATA gets lost, the receiver can only retroactively realize
> there was a jump.  Having an octet that is never transmitted ensures the
> receiver actually sees a gap, which means that an in-order API will not
> proceed until it has received either the missing octet or an
> EXPIRED_STREAM_DATA informing it the octet will never arrive.
> >
> >
> >
> > This simplification (versus -02) comes at a price:  If an API were
> exposing stream data out-of-order, then in your example, the receiver knows
> that a message always begins on a ten-byte boundary.  A receiver can no
> longer find ten-byte boundaries, because the offset on the read side
> doesn’t match the offset on the send side.  I agree with you that this
> seems like a reasonable trade-off for the simpler flow control.
> >
> >
> >
> > ·         Section 3 says:  Receipt of an EXPIRED_STREAM_DATA does not
> advance the largest received offset for the stream.
> >
> > Section 5 says:  A receiver SHOULD discard any stream data received for
> an offset smaller than the new smallest receive offset, possibly advancing
> the largest received offset for the stream.
> >
> >
> >
> > I’ve just uploaded a new draft for partially reliable QUIC streams.
> Note: this feature is likely not in scope for V1, but it can be an
> extension for V1 and/or a part of V2.
> >
> >
> >
> > The new version 03 (https://urldefense..proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_draft-2Dlubashev-2Dquic-2Dpartial-
> 2Dreliability-2D03&d=DwIFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=
> C0sUo-LFNBaYfyoaCsf6TA&m=yyRpbe14kTYgY413gjVkOalWE_UC3xYerKpJqS8HiQo&s=
> XZ295xiZsN11yP8GLcdMs1jf7z0b2_WhiIxMLLorZXo&e=
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dlubashev-2Dquic-2Dpartial-2Dreliability-2D03&d=DwIFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=yyRpbe14kTYgY413gjVkOalWE_UC3xYerKpJqS8HiQo&s=XZ295xiZsN11yP8GLcdMs1jf7z0b2_WhiIxMLLorZXo&e=>)
> no longer needs complex flow control changes and removes the need to
> transmit multiple frames in the same packet.
> >
> >
> >
> > Igor
> >
> >
> >
> > P.S.
> >
> >   There is also a new version 02, which includes a more complex
> algorithm with more features and different trade-offs.  But I think version
> 03 is a better match for the needs so far.
>

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 31 May 2018 23:22:32 +0000

Thanks for the review, Martin!

Yes, all the shenanigans are to limit the _connection_ flow control wastage to a single octet per stream [*].  If you allow for unbounded wastage, you have to actively deal with this (that's what version 02 of the draft does, but it is necessarily more complex for it).

> If the octets that cover the gap were sent, then you have a problem because now you have sent two versions of the same octets

You would not create a gap, if octets right before the "expiration point" were sent.  Because the application only expires on message boundaries, there is no problem if all octets before the "expiration point" (aka. "start of a new message") are actually received.  It would just mean that the receiver application got the expired message; it would not get confused about where a new message starts.

> [...] identifying a gap [...]

For the receiver, identifying a gap (to notify the application of the gap) is actually easy.  Got EXPIRED_STREAM_DATA that covers some offsets you never received?  You've got a gap to notify the application about!

-----Original Message-----
From: Martin Thomson [mailto:[email-masked]] 
Sent: Wednesday, May 30, 2018 8:51 PM
To: Mike Bishop <[email-masked]>
Cc: Lubashev, Igor <[email-masked]>; QUIC WG <[email-masked]>
Subject: Re: Partially Reliable Message Stream

Like Mike, I find this gap thing confusing.  And Mike's message helped, but not a lot.

It took a while to realize that forcing a gap creates a new starting point for the next message that forces the recipient to recognize and deal with.  However, the logic for identifying that is just hard and it's an unnecessary imposition in my view.

Say I send a message with a length of 10 and only two octets make it onto the wire (as with your example), then it is immediately abandoned.  Your solution would have the EXPIRED_STREAM_DATA force a gap after the second octet, so that the next message could start at offset 3 rather than offset 10.  That saves the remaining 7 octets of flow control credit.  (Use bigger numbers and perhaps it seems more motivating).

If the octets that cover the gap were sent, then you have a problem because now you have sent two versions of the same octets, so you can only do this if the octets were never sent.  It starts to get closer and closer to the point where a new stream is just easier.

Worst, I think is that the receiver has to have complicated logic for identifying that a gap exists and dealing with it.  It has to read up to what it has, observe that the data has expired, then learn where the gap ends, then resynchronize based on 1+gap_end for the next message.

Better to eat the flow control cost in my view.  Or send RST_STREAM and make another stream.

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 31 May 2018 23:37:11 +0000

Version 02 is certainly a more complete solution, but it is more complex.  I am very interested in hearing different people views on this trade-off.

I am curious of the specifics of an application you have in mind for this.  I.e. how keeping offsets the same on the sender and receiver helps with a message stream.  Are offsets themselves contained in the messages?  Does the application communicate offsets outside of the message stream (on a different stream, for example)?

(It will probably work better to explore this next week in person.)

  *   Igor

From: Roberto Peon [mailto:[email-masked]]
Sent: Thursday, May 31, 2018 11:21 AM
To: Mike Bishop <[email-masked]>; Martin Thomson <[email-masked]>
Cc: QUIC WG <[email-masked]>; Lubashev, Igor <[email-masked]>
Subject: Re: Partially Reliable Message Stream

-=R

-------- Original message --------
From: Mike Bishop <[email-masked]<mailto:[email-masked]>>
Date: 5/30/18 8:33 PM (GMT-08:00)
To: Roberto Peon <[email-masked]<mailto:[email-masked]>>, Martin Thomson <[email-masked]<mailto:[email-masked]>>
Cc: QUIC WG <[email-masked]<mailto:[email-masked]>>, "Lubashev, Igor" <ilubashe=[email-masked]<mailto:ilubashe=[email-masked]>>
Subject: RE: Partially Reliable Message Stream

From: Roberto Peon <[email-masked]<mailto:[email-masked]>>
Sent: Wednesday, May 30, 2018 6:03 PM
To: Mike Bishop <[email-masked]<mailto:[email-masked]>>; Martin Thomson <[email-masked]<mailto:[email-masked]>>
Cc: QUIC WG <[email-masked]<mailto:[email-masked]>>; Lubashev, Igor <ilubashe=[email-masked]<mailto:ilubashe=[email-masked]>>
Subject: Re: Partially Reliable Message Stream

Better to eat the flow control cost in my view.  Or send RST_STREAM
and make another stream.
On Thu, May 31, 2018 at 8:48 AM Mike Bishop <[email-masked]<mailto:[email-masked]>> wrote:
> The new version 03 (https://urldefense..proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dlubashev-2Dquic-2Dpartial-2Dreliability-2D03&d=DwIFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=yyRpbe14kTYgY413gjVkOalWE_UC3xYerKpJqS8HiQo&s=XZ295xiZsN11yP8GLcdMs1jf7z0b2_WhiIxMLLorZXo&e=<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dlubashev-2Dquic-2Dpartial-2Dreliability-2D03&d=DwIFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=yyRpbe14kTYgY413gjVkOalWE_UC3xYerKpJqS8HiQo&s=XZ295xiZsN11yP8GLcdMs1jf7z0b2_WhiIxMLLorZXo&e=>) no longer needs complex flow control changes and removes the need to transmit multiple frames in the same packet.
>
>
>
> Igor
>
>
>
> P.S.
>

---

From: "Lubashev, Igor" <[email-masked]>
Date: Fri, 1 Jun 2018 00:18:15 +0000

Thanks, Jana.

Stream Groups is an interesting idea that would need to be flushed out further to see where it leads us.  I was trying to minimize the changes to QUIC and limiting the design to applications that want in-order message delivery but prefer to drop old messages in favor of a more timely receipt of newer messages.  I.e. things like real-time audio/video, per-character game state updates, telemetry updates.

1st Class Messages could accomplish more, but, as you said, you’d need to design flow control for these.  And that was the hardest part, as I discovered!

(In fact, version 03 flow control accounting is wrong, as I think about it.  Instead of “Receipt of an EXPIRED_STREAM_DATA does not advance the largest received offset for the stream.”, it should say “If the largest received offset is smaller than Minimum Stream Offset for the stream, it is advanced to Minimum Stream Offset minus 1.”  This will catch a rogue sender violating flow control by sending a huge Minimum Stream Offset, in an attempt to get the receiver to increase its STREAM_MAX_DATA to “Minimum Stream Offset” + stream_buffer_size and then to effect a corresponding huge increase in MAX_DATA.  That would allow the sender to send data on other streams effectively without a connection flow control limit.)

From: Jana Iyengar [mailto:[email-masked]]
Sent: Thursday, May 31, 2018 4:37 PM
To: Roberto Peon <[email-masked]>
Cc: Mike Bishop <[email-masked]>; Martin Thomson <[email-masked]>; QUIC WG <[email-masked]>; Lubashev, Igor <[email-masked]>
Subject: Re: Partially Reliable Message Stream

I'm not keen to get into the weeds of this discussion at this time (as you say, it's for v2), but I understand the overall design. A few higher-order points with this design direction:

1. There's clear demarcation of app messages, but they aren't really made first class citizens. If you did that, I imagine that you might have a simpler design. For instance, define a MSG_STREAM frame, which supports message streams (instead of the current byte streams), and then you can cancel messages easily without having to do stream offset shenanigans.

2. Your design has exactly one ordering capability, which is limiting. Generally, apps that can drop messages and read subsequent ones are also capable of handling messages out of sequence. You'll want to support partial ordering as well and not constrain messages to be only delivered in strictly linear order.

3. I'm keen to see what we can do with using streams as messages. We don't need to punt everything to the app (as your text suggests), but we can start with a grouping of streams (or messages) and building flow control mechanisms for these groups. I suspect this might be cleaner, and it gives you flexibility to consider various orderings of messages. When we're closer to v2, I'm happy to spin up a draft exploring this direction.

On Thu, May 31, 2018 at 8:21 AM, Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:
Yes. You may want a single resource for an output which you may be generating in parallel. Even if a single thread 'owns' the stream, you've lost the ability to coordinate the data it recieves by monotonically increasing offset.

-=R

-------- Original message --------
From: Mike Bishop <[email-masked]<mailto:[email-masked]>>
Date: 5/30/18 8:33 PM (GMT-08:00)
To: Roberto Peon <[email-masked]<mailto:[email-masked]>>, Martin Thomson <[email-masked]<mailto:[email-masked]>>
Cc: QUIC WG <[email-masked]<mailto:[email-masked]>>, "Lubashev, Igor" <ilubashe=[email-masked]<mailto:[email-masked]>>
Subject: RE: Partially Reliable Message Stream

Cc: QUIC WG <[email-masked]<mailto:[email-masked]>>; Lubashev, Igor <ilubashe=[email-masked]<mailto:[email-masked]>>

I find it confusing as well..  I far prefer keeping the offsets monotonically increasing as it allows for real-world use of a partially reliable stream (where you may not recieve that data but you can at least figure out what you did get and interpret it).

---

From: Roberto Peon <[email-masked]>
Date: Fri, 1 Jun 2018 00:20:02 +0000

Background (boring for folks who know about interactive video):

In a real-time/low-jitter use-case, the player may skip over some (older) part of the bitstream in favor of decoding/interpreting more recent data.
When that happens, it will be uninterested in any older data.

How does it find the most appropriate next data? It typically is not the data that is the most recent—even for interactive ‘playback’, you want a ~100ms-200ms buffer to deal with network jitter.
To decode the video bitstream, we need to know where the frames are in it. It (the bistream) doesn’t always have framing of its own.

If there is an ‘index’ stream into which we put the per-frame/per-time-unit metadata about the video bitstream (which we put on another stream), we can cheaply understand the framing for the bitstream.
That enables the player to seek ahead to the parts that it can interpret, as the reader of the index stream knows that if it has k bytes starting at a position where n % k ==0, it is a valid record. You can’t do that on the bitstream, as it has variable length. You could ‘escape’ on the bitstream to ensure a unique framing token, but that would expand the size of the bitstream, and be a pain in others.

You can then play even more fun games, for instance putting the framing in both fixed-size and non-fixed size streams. In such a case you can recover framing if you have received that part of the index stream that describes the bitstream that you wish to interpret, or if the bitstream framing is intact from any point where you understood it in the past.

The latter “even more fun games” is a reasonable deployment scenario for video stuff, as it would allow a user to continue to use the standard playback path with the standard containers, but also affect a more-likely-to-be-interpretable-with-low-jitter playback using partial reliability by having the container data in the separate file.

-=R

From: QUIC <[email-masked]> on behalf of "Lubashev, Igor" <ilubashe=[email-masked]>
Date: Thursday, May 31, 2018 at 4:37 PM
To: Roberto Peon <[email-masked]>, Mike Bishop <[email-masked]>, Martin Thomson <[email-masked]>
Cc: QUIC WG <[email-masked]>
Subject: RE: Partially Reliable Message Stream

  *   Igor

-=R

-------- Original message --------
From: Mike Bishop <[email-masked]<mailto:[email-masked]>>
Date: 5/30/18 8:33 PM (GMT-08:00)
To: Roberto Peon <[email-masked]<mailto:[email-masked]>>, Martin Thomson <[email-masked]<mailto:[email-masked]>>
Cc: QUIC WG <[email-masked]<mailto:[email-masked]>>, "Lubashev, Igor" <ilubashe=[email-masked]<mailto:ilubashe=[email-masked]>>
Subject: RE: Partially Reliable Message Stream

Cc: QUIC WG <[email-masked]<mailto:[email-masked]>>; Lubashev, Igor <ilubashe=[email-masked]<mailto:ilubashe=[email-masked]..org>>

Better to eat the flow control cost in my view.  Or send RST_STREAM
and make another stream.
On Thu, May 31, 2018 at 8:48 AM Mike Bishop <[email-masked]<mailto:[email-masked]>> wrote:
> The new version 03 (https://urldefense...proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dlubashev-2Dquic-2Dpartial-2Dreliability-2D03&d=DwIFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=yyRpbe14kTYgY413gjVkOalWE_UC3xYerKpJqS8HiQo&s=XZ295xiZsN11yP8GLcdMs1jf7z0b2_WhiIxMLLorZXo&e=<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dlubashev-2Dquic-2Dpartial-2Dreliability-2D03&d=DwIFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=yyRpbe14kTYgY413gjVkOalWE_UC3xYerKpJqS8HiQo&s=XZ295xiZsN11yP8GLcdMs1jf7z0b2_WhiIxMLLorZXo&e=>) no longer needs complex flow control changes and removes the need to transmit multiple frames in the same packet.
>
>
>
> Igor
>
>
>
> P.S.
>