Subject: Implicitly opened streams and exposing stream IDs

From: Marten Seemann <[email-masked]>
Date: Mon, 02 Apr 2018 11:43:23 +0000

Recently, the implicit opening of streams (i.e. that when a receiver
receives a frame for stream N+4, it can assume that stream N was already
opened, but the packet might have been reordered) was removed from the
draft. I think this change has some consequences that we haven't discussed
so far.

For the purpose of writing some pseudocode, I'm assuming a QUIC API that
provides an *AcceptStream()* method, but the conclusions will be the same
for a callback-based API.

   - If QUIC has implicit stream opening, *AcceptStream()* would return the
   streams in order (and if a frame opening stream N+4 is received before
   stream n is opened, *AcceptStream()* will first return stream N and then
   stream N+4).
   - Without implicit stream opening, *AcceptStream()* just returns
   whatever stream is received first. Streams might be returned in arbitrary
   order, if the peer doesn't open streams consecutively or if packets are
   reordered.

Now imagine an application protocol where the first unidirectional stream
opened by the client is a control stream, and all higher unidirectional
streams are data streams. The application on the server side needs to find
out which stream is the control stream, because it needs to be handled
separately.

With implicit stream opening, the server code would be:

    control_stream = AcceptStream() // is guaranteed to open the first
stream
    // handle the control stream
     while true:
         stream = AcceptStream()
         // handle the data stream

and without implicit stream opening:

    while true:
        stream = AcceptStream()
        if stream.ID() == kControlStreamID:
            // handle the control stream
        else:
            // handle the data stream

In this case, after accepting a stream, we first have to check the stream
ID, since there's no guarantee if the control stream will actually be
received first.

For this stream mapping, it seems like the removal of implicitly opened
streams implies that QUIC has to expose stream IDs to the application
layer. I'm not sure if this was intended when making the change, especially
since we're considering to change HQ such that it doesn't rely on QUIC
stream IDs any more.
We only manage to avoid the problem described here in our HTTP mapping,
because the HTTP control streams are unidirectional and the request streams
are bidirectional, and can therefore be told apart by their directionality.
However, as a general transport protocol, other applications built on top
of QUIC will have to find some way to deal with it.

---

From: Lucas Pardue <[email-masked]>
Date: Mon, 2 Apr 2018 12:05:51 +0000

Hi Marten,

Would Stream headers fix this problem? I.e. anything that requires special behaviour other than "bulk data" has some bytes of of magic at the start of the stream.

Regards
Lucas
________________________________________
From: QUIC [[email-masked]] on behalf of Marten Seemann [[email-masked]]
Sent: 02 April 2018 12:43
To: QUIC WG
Subject: Implicitly opened streams and exposing stream IDs

Recently, the implicit opening of streams (i.e. that when a receiver receives a frame for stream N+4, it can assume that stream N was already opened, but the packet might have been reordered) was removed from the draft. I think this change has some consequences that we haven't discussed so far.

For the purpose of writing some pseudocode, I'm assuming a QUIC API that provides an AcceptStream() method, but the conclusions will be the same for a callback-based API.

  *   If QUIC has implicit stream opening, AcceptStream() would return the streams in order (and if a frame opening stream N+4 is received before stream n is opened, AcceptStream() will first return stream N and then stream N+4).
  *   Without implicit stream opening, AcceptStream() just returns whatever stream is received first. Streams might be returned in arbitrary order, if the peer doesn't open streams consecutively or if packets are reordered.

Now imagine an application protocol where the first unidirectional stream opened by the client is a control stream, and all higher unidirectional streams are data streams. The application on the server side needs to find out which stream is the control stream, because it needs to be handled separately.

    control_stream = AcceptStream() // is guaranteed to open the first stream
    // handle the control stream
     while true:
         stream = AcceptStream()
         // handle the data stream

    while true:
        stream = AcceptStream()

In this case, after accepting a stream, we first have to check the stream ID, since there's no guarantee if the control stream will actually be received first.

For this stream mapping, it seems like the removal of implicitly opened streams implies that QUIC has to expose stream IDs to the application layer. I'm not sure if this was intended when making the change, especially since we're considering to change HQ such that it doesn't rely on QUIC stream IDs any more.
We only manage to avoid the problem described here in our HTTP mapping, because the HTTP control streams are unidirectional and the request streams are bidirectional, and can therefore be told apart by their directionality. However, as a general transport protocol, other applications built on top of QUIC will have to find some way to deal with it.

-----------------------------
http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and
may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in
error, please delete it from your system.
Do not use, copy or disclose the
information in any way nor act in reliance on it and notify the sender
immediately.
Please note that the BBC monitors e-mails
sent or received.
Further communication will signify your consent to
this.
-----------------------------

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 2 Apr 2018 14:39:31 -0300

Isn’t this between QUIC, the QUIC application API, and the application
protocol?

If you tell QUIC to feed streams in order, just like you it to feed data
inside a stream in order, then it can virtually open streams in order
stream towards the API without affecting the network.

QUIC is also permitted to deliver data out of order inside a stream AFAIR,
although this would not be the usual API behaviour.

In the end, specific broadly accepted app protocols like QUIC/HTTP will
drive the API design.

For other use cases such as some small brief message streams and some heavy
data loaded streams, you might very well want to have the latest stream
arrive first, to process priority messages. Saying that QUIC can never, as
opposed to usually, expose the stream ID seems pointless. It is like saying
a QUIC api can never expose holes in a single streams data stream.

I don’t really see an issue on the wire, and I believe removing implicit
open takes a way some fear, uncertainty and doubt about stream resource
handling - at least I never felt completely comfortable with the stream
design until now.

Regards,
Mikkel

On 2 April 2018 at 14.06.04, Lucas Pardue ([email-masked]) wrote:

Hi Marten,

Would Stream headers fix this problem? I.e. anything that requires special
behaviour other than "bulk data" has some bytes of of magic at the start of
the stream.

Regards
Lucas
________________________________________
From: QUIC [[email-masked]] on behalf of Marten Seemann [
[email-masked]]
Sent: 02 April 2018 12:43
To: QUIC WG
Subject: Implicitly opened streams and exposing stream IDs

For the purpose of writing some pseudocode, I'm assuming a QUIC API that
provides an AcceptStream() method, but the conclusions will be the same for
a callback-based API.

* If QUIC has implicit stream opening, AcceptStream() would return the
streams in order (and if a frame opening stream N+4 is received before
stream n is opened, AcceptStream() will first return stream N and then
stream N+4).
* Without implicit stream opening, AcceptStream() just returns whatever
stream is received first. Streams might be returned in arbitrary order, if
the peer doesn't open streams consecutively or if packets are reordered.

while true:
stream = AcceptStream()

may contain personal views which are not the views of the BBC unless
specifically stated.

---

From: Dmitri Tikhonov <[email-masked]>
Date: Mon, 2 Apr 2018 13:48:11 -0400

On Mon, Apr 02, 2018 at 11:43:23AM +0000, Marten Seemann wrote:
> layer.

I don't see a problem with that -- HTTP is the application layer, and
the draft specifies that streams 2 and 3 are the control streams.

> I'm not sure if this was intended when making the change, especially

Are you referring to a specific pull request or an issue on GitHub?
(I must have missed it.)

  - Dmitri.

---

From: Marten Seemann <[email-masked]>
Date: Tue, 03 Apr 2018 07:10:33 +0000

Hello,

Sure, the application *could* deal with it by starting every stream with
some kind of header, to signal what kind of stream type this is. I'm not
sure if I like this solution, since it creates additional complexity, and
it is only necessary because we removed a useful feature from the spec.
What did we actually gain from removing implicit stream opening from the
protocol? As far as I can see, we were able to relax the requirement
"endpoints MUST open the send side of streams for each type in order" to
"endpoints SHOULD open the send side of streams for each type in order".
This might seem nice from a conceptual viewpoint, but honestly, I don't
really see why anyone would actually want to do this. I think the main
argument against the MUST was that it's not enforceable anyway, but this
applies to a lot of other MUSTs in the spec as well (e.g. that flow control
limits can't be decreased).

The fix for this is straightforward: We should REQUIRE a peer to open
streams in order. How the receiver handles out of order streams then is an
implementation and API decision. I see two ways that an implementation
could reasonably deal with this:

   1. Return streams in the order they are received, e.g. return N+4 before
   N, if packets are reordered.
   2. Return streams ordered by stream ID, i.e. return first N and then N+4
   if a frame for N+4 is received.

This way, we wouldn't need to reintroduce implicitly opened streams, but
leave this up to implementations. I should have named this thread
differently, if only I had realized this earlier ;)

Regards,
Marten

On Mon, Apr 2, 2018 at 7:05 PM Lucas Pardue <[email-masked]> wrote:

>   *   Without implicit stream opening, AcceptStream() just returns

---

From: Martin Thomson <[email-masked]>
Date: Tue, 3 Apr 2018 17:28:21 +1000

Requiring in-order opening is harder than it sounds.  Say you have a
multithreaded application that is initiating requests.  The usual
process is something like:

s = connection.OpenStream()
s.Write(request.Headers.Encode())

That is, there is time between acquisition of the stream and its use.
Do that with enough concurrency and you end up having to add late
binding of stream identifiers to streams, which complicates the design
of the stack considerably.  It also complicates usage because now
there is a period where s.Id() returns an undefined result.

Alternatively, you could track the highest stream ID on which data has
been sent and send an empty, offset 0 STREAM frame rather than create
a gap.

Or, you could decide that a MUST here is unenforceable (even more so
than the flow control limits one) and just wantonly ignore that
requirement if this race condition happens.  While it is in theory
possible to catch someone in a violation, it requires some interesting
conditions (like zero packet loss).

BTW, I also wish to make it possible to avoid relying on specific
stream IDs for special functions.  I hadn't considered opening order
as a way to do that though and had assumed that we'd use an in-stream
header for that.  I don't think that makes the complexity worse, or at
least no worse than having to deal with different frame types for
different functions.

On Tue, Apr 3, 2018 at 5:10 PM, Marten Seemann <[email-masked]> wrote:
> Hello,
>
> Sure, the application could deal with it by starting every stream with some
> kind of header, to signal what kind of stream type this is. I'm not sure if
> I like this solution, since it creates additional complexity, and it is only
> necessary because we removed a useful feature from the spec.
> implementation and API decision. I see two ways that an implementation could
> reasonably deal with this:
>
> Return streams in the order they are received, e.g. return N+4 before N, if
> packets are reordered.
> Return streams ordered by stream ID, i.e. return first N and then N+4 if a
> frame for N+4 is received.
>
>> streams implies that QUIC has to expose stream IDs to the application layer.
>> I'm not sure if this was intended when making the change, especially since
>> we're considering to change HQ such that it doesn't rely on QUIC stream IDs
>> any more.
>> However, as a general transport protocol, other applications built on top of
>> QUIC will have to find some way to deal with it.
>>
>>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 3 Apr 2018 00:50:16 -0700

Say you deal with 10K outbound bidi streams.
As Martin suggests, you can make this heavily concurrent and also reserve
certain patterns for certain use.

With implicit bi-directional streams you must be prepared to handle
connection state for all 10K streams that is supposed to be started by you.
You cannot create an error if you receive data, at least not at transport
level, so you must open and track state. This in turn will limit you number
of streams you are willing to handle at once, or force you to split streams
into linked uni-streams.

Further, there might be zero application logic that understands what data
on such a stream means so when an unfortunate subsystem decides to open a
stream, it has to ponder whatever the data already pending on that stream
means.

Opening strictly in order (whether required or not) reduces those problems,
but it also severely constrains the application. For example you cannot
send a stream id internally to a subsystem and ask it to produce data once
ready to do so. You need to invent a complex mapping that is difficult to
coordinate in a distributed system. It is much simpler to deal with the
stream id’s directly. With multi-path you could even have multiple hosts
managing a single connection, and if not, you could still do it with proper
address mapping.

So implicit streams and strict opening order might simplify very simple
problems, but also make very hard problems even harder.

Also, the entire idea of linearising QUIC seems to go against the spirit of
independent non-blocking streams.

Still, having a maximum stream ID is very useful in limiting the resources
that must be managed.

Mikkel

On 3 April 2018 at 09.28.38, Martin Thomson ([email-masked])
wrote:

> implementation and API decision. I see two ways that an implementation
could
layer.
>> I'm not sure if this was intended when making the change, especially
since

---

From: Marten Seemann <[email-masked]>
Date: Tue, 03 Apr 2018 08:40:53 +0000

Hi Martin,

I don't think delayed stream usage is a problem here. To be precise, what I
mean is not opening the stream (as in actually sending a frame on this
stream), but creation of the stream. So all I want is to REQUIRE here is
that peers *create* streams in order.

There's an important difference in how the receiving peer is allowed to
act. Assume for example the receiver wants to grant a higher flow control
limit than it advertised during the handshake:

   - If streams MUST be created in order, the receiver can send a
   MAX_STREAM_DATA for streams N and N+4 as soon as it receives a frame for
   N+4.
   - If streams only SHOULD be created in order, sending a MAX_STREAM_DATA
   for stream N (after receiving a frame for N+4) will cause a connection
   error, if the sender decided to skip stream N.

I'm not saying that this is an important use case, but this example shows
that implementations *only* have the two options for handling out of order
streams described in my last email *if* we require in order stream creation.

I'd therefore suggest to change the text in the spec to something like
this: "Endpoints MUST create the send side of streams for each type in
order".

Best,
Marten

---

From: Martin Thomson <[email-masked]>
Date: Tue, 3 Apr 2018 19:23:41 +1000

Creation isn't an observable action, so we need to use observable
actions to drive what the spec says.  So that would seem to be
equivalent to restoring implicit open in my mind.  Which - for the
record - I was a proponent of.  But you don't need to convince me :)

As you say, the observable difference is in whether sending
MAX_STREAM_DATA on stream N is permitted once stream N+4 has been
opened.  (And similar such things.)

I found the rather strident arguments in favour of dropping implicit
open convincing, in the sense that a strict requirement for in order
creation was onerous for those who want to parallelize stream
consumption.  But it's entirely possible that that could be dealt with
by saying that the streams are still created in order, but used out of
order.  I don't know.  Those who argued for a lifting of the
requirement should weigh in.

On Tue, Apr 3, 2018 at 6:40 PM, Marten Seemann <[email-masked]> wrote:
> that peers create streams in order.
>
> There's an important difference in how the receiving peer is allowed to act.
> Assume for example the receiver wants to grant a higher flow control limit
> than it advertised during the handshake:
>
> If streams MUST be created in order, the receiver can send a MAX_STREAM_DATA
> for streams N and N+4 as soon as it receives a frame for N+4.
> If streams only SHOULD be created in order, sending a MAX_STREAM_DATA for
> stream N (after receiving a frame for N+4) will cause a connection error, if
> the sender decided to skip stream N.
>
> I'd therefore suggest to change the text in the spec to something like this:
> "Endpoints MUST create the send side of streams for each type in order".
>
> Best,
> Marten
>

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 3 Apr 2018 08:07:18 -0400

On Tue, Apr 03, 2018 at 05:28:21PM +1000, Martin Thomson wrote:
> Requiring in-order opening is harder than it sounds.

Some more discussion here:

  https://github.com/quicwg/base-drafts/issues/634

  - Dmitri.

---

From: Nick Banks <[email-masked]>
Date: Tue, 3 Apr 2018 13:47:36 +0000

+1 for keeping the implicit opening of streams.

It does greatly simplify the tracking of idle/opened/closed streams. Also I agree that it simplifies the application layer (IMO), for things like HTTP control stream(s). If all streams are just opened in order, then the application layer really doesn't need (though it can, if it wants to link information across streams) to care about stream ID at all. The only down side to me about implicit stream opening is the attack surface. It only takes a few bytes in a single packet from the peer to open all streams you allotted for them. But I still don't feel that is that big of a deal. Just don't allot for streams for the peer that you are willing to maintain at one time.

To Martin's comment of "creation isn't an observable action" why do you say that? I can't find the PR right now, but I thought there was a PR recently that allows for sending the initial STREAM frame with offset=0 and length=0. That seems to be a way to directly indicate stream opening, and in my mind, make it an observable action.

Thanks,
- Nick

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Dmitri Tikhonov
Sent: Tuesday, April 3, 2018 5:07 AM
To: Martin Thomson <[email-masked]>
Cc: Marten Seemann <[email-masked]>; Lucas Pardue <[email-masked]>; QUIC WG <[email-masked]>
Subject: Re: Implicitly opened streams and exposing stream IDs

  https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fquicwg%2Fbase-drafts%2Fissues%2F634&data=02%7C01%7Cnibanks%40microsoft.com%7C8d1fc56b3fba4a97335308d5995b7d97%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636583540596473225&sdata=0b6u8%2FntdWlkQInAefy0%2BYOaLH3%2BnEGmmUmVZ%2BjtmSo%3D&reserved=0

  - Dmitri.

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 3 Apr 2018 09:56:45 -0400

On Tue, Apr 03, 2018 at 01:47:36PM +0000, Nick Banks wrote:
> It does greatly simplify the tracking of idle/opened/closed streams.

I don't see how this is a simplification.  You will need a "set" data
structure of some sort in both cases:

    A. Implicit opening.  Opened streams can be tracked with a single
       number -- *but* closed streams need to be tracked one by one
       (using a set or a hash).

    B. Explicit opening.  Opened streams are tracked using a hash.

One can argue that (B) is actually simpler.

  - Dmitri.

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 3 Apr 2018 10:00:09 -0400

I feel I am in some sort of discussion hole -- where was it first
discussed to get rid of relying on Stream IDs in the application
layer?  I'd like to get some background.

  -Dmitri.

---

From: Nick Banks <[email-masked]>
Date: Tue, 3 Apr 2018 14:02:28 +0000

For implicit opening, you only need to track the following, and you know the state of all stream IDs:

 - Set of all currently opened stream objects
 - Largest stream ID ever opened

For explicit opening, since the streams below the largest stream ID ever opened, may never have been opened, you need to additionally track the stream IDs that have been closed (or inversely the set of stream IDs that have ever been opened).

- Nick

-----Original Message-----
From: Dmitri Tikhonov <[email-masked]> 
Sent: Tuesday, April 3, 2018 6:57 AM
To: Nick Banks <[email-masked]>
Cc: Martin Thomson <[email-masked]>; Marten Seemann <[email-masked]>; Lucas Pardue <[email-masked]>; QUIC WG <[email-masked]>
Subject: Re: Implicitly opened streams and exposing stream IDs

> It does greatly simplify the tracking of idle/opened/closed streams.

I don't see how this is a simplification.  You will need a "set" data structure of some sort in both cases:

  - Dmitri.

---

From: Nick Banks <[email-masked]>
Date: Tue, 3 Apr 2018 14:05:04 +0000

As to requiring the application rely on stream IDs, I was referring to Marten's first email that started this thread:

" For this stream mapping, it seems like the removal of implicitly opened streams implies that QUIC has to expose stream IDs to the application layer. I'm not sure if this was intended when making the change, especially since we're considering to change HQ such that it doesn't rely on QUIC stream IDs any more."

I agree with this statement. For application protocols where they specific the "first stream" as the control stream, if you guarantee the first stream the application layer gets is always the first ordered stream, by ID, then they don't need to know the ID. Otherwise, then need to get the stream's ID to do the enforcement themselves.

- Nick

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Dmitri Tikhonov
Sent: Tuesday, April 3, 2018 7:00 AM
To: Marten Seemann <[email-masked]>; QUIC WG <[email-masked]>
Subject: Re: Implicitly opened streams and exposing stream IDs

I feel I am in some sort of discussion hole -- where was it first discussed to get rid of relying on Stream IDs in the application layer?  I'd like to get some background.

  -Dmitri.

On Mon, Apr 02, 2018 at 01:48:10PM -0400, Dmitri Tikhonov wrote:
> > In this case, after accepting a stream, we first have to check the 
> > stream ID, since there's no guarantee if the control stream will 
> > actually be received first.
> > 
> > For this stream mapping, it seems like the removal of implicitly 
> > opened streams implies that QUIC has to expose stream IDs to the 
> > application layer.
> 
> > since we're considering to change HQ such that it doesn't rely on 
> > QUIC stream IDs any more.
>

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 3 Apr 2018 10:07:13 -0400

On Tue, Apr 03, 2018 at 02:02:28PM +0000, Nick Banks wrote:
> For explicit opening, since the streams below the largest stream ID
> ever opened, may never have been opened, you need to additionally
> track the stream IDs that have been closed (or inversely the set of
> stream IDs that have ever been opened).

Ah yes, that's true -- I remember adding this code now.

The tracking is simpler with implicit stream opening.

  - Dmitri.

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 3 Apr 2018 10:22:21 -0400

On Tue, Apr 03, 2018 at 02:05:04PM +0000, Nick Banks wrote:
> " For this stream mapping, it seems like the removal of implicitly
> application layer. I'm not sure if this was intended when making the
> change, especially since we're considering to change HQ such that it
> doesn't rely on QUIC stream IDs any more."

I think I found it:

  https://github.com/quicwg/base-drafts/issues/910

Thank you.

> I agree with this statement. For application protocols where they
> specific the "first stream" as the control stream, if you guarantee the
> first stream the application layer gets is always the first ordered
> stream, by ID, then they don't need to know the ID. Otherwise, then
> need to get the stream's ID to do the enforcement themselves.

I agree with the implication (no implicit open -> need Stream IDs),
but I do not have a problem with the implication itself.  I think
exposing Stream IDs is fine -- I will comment elsewhere in this
thread.

  - Dmitri.

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 3 Apr 2018 10:27:36 -0400

> problems, but also make very hard problems even harder.

+1.  Very well put.

> independent non-blocking streams.

+1.  (HoL!)

> Still, having a maximum stream ID is very useful in limiting the resources

Note that QUIC does have it -- MAX_STREAM_ID.  The in-order stream
creation is unenforceable; faking it on the receiver side introduces
head-of-line blocking.  As long as the incoming stream has ID smaller
than MAX_STREAM_ID, there is no reason not to handle the stream
immediately.

  - Dmitri.

---

From: Ian Swett <[email-masked]>
Date: Tue, 03 Apr 2018 18:56:01 +0000

I don't like implicitly opened streams, for the reasons above and I'd
prefer to avoid relying on implicit anything in QUIC.  However, I hadn't
considered ordering of stream opening as a way to avoid exposing stream
id.  It's interesting, though I think it's probably easier to just expose
Stream ID.

> > stream IDs that have ever been opened).

As discussed on the issue, the tracking data structure is the same, but the
state of the stream is slightly different.  In one case, you have a marker
for 'not yet open'(maybe a null as the value in your hashmap) and in the
other case it's open and writable.

To avoid allocating a large amount of memory with a small number of bytes,
one may want to not create a stream object until it's actually needed.
That would prevent writing, but as we've said, it's not necessary
implementations support writing on implicitly opened streams.

Chrome does this today with stream tracking, see:
https://cs.chromium.org/chromium/src/net/quic/core/quic_session.cc?sq=package:chromium&l=845

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 5 Apr 2018 12:14:09 +0000

This is not a QUIC transport draft concern. This is a question of QUIC library API design. Your API may choose to have Accept return streams in order. My API may choose to have Accept return streams as they come in (and expose steam IDs). As long as the transport draft requires that streams are CREATED in order (as it does now), both API designs are valid.

It would be a matter of a very different draft to standardize QUIC APIs (like we have for POSIX) so an application can be easily ported to use a different vendor's QUIC libraries.

- Igor

-----Original Message-----
From: Marten Seemann [[email-masked]]
Received: Monday, 02 Apr 2018, 7:43AM
To: QUIC WG [[email-masked]]
Subject: Implicitly opened streams and exposing stream IDs

    while true:
        stream = AcceptStream()

---

From: Marten Seemann <[email-masked]>
Date: Thu, 05 Apr 2018 12:51:32 +0000

Hi Igor,

This is exactly the change I'm proposing. Currently, the draft says that
you SHOULD create streams in order (section 9.2). I'd like to change that
SHOULD to a MUST.
Only if in order stream creation is a requirement, this question becomes an
API design question.

Regards,
Marten

On Thu, Apr 5, 2018 at 7:14 PM Lubashev, Igor <[email-masked]> wrote:

> This is not a QUIC transport draft concern. This is a question of QUIC
> library API design. Your API may choose to have Accept return streams in
> order. My API may choose to have Accept return streams as they come in (and
> expose steam IDs). As long as the transport draft requires that streams are
> CREATED in order (as it does now), both API designs are valid.
>
> It would be a matter of a very different draft to standardize QUIC APIs
> (like we have for POSIX) so an application can be easily ported to use a
> different vendor's QUIC libraries.
>
>
> - Igor
>
>
>
> *From:* Marten Seemann [[email-masked]]
> *Received:* Monday, 02 Apr 2018, 7:43AM
> *To:* QUIC WG [[email-masked]]
> *Subject:* Implicitly opened streams and exposing stream IDs
>
>    - If QUIC has implicit stream opening, *AcceptStream()* would return
>    the streams in order (and if a frame opening stream N+4 is received before
>    stream n is opened, *AcceptStream()* will first return stream N and
>    then stream N+4).

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 5 Apr 2018 14:06:12 +0000

I agree. I would like to see MUST there as well.

-----Original Message-----
From: Marten Seemann [[email-masked]]
Received: Thursday, 05 Apr 2018, 8:51AM
To: Lubashev, Igor [[email-masked]]
CC: [email-masked] [[email-masked]]
Subject: Re: Implicitly opened streams and exposing stream IDs

Hi Igor,

This is exactly the change I'm proposing. Currently, the draft says that you SHOULD create streams in order (section 9.2). I'd like to change that SHOULD to a MUST.
Only if in order stream creation is a requirement, this question becomes an API design question.

Regards,
Marten

On Thu, Apr 5, 2018 at 7:14 PM Lubashev, Igor <[email-masked]<mailto:[email-masked]>> wrote:
This is not a QUIC transport draft concern. This is a question of QUIC library API design. Your API may choose to have Accept return streams in order. My API may choose to have Accept return streams as they come in (and expose steam IDs). As long as the transport draft requires that streams are CREATED in order (as it does now), both API designs are valid.

- Igor

-----Original Message-----
From: Marten Seemann [[email-masked]<mailto:[email-masked]>]
Received: Monday, 02 Apr 2018, 7:43AM
To: QUIC WG [[email-masked]<mailto:[email-masked]>]
Subject: Implicitly opened streams and exposing stream IDs

    while true:
        stream = AcceptStream()

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 5 Apr 2018 14:11:47 +0000

> Creation isn't an observable action, so we need to use observable actions to drive what the spec says.

Creation is a wire-observable action, if an endpoint creates just a single stream, when MAX_STREAMS allows creation of streams n through n+m.  If a single stream is created, there will only be a single STREAM(id=x, off=0) on the wire (possibly retransmitted).  It will be very observable if x != n.

- Igor

-----Original Message-----
From: Martin Thomson [mailto:[email-masked]] 
Sent: Tuesday, April 03, 2018 5:24 AM
To: Marten Seemann <[email-masked]>
Cc: Lucas Pardue <[email-masked]>; QUIC WG <[email-masked]>
Subject: Re: Implicitly opened streams and exposing stream IDs

Creation isn't an observable action, so we need to use observable actions to drive what the spec says.  So that would seem to be equivalent to restoring implicit open in my mind.  Which - for the record - I was a proponent of.  But you don't need to convince me :)

As you say, the observable difference is in whether sending MAX_STREAM_DATA on stream N is permitted once stream N+4 has been opened.  (And similar such things.)

I found the rather strident arguments in favour of dropping implicit open convincing, in the sense that a strict requirement for in order creation was onerous for those who want to parallelize stream consumption.  But it's entirely possible that that could be dealt with by saying that the streams are still created in order, but used out of order.  I don't know.  Those who argued for a lifting of the requirement should weigh in.

On Tue, Apr 3, 2018 at 6:40 PM, Marten Seemann <[email-masked]> wrote:
> Hi Martin,
>
> I don't think delayed stream usage is a problem here. To be precise, 
> what I mean is not opening the stream (as in actually sending a frame 
> on this stream), but creation of the stream. So all I want is to 
> REQUIRE here is that peers create streams in order.
>
> Assume for example the receiver wants to grant a higher flow control 
> If streams MUST be created in order, the receiver can send a 
> MAX_STREAM_DATA for streams N and N+4 as soon as it receives a frame for N+4.
> If streams only SHOULD be created in order, sending a MAX_STREAM_DATA 
> I'm not saying that this is an important use case, but this example 
> shows that implementations *only* have the two options for handling 
> out of order streams described in my last email *if* we require in order stream creation.
>
>> binding of stream identifiers to streams, which complicates the 
>> design of the stack considerably.  It also complicates usage because 
>> now there is a period where s.Id() returns an undefined result.
>>
>> Alternatively, you could track the highest stream ID on which data 
>> has been sent and send an empty, offset 0 STREAM frame rather than 
>> create a gap.
>>
>> possible to catch someone in a violation, it requires some 
>> interesting conditions (like zero packet loss).
>>
>> header for that.  I don't think that makes the complexity worse, or 
>> at least no worse than having to deal with different frame types for 
>> different functions.
>>
>>
>> > Sure, the application could deal with it by starting every stream 
>> > with some kind of header, to signal what kind of stream type this 
>> > is. I'm not sure if I like this solution, since it creates 
>> > additional complexity, and it is only necessary because we removed 
>> > a useful feature from the spec.
>> > What did we actually gain from removing implicit stream opening 
>> > from the protocol? As far as I can see, we were able to relax the 
>> > requirement "endpoints MUST open the send side of streams for each 
>> > type in order" to "endpoints SHOULD open the send side of streams for each type in order".
>> > This might seem nice from a conceptual viewpoint, but honestly, I 
>> > don't really see why anyone would actually want to do this. I think 
>> > the main argument against the MUST was that it's not enforceable 
>> > anyway, but this applies to a lot of other MUSTs in the spec as 
>> > well (e.g. that flow control limits can't be decreased).
>> >
>> > The fix for this is straightforward: We should REQUIRE a peer to 
>> > open streams in order. How the receiver handles out of order 
>> > streams then is an implementation and API decision. I see two ways 
>> > that an implementation could reasonably deal with this:
>> >
>> > Return streams in the order they are received, e.g. return N+4 
>> > before N, if packets are reordered.
>> > Return streams ordered by stream ID, i.e. return first N and then 
>> > N+4 if a frame for N+4 is received.
>> >
>> > This way, we wouldn't need to reintroduce implicitly opened 
>> > streams, but leave this up to implementations. I should have named 
>> > this thread differently, if only I had realized this earlier ;)
>> >
>> > Regards,
>> > Marten
>> >
>> > On Mon, Apr 2, 2018 at 7:05 PM Lucas Pardue 
>> > <[email-masked]>
>> > wrote:
>> >>
>> >> Hi Marten,
>> >>
>> >> Would Stream headers fix this problem? I.e. anything that requires 
>> >> special behaviour other than "bulk data" has some bytes of of 
>> >> magic at the start of the stream.
>> >>
>> >> Regards
>> >> Lucas
>> >> Recently, the implicit opening of streams (i.e. that when a 
>> >> receiver receives a frame for stream N+4, it can assume that 
>> >> stream N was already opened, but the packet might have been 
>> >> reordered) was removed from the draft. I think this change has 
>> >> some consequences that we haven't discussed so far.
>> >>
>> >> For the purpose of writing some pseudocode, I'm assuming a QUIC 
>> >> API that provides an AcceptStream() method, but the conclusions 
>> >> will be the same for a callback-based API.
>> >>
>> >> streams in order (and if a frame opening stream N+4 is received 
>> >> before stream n is opened, AcceptStream() will first return stream 
>> >> N and then stream N+4).
>> >> whatever stream is received first. Streams might be returned in 
>> >> arbitrary order, if the peer doesn't open streams consecutively or 
>> >> if packets are reordered.
>> >>
>> >> Now imagine an application protocol where the first unidirectional 
>> >> stream opened by the client is a control stream, and all higher 
>> >> unidirectional streams are data streams. The application on the 
>> >> server side needs to find out which stream is the control stream, 
>> >> because it needs to be handled separately.
>> >>
>> >> I'm not sure if this was intended when making the change, 
>> >> especially since we're considering to change HQ such that it 
>> >> doesn't rely on QUIC stream IDs any more.
>> >> We only manage to avoid the problem described here in our HTTP 
>> >> mapping, because the HTTP control streams are unidirectional and 
>> >> the request streams are bidirectional, and can therefore be told 
>> >> apart by their directionality.
>> >> However, as a general transport protocol, other applications built 
>> >> on top of QUIC will have to find some way to deal with it.
>> >>
>> >>
>> >> This e-mail (and any attachments) is confidential and may contain 
>> >> personal views which are not the views of the BBC unless 
>> >> specifically stated.
>> >> information in any way nor act in reliance on it and notify the 
>> >> sender immediately.
>> >> Please note that the BBC monitors e-mails sent or received.
>> >> Further communication will signify your consent to this.

---

From: Dmitri Tikhonov <[email-masked]>
Date: Thu, 5 Apr 2018 10:34:37 -0400

The switch to MUST would cater to a single use case: a control stream
opened at the beginning of the connection.  Presumably, the following
streams would not be useable without the control stream (as is the case
with HQ), so waiting for the control stream frames is OK.

The downside is that the stream creator no longer has the flexibility
to use its legit set of Stream IDs as it sees fit, precluding some use
cases (see issue #634 [1]) -- perhaps even the very use of QUIC for
some protocols!

The API problem Marten describes seems like a non-issue:  There are
two sets of APIs:
    - User code / HQ; and
    - HQ / QUIC.

Stream IDs do not have to be exposed in the former:  The user does not
care about them, as these things are just your regular HTTP messages.
In the HQ / QUIC API, on the other hand, expose away!  Streams have
IDs, the QUIC transport draft talks about them, the IDs have meanings --
it only makes sense that the layer that uses the QUIC objects directly
can use Stream IDs.  Why not?!

  - Dmitri.

1. https://github.com/quicwg/base-drafts/issues/634

---

From: Dmitri Tikhonov <[email-masked]>
Date: Thu, 5 Apr 2018 10:34:37 -0400

- Dmitri.