Subject: Split error codes in two

From: Martin Thomson <[email-masked]>
Date: Mon, 4 Sep 2017 14:55:02 +1000

Input sought.

I've a proposal in that splits the error code space:

   https://github.com/quicwg/base-drafts/pull/722

The basic idea here - one that I believe that we have agreement on,
but would like to confirm - is that only applications should be
cancelling streams.  In short: if the transport kills a stream that
the application was relying on, the whole application protocol state
becomes indeterminate.

In order to ensure no possibility of confusion, I think that it is
cleaner to have transport errors and application errors.  The above PR
makes that split.

It's a pretty clean cut, but there are a few costs: CONNECTION_CLOSE
needs to be split because both application and transport might need to
kill a connection.  The NO_ERROR code is then duplicated.  There is
surprisingly little need for overlap on other error codes.

In doing this, I realized two further things:

1. STOP_SENDING is a purely application-layer construct, but it's a
transport-layer frame.  That makes it much harder to manage with this
change.  Moving it to HTTP makes a lot of sense (to me).

   https://github.com/quicwg/base-drafts/pull/759

2. We are now nudging 20 error codes (plus the per-frame-type ones).
4 billion might be a little more space than we need, so halving the
number of bits with give error cords seems reasonable:

   https://github.com/quicwg/base-drafts/pull/723

---

From: Christian Huitema <[email-masked]>
Date: Mon, 4 Sep 2017 13:03:11 -0700

>    https://github.com/quicwg/base-drafts/pull/759

Arguably, MAX STREAM DATA is also an application-layer construct. If an
application does not want to receive more bytes on a stream, it could
stop increasing that counter. That would have more or less the same
effect as STOP SENDING, in a passive aggressive way...

>    https://github.com/quicwg/base-drafts/pull/723
Is that really a priority? I mean, we probably do not really need to
save 2 bytes in error messages.

And then, how do we deal with extensions? Suppose that I define an
experimental version of the protocol that brings both new functionality
and new error codes. What number range shall I use for these extended
error codes?

-- 
Christian Huitema

---

From: Martin Thomson <[email-masked]>
Date: Tue, 5 Sep 2017 09:55:50 +1000

> effect as STOP SENDING, in a passive aggressive way...

MAX_STREAM_DATA definitely has to be visible to the transport.  But
having STOP_SENDING visible is actually more of a hazard than a help.
If the transport cancels a stream, then it might destroy application
state.

> save 2 bytes in error messages.

No, it's not a priority, but it was easy to do.  And I can't see any
way that 4 billion helps us.

> and new error codes. What number range shall I use for these extended

We've a fair bit of experience with that.  Provided that we don't put
an insane policy on the registry, new values are easy to acquire.  I
prefer the "provisional registration" approach, but we also have the
policy TLS uses, and there is the option of a free-for-all chunk.
None of that requires 4 billion codes.

---

From: Victor Vasiliev <[email-masked]>
Date: Tue, 5 Sep 2017 03:12:07 -0700

> the application was relying on, the whole application protocol state

I assume this means only on the wire?  Because you have to close
the streams both when your connection goes down, and when you get
a 0-RTT reject.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 5 Sep 2017 21:18:53 +1000

> the streams both when your connection goes down, and when you get

I forget where this discussion happened, but the current thinking was
that 0-RTT reject doesn't result in streams being closed.  Instead, it
causes any data that was sent to need to be retransmitted.  Basically,
treat it as lost (though not for the purposes of congestion signals,
waves hands a little).  Does that make sense?

---

From: Victor Vasiliev <[email-masked]>
Date: Tue, 5 Sep 2017 07:31:36 -0400

On Tue, Sep 5, 2017 at 7:18 AM, Martin Thomson <[email-masked]>
wrote:

> > On Sun, Sep 3, 2017 at 9:55 PM, Martin Thomson <[email-masked]
> >
> > wrote:
> >>

That is what the current gQUIC code does, and it has been historically
sufficiently problematic that I've been trying to switch us away from that
behavior.  TLS prohibits automatic resend without explicit opt-in from the
application, and even then, it's a bad idea.  Application payload may vary
substantially depending on the state of the TLS connection, so if the
connection changes, client's presented identity, the exporter secrets used,
etc, may become different (and in case of exporter secrets, they will).

---

From: Nick Harper <[email-masked]>
Date: Tue, 5 Sep 2017 17:07:15 -0700

The approach of treating rejected 0-RTT data like it was lost and
retransmitting at the transport layer causes problems. As Victor
mentioned, one of the problems is that the exporter secret changes,
which is a problem for Token Binding if a client attempts 0-RTT Token
Binding and 0-RTT is rejected. In this case, the Token Binding HTTP
header needs to be rewritten to be a signature over the new exporter
value, when the original stream had a Token Binding header that
covered the old exporter value. I agree with Victor that on 0-RTT
reject, streams should be closed.

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 5 Sep 2017 17:59:32 -0700

I think this (0-RTT) is a non-issue, but correct me if I'm wrong. This is
really about sending RST_STREAMs, not about anything else. I agree with
Victor and Nick that 0-RTT data cannot simply be retransmitted, but I think
this set of PRs is really about the wire protocol. The way I read it, QUIC
should never be sending RST_STREAMs of its own accord; that should be an
application-controlled mechanism. And to be clear, the only place that's
not true in gQUIC is when a RST_STREAM is received. gQUIC responds
immediately with a RST_STREAM.

In the current draft, we've already changed that requirement: the
application is expected to generate a RST_STREAM in response to a received
RST_STREAM. That leaves the transport requirement of responding with a
RST_STREAM to a STOP_SENDING. I believe STOP_SENDING really is an HTTP-ism
and belongs in HTTP. A transport that wants to throttle the sender on a
particular stream can use flow control to do it.

I think reducing error code space is innocuous, and I'm fine with it.

On Tue, Sep 5, 2017 at 5:07 PM, Nick Harper <[email-masked]> wrote:

> > On Tue, Sep 5, 2017 at 7:18 AM, Martin Thomson <[email-masked]
> >
> > wrote:
> >>

---

From: Martin Thomson <[email-masked]>
Date: Wed, 6 Sep 2017 11:10:26 +1000

On Wed, Sep 6, 2017 at 10:59 AM, Jana Iyengar <[email-masked]> wrote:
> application-controlled mechanism. And to be clear, the only place that's not
> true in gQUIC is when a RST_STREAM is received. gQUIC responds immediately
> with a RST_STREAM.

I've spun the 0-RTT discussion off onto another thread.  I think that
there are issues, but hopefully they are minor.

> RST_STREAM.

That text was removed with the addition of STOP_SENDING.  (Or at least
I couldn't find it).  I think that's the right answer, because this
will be entirely discretionary on the part of applications.

(I'm not disagreeing with any of your conclusions, just pointing that
out in case you thought that it was an important consideration.)

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 5 Sep 2017 18:12:52 -0700

On Tue, Sep 5, 2017 at 6:10 PM, Martin Thomson <[email-masked]>
wrote:

> not

I think they're orthogonal, but I'll go look at the other thread.

> will be entirely discretionary on the part of applications.

Yup, I was just pointing out the reasons that we don't need RST_STREAM to
be a transport directive.

---

From: "Philipp S. Tiesel" <[email-masked]>
Date: Wed, 6 Sep 2017 11:16:06 +0200

> On 6. Sep 2017, at 03:12, Jana Iyengar <[email-masked]> wrote:
> On Tue, Sep 5, 2017 at 6:10 PM, Martin Thomson <[email-masked]> wrote:
> Yup, I was just pointing out the reasons that we don't need RST_STREAM to be a transport directive.

RST_STREAM is also referenced in the "Stream Errors” section and still makes sense for transport errors only effecting a single stream

Removing RST_STREAM also means making all errors effecting just a single stream connection errors leading to teardown of the whole connection.

AVE!
  Philipp S. Tiesel / phils…

---

From: Martin Thomson <[email-masked]>
Date: Wed, 6 Sep 2017 19:55:42 +1000

> RST_STREAM is also referenced in the "Stream Errors” section and still makes sense for transport errors only effecting a single stream

The problem here is that a stream reset at the transport layer causes
loss of application layer state.  There might be some application data
that is critical (the opposite of your partially reliability use
case).  The transport dropping it would leave the application in an
indeterminate state.

---

From: Mike Bishop <[email-masked]>
Date: Wed, 6 Sep 2017 17:55:14 +0000

What transport-level single-stream errors are there which shouldn't kill the connection?  The only ones I can think of are flow control violations and max-stream-ID violations.  Perhaps "frame for stream in wrong state," though you have to be somewhat gracious about that one given reordering.  I think I'm okay with making those fatal.

I'm less convinced that STOP_SENDING is HTTP-specific.  To my mind, it's the wire expression of a read_close(), just as RST_STREAM is the wire expression of a write_close().  If we'd gone the advisory route, it would be an HTTP-ism.  (Or at least, the mandated behavior would be at the HTTP layer.)  However, if we think other applications aren't ever going to want to close a read stream....

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Philipp S. Tiesel
Sent: Wednesday, September 6, 2017 2:16 AM
To: Jana Iyengar <[email-masked]>
Cc: Victor Vasiliev <[email-masked]>; QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>; Nick Harper <[email-masked]>
Subject: Re: Split error codes in two

>>> application is expected to generate a RST_STREAM in response to a 
>>> received RST_STREAM.
>>> 
>> That text was removed with the addition of STOP_SENDING.  (Or at 
>> least I couldn't find it).  I think that's the right answer, because 
>> this will be entirely discretionary on the part of applications.
>> 

AVE!

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 6 Sep 2017 18:59:17 +0000

Whether STOP_SENDING is in transport is a matter of picking the guiding principle for what one would consider putting in transport vs application.  Some of the alternatives for the "guiding principle" are:
* "Is likely to be used by many apps?"
* "Will it likely perform better in transport?"
 * "Is it a 'cleaner' design due to all sorts of symmetries?"

I think that STOP_SENDING "may perform better in transport" due to the rule of thumb that "stop" signals should take priority over "ordinary" data.  I.e. having it as a part of an application protocol has a risk that STOP_SENDING would be blocked by flow control, HOL blocking, transport/app interface, etc.  The "symmetry" argument, however, does put STOP_SENDING in the application, since RST_STREAM is never initiated by transport otherwise.  And I have little intuition as far as "use by many apps" is concerned.

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Philipp S. Tiesel
Sent: Wednesday, September 6, 2017 2:16 AM
To: Jana Iyengar <[email-masked]>
Cc: Victor Vasiliev <[email-masked]>; QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>; Nick Harper <[email-masked]>
Subject: Re: Split error codes in two

AVE!

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 7 Sep 2017 12:02:27 -0700

>
> I'm less convinced that STOP_SENDING is HTTP-specific.  To my mind, it's
> the wire expression of a read_close(), just as RST_STREAM is the wire
> expression of a write_close().  If we'd gone the advisory route, it would
> be an HTTP-ism.  (Or at least, the mandated behavior would be at the HTTP
> layer.)  However, if we think other applications aren't ever going to want
> to close a read stream....

TCP allows for read_close() with shutdown(socket, RD), and it basically
means that TCP will send a RST to subsequent incoming data on that socket
from the peer. The expectation is that the application knows to not send
data to a peer that has issued a read_close().

What we're talking about here is different than a read_close(), since it's
an instruction to the peer to stop sending. This is only useful if I don't
want to stop sending but want only the peer to stop sending.... and the
driving use case was RST_STREAM(NO_ERROR) in HTTP/2. I still find that use
case odd, but I don't know that there's really a need for this otherwise.
I'm not very married on this point, but I'm wary of providing a knob that
really doesn't have a strong use case.

---

From: Subodh Iyengar <[email-masked]>
Date: Thu, 7 Sep 2017 23:25:53 +0000

If an app wanted to send both the stop sending and the reset stream, would it have to wait till stop sending was sent on the wire before sending reset streams? For example a API might reasonably be something like

   transport->write(StopSendingFrame)

   transport->reset();

If the app calls reset, then the transport might dump all data associated with the stream immediately so the stop sending might never even be sent.

How do people intend to use stop sending from the app protocols that don't send RST in response to RST?

Subodh

________________________________
From: QUIC <[email-masked]> on behalf of Jana Iyengar <[email-masked]>
Sent: Thursday, September 7, 2017 12:02:27 PM
To: Mike Bishop
Cc: Philipp S. Tiesel; QUIC WG; Martin Thomson; Nick Harper; Victor Vasiliev
Subject: Re: Split error codes in two

TCP allows for read_close() with shutdown(socket, RD), and it basically means that TCP will send a RST to subsequent incoming data on that socket from the peer. The expectation is that the application knows to not send data to a peer that has issued a read_close().

What we're talking about here is different than a read_close(), since it's an instruction to the peer to stop sending. This is only useful if I don't want to stop sending but want only the peer to stop sending.... and the driving use case was RST_STREAM(NO_ERROR) in HTTP/2. I still find that use case odd, but I don't know that there's really a need for this otherwise. I'm not very married on this point, but I'm wary of providing a knob that really doesn't have a strong use case.

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 7 Sep 2017 16:42:52 -0700

So, a RST_STREAM is supposed to be an indication that everything about the
stream is gone or going away. A sender of a RST_STREAM may not deliver data
pending on that stream up to the application, so I don't think it makes
sense to design an app protocol that doesn't send a RST in response to a
RST.

But this brings up an interesting issue: Here we have transport behavior
that requires a particular application behavior, which I'm not a fan of. It
may make sense to require the transport to send a RST in response to a RST,
with a special error code that means "sent in response to a received RST".
That ought to make the semantics of RST clear to apps.

On Thu, Sep 7, 2017 at 4:25 PM, Subodh Iyengar <[email-masked]> wrote:

> If an app wanted to send both the stop sending and the reset stream, would
> it have to wait till stop sending was sent on the wire before sending reset
> streams? For example a API might reasonably be something like
>
>
> If the app calls reset, then the transport might dump all data associated
> with the stream immediately so the stop sending might never even be sent.
>
>
> ------------------------------
> *From:* QUIC <[email-masked]> on behalf of Jana Iyengar <
> [email-masked]>
> *Sent:* Thursday, September 7, 2017 12:02:27 PM
> *To:* Mike Bishop
> *Cc:* Philipp S. Tiesel; QUIC WG; Martin Thomson; Nick Harper; Victor
> Vasiliev
> *Subject:* Re: Split error codes in two
>
>

---

From: Martin Thomson <[email-masked]>
Date: Fri, 8 Sep 2017 09:52:03 +1000

> with the stream immediately so the stop sending might never even be sent.

STOP_SENDING is sent on another stream, one that won't be killed by the reset.

---

From: Subodh Iyengar <[email-masked]>
Date: Thu, 7 Sep 2017 23:54:23 +0000

> STOP_SENDING is sent on another stream, one that won't be killed by the reset.
Ah I missed that in the PR. Makes sense now.

Subodh

________________________________
From: Martin Thomson <[email-masked]>
Sent: Thursday, September 7, 2017 4:52:03 PM
To: Subodh Iyengar
Cc: Jana Iyengar; Mike Bishop; Philipp S. Tiesel; QUIC WG; Nick Harper; Victor Vasiliev
Subject: Re: Split error codes in two

---

From: "Lubashev, Igor" <[email-masked]>
Date: Fri, 8 Sep 2017 02:14:31 +0000

> I don't think it makes sense to design an app protocol that doesn't send a RST in response to a RST

Currently, the transport spec does not require RST_STREAM in response to RST_STREAM. It treats RST_STREAM like STREAM-with-FIN-except-don't-expect-transmittions (and it allows the receiver to drop buffered data not yet delivered to the application, although I would be very careful with that if the library API provides "peek" or "getAvailableDataSize" functions).

I can think of a number of applications for which send-data-after-receiving-RST is a natural design alternative. For example, you can have a print sever that talks to a printer via QUIC with a stream-per-job. Once the server-to-printer stream direction is closed (normally or job aborted via RST), the printer may wish to tell the server how many actual pages were printed by that job.

-----Original Message-----
From: Jana Iyengar [[email-masked]]
Received: Thursday, 07 Sep 2017, 7:43PM
To: Subodh Iyengar [[email-masked]]
CC: Mike Bishop [[email-masked]]; Philipp S. Tiesel [[email-masked]]; Victor Vasiliev [[email-masked]]; QUIC WG [[email-masked]]; Martin Thomson [[email-masked]]; Nick Harper [[email-masked]]
Subject: Re: Split error codes in two

So, a RST_STREAM is supposed to be an indication that everything about the stream is gone or going away. A sender of a RST_STREAM may not deliver data pending on that stream up to the application, so I don't think it makes sense to design an app protocol that doesn't send a RST in response to a RST.

But this brings up an interesting issue: Here we have transport behavior that requires a particular application behavior, which I'm not a fan of. It may make sense to require the transport to send a RST in response to a RST, with a special error code that means "sent in response to a received RST". That ought to make the semantics of RST clear to apps.

On Thu, Sep 7, 2017 at 4:25 PM, Subodh Iyengar <[email-masked]<mailto:[email-masked]>> wrote:

   transport->reset();

Subodh

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Jana Iyengar <[email-masked]<mailto:[email-masked]>>

---

From: Martin Thomson <[email-masked]>
Date: Fri, 8 Sep 2017 12:50:34 +1000

On Fri, Sep 8, 2017 at 9:42 AM, Jana Iyengar <[email-masked]> wrote:
> I don't think it makes sense to design an app protocol that doesn't send a
> RST in response to a RST.

I've been told not to invoke this particular demon, but you just
invoked the Unidirectional streams problem.  I think that we get there
because of this meme that says that data in the one direction is
somehow necessarily bound to data in the other direction.  That's an
entirely constructed notion.  A useful construct at times, certainly,
but that's not the point here.

So I disagree.  There are many protocols in which you send messages
(== streams) in one direction but not another.  One of the ways you
get into a unidirectional state in the current draft is to end one
side.  A FIN is only one way to do that, a unidirectional RST can be
faster and even superior in other ways.  The HTTP use case clearly
illustrates that.

Also, as Igor observes, we don't require a reciprocal RST_STREAM in
the current draft.

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 8 Sep 2017 17:33:57 -0700

> but that's not the point here.

I think that's the most common use of a RST -- to close both directions.

> faster and even superior in other ways.  The HTTP use case clearly

FIN is the primary way in which half-close is achieved, which is what
you're talking about. The HTTP use case is a complete one-off --
STOP_SENDING is an odd enough use case that I'm always left wondering who
actually uses it.

> Also, as Igor observes, we don't require a reciprocal RST_STREAM in

Yes, we don't. Recall that the RST_STREAM was, prior to the change that
doesn't require reciprocity, a bidirectional signal, which was simplified
to a unidirectional one, with the general expectation that apps would close
the other side when they receive this signal from QUIC. It's surely going
to be true of HTTP. It's possible for an app to receive a RST and then keep
sending, but that is a use case I haven't seen in practice.

---

From: Mike Bishop <[email-masked]>
Date: Wed, 13 Sep 2017 00:04:20 +0000

Jana and I were discussing this today, and part of the difference comes down to how you conceive of the API a QUIC implementation will expose.  My mental model has been that a stream/stream-pair consists of an InputStream and an OutputStream; you can read on one, you can write on the other.  For the write stream, you can write your data to completion and close the stream, or you can decide to abort writing.  The wire expressions of these are FIN and RST_STREAM, respectively.  For the read stream, you can read the incoming data to the end or you can abort reading and discard the remainder of the stream; the wire expression of this would be flow control updates and STOP_SENDING, respectively.  On the other side, receipt of a RST_STREAM surfaces as a read error and receipt of a STOP_SENDING surfaces as a write error.  (Jana commented that in this mental model it might be clearer to call the frames WRITE_ABORT and READ_ABORT.)

If we take Martin’s proposition that streams only close by application action, there would never be a write error surfaced to the application unless the underlying transport went away entirely.  Instead, it’s the responsibility of the application protocol to communicate (on another stream, in this case) that you should abort writing, even if you’ve already written to completion, and then to do so.  That implies that the application needs the ability to return to streams on which it’s done writing, in case it’s later asked to go back and reset them.

Much of our discussions around unidirectional streams, stream state transitions, etc. including the question of whether STOP_SENDING belongs at the transport layer, stem from the fact that we all have different implicit mental models of the functional surface QUIC exposes.  Perhaps we should take a step back and agree on that first, as I believe Christian has suggested in the past?

From: Jana Iyengar [mailto:[email-masked]]
Sent: Friday, September 8, 2017 5:34 PM
To: Martin Thomson <[email-masked]>
Cc: Subodh Iyengar <[email-masked]>; Mike Bishop <[email-masked]>; Philipp S. Tiesel <[email-masked]>; QUIC WG <[email-masked]>; Nick Harper <[email-masked]>; Victor Vasiliev <[email-masked]>
Subject: Re: Split error codes in two

On Thu, Sep 7, 2017 at 7:50 PM, Martin Thomson <[email-masked]<mailto:[email-masked]>> wrote:

FIN is the primary way in which half-close is achieved, which is what you're talking about. The HTTP use case is a complete one-off -- STOP_SENDING is an odd enough use case that I'm always left wondering who actually uses it.

Yes, we don't. Recall that the RST_STREAM was, prior to the change that doesn't require reciprocity, a bidirectional signal, which was simplified to a unidirectional one, with the general expectation that apps would close the other side when they receive this signal from QUIC. It's surely going to be true of HTTP. It's possible for an app to receive a RST and then keep sending, but that is a use case I haven't seen in practice.

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 13 Sep 2017 04:23:35 +0000

This is a very good idea to decide what we want to model streams on.  Starting with well-known, simple abstractions seems like the best path to adoption.

There are two such abstractions that come to mind:

  1.  “TCP Stream sockets”.  These only support signaling “orderly” closing of the write direction.  Closing of the read direction can only be done via close(), but that may leave the write direction in an undefined state (writing will complete, unless there was unread data in the read buffer or any data is received after close – see RFC 1122 4.2.2.13 $3).

I think the current spec allows for QUIC API to emulate TCP stream socket abstraction.  Current spec  also adds signaling “abort” closing of the write direction (RST_STREAM does not require an immediate RST_STREAM in the opposite direction), although there was some talk of removing this feature.

  1.  “A POSIX pipe pair”, as Mike is proposing (one pair in each direction).  Connected pipes are very clean (‘symmetric’) abstractions, and they communicate close events (you close the ‘write’ pipe, ‘read’ pipe get EOS; you close the ‘read’ pipe, ‘write’ pipe gets an exception).

With read-direction close signaling, however, it is important to not to cause often-unnecessary STOP_SENDING frames sent for every common-case stream.close(). I think that could be done via one of the following ways:

  1.  Unidirectional streams. Send STOP_SENDING on close() by the receiver (if the steam is open).
  2.  API convention: “only send STOP_SENDING on stream.inputStream().close()”. On stream.close(), only send STOP_SENDING if there is unread data or new data comes in later.
  3.  Actually signal that read direction is closed in stream.close() by defining STREAM-with-FIN frame as “close both write and read directions” (common case).  Have separate frame types for WRITE_CLOSE, WRITE_ABORT (aka RST_STREAM), and READ_ABORT (aka STOP_SENDING).

From: Mike Bishop [mailto:[email-masked]]
Sent: Tuesday, September 12, 2017 8:04 PM
To: Jana Iyengar <[email-masked]>; Martin Thomson <[email-masked]>
Cc: Philipp S. Tiesel <[email-masked]>; Subodh Iyengar <[email-masked]>; Nick Harper <[email-masked]>; QUIC WG <[email-masked]>; Victor Vasiliev <[email-masked]>
Subject: RE: Split error codes in two

From: Jana Iyengar [mailto:[email-masked]]
Sent: Friday, September 8, 2017 5:34 PM
To: Martin Thomson <[email-masked]<mailto:[email-masked]>>
Cc: Subodh Iyengar <[email-masked]<mailto:[email-masked]>>; Mike Bishop <[email-masked]<mailto:[email-masked]>>; Philipp S. Tiesel <[email-masked]<mailto:[email-masked]>>; QUIC WG <[email-masked]<mailto:[email-masked]>>; Nick Harper <[email-masked]<mailto:[email-masked]>>; Victor Vasiliev <[email-masked]<mailto:[email-masked]>>
Subject: Re: Split error codes in two

---

From: Christian Huitema <[email-masked]>
Date: Tue, 12 Sep 2017 21:56:37 -0700

There is another abstraction -- streams as messages. Application pushes
a message that is the entire content of the stream, and receives a
message once the entire stream has been received. That's the model used
by DNS over QUIC.

On 9/12/2017 9:23 PM, Lubashev, Igor wrote:
>
> This is a very good idea to decide what we want to model streams on. 
> Starting with well-known, simple abstractions seems like the best path
> to adoption.
>
>  
>
>  1. “TCP Stream sockets”.  These only support signaling “orderly”
>     closing of the write direction.  Closing of the read direction can
>     only be done via close(), but that may leave the write direction
>     in an undefined state (writing will complete, unless there was
>     unread data in the read buffer or any data is received after close
>     – see RFC 1122 4.2.2.13 $3).
>
> I think the current spec allows for QUIC API to emulate TCP stream
> socket abstraction.  Current spec  also adds signaling “abort” closing
> of the write direction (RST_STREAM does not require an immediate
> RST_STREAM in the opposite direction), although there was some talk of
> removing this feature.
>
>  
>
>  2. “A POSIX pipe pair”, as Mike is proposing (one pair in each
>     direction).  Connected pipes are very clean (‘symmetric’)
>     abstractions, and they communicate close events (you close the
>     ‘write’ pipe, ‘read’ pipe get EOS; you close the ‘read’ pipe,
>     ‘write’ pipe gets an exception).
>
>  
>
> With read-direction close signaling, however, it is important to not
> to cause often-unnecessary STOP_SENDING frames sent for every
> common-case stream.close(). I think that could be done via one of the
> following ways:
>
>  2. API convention: “only send STOP_SENDING on
>     stream.inputStream().close()”. On stream.close(), only send
>     STOP_SENDING if there is unread data or new data comes in later.
>  3. Actually signal that read direction is closed in stream.close() by
>     defining STREAM-with-FIN frame as “close both write and read
>     directions” (common case).  Have separate frame types for
>     WRITE_CLOSE, WRITE_ABORT (aka RST_STREAM), and READ_ABORT (aka
>     STOP_SENDING).
>
>  
>
>  
>
> *From:* Mike Bishop [mailto:[email-masked]]
> *Sent:* Tuesday, September 12, 2017 8:04 PM
> *To:* Jana Iyengar <[email-masked]>; Martin Thomson
> <[email-masked]>
> *Cc:* Philipp S. Tiesel <[email-masked]>; Subodh Iyengar
> <[email-masked]>; Nick Harper <[email-masked]>; QUIC WG
> <[email-masked]>; Victor Vasiliev <[email-masked]>
> *Subject:* RE: Split error codes in two
>
>  
>
> Jana and I were discussing this today, and part of the difference
> comes down to how you conceive of the API a QUIC implementation will
> expose.  My mental model has been that a stream/stream-pair consists
> of an InputStream and an OutputStream; you can read on one, you can
> write on the other.  For the write stream, you can write your data to
> completion and close the stream, or you can decide to abort writing. 
> The wire expressions of these are FIN and RST_STREAM, respectively. 
> For the read stream, you can read the incoming data to the end or you
> can abort reading and discard the remainder of the stream; the wire
> expression of this would be flow control updates and STOP_SENDING,
> respectively.  On the other side, receipt of a RST_STREAM surfaces as
> a read error and receipt of a STOP_SENDING surfaces as a write error. 
> (Jana commented that in this mental model it might be clearer to call
> the frames WRITE_ABORT and READ_ABORT.)
>
>  
>
> If we take Martin’s proposition that streams only close by application
> action, there would never be a write error surfaced to the application
> unless the underlying transport went away entirely.  Instead, it’s the
> responsibility of the application protocol to communicate (on another
> stream, in this case) that you should abort writing, even if you’ve
> already written to completion, and then to do so.  That implies that
> the application needs the ability to return to streams on which it’s
> done writing, in case it’s later asked to go back and reset them.
>
>  
>
> Much of our discussions around unidirectional streams, stream state
> transitions, etc. including the question of whether STOP_SENDING
> belongs at the transport layer, stem from the fact that we all have
> different implicit mental models of the functional surface QUIC
> exposes.  Perhaps we should take a step back and agree on that first,
> as I believe Christian has suggested in the past?
>
>  
>
> *From:* Jana Iyengar [mailto:[email-masked]]
> *Sent:* Friday, September 8, 2017 5:34 PM
> *To:* Martin Thomson <[email-masked]
> <mailto:[email-masked]>>
> *Cc:* Subodh Iyengar <[email-masked] <mailto:[email-masked]>>; Mike
> Bishop <[email-masked]
> <mailto:[email-masked]>>; Philipp S. Tiesel
> <[email-masked] <mailto:[email-masked]>>; QUIC WG <[email-masked]
> <mailto:[email-masked]>>; Nick Harper <[email-masked]
> <mailto:[email-masked]>>; Victor Vasiliev <[email-masked]
> <mailto:[email-masked]>>
> On Thu, Sep 7, 2017 at 7:50 PM, Martin Thomson
> <[email-masked] <mailto:[email-masked]>> wrote:
>
> STOP_SENDING is an odd enough use case that I'm always left wondering
> who actually uses it.
>
>  
>
> Yes, we don't. Recall that the RST_STREAM was, prior to the change
> that doesn't require reciprocity, a bidirectional signal, which was
> simplified to a unidirectional one, with the general expectation that
> apps would close the other side when they receive this signal from
> QUIC. It's surely going to be true of HTTP. It's possible for an app
> to receive a RST and then keep sending, but that is a use case I
> haven't seen in practice.
>
>  
>

-- 
Christian Huitema

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 13 Sep 2017 05:27:19 +0000

This can be accomplished in either "tcp socket" or "pipe pairs" model, given a sufficiently large buffer and an ability of the API to poll on "EOS received" before consuming all buffered bytes (not normally a part of traditional APIs).

-----Original Message-----
From: Christian Huitema [[email-masked]]
Received: Wednesday, 13 Sep 2017, 12:57AM
To: [email-masked] [[email-masked]]
Subject: Re: Split error codes in two

There is another abstraction -- streams as messages. Application pushes a message that is the entire content of the stream, and receives a message once the entire stream has been received. That's the model used by DNS over QUIC.

From: Mike Bishop [mailto:[email-masked]]
Sent: Tuesday, September 12, 2017 8:04 PM
To: Jana Iyengar <[email-masked]><mailto:[email-masked]>; Martin Thomson <[email-masked]><mailto:[email-masked]>
Cc: Philipp S. Tiesel <[email-masked]><mailto:[email-masked]>; Subodh Iyengar <[email-masked]><mailto:[email-masked]>; Nick Harper <[email-masked]><mailto:[email-masked]>; QUIC WG <[email-masked]><mailto:[email-masked]>; Victor Vasiliev <[email-masked]><mailto:[email-masked]>
Subject: RE: Split error codes in two

--
Christian Huitema

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 13 Sep 2017 14:40:16 +0000

Thinking about this a bit more, it seems that the key difference between "tcp socket" and "two pipe pairs" models is independence of the send and receive directions.

In "two pipe pairs", send and receive directions are completely independent (they are just two unidirectional streams with some syntactic sugar that allows them to be opened and closed together, if needed).  To be faithful to this model, we'd need to support each direction getting closed independently by either endpoint.

For "tcp socket", the two directions are not independent, and streams in both directions are reset if some data sent in either direction cannot be delivered (including when the receiver decided it does not want that data).  To be faithful to this model, we'd need transport to send RST_STREAM in response to RST_STREAM automatically, not because it needs to reset the other direction (the other direction is already semantically reset) but just for flow control accounting.  And there is also no STOP_SENDING, as this is just asking for a complete reset.

I am not yet seeing what "middle ground" can be here, while maintaining a coherent model/philosophy behind the choices.

From: Lubashev, Igor [mailto:[email-masked]]
Sent: Wednesday, September 13, 2017 1:27 AM
To: [email-masked]; [email-masked]
Subject: RE: Split error codes in two

-----Original Message-----
From: Christian Huitema [[email-masked]]
Received: Wednesday, 13 Sep 2017, 12:57AM
To: [email-masked]<mailto:[email-masked]> [[email-masked]]
Subject: Re: Split error codes in two

  1.  "TCP Stream sockets".  These only support signaling "orderly" closing of the write direction.  Closing of the read direction can only be done via close(), but that may leave the write direction in an undefined state (writing will complete, unless there was unread data in the read buffer or any data is received after close - see RFC 1122 4.2.2.13 $3).

I think the current spec allows for QUIC API to emulate TCP stream socket abstraction.  Current spec  also adds signaling "abort" closing of the write direction (RST_STREAM does not require an immediate RST_STREAM in the opposite direction), although there was some talk of removing this feature.

  1.  "A POSIX pipe pair", as Mike is proposing (one pair in each direction).  Connected pipes are very clean ('symmetric') abstractions, and they communicate close events (you close the 'write' pipe, 'read' pipe get EOS; you close the 'read' pipe, 'write' pipe gets an exception).

With read-direction close signaling, however, it is important to not to cause often-unnecessary STOP_SENDING frames sent for every common-case stream.close(). I think that could be done via one of the following ways:
a.       Unidirectional streams. Send STOP_SENDING on close() by the receiver (if the steam is open).
b.       API convention: "only send STOP_SENDING on stream.inputStream().close()". On stream.close(), only send STOP_SENDING if there is unread data or new data comes in later.
c.       Actually signal that read direction is closed in stream.close() by defining STREAM-with-FIN frame as "close both write and read directions" (common case).  Have separate frame types for WRITE_CLOSE, WRITE_ABORT (aka RST_STREAM), and READ_ABORT (aka STOP_SENDING).

If we take Martin's proposition that streams only close by application action, there would never be a write error surfaced to the application unless the underlying transport went away entirely.  Instead, it's the responsibility of the application protocol to communicate (on another stream, in this case) that you should abort writing, even if you've already written to completion, and then to do so.  That implies that the application needs the ability to return to streams on which it's done writing, in case it's later asked to go back and reset them.

--

Christian Huitema

---

From: Roberto Peon <[email-masked]>
Date: Wed, 13 Sep 2017 19:03:33 +0000

So many RPC models have failed utterly because they couldn’t handle receiving the data in a streaming fashion.

-=R

From: QUIC <[email-masked]> on behalf of Christian Huitema <[email-masked]>
Date: Tuesday, September 12, 2017 at 9:57 PM
To: "[email-masked]" <[email-masked]>
Subject: Re: Split error codes in two

--

Christian Huitema

---

From: Martin Thomson <[email-masked]>
Date: Mon, 25 Sep 2017 14:03:10 +1000

On Thu, Sep 14, 2017 at 12:40 AM, Lubashev, Igor <[email-masked]> wrote:
> Thinking about this a bit more, it seems that the key difference between
> “tcp socket” and “two pipe pairs” models is independence of the send and
> receive directions.

I prefer Christian's analogy of streams as messages.  That's what
motivated the design that I proposed.  Then you don't get "pipe pairs"
you get "messages and (optionally) response messages".

On Thu, Sep 14, 2017 at 5:03 AM, Roberto Peon <[email-masked]> wrote:
> So many RPC models have failed utterly because they couldn’t handle
> receiving the data in a streaming fashion.

That doesn't invalidate the streams-are-messages model, it just
recognizes that the messages cannot be sent and received as a single
unit (unless they are 0 length, I guess).  Flow control, multiplexing,
etc, etc...

~

I want to come back to the original question in light of what we learned:

 -> what is the transport going to **do** with a STOP_SENDING?

I can't think of anything other than to tell the application.  That
makes the answer pretty clear to me.

I don't think that this discussion - valuable as it is been - has
changed my position on this much.  Ideally I would like to enable all
the various viewpoints on the question to be equally valid.  Building
a protocol that can provide something that looks like a TCP socket is
challenging though, because the complete close of a coupled pair
cannot be done generically - I think that we all agree that this is
impossible in the general case.

I can appreciate the desire to build some of the pieces that might be
needed to build a bidirectional close, but I think that is more
dangerous than useful and would prefer to leave this to application
protocols.

---

From: "Lubashev, Igor" <[email-masked]>
Date: Mon, 25 Sep 2017 23:10:37 +0000

I agree that in many cases, "streams-as-messages" model fits well in the application use case (not HTTP Stream 1, though).  It seems reasonable to suppose that an application would occasionally like to send a very large (100MB?) message, and it gives it to the transport in one shot -- maybe using something like Linux sendfile() API call.

>   -> what is the transport going to **do** with a STOP_SENDING?

The answer seems orthogonal to bidirectionality.  It would make sense for the transport to stop sending the message and notify the application that it did so.  The alternative is to continue sending the large file, spending significant network and CPU resources doing so, and hoping the application would be prompt at receiving the notification and asking the transport to cancel the transfer.  Both alternatives achieve the same semantics: "an app is notified that that the receiver refused to receive this message in its entirety".  The former option allows saving network and systems resources on the protocol level.

In the "pipe" model (unidirectional or bidirectional), this is a question of whether only the sender is allowed to close the pipe, or the receiver is also allowed to close.

-----Original Message-----
From: Martin Thomson [mailto:[email-masked]] 
Sent: Monday, September 25, 2017 12:03 AM
To: Roberto Peon <[email-masked]>
Cc: [email-masked]; Christian Huitema <[email-masked]>
Subject: Re: Split error codes in two

On Thu, Sep 14, 2017 at 12:40 AM, Lubashev, Igor <[email-masked]> wrote:
> Thinking about this a bit more, it seems that the key difference 
> between “tcp socket” and “two pipe pairs” models is independence of 
> the send and receive directions.

I prefer Christian's analogy of streams as messages.  That's what motivated the design that I proposed.  Then you don't get "pipe pairs"
you get "messages and (optionally) response messages".

> receiving the data in a streaming fashion.

That doesn't invalidate the streams-are-messages model, it just recognizes that the messages cannot be sent and received as a single unit (unless they are 0 length, I guess).  Flow control, multiplexing, etc, etc...

~

I can't think of anything other than to tell the application.  That makes the answer pretty clear to me.

I don't think that this discussion - valuable as it is been - has changed my position on this much.  Ideally I would like to enable all the various viewpoints on the question to be equally valid.  Building a protocol that can provide something that looks like a TCP socket is challenging though, because the complete close of a coupled pair cannot be done generically - I think that we all agree that this is impossible in the general case.

I can appreciate the desire to build some of the pieces that might be needed to build a bidirectional close, but I think that is more dangerous than useful and would prefer to leave this to application protocols.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 26 Sep 2017 09:35:06 +1000

We've drifted dangerously close to defining an API.

You are suggesting an upcall that says "I've received a WRITE_ABORT on
stream X.  (optionally: I've stopped sending stream X.)  Should I (a)
reset, or (b) continue?"

Alternatively, you could configure the answer on streams with
defaults, or when you start writing.

That's a pretty well-defined notion in the abstract, but if we had a
transport-layer message, every generic QUIC stack would need to have
one or other mechanism.

---

From: Mike Bishop <[email-masked]>
Date: Mon, 25 Sep 2017 23:43:55 +0000

Yes, we have -- which is why I suggested we go ahead and define one, albeit perhaps just a reference API.  I'm not convinced an upcall is necessary -- swallowing unsent data and notifying the app if it attempts to send more is probably sufficient.  But the distinction between the two *would* be an API design decision.

And I would argue that if a generic QUIC stack needs to have a mechanism to transfer multi-gigabyte messages, it also needs to have a way to forestall reading the entirety of those messages.  (Although we already do have the ability to choke off flow control until the application realizes it's supposed to RST_STREAM, I suppose.)

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Martin Thomson
Sent: Monday, September 25, 2017 4:35 PM
To: Lubashev, Igor <[email-masked]>
Cc: [email-masked]; huitema <[email-masked]>; Roberto Peon <[email-masked]>
Subject: Re: Split error codes in two

You are suggesting an upcall that says "I've received a WRITE_ABORT on stream X.  (optionally: I've stopped sending stream X.)  Should I (a) reset, or (b) continue?"

Alternatively, you could configure the answer on streams with defaults, or when you start writing.

That's a pretty well-defined notion in the abstract, but if we had a transport-layer message, every generic QUIC stack would need to have one or other mechanism.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 26 Sep 2017 09:53:50 +1000

I think that you just managed to change my mind...

> Yes, we have -- which is why I suggested we go ahead and define one, albeit perhaps just a reference API.  I'm not convinced an upcall is necessary -- swallowing unsent data and notifying the app i…

Actually, that would be a protocol decision.  STOP_SENDING could cause
a state transition and cause the stream to become reset, but it
creates some nasty side effects.

> And I would argue that if a generic QUIC stack needs to have a mechanism to transfer multi-gigabyte messages, it also needs to have a way to forestall reading the entirety of those messages.  (Alth…

This is the point that I think is convincing.  Ignoring what we might
ask an application layer to do (or not), operating purely at the
transport layer, if the reader closes, then it won't send flow control
credits.

That is, it could (and direct the pipe to /dev/null), but I don't
think that we've provided the right incentives.  Redirecting to
/dev/null is costly for both sides, so I see little reason for them to
do that.  The result is therefore most likely a deadlock.

That's the clincher for me.  If the transport at the reader doesn't
read, then the writer can't write and we end up with streams that are
deadlocked until the application resolves it.  I don't mind if the
application wants to insert itself (the upcall suggestion), but I
think that a transport-layer function forces the transport to
recognize this problem and deal with it.