Subject: Re: [hybi] Proposal: require non-final websocket frames to have a minimum payload length of X bytes

From: William Budd <[email-masked]>
Date: Fri, 21 Jun 2013 14:58:38 +0900

Peter Thorson wrote:
> I would argue that a WebSocket server that dropped connections attempting
> to send excessive numbers of fragments (for some application defined value
> of excessive) or fragments that were too small (for some application defined
> value of small) with a private code or code 1008/Policy Violation would be
> compliant with RFC6455 without any modifications.

That would be nice, but I'm not sure I agree with that. Quoting the
relevant section of RFC 6455:

  "Unless specified otherwise by an extension, frames have no semantic
   meaning.  An intermediary might coalesce and/or split frames, if no
   extensions were negotiated by the client and the server or if some
   extensions were negotiated, but the intermediary understood all the
   extensions negotiated and knows how to coalesce and/or split frames
   in the presence of these extensions.  One implication of this is that
   in absence of extensions, senders and receivers must not depend on
   the presence of specific frame boundaries."

Assume a server considers a certain amount of fragments per message
excessive, and documents this fact as part of the API description on
its website. It's clients take note and make sure that they don't
fragment beyond the documented threshold.

Despite the best of intentions by both endpoints, the connections can
still fail here in unpredictable ways, because any intermediaries
(otherwise uninvolved 3rd parties) are free to fragment messages
further as per RFC 6455. The consensus between endpoints becomes
irrelevant because the intermediate didn't know about the agreement
and really had no reason to care anyway. In that sense, senders and
receivers are in fact "depending here on the presence of specific
frame boundaries", and thus in violation of the spec quoted above.
(Please correct me if my interpretation is misguided.)

Zhong Yu wrote:
> But why would empty frames cause any trouble for the receiver?
> Can't the receiver simply discard them?

Let me repeat just in case: I'm only referring to empty frames that do
*not* have the FIN bit set. In other words, these are frames that
specify that the message is incomplete and will continue in an
upcoming continuation frame. So why can't I simply discard those?
Because as long as I haven't yet received any (or enough) of the
payload of said message, I can't yet determine whether the payload
will turn out to be valid or not from the p.o.v. off my application.
To discard the message merely on the basis of frame boundaries seems
to me like a violation of the spec ("senders and receivers must not
depend on the presence of specific frame boundaries."). Or at least
that's how I interpreted it.

Joakim Erdfelt wrote:
> alternative behavior would be for the Client to state that it supports fragment
> rules by simply saying
>  fragment
>and the server responding with the rules it expects
>   fragment; min=0; max=1234567
>
> thoughts?

I think this would be an excellent idea, actually. My "min. size for
non-final fragments idea" might be less work to implement, but if it
turns out to be too controversial or problematic in some use cases,
I'd rally behind this. Perhaps this kind of negotiation during
handshake would be best to keep everyone happy.

That said, I'm again not sure what the role of intermediaries would be
here. What should happen if intermediaries don't like the values
negotiated by the endpoints?

Joakim Erdfelt wrote:
> The library however has no prior knowledge of the size of the message, so the
> flush could occur, then the user application determines it is done streaming data
> and simply closes that message stream, resulting in a need for a final 0 payload
> length FIN frame.
Tobias Oberstein wrote:
> Yep. That's exactly what Autobahn does when using it's streaming API: complete
> the message by sending an empty FIN frame.

Sorry to repeat myself again, but my proposal would explicitly allow 0
payload length FIN frames. Reread the subject line of these emails?

Anyway, I appreciate you all for being open minded enough to (at
least) consider changes like these at this late stage of the game.

Best,
Will

---

From: Zhong Yu <[email-masked]>
Date: Fri, 21 Jun 2013 09:00:26 -0500

> To discard the message merely on the basis of frame boundaries seems

Hi William, I meant that these empty *frames* can be discarded by server.

My understanding is that you want one message correspond to one frame
and vice versa in your application where all messages are tiny, this
will make coding a little easier. I think that you have to bite the
bullet and handle the complexity of buffering fragmented messages:)

Zhong Yu

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: William Budd <[email-masked]>
Date: Sat, 22 Jun 2013 00:01:32 +0900

> bullet and handle the complexity of buffering fragmented messages:)

Ah, sorry, I see. Right, of course I can discard those frames, but we
all know that doing so isn't free.

It requires a couple of CPU instructions to parse the extra frames.
The likelihood of a read buffer containing partial payloads goes up.
State machines become slightly larger. Code complexity increases a
bit. More per-client payload buffers means more cache misses.
Etcetera. None of these things are a terribly big deal if there really
is a good reason to be doing so, but the thing is; it is not at all
clear to me that there is such a reason.

As of yet nobody has replied with a reason (as far as I can tell) why
anyone would want to send any non-FIN fragment of less than 125 bytes
(other than for malicious purposes). Especially so since the websocket
protocol is designed to allow control frames (with arbitrary payloads
of up to 125 bytes) to be inserted in between fragments whenever
needed. And if there is no reason to do so, why not make that
threshold official and make the job of implementers out there a little
easier, and allow his or her software to run a little faster? (And I
address this question not to you only, but to everyone reading this
email).

Apologies for the --vverbosity,
Will

---

From: Tobias Oberstein <[email-masked]>
Date: Fri, 21 Jun 2013 08:18:13 -0700

> As of yet nobody has replied with a reason (as far as I can tell) why anyone
> would want to send any non-FIN fragment of less than 125 bytes (other than for
> malicious purposes). Especially so since the websocket protocol is designed to
> allow control frames (with arbitrary payloads of up to 125 bytes) to be inserted
> in between fragments whenever needed. And if there is no reason to do so, why

Sending of control frames might not be exposed in API, but an implementation exposing a frame-based API can allow apps to do app-level keep-alive by sending empty payload frames (I am not saying that this is a terrible good idea, but it shows that it restricts what can be done ..)

> not make that threshold official and make the job of implementers out there a

Because it's
a) arbitrary (why 125?) or requires negotiation and
b) an implementation with frame-based (or streaming) API might not want to buffer up stuff up to 125 before it sends out a frame.

> little easier, and allow his or her software to run a little faster? (And I address
> this question not to you only, but to everyone reading this email).

It makes implementations more complex.

It's one thing to forbid empty payload non-FIN frames and another to require an arbitrary minimal payload size. 

/Tobias

---

From: Peter Thorson <[email-masked]>
Date: Fri, 21 Jun 2013 10:21:14 -0500

> Apologies for the --vverbosity,

I see two cases where framing related limitations come into play.

1. An endpoint that needs to apply limitations based on the type of application is is running. For example a service that processed large volumes of short text messages might want to choose a fixed message buffer size for more efficient operation and issue a 1008 error if a larger message came in.

In this case of application specific limitations I think informal consensus between endpoints or explicitly defined extensions detailing the requirements and/or allowing negotiation of them are both excellent solutions. I don't think defining these types of limitations makes sense in the core protocol as the limitations and their reasoning is very much time and application specific. A non-malicious reason for sending small frames would be a case of streaming API that wanted to send character by character "as you type" data as is commonly done in chat programs or interactive terminals.

As you have pointed out above, one limitation of informal consensus is the ability for intermediaries to gum things up. Extensions would get around that, as the rules stated in RFC6455 only apply when the intermediary understands all extensions being used, perhaps not from a technical standpoint, as the intermediary may still not be able to work under the given constraints but the reason reported to the endpoints about what is wrong would be more clear.

2. Security against resource exhaustion / DoS attacks. Examples: Very large frames (i.e. frames too large for any endpoint to reasonably buffer in memory), excessive numbers of small or zero length frames, etc. In these cases you cannot rely on the remote endpoint or any intermediaries to behave according to a specification. No matter what the spec says, your server will need to unilaterally impose security policies about the various limits that it can handle without regard to what is officially allowed. Adding such a requirement to the official specification would not prevent implementers from having to deal with small frames in malicious situations.

---

From: William Budd <[email-masked]>
Date: Sat, 22 Jun 2013 01:27:38 +0900

> A non-malicious reason for sending small frames would be a case of streaming API that wanted to send character by character "as you type" data as is commonly done in chat programs or interactive terminals.

Any such input character frames really don't need to be fragmented
though. Just set the final bit on every frame, and treat them all as
separate messages, without incurring any performance penalty or other
downside. (If anything, it could be speed up the implementation,
because any layer unaware of the chat API might otherwise needlessly
check for continuations instead of just blindly handing it over to the
app.)

While you might argue that on some semantical level individual
characters don't make for separate messages, I don't believe such a
consideration has anything to do with why websocket fragmentation was
implemented in the first place, as the distinction is of no practical
consequence. As RFC 6455 points out, fragmentation in websockets is
merely a mechanism intended to make large blobs of data easier to
handle.

> As you have pointed out above, one limitation of informal consensus is the ability for intermediaries to gum things up. Extensions would get around that, as the rules stated in RFC6455 only apply w…

True, but it seems to me that the elephant in the room with regard to
websocket extensions is that you can forget about your clients using
their browser of choice to connect to your server and have them
understand the extension, unless you're a huge company like Google
with the clout to influence said browsers to support doing so.

> 2. Security against resource exhaustion / DoS attacks. Examples: Very large frames (i.e. frames too large for any endpoint to reasonably buffer in memory), excessive numbers of small or zero length…

That's true as well, but I'd prefer to defend my server's policy by
saying "you violated the spec", rather than by saying "the spec allows
what you did, but it's actually still always a horrible thing to do,
and might have brought down my server if I hadn't refused it". :)

Will

---

From: William Budd <[email-masked]>
Date: Sat, 22 Jun 2013 01:40:41 +0900

> Sending of control frames might not be exposed in API, but an implementation exposing a frame-based API can allow apps to do app-level keep-alive by sending empty payload frames (I am not saying th…

Alright, but how would that work though? Since the empty payload frame
would be part of a larger fragmented message, I would expect any
response to that frame to come back no sooner than the response to the
entire message it's part of, given that it technically isn't a control
frame. In other words, wouldn't this "app-level keep-alive" better be
sent as a separate message?

> b) an implementation with frame-based (or streaming) API might not want to buffer up stuff up to 125 before it sends out a frame.

It's no more arbitrary than what's already in the spec. I'm
deliberately mirroring the maximum size of control frames in my
suggestion. That is, the spec says control frames can be no more than
125 bytes. I say, non-FIN fragment frames can be no less than 125
bytes. Their use cases are in many ways complimentary (and because 125
is the max value you can specify in the first pay load size byte).
Just trying to be practical and pragmatic here.

Will

---

From: Tobias Oberstein <[email-masked]>
Date: Fri, 21 Jun 2013 10:50:34 -0700

> > Sending of control frames might not be exposed in API, but an
> > implementation exposing a frame-based API can allow apps to do
> > app-level keep-alive by sending empty payload frames (I am not saying
> > that this is a terrible good idea, but it shows that it restricts what
> > can be done ..)
> 
> Alright, but how would that work though? Since the empty payload frame would
> be part of a larger fragmented message, I would expect any response to that
> frame to come back no sooner than the response to the entire message it's part

There is no response expected .. it's just about pushing some octets onto the wire to
keep the endpoints (e.g. radio) or any intermediaries from sending the TCP to sleep.

> of, given that it technically isn't a control frame. In other words, wouldn't this
> "app-level keep-alive" better be sent as a separate message?

An app that uses a frame-based API might not want to start a new message
just to do app-level keep alive.

> > b) an implementation with frame-based (or streaming) API might not want to
> buffer up stuff up to 125 before it sends out a frame.
> 
> It's no more arbitrary than what's already in the spec. I'm deliberately
> mirroring the maximum size of control frames in my suggestion. That is, the
> spec says control frames can be no more than
> 125 bytes. I say, non-FIN fragment frames can be no less than 125 bytes. Their

Obviously, "no more" is something different than "no less", and regarding data
messages, there is only 1 "arbitrary" limit in the spec: a data frame may have
a payload of at most 2^63 octets (which amounts to "infinity" in practice).

And what about b)?

At this point, my point of view is: your original worry about DoS is valid, but better
served via implementation knobs. Given that, I just fail to see value in imposing
a lower bound on non-FIN frame payload length. It seems to make implementations
more complex and restrict apps.
 
/Tobias

---

From: Zhong Yu <[email-masked]>
Date: Fri, 21 Jun 2013 15:26:06 -0500

A websocket message is not necessarily an atomic thing for
applications. Client and server can stream bytes within a message, and
if it's important to transmit bytes as soon as available, it's
necessary to support tiny frames.

I fail to see why tiny frame can cause any problem for the receiver.
If it's a problem, we should have heard of the same thing on other
protocols, for example, 1 byte segments in TCP. It does not seems to
be a very profitable attack vector.

Zhong Yu

---

From: "John A. Tamplin" <[email-masked]>
Date: Fri, 21 Jun 2013 17:06:59 -0400

> address this question not to you only, but to everyone reading this

I don't object to requiring any non-initial, non-final frame to have >0
length.  However, I do object to a minimum requirement above 1.

For example, let's say I have a streaming API, and 1 byte has been written
since the last time the buffer was flushed and a frame sent (whether full
or other reasons).  The client calls flush - if I can't send a 1-byte
frame, I have to ignore the client's request to flush the buffer, breaking
my guarantees.

-- 
John A. Tamplin

---

From: William Budd <[email-masked]>
Date: Sat, 22 Jun 2013 13:32:44 +0900

> since the last time the buffer was flushed and a frame sent (whether full or
> other reasons).  The client calls flush - if I can't send a 1-byte frame, I
> have to ignore the client's request to flush the buffer, breaking my
> guarantees.

Alright. Tobias, Zhong and you have convinced me that using websocket
fragmentation is an established way of implementing a streaming API,
and that requiring said minimum size would interfere with that.

I personally think you could just as well stream by sending separate
websocket messages. That is, whenever you want to flush the buffer,
just send whatever you have with the FIN flag set. The presence of the
FIN bit would then just indicate a flush. In that sense, new messages
would continue to arrive as part of the same stream.

The benefit of this approach is that anyone API handling large opaque
data could use the fragmentation mechanism in the intended ways
specified in RFC 6455 (note that streaming is *not* mentioned in the
spec as rationale for justifying fragmentation), while my suggestion
of minimum size would allow everyone who only handles small data and
doesn't need fragmentation to be totally oblivious of fragmentation
for their purposes.

Realistically though, I realize it's way too late to convince people
of how I think websocket fragmentation should be used or not be used.
These emails are simply a few years too late for that...

I still see potential though in what Joakim suggested earlier for a

>   fragment; min=0; max=1234567

Does anyone else like this idea? And if so, any thoughts on "what
should happen if intermediaries don't like the values
negotiated by the endpoints"?

Thanks,
Will

---

From: "John A. Tamplin" <[email-masked]>
Date: Sat, 22 Jun 2013 00:40:22 -0400

> would continue to arrive as part of the same stream.

Then to maintain the message-oriented WebSocket approach, you would have to
have another framing level above to say whether this was the last frame of
the message or not, since right now FIN means "the message is complete",
signal that to the WebSocket client.  If you now want FIN to mean "end of
this arbitrary buffer boundry" (which btw, the client doesn't actually care
to know), then you need to invent something new for the original meaning.

> These emails are simply a few years too late for that...

You have no idea how contentious it was to come up with the current framing
approach that nobody strongly objected to.  It took well over a year to get
agreement.

> negotiated by the endpoints"?

If it is something the intermediaries need to care about, it has to be an
extension -- otherwise, intermediaries are allowed to arbitrarily change
fragmentation to suit their needs.  If it is an extension, then it sounds
reasonable, though I don't know that there is enough value for there to be
enough implementations of it to be useful.

-- 
John A. Tamplin

---

From: William Budd <[email-masked]>
Date: Sat, 22 Jun 2013 14:28:37 +0900

> to know), then you need to invent something new for the original meaning.

I'd argue that what does or doesn't constitute the end of a message
should ideally be handled on the (web)application layer anyway, since
the semantics of a "message" are very much domain dependent. The
websocket layer only needs to care about sending frames back and fro.
In other words, I see websocket fragmentation as a traffic shaping
tool for intermediaries, rather than a means for endpoints to express
message semantics with (as that can be done with the API semantics of
your payload).

> approach that nobody strongly objected to.  It took well over a year to get

Actually, Google did provide me with some glimpses of that here and
there. If I had known of websockets back then I probably wouldn't have
hesitated to pour some more oil on that fire.

> enough implementations of it to be useful.

Yeah, you're probably right......... D:

thanks,
Will

---

From: "John A. Tamplin" <[email-masked]>
Date: Sat, 22 Jun 2013 08:24:44 -0400

> message semantics with (as that can be done with the API semantics of

So basically, you just want a streaming protocol that you can write
whatever bytes you like, and then the application decides what messages
are?  Well, that protocol already exists, and is just plain TCP.  Every app
would have to implement its own framing, and then connecting to a
third-party WebSocket server would require implementing that server's
protocol, which is going to have the same complications as writing each
protocol on top of TCP today.  One of the goals was to make it easy to send
messages from a simply JS client in the browser, and having to implement
all your own framing in JS doesn't sound like it meets that goal.   There
are also security issues involved in having potentially hostile JS
applications running in the browser (which itself might be behind a
firewall with access to sensitive resources) having direct access to TCP,
such as exploiting bugs in UTF8 decoders by sending arbitrary bytes to what
is intended to be a text-based endpoint.

Fragmentation is also required to support streaming without arbitrarily
large buffers.

Basically, it sounds like you want a different protocol from WebSockets,
rather than a simple modification to prevent a DoS attack.  I think you
should go back and read the archives of the discussions when WebSockets was
created to understand the rationale behind the decisions.   In any case, it
doesn't seem very likely to me that we would revisit the basic idea of what
WebSockets is at this point.

-- 
John A. Tamplin