Subject: Re: [hybi] Additional HTTP headers on upgrade request?

From: Ian Hickson <[email-masked]>
Date: Wed, 21 Jul 2010 00:57:19 +0000 (UTC)

On Fri, 7 May 2010, Doug Simpkinson wrote:
>
> In draft 76, the upgrade request is said to include cookies that may be 
> relevant, but no mention of other HTTP headers is made.
> 
> What about other relevant HTTP headers such as User-Agent and 
> Accept-Language?  Shouldn't these also be sent?

What's the use case?

If there are clear and important use cases that are best handled by adding 
new fields to the handshake, then we should add them.

> My concern is that if these headers are not explicitly mentioned in the 
> draft spec that they won't be sent by all implementations, and that will 
> make life more difficult for webmasters.

I've clarified the spec to make it clearer that these headers are not 
included.

On Sat, 8 May 2010, Greg Wilkins wrote:
> 
> the 76 pre draft actually exclude header other than those mentioned with 
> phrase:
> 
>   The only options available in this version are the ...
> 
> While I don't think this text is very clear, I did ask the editor if it 
> was meant to exclude headers not mentioned and he indicated that this 
> was the case.

The text above is non-normative, but the requirements are pretty explicit 
and do indeed not include any provision for including other fields.

On Sun, 9 May 2010, Doug Simpkinson wrote:
>
> Do you know what's the reasoning behind omitting any HTTP headers?
> What advantage is there to not sending User-Agent and Accept-Language?

That's the wrong question -- the question is, what's the advantage to 
sending them? The more we omit, the simpler the protocol is. The default 
should be to omit.

In the case of User-Agent, for instance, one could put forward analytics 
as a use case. However, for that use case it seems best for the 
application-layer protocol to opt into sending the information. That way 
the author is in complete control as to what is logged.

> It is quite common for different browsers to have different behavior, so 
> omitting User-Agent removes some flexibility in dealing with browser 
> quirks.

Having the spec be trivial to implement makes it less likely that there 
will be differences, making this moot.

In practice, browser sniffing has been as much of a problem as a help -- 
people sniff for a browser, then rely on that browser's behaviour, and 
when the browser vendor tries to fix the underlying problems, or tries to 
converge with other browsers, they find they cannot because pages send 
them different content than other browsers. This is why, for instance, 
every successful browser in the world claims to be "Mozilla".

On Mon, 10 May 2010, Maciej Stachowiak wrote:
> 
> Likewise for Accept-Language. I would be tempted say that language 
> selection could just be handled as part of subprotocol negotiation, but 
> at least currently JS in the browser does not have easy access to the 
> user's set of preferred languages.

Nor does the user, in many cases...

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 21 Jul 2010 02:12:21 +0100

> new fields to the handshake, then we should add them.

The use case for User-Agent, normally determined by the client
implementation (not the application running on top of the WebSocket
API), is to provide servers with unreliable but pragmatically useful
information about what client implementations are using the server.

It is used for profiling, for general interest, for tracking what
clients may need to be supported if there are particular issues to be
avoided with some clients (such as a particular message that breaks
one of them), for tracking when clients with particular workarounds on
the server have fallen into sufficient misuse that the workarounds can
be removed, and for implementing specific workarounds for particular
recognised clients.

Hopefully workarounds at the WebSocket framing level are unlikely to
be commonly needed (although that might not be true if some clients
have a small message size limit and particular server applications
have to format their messages to accomodate this); but User-Agent
turned out to be quite pragmatic in HTTP for dealing with
client-specific issues that nobody had foreseen until they were too
widely deployed to ignore or fix.

-- Jamie

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 21 Jul 2010 11:38:17 +1000

> > Accept-Language?  Shouldn't these also be sent?

Ian,

it is a little disingenuous of you to fall back on the "what's the use
case?" stone wall
on topics like this that have been long discussed and people have given many
use
cases at length.   You might not understand and/or agree with the use cases
presented,
but that is entirely different to pretending that others have not motivated
their contributions
to the discussion with good reasoning.

The result of continually asking "what is the use-case" when it has already
been explained
many times, is that the list just goes around and around in endless circles.

In this particular case, it has been long argued that arbitrary headers
should be allowed in
the handshake for flexibility and anticipation of future needs.   You've
refuted the need for
generally extensible header.  Now  Doug motivated his comment with the
specific
use-case of using user-agent for dealing with browser quirks.

Let's flip this around the otherway... because allowing arbitrary headers is
not really
a feature of websocket is not really a feature because before the upgrade
the connection
is HTTP and thus arbitrary headers is allowed.

In essence, you should be making the use-case for adding a "feature" to the
websocket
handshake to prevent the addition of arbitrary headers.

So what is the use-case for that?  I know you have previously argued that
somehow it
would be easier for amateur not to have to write code to ignore headers
(surely they
just don't write code to handle them?), but the amateur programmer
requirement has been
widely rejected.     Is there any other use-case that motivates the
rejection of websocket
handshakes due to the presence of additional ignorable headers?

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 21 Jul 2010 11:49:27 +1000

I'll try writing that again in fully formed English sentences this time
(sorry too much in a rush):

Ian,

Let's flip this around the otherway... because allowing arbitrary headers is
not really
a feature to be added to  websocket  because before the upgrade the
connection
is HTTP and thus arbitrary headers are by default allowed.

In essence, you are trying to add a "feature" to the websocket handshake to
prevent
the addition of arbitrary headers.

would be easier for amateurs not to have to write code to ignore headers

regards

On 21 July 2010 11:38, Greg Wilkins <[email-masked]> wrote:

> on topics like this that have been long discussed and people have given
> many use
> Let's flip this around the otherway... because allowing arbitrary headers
> is not really

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 21 Jul 2010 07:19:10 +0200

> be removed, and for implementing specific workarounds for particular

And it's used a lot to act differently with smartphones. Maybe that
could be used on the server side to try to group outgoing data into
fewer packets in order to improve user experience or reduce communication
costs.

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 21 Jul 2010 07:25:17 +0200

> use-case of using user-agent for dealing with browser quirks.

And I see a very important one concerning cookies. Cookies are important
to the server but also to the infrastructure. They're used a lot to
enable "persistence" or "stickiness", depending how it's called. They
make it possible for server-side load balancers to direct the users's
connection to the same server he was on previous connection. This is
of particular importance here because most often we'll need the WS
connection to reach the same server as the one which initiated a local
context. And for this it is important to be able to match reliable
content in the request. This is normally performed using cookies that
are assigned by either the server, the load balancer or sometimes even
some JS on the client.

This is one more reason to support standard headers with standard HTTP
in the handshake.

> handshakes due to the presence of additional ignorable headers?

Indeed, why break something which works well, is well understood, well
mastered and well debugged ?

Willy

---

From: Ian Hickson <[email-masked]>
Date: Wed, 21 Jul 2010 22:38:39 +0000 (UTC)

On Wed, 21 Jul 2010, Jamie Lokier wrote:
> avoided with some clients (such as a particular message that breaks one 
> of them), for tracking when clients with particular workarounds on the 
> server have fallen into sufficient misuse that the workarounds can be 
> removed, and for implementing specific workarounds for particular 
> recognised clients.
> 
> Hopefully workarounds at the WebSocket framing level are unlikely to be 
> commonly needed (although that might not be true if some clients have a 
> small message size limit and particular server applications have to 
> format their messages to accomodate this); but User-Agent turned out to 
> be quite pragmatic in HTTP for dealing with client-specific issues that 
> nobody had foreseen until they were too widely deployed to ignore or 
> fix.

This seems reasonable, but it also seems like something that we should 
just push to the application layer, and that we can push to the 
application layer with minimal incremental cost -- and indeed with some 
side-benefits, like making the handshake simpler for people who don't need 
this information, like preventing the problems that occured with HTTP's 
User-Agent header (such as everything saying they're Mozilla), and like 
giving authors more control over exactly what they want to check for (e.g. 
some might only care about the rendering engine, others might care only 
about the user's screen size, and by pushing it into the application layer 
we allow the subprotocol designers to decide what is important to them).

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 22 Jul 2010 00:04:24 +0100

> we allow the subprotocol designers to decide what is important to them).

I'm thinking of when the people writing the server have no
relationship with the people writing applications (so they can't fix
applications), and need to handle client implemention issues (not
application issues).

For example suppose Yahoo provided a WebSocket-based access to their
public YQL query system, and suppose ten thousand authors wrote client
applications to run in the major browsers, and most of those
applications were cut-and-pasted by other authors who used them
without understanding how they work.

Then suppose it was discovered that certain rare byte sequences or
other low-level thing triggered a fault in Firefox 4.2.1, due to a
Firefox implementation bug, fixed in 4.2.2.  (That's the sort of ugly
thing which HTTP User-Agent is used to work around).  And suppose
4.2.1 was widely distributed and despite 4.2.2's availability,
millions of users continued to use 4.2.1 for some years.

If that happened, it would be in Yahoo's interest to avoid the
problematic byte sequences somehow, as transparently as possible.
They couldn't alter the client applications to send the browser
version; issuing advice to authors would be slow to have effect, and
anyway many deployed clients would have been cut-and-pasted, so even
authors updating their code would be slow to have an effect.

So in the absence of anything like User-Agent, Yahoo would have to
implement some kind of dirty low-level workaround affecting all
clients, despite only being needed for Firefox 4.2.1.

I agree there's plenty of issues with User-Agent in the HTTP world;
it's no panacea.  But it has been pragmatically useful for working
around HTTP low-level issues, such as breaking at certain response
lengths, or when claiming to support a negotiated feature but actually
being broken so the offer has to be ignored.

I've no correct answer for this, only the observation that User-Agent
has been pragmatically helpful in HTTP, despite the "everything
pretends to be Mozzilla" .  However I think it's more relevant if the
handshake acquires feature negotiation, specifically to ignore feature
offers from WebSocket implementations that are found to have buggy
implementations of features.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 22 Jul 2010 00:13:08 +0100

> fewer packets in order to improve user experience or reduce communication

It would be nicer if the server is told what kind of communication
channel is being used, rather than what kind of device it's
communicating with.  It's not easy, e.g. if your laptop is going via
your phone's 3g, then its a 3g channel but the laptop probably doesn't
know it.  In the scheme of things it would be nice if proxies, such as
at mobile base stations, were permitted to annotate the handshake.
They _can_ do that with HTTP, using Via.  It's informal but that
doesn't stop it being useful - in principle.  I don't know if anyone
mobile system actually does that, or if any server uses the data.

We really don't want to overdesign WebSocket, but ought to be room for
certain particularly simple, maybe optional, things that have proven
pragmatically useful before.  I suspect User-Agent and Via fit into
that category, even though they are messy and unreliable.  But I'm not
really sure.

-- Jamie

---

From: Ian Hickson <[email-masked]>
Date: Thu, 22 Jul 2010 20:55:26 +0000 (UTC)

On Thu, 22 Jul 2010, Jamie Lokier wrote:
> > > clients may need to be supported if there are particular issues to 
> > > be avoided with some clients (such as a particular message that 
> > > breaks one of them), for tracking when clients with particular 
> > > workarounds on the server have fallen into sufficient misuse that 
> > > the workarounds can be removed, and for implementing specific 
> > > workarounds for particular recognised clients.
> > > 
> > application layer with minimal incremental cost -- and indeed with 
> > some side-benefits, like making the handshake simpler for people who 
> > don't need this information, like preventing the problems that occured 
> > with HTTP's User-Agent header (such as everything saying they're 
> > Mozilla), and like giving authors more control over exactly what they 
> > want to check for (e.g. some might only care about the rendering 
> > engine, others might care only about the user's screen size, and by 
> > pushing it into the application layer we allow the subprotocol 
> > designers to decide what is important to them).
> 
> I'm thinking of when the people writing the server have no relationship 
> with the people writing applications

Writing a server for Web Sockets is trivial enough that I really don't see 
why that needs to be the case.

> applications were cut-and-pasted by other authors who used them without 
> understanding how they work.
> 
> Then suppose it was discovered that certain rare byte sequences or other 
> low-level thing triggered a fault in Firefox 4.2.1, due to a Firefox 
> implementation bug, fixed in 4.2.2.  (That's the sort of ugly thing 
> which HTTP User-Agent is used to work around).  And suppose 4.2.1 was 
> widely distributed and despite 4.2.2's availability, millions of users 
> continued to use 4.2.1 for some years.
> 
> problematic byte sequences somehow, as transparently as possible. They 
> couldn't alter the client applications to send the browser version; 
> issuing advice to authors would be slow to have effect, and anyway many 
> deployed clients would have been cut-and-pasted, so even authors 
> updating their code would be slow to have an effect.
> 
> implement some kind of dirty low-level workaround affecting all clients, 
> despite only being needed for Firefox 4.2.1.

Sure, one can come up with hypothetical scenarios in both directions. 

For example, suppose that everything you describe was done, except that 
Yahoo! fixed the problem before Firefox 4.2.2 was released, and just 
implemented the fix by checking for Firefox instead of just 4.2.1. And 
further suppose that the fix actually requires doing something 
incompatible with what the spec actually says. (None of this is 
far-fetched, it's extremely common in fact.) Now we have a situation where 
Firefox can't update their implementation because doing so would break 
all thees thousands of Yahoo!-based applications.

I think in the end it turns out to be a wash, and the benefits of the 
features are equal to the downsides. In that kind of situation, it's best 
to keep the protocol simple and not have the feature, IMHO.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 22 Jul 2010 23:40:58 +0200

> why that needs to be the case.

Because that's how that happens in any enterprise. As soon as there
are developer teams, the work is split depending on various areas.
There are often teams working at various levels, some implementing
frameworks, other ones writing servers, and other ones writing
modular application components that are assembled to form applications.

Willy

---

From: John Tamplin <[email-masked]>
Date: Thu, 22 Jul 2010 18:03:05 -0400

> why that needs to be the case.

First, there appears to be significant disagreement over whether the server
should have to be trivial enough for an amateur to write in the first place.

Second, why write new code that duplicates existing code rather than use a
well-tested implementation?  Especially if things like keep-alives are left
out of the protocol, frameworks (for both client and server) will be written
to implement them, and the server is no longer trivial.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Simone Bordet <[email-masked]>
Date: Fri, 23 Jul 2010 00:57:29 +0200

Hi,

> why that needs to be the case.

I am wondering why application developers that know banking,
financial, etc should all of a sudden read the websocket protocol spec
to parse 0x00 and 0xFF and do UTF-8 decoding.
Unless you are assuming that they instead get exposed to a WebSocket
API provided by a library, like they are in the client with the
JavaScript WebSocket API. *That* would make their programming simpler.

But if this holds, then the server-side WebSocket library has to be
written carefully to handle basic features (like protocol decoding and
UTF-8 decoding), and maybe could even offer additional features and
this, I guess, is not in the capability range of an amateur
programmer, nor in the interests of an application programmer. So the
library will be written by a "decent programmer".

Then, maybe, we can make different decisions about the protocol that
do not involve "amateur programmers" anymore, for example having the
frame length expressed before the UTF-8 encoded message.

Simon
-- 
http://bordet.blogspot.com
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 26 Jul 2010 10:29:40 +1000

Ian,

It is not an issue of triviality.  It is an issue of security,
maintainability, traceability, software engineering, software reuse and
living in the real world.

I have never worked in an environment where it would be acceptable for an
application to crack open any port on the internet (let alone port 80) and
start sending/receiving arbitrary bytes.

These "in the future there will be no monolithic servers" and "everything
will be implemented by the same developers" arguments are adjuncts to the
"amateur programmers" one.  They've been responsible for much stonewalling
and endless iterations in this WG.     It's really great that you have a
vision for the future of web application development, but I do not see that
using the  websocket protocol as a vehicle to advance your architectural
agenda is appropriate.      If you want to advance your architectural ideas,
I suggest implementing some non-trivial solutions using them and then seeing
if others will follow your example.

I do not believe it is acceptable to have any solution in this space that
requires intimate implementation details to be shared between the server,
the server-side application and the client side application.   It is not
acceptable to continue to say things like: "they will know the binary frame
is B67 encoded ISO-12345-37 because the same developers will write the
client application, the server application and the server itself".