Subject: [http-state] non-ASCII cookie values (was Re: Closing Ticket 3: Public Suffixes)

From: Adam Barth <[email-masked]>
Date: Mon, 1 Feb 2010 10:14:17 -0800

On Sun, Jan 31, 2010 at 2:37 PM, eric bianchetti
<[email-masked]> wrote:
> That part does not please :
>
> The cookie-value is opaque to the user agent and MAY be anything the
>    origin server chooses to send, possibly in a server-selected
>    printable ASCII encoding.
>
> Livng and working in a non ASCII country, I tend to think we shall prepare for the coming of the other languages (Thai, Chines, Korean ....), IF a person get a cookie from a Thai server , can we securely suppose that person(computer) went to a thai site, and that person is using Thai on a daily basis? (Replace Thai by any multi bytes languages).

The part of that sentence after the "possibly" doesn't haven any
normative force (it's just advice that the server can take or leave).
I can remove the reference to ASCII here if you like.  Julian please
correct me if I'm wrong, but I believe that HTTP headers typically
contain only ASCII characters.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Mon, 01 Feb 2010 19:44:31 +0100

> contain only ASCII characters.

I think the current thinking is: "it is opaque data, but for anything 
non-ASCII you need to negotiate it out-of-band between client and 
server, and furthermore intermediaries and libraries may screw things up".

So if the cookie is supposed to carry information that can't directly be 
encoded in ASCII, the best way is to use an encoding on top of it, such 
as base64.

BR, Julian

---

From: Maciej Stachowiak <[email-masked]>
Date: Mon, 01 Feb 2010 17:25:05 -0800

> contain only ASCII characters.

We do have to define what happens if there are non-ASCII characters in a header value. Presumably, if such bytes are set via Set-Cookie, then the Cookie header just echoes them back, so we can treat them as opaque. (That's assuming there is no requirement to reject non-ASCII cookie values.) But there are two ways in which it matters:

1) A header with non-ASCII bytes get set via Set-Cookie, then read through a JS API such as document.cookie. document.cookie gives a UTF-16 encoded string, so at this point the server has to decide how to interpret non-ASCII bytes in the cookie value.

2) If you set a cookie via document.cookie and include non-ASCII characters in the value, what bytes get sent?

Regards,
Maciej

---

From: David Morris <[email-masked]>
Date: Mon, 1 Feb 2010 21:07:28 -0800 (PST)

> 2) If you set a cookie via document.cookie and include non-ASCII characters in the value, what bytes get sent?

Seems to me that the platform providing the document.cookie object is 
responsible for making any value placed in the cookie: header correct.

I think Julian provided a sensible answer for non-ascii characters ... 
certainly for this iteration of document what exists ... unless there 
is a different common behavior to document.

---

From: Adam Barth <[email-masked]>
Date: Mon, 1 Feb 2010 22:49:42 -0800

On Mon, Feb 1, 2010 at 10:44 AM, Julian Reschke <[email-masked]> wrote:
>>> Livng and working in a non ASCII country, I tend to think we shall
>>> prepare for the coming of the other languages (Thai, Chines, Korean ....),
>>> IF a person get a cookie from a Thai server , can we securely suppose that
>>> person(computer) went to a thai site, and that person is using Thai on a
>>> daily basis? (Replace Thai by any multi bytes languages).
>>
> non-ASCII you need to negotiate it out-of-band between client and server,
> and furthermore intermediaries and libraries may screw things up".
>
> encoded in ASCII, the best way is to use an encoding on top of it, such as
> base64.

I've reworded this sentence to clarify that the intent is as you describe:

          <t>To maximize compatibility with user agents, servers that wish to
          store non-ASCII data in a cookie-value SHOULD encode that data using
          a printable ASCII encoding, such as base64.</t>

Adam

---

From: Adam Barth <[email-masked]>
Date: Mon, 1 Feb 2010 22:54:12 -0800

On Mon, 1 Feb 2010, Maciej Stachowiak wrote:
> We do have to define what happens if there are non-ASCII characters in a header value. Presumably, if such bytes are set via Set-Cookie, then the Cookie header just echoes them back, so we can treat them as opaque. (That's assuming there is no requirement to reject non-ASCII cookie values.)

That's correct.  I've filed a ticket for adding a test case for
non-ASCII cookie values.

On Mon, Feb 1, 2010 at 9:07 PM, David Morris <[email-masked]> wrote:
>> But there are two ways in which it matters:
>>

That seems like a wise course of action.  This document understands
octet sequences.  HTML5 should define how to translate those octet
sequences into JavaScript characters (when reading document.cookie)
and how to perform the reverse translation (when writing
document.cookie).

Adam

---

From: Maciej Stachowiak <[email-masked]>
Date: Tue, 02 Feb 2010 16:27:14 -0800

> and how to perform the reverse translation (when writing

HTML5 does not spec this detail and apparently expects the cookie spec to expose a string interface, not an octet-sequence interface:
http://dev.w3.org/html5/spec/Overview.html#dom-document-cookie

I think defining conversion between octet sequence and string could plausibly go in either spec. I think the cookie spec would be a better place, because other string-oriented interfaces from web platform specs to cookies (if any) should probably use the same conversion, and browser UI for managing cookies should probably use the same conversion too. So it would be a useful thing to define even if it's not used by the network protocol. However, if you don't think it should be in the cookie spec, I can file a bug against HTML5.

Regards,
Maciej

---

From: Adam Barth <[email-masked]>
Date: Tue, 2 Feb 2010 16:30:46 -0800

> I think defining conversion between octet sequence and string could plausibly go in either spec. I think the cookie spec would be a better place, because other string-oriented interfaces from web p…

I'm happy to do whatever you and Ian think is best.  If the algorithm
needed for HTML5 is sensible enough to be re-used by other string
consumers, that would be a reason to put it in this document.  If the
algorithm is non-sensical, we might be doing other applications a
service keeping it HTML specific.

Adam

---

From: Maciej Stachowiak <[email-masked]>
Date: Tue, 02 Feb 2010 17:19:04 -0800

> service keeping it HTML specific.

I don't actually know what algorithms current UAs use. In fact, I don't even know for sure whether non-ASCII characters in cookies are allowed at all. I guess someone needs to do some testing. I'll ask Ian if he has an opinion on which spec should define this.

Regards,
Maciej

---

From: Adam Barth <[email-masked]>
Date: Tue, 2 Feb 2010 17:37:53 -0800

> I don't actually know what algorithms current UAs use. In fact, I don't even know for sure whether non-ASCII characters in cookies are allowed at all. I guess someone needs to do some testing. I'll…

I've added some basic tests using non-ASCII characters (in this case
Chinese): charset0001, charset0002, charset0003.  IE, Firefox, Chrome,
and Opera handle these cookies fine.  Safari does something strange
that I haven't tracked down yet.

There's a long tail of things to test here, including invalid unicode
in various encodings, but basic non-ASCII handling appears to do
something somewhat sensible.  I haven't tested the document.cookie API
at all yet.

Adam

---

From: Maciej Stachowiak <[email-masked]>
Date: Tue, 02 Feb 2010 18:06:22 -0800

> something somewhat sensible.  I haven't tested the document.cookie API

Invalid unicode probably will not be an issue if browsers either treat these as raw bytes or treat it as a non-unicode encoding like Latin1.

Regards,
Maciej

---

From: Dan Winship <[email-masked]>
Date: Tue, 02 Feb 2010 21:21:53 -0500

On 02/02/2010 07:27 PM, Maciej Stachowiak wrote:
> HTML5 does not spec this detail and apparently expects the cookie
> spec to expose a string interface, not an octet-sequence
> interface:

Slight tangent...

One of the major reasons cookies are such a disaster is that web site
frameworks and document.cookie both expose cookies to web site authors
as just strings, and expect the web site authors to get all the nasty
syntax details correct. Which they of course don't.

Although in the short term we need to nail down exactly how
document.cookie works for backward-compatibility purposes, in the long
run, we might be able to make the world a better place if we helped
design a fabulous new higher-level cookie API for HTML5, where the
browser would handle the tricky syntax bits, and would just throw an
exception if the page tried to use an illegal cookie name, etc. This new
API could even fix some issues that we aren't currently able to fix in
the spec, eg, by setting the "Secure" flag on the cookie by default if
the page had been loaded over https, etc.

(And likewise, the spec could recommend that web site
frameworks/libraries SHOULD provide similar idiot-proof high-level
cookie APIs, rather than expecting authors to generate valid Set-Cookie
headers by themselves.)

-- Dan

---

From: Adam Barth <[email-masked]>
Date: Tue, 2 Feb 2010 19:59:51 -0800

> Invalid unicode probably will not be an issue if browsers either treat these as raw bytes or treat it as a non-unicode encoding like Latin1.

Yes, but testing makes perfect.  I don't know that much about crazy
unicode cases.  If you have a link to a resource of unicode test
vectors, I can adapt them into cookie tests.

Adam

---

From: Ian Hickson <[email-masked]>
Date: Wed, 3 Feb 2010 04:38:14 +0000 (UTC)

> service keeping it HTML specific.

I'm fine with either; if it's to be in HTML5, just let me know what to put 
in. If there's anything I can do to help either way, let me know.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Julian Reschke <[email-masked]>
Date: Wed, 03 Feb 2010 08:34:20 +0100

Adam Barth wrote:
> ...
> ...

Could you please elaborate on what means "handles fine"? After all, it's 
not specified, right?

Best regards, Julian

---

From: Maciej Stachowiak <[email-masked]>
Date: Tue, 02 Feb 2010 23:38:41 -0800

On Feb 2, 2010, at 11:34 PM, Julian Reschke wrote:

> Could you please elaborate on what means "handles fine"? After all, it's not specified, right?

I assume he means that the same bytes that come in as part of the value in a Set-Cookie header get sent out in a future Cookie header, even if they are non-ASCII.

I am curious if the Cookie name, and not just the value, can contain non-ASCII in existing implementations.

Regards,
Maciej

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 3 Feb 2010 08:42:38 +0100 (CET)

On Tue, 2 Feb 2010, Dan Winship wrote:

> (And likewise, the spec could recommend that web site frameworks/libraries 
> SHOULD provide similar idiot-proof high-level cookie APIs, rather than 
> expecting authors to generate valid Set-Cookie headers by themselves.)

Don't we pretty much assume and wish for this for every spec and protocol? I 
mean, we make a smaller set of libraries and frameworks for everything that 
then a large amount of users can use.

It is really the job of this spec to spell it out?

(I'm not really against the idea, I'm just curious.)

-- 

  / daniel.haxx.se

---

From: Adam Barth <[email-masked]>
Date: Tue, 2 Feb 2010 23:48:49 -0800

> Could you please elaborate on what means "handles fine"? After all, it's not

I mean the bytes are treated opaquely, which is what the draft
specifies.  The draft specifies what to do with every sequence of
bytes a user agent can receive from a server.

> I assume he means that the same bytes that come in as part of the value in a Set-Cookie header get sent out in a future Cookie header, even if they are non-ASCII.

Precisely.

> I am curious if the Cookie name, and not just the value, can contain non-ASCII in existing implementations.

Yes, it can.  charset0002 tests a non-ASCII name and value.
charset0003 tests a non-ASCII name, value, and custom attribute, all
of which acts naturally in IE/Firefox/Chrome/Opera:

== charset0001 ==
Set-Cookie: foo=春节回家路・春运完全手册
Cookie: foo=春节回家路・春运完全手册

== charset0002 ==
Set-Cookie: 春节回=家路・春运完全手册
Cookie: 春节回=家路・春运完全手册

== charset0003 ==
Set-Cookie: 春节回=家路・春运; 完全手册
Cookie: 春节回=家路・春运

Adam

---

From: Adam Barth <[email-masked]>
Date: Tue, 2 Feb 2010 23:52:15 -0800

> It is really the job of this spec to spell it out?

I tried to move the discussion of Dan's email to a different thread
(even he pointed out that it was tangential to this thread):

http://www.ietf.org/mail-archive/web/http-state/current/msg00500.html

Adam

---

From: Julian Reschke <[email-masked]>
Date: Wed, 03 Feb 2010 09:02:04 +0100

> bytes a user agent can receive from a server.

Understood (when I was saying "not specified" I was thinking of historic
Cookie specs).

> Set-Cookie: 春节回=家路・春运; 完全手册

So what does happen with Safari?

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Wed, 3 Feb 2010 00:06:58 -0800

> Understood (when I was saying "not specified" I was thinking of historic

Honestly, the historical cookie specs are so far from the truth that
it's easier to act as they don't exist.

>> of which acts naturally in IE/Firefox/Chrome/Opera:
[...]
> So what does happen with Safari?

I knew someone would ask.  :)

It looks like Safari does not return a Cookie header in those cases.

Adam

---

From: Adam Barth <[email-masked]>
Date: Wed, 3 Feb 2010 00:08:11 -0800

> It looks like Safari does not return a Cookie header in those cases.

Those cases being charset0001, charset0002, and charset0003.  One
possible explanation is that Safari rejects cookies that have
non-ASCII characters in their values.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Wed, 03 Feb 2010 09:11:11 +0100

> non-ASCII characters in their values.

...which I think is not really a bug, according to the specs.

Unless that affects Safari's interoperability, I'd strongly recommend to 
consider making this the recommended behavior.

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Wed, 3 Feb 2010 00:34:54 -0800

> ...which I think is not really a bug, according to the specs.

I'm not sure that's the most important consideration.

> consider making this the recommended behavior.

Why?  Ideally, a justification would be an evidence-based argument we
could use to convince the implementors of IE, Firefox, Chrome, and
Opera that the benefits of changing their behavior out-weighs the
costs.

In general, I'm very reluctant to specify a behavior that conflicts
with IE, Firefox, Chrome, and Opera.  In specific, IE has something
like 99% market share in some Asian markets.  The fact that this issue
affects Asian markets disproportionately means the pressure to be
IE-like is even greater than usual.

Adam

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 03 Feb 2010 00:39:16 -0800

On Feb 3, 2010, at 12:11 AM, Julian Reschke wrote:

> Unless that affects Safari's interoperability, I'd strongly recommend to consider making this the recommended behavior.

I don't have information one way or another on this. I don't see any bugs that are obviously this issue, but it could be the cause of less obvious compatibility bugs. Perhaps the thing to do is a study to determine the frequency with which non-ASCII cookies are sent.

BTW from code inspection, it looks to me like Safari *will* allow non-ASCII characters in the cookie value if it is a quoted string.

Regards,
Maciej

---

From: Adam Barth <[email-masked]>
Date: Wed, 3 Feb 2010 00:41:57 -0800

> I don't have information one way or another on this. I don't see any bugs that are obviously this issue, but it could be the cause of less obvious compatibility bugs. Perhaps the thing to do is a s…

Do you see a particular benefit to dropping non-ASCII cookies?  I
don't understand what we're trying to gain here.

> BTW from code inspection, it looks to me like Safari *will* allow non-ASCII characters in the cookie value if it is a quoted string.

Thanks.  I'll add a test for this case.

Adam

---

From: Adam Barth <[email-masked]>
Date: Wed, 3 Feb 2010 00:46:31 -0800

> Thanks.  I'll add a test for this case.

I was not able to reproduce this in Snow Leopard using this test case:

== charset0004 ==
Set-Cookie: foo="春节回家路・春运完全手册"
Cookie: foo="春节回家路・春运完全手册"

Can you explain what you mean in the form of a test case?

Adam

---

From: Julian Reschke <[email-masked]>
Date: Wed, 03 Feb 2010 09:50:57 +0100

> IE-like is even greater than usual.

On the other hand, why require something that's known not be in use (if 
that is the case)?

If you allow non-ASCII characters (or actually require support), you'll 
also have to figure out how existing server and client based APIs are to 
deal with them (so what encoding is it?).

Best regards, Julian

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 03 Feb 2010 01:53:24 -0800

> don't understand what we're trying to gain here.

I don't care much either way, and it would not be hard to change our implementation. I'm just trying to clarify the level of evidence we have on whether this affects compat. Other things being equal, I would prefer to see Safari change to match the other browsers.

I will note that Safari (and Mac OS) are pretty rarely used in some asian markets, such as China and India, so there could be issues we are unaware of. iPhone has significant penetration into some of those markets however.

Regards,
Maciej

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 03 Feb 2010 01:54:12 -0800

On Feb 3, 2010, at 12:46 AM, Adam Barth wrote:

> Set-Cookie: foo="$)A4:=Z;X<RB7$)G!&QN$)ATKMjH+JV2a"
> Cookie: foo="$)A4:=Z;X<RB7$)G!&QN$)ATKMjH+JV2a"
> 

It looks like the code *should* handle this, but I have not done any testing.

Regards,
Maciej

---

From: Adam Barth <[email-masked]>
Date: Wed, 3 Feb 2010 02:01:55 -0800

> It looks like the code *should* handle this, but I have not done any testing.

It's entirely possible I've screwed up something on my end.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Wed, 03 Feb 2010 13:41:15 +0100

> deal with them (so what encoding is it?).

I just did a quick test with an ISO-8859-1 encoded cookie value, 
client-side javascript and "alert(document.cookie)":

- IE and Firefox appear to treat the cookie as ISO-8859-1
- Safari appears to ignore the cookie
- Chrome and Opera appear to try to decode as UTF-8 (and returns a 
REPLACEMENT CHAR in place of the umlaut I tried)

Best regards, Julian

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 03 Feb 2010 13:49:12 +0100

On Wed, 03 Feb 2010 13:41:15 +0100, Julian Reschke <[email-masked]>  
wrote:
>>  If you allow non-ASCII characters (or actually require support),  
>> you'll also have to figure out how existing server and client based  
>> APIs are to deal with them (so what encoding is it?).
>> ...
>

Hmm, at least Chrome is consistent with how it does things for  
XMLHttpRequest...

Since not everyone on this list may know this, for XMLHttpRequest it is  
currently specified that trying to set headers/values with characters  
higher than U+00FF fails. Otherwise setting succeeds with the higher-order  
byte of each character removed. On reading a code point is created for  
each byte with the higher-order byte set to zero. It would make sense if  
these APIs behaved identical I think.

-- 
Anne van Kesteren
http://annevankesteren.nl/

---

From: Adam Barth <[email-masked]>
Date: Tue, 9 Feb 2010 08:47:58 -0800

On Wed, Feb 3, 2010 at 12:50 AM, Julian Reschke <[email-masked]> wrote:
> On the other hand, why require something that's known not be in use (if that
> is the case)?

As Maciej points out, this is not the standard of evidence that user
agent implementors usually use when making compatibility decisions.  I
understand that this is a bit of a cultural divide in this group, but
I'll try to give you a flavor of why UA folks think this way.

A number of months ago, I helped change the way JavaScript prototypes
were configured in WebKit.  Instead of using the lexical or dynamic
scope (which is what WebKit was using a mix of at the time), we
changed them to some other esoteric rule that matched Firefox and IE.
At the time, we had zero evidence that this change had any
compatibility because the cases where these things are different are
extremely obscure.  So, why did we change it?  Well, because there's
no value in being different from other browsers and some (potential)
value in being the same.  That means the benefit (potentially
non-zero) outweighs the cost (essentially zero).

Now, as far as I know, the project still has zero evidence that this
change actually improved compatibility in the real world.  However, I
happen to know by some random coincidence that this change was hugely
beneficial.  It just so happens that when folks use both Selenium (a
popular web application testing harness) and Prototype.js (a popular
JavaScript library), they arrive in precisely the situation where
these computations give different answers.  Without the change,
developers have a lot of trouble testing a Prototype.js-using web site
with Selenium, but, with the change, it works fine.

So what are the consequences of the change?  Well, it means developers
are much more likely to test their web sites in Safari because they
can just re-use their existing Selenium tests instead of having to
build or find another testing harness.  That means they're more likely
to fix bugs that affect Safari, which means their web sites will work
better in Safari.  That, in turns, means users of Safari will have
better experiences at web sites and the marketshare of Safari will
grow.

If the WebKit project had required actual examples of the change being
beneficial, they probably would not have made that change and would
not have enjoyed the benefits.  Now, not every change pays off as well
as the one I describe, and, in many cases, we never find out how well
they pay off.  The point is more that user agent implementing are
being rational when they make these sorts of decisions.

> If you allow non-ASCII characters (or actually require support), you'll also
> have to figure out how existing server and client based APIs are to deal
> with them (so what encoding is it?).

For our purposes, it doesn't really matter how existing servers handle
the encodings.  In the end, they do what they do.  As long as we send
the "right" bytes on the wire, they'll do the "right" thing, whatever
that is.  As for client APIs, yes, we'll need to get that right,
either in this specification or in the specification of those APIs (as
we've discussed elsewhere in this thread).

Adam