Subject: Re: [http-state] Ticket 5: Cookie ordering

From: "Yngve Nysaeter Pettersen" <[email-masked]>
Date: Thu, 04 Feb 2010 23:58:32 +0100

On Tue, 19 Jan 2010 23:10:48 +0100, Adam Barth <[email-masked]> wrote:

> Ticket 3 is still open for discussion, but I'd like to get started
> talking the next ticket:
> http://trac.tools.ietf.org/wg/httpstate/trac/ticket/5
>
> == Overview ==
>
> Currently the draft defines the order in which cookies should appear
> in the Cookie header.  In particular, cookies are ordered first by the
> length of the Path attribute (longest first) and then by creation date
> (earliest first).  The majority of the most widely used user agents
> use this ordering. (I can look up exactly which browsers follow the
> ordering if that's important.)
>
> Sending cookies with longer (i.e., more specific) paths first is
> important for compatibly because some servers host multiple (mutually
> trusting) web applications at different (possibly overlapping) paths.
> For example, mail.google.com hosts a public instance of Gmail at
> https://mail.google.com/ and corporate instances of Gmail at
> https://mail.google.com/a/example.com/.  These web applications expect
> their own cookies (with the more specific path) to be presented before
> cookies for other web applications (even if the cookies have the same
> names).  In fact, this behavior is required by the original Netscape
> cookie spec.

For reference, Opera's sorting is this way:

  - Cookies with the most specific domain are sent first. that is, The  
host's cookies are first, then the parent domain, then the grandparent,  
and so on
  - For a specific domain the cookies are organized in most specific  
(longest) path being listed first, then the parent path, and so on.
  - For a specific path it is oldest cookie first.

IMO the cookie with the more specific domain should be sent first, since  
it will have been set by a server that is closer to the receiving server,  
if it wasn't the one setting the cookie.

Our way of selecting and sending cookies follows naturally from how the  
cookie database is organized: As a Tree.

Regarding the specification, I would suggest that it continues to specify  
ordering, at least for path.

AFAIK it will not be possible to define an inter-operable domain ordering  
specification, so instead the specfication should say that servers must  
not rely on a specific ordering for cookies with the same name set for  
different domains and the same path (The path specification means that it  
is perhaps possible to rely on it for same name cookies with different  
name, even for different domains).

Short of using RFC2965 cookies it is not possible to discover the domains  
or paths for two cookies with the same name (unless it is coded in the  
value)

-- 
Sincerely,
Yngve N. Pettersen
********************************************************************
Senior Developer		     Email: [email-masked]
Opera Software ASA                   http://www.opera.com/
Phone:  +47 24 16 42 60              Fax:    +47 24 16 40 01
********************************************************************

---

From: Daniel Stenberg <[email-masked]>
Date: Fri, 5 Feb 2010 10:59:09 +0100 (CET)

On Thu, 4 Feb 2010, Yngve Nysaeter Pettersen wrote:

> - Cookies with the most specific domain are sent first. that is, The host's 
> cookies are first, then the parent domain, then the grandparent, and so on
> - For a specific domain the cookies are organized in most specific (longest) 
> path being listed first, then the parent path, and so on.

Thank you, this tells us the current wording of how to sort cookies is not 
quite what Opera uses. I find it interesting that you use the domain before 
the path length, as that will actually make it non-compliant with how the spec 
is currently phrased.

> IMO the cookie with the more specific domain should be sent first, since it 
> will have been set by a server that is closer to the receiving server, if it 
> wasn't the one setting the cookie.

"Make sense" and what we write in the spec isn't necessarily a fit ;-) If the 
major implementations don't sort based on the domain, then we really can't say 
that it is done so very much in the wild.

Did anyone make a test if the domain length has any impact on the cookie sort 
order in other implementations?

> ordering, at least for path.

... right, but then you just said that Opera isn't adhering to that spec...

> specification, so instead the specfication should say that servers must not 
> rely on a specific ordering for cookies with the same name set for different 
> domains and the same path (The path specification means that it is perhaps 
> possible to rely on it for same name cookies with different name, even for 
> different domains).

Not only that. Your way even makes sure servers can't unconditionally assume 
the order of cookies with different path lengths, as they must first take into 
account if the cookies have different domains or not.

I think the spec need to say something about this, if nothing else in an 
explanatory way as to why or how different clients may send cookies.

All this cookie ordering taken into account, I think a wise recommendation in 
the spec might be to just advice that cookies using the same name with 
different paths or domains are simply not a good idea to use at all if you 
want to inter-operate.

-- 

  / daniel.haxx.se

---

From: "Yngve Nysaeter Pettersen" <[email-masked]>
Date: Fri, 05 Feb 2010 13:56:58 +0100

On Fri, 05 Feb 2010 10:59:09 +0100, Daniel Stenberg <[email-masked]> wrote:

>> AFAIK it will not be possible to define an inter-operable domain  
>> ordering specification, so instead the specfication should say that  
>> servers must not rely on a specific ordering for cookies with the same  
>> name set for different domains and the same path (The path  
>> specification means that it is perhaps possible to rely on it for same  
>> name cookies with different name, even for different domains).
>
> Not only that. Your way even makes sure servers can't unconditionally  
> assume the order of cookies with different path lengths, as they must  
> first take into account if the cookies have different domains or not.
>
> All this cookie ordering taken into account, I think a wise  
> recommendation in the spec might be to just advice that cookies using  
> the same name with different paths or domains are simply not a good idea  
> to use at all if you want to inter-operate.

There is also the issue that if ordering, even at the domain level is not  
specified, then there will be problems if a server treats ordering as  
significant, such as only checking the first cookie with the name it is  
looking for.

If we call this server bar.example.com, and assumes it is looking for and  
sets the cookie MyCookie=bar for the host itself (no domain), and we also  
have a server foo.example.com that sets MyCookie=foo for the domain  
.example.com the following can occur if ordering is that first received  
cookie is first in line

If the client visits bar.example.com first, then foo.example.com, order by  
creation date will produce the MyCookie sequence "bar, foo", if  
foo.example.com is visited first then the sequence will be "foo, bar". In  
the latter case bar.example.com will not find its cookie, and may  
repeatedly try to set it. Any update of the cookie (unless original  
creation date is preserved) will move the newest cookie to the last  
position in the list.

If the ordering is most recently set/updated the sequences would be  
reversed (bar first gives "foo, bar", foo first gives "bar, foo"), and the  
bar server would run into problems on the first access after the client  
visited foo, but that would be fixed with sending a new cookie, but the  
session would be lost.

In Opera the sequence would always be "bar,foo" since the foo cookie is  
set for the parent domain which is listed later.

IOW, if ordering is determined by anything but the domain and path the  
sequence of cookie is going to vary depending on which servers the clients  
visits and the sequence it visits them, and this might cause significant  
problems for a server that considers ordering significant.

---

From: Dan Witte <[email-masked]>
Date: Fri, 5 Feb 2010 16:04:35 -0800 (PST)

> problems for a server that considers ordering significant.

Indeed. In Firefox, we sort by path length and then creation time. I can't comment as to whether sites depend on it now, but at one point we did drop the creation time sort - and sites broke. https://bugzilla.mozilla.org/show_bug.cgi?id=236772 has some details.

We also found that creation time should be persisted when cookies are modified. I don't have a bug reference handy for this one, but it could be dug up.

I like the idea of ordering by domain. Ultimately, having this stuff in a more storage-esque API where domain and path info are available in the Cookie header, or document.cookie, would make this much less of an issue...

Regards,
Dan.

---

From: "Yngve Nysaeter Pettersen" <[email-masked]>
Date: Mon, 08 Feb 2010 17:19:07 +0100

On Fri, 05 Feb 2010 13:56:58 +0100, Yngve Nysaeter Pettersen  
<[email-masked]> wrote:
> sequence of cookie is going to vary depending on which servers the  
> clients visits and the sequence it visits them, and this might cause  
> significant problems for a server that considers ordering significant.

Some testing by a couple of my colleagues setting two cookies with the  
same name (and path) "host-only" and "domain-wide" have found the  
following in browsers other than Opera:

-----
Visit order: Host-only, domain-wide
Cookie order:  "host-only", "domain-wide"
-----

-----
Visit order: domain-wide, Host-only
Cookie order (IE):  "host-only", "domain-wide"
Cookie order (Others): "domain-wide", "host-only"
-----

To me it looks like IE is sorting by domain, at the same path level, with  
FF and Safari (the two tested) sort on creation data.

The consequence is that there is apparently three deployed ways to send  
cookies:

    - Cookies at the same path level are grouped and sorted by creation  
date, earliest first (FF&co)
    - Cookies at the same path level are grouped and sorted by domain, most  
specific first (IE)
    - Cookies are grouped by domain (most specific first), then sorted by  
path (most specific first) within each domain (Opera)

IMO the creation date method is less predictable than the other two, and  
will cause problems for sites depending on a specific sequence of cookies.

My suggestion would be that the spec should recommend ordering an ordering  
based on on both domain and path (order of preference to be decided), as  
that will be more predictable for sites using multiple cookies with the  
same name at various domain and path levels.

---

From: Adam Barth <[email-masked]>
Date: Mon, 8 Feb 2010 08:20:52 -0800

Would you be willing to share your test cases?  I'd like to add them
to the test suite.

Thanks,
Adam

On Mon, Feb 8, 2010 at 8:19 AM, Yngve Nysaeter Pettersen
<[email-masked]> wrote:
> Some testing by a couple of my colleagues setting two cookies with the same
> name (and path) "host-only" and "domain-wide" have found the following in
> browsers other than Opera:
>
> -----
> To me it looks like IE is sorting by domain, at the same path level, with FF
> and Safari (the two tested) sort on creation data.
>
>   - Cookies at the same path level are grouped and sorted by creation date,
> earliest first (FF&co)
>   - Cookies are grouped by domain (most specific first), then sorted by path
> (most specific first) within each domain (Opera)
>
> that will be more predictable for sites using multiple cookies with the same
> name at various domain and path levels.
>
> --
> Sincerely,
> Yngve N. Pettersen
> _______________________________________________
> http-state mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/http-state
>

---

From: "Paul E. Jones" <[email-masked]>
Date: Mon, 8 Feb 2010 11:49:09 -0500

Yngve,

> same name at various domain and path levels.

Should we rely on the client to do this?  Your proposal is good, but I'm
skeptical that I can rely on this as a developer on the server side.

Personally, I'd prefer that the client never send two cookies with the same
name.  There are two or more cookies in the browser with the same name, I
would think the browser should only send the cookie with the most precise
path match.  So, if there is a cookie at / and a cookie at /foo, and the
user is accessing /foo, only the cookie for that path would be provided.  Is
there a good reason to send multiple cookies with the same name?

Paul

---

From: Adam Barth <[email-masked]>
Date: Mon, 8 Feb 2010 09:00:43 -0800

> skeptical that I can rely on this as a developer on the server side.

I would recommend against relying on this behavior.  We can explicitly
admonish servers not to rely on this behavior.

> there a good reason to send multiple cookies with the same name?

We have to send multiple cookies with the same name because that's
what user agents do today.  We're not permitted to alter the syntax of
the protocol in Phase 1.  In Phase 2, these sorts of changes will be
on the table.

Adam

---

From: Achim Hoffmann <[email-masked]>
Date: Mon, 08 Feb 2010 18:45:36 +0100

Hi all,

even I'm listening to the list since the beginning, I'd kept quit 'cause
my main interest is about cookies and web application security.

Anyway, it's time to speak as the cookie oder/sequence seems to be a
mystery since ages. See my question years back and still no answer
(except those in this list now:)
	http://www.webappsec.org/lists/websecurity/archive/2005-11/msg00097.html

'til now we got a clear statement from Opera (thanks Yngve), but there're
more  browser vendors on the list. What is so difficult to make a clear
and undoubtfull statement how it works in each browser?

The cookie sequence is really important as I'm aware of some serious
attacks (miss)using this sequence.
It needs to be fixed, so we can tell application developers how to deal
with a list of cookies with the same name (unless someone implements
Cookie2 in major browsers again:)

My vote for a sequence (with security in mind) would be:
  1. most qualified first
    a) protocol, always all HTTPS before HTTP (exception see e))
    b) FQDN (domains MUST be IDN, other characters MUST be transformed to IDN)
    c) port
    d) path
    e) secure Flag
    f) timestamp
    g) wildcard (domain) cookies at very last
  2. above order MUST not change if the browser's cookie store is
     overflowed

Do I miss something?
Cheers,
Achim

Yngve Nysaeter Pettersen wrote on 08.02.2010 17:19:
> To me it looks like IE is sorting by domain, at the same path level,
> with FF and Safari (the two tested) sort on creation data.
> 
>    - Cookies at the same path level are grouped and sorted by domain,
> most specific first (IE)
> My suggestion would be that the spec should recommend ordering an
> ordering based on on both domain and path (order of preference to be
> decided), as that will be more predictable for sites using multiple
> cookies with the same name at various domain and path levels.
>

---

From: Adam Barth <[email-masked]>
Date: Mon, 8 Feb 2010 09:50:05 -0800

> my main interest is about cookies and web application security.

Thanks for participating.  Web application security is definitely an
important consideration for this group.

> and undoubtfull statement how it works in each browser?

That is indeed what we're attempting to do.  The current draft
represents my understanding about how the cookie protocol works in
browsers.  Together with the deviation description document, that
should provide a detailed answer to your question.

> My vote for a sequence (with security in mind) would be:

Unfortunately, this discussion is not about voting for our favorite
cookie orderings.  We're merely documenting the cookie ordering that
already exists in implementations.

Adam

---

From: Achim Hoffmann <[email-masked]>
Date: Mon, 08 Feb 2010 19:30:22 +0100

> already exists in implementations.

No worries:)
I'd be happy if the vendor people simply tell us what the status is today.

Achim

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 9 Feb 2010 10:16:14 +0100 (CET)

On Mon, 8 Feb 2010, Adam Barth wrote:

> We have to send multiple cookies with the same name because that's what user 
> agents do today.  We're not permitted to alter the syntax of the protocol in 
> Phase 1.  In Phase 2, these sorts of changes will be on the table.

As this topic twists on, we all see that no server can rely on any specific 
order on cookies that are sent by clients, so thus we can in fact discourage 
the use of multiple cookies with the same name as there's basically no way for 
a server to know in which order it'll get the cookies. We can _perhaps_ find a 
sort order we can say clients should use, but as seen here that wouldn't 
change how a large amount of existing clients behave.

Also, allow me to re-iterate that the 5 major browsers are not even close to 
100% of the cookie using HTTP clients and we can find several other widely 
used implementations that will send the cookies in a different sort order than 
the big browsers.

-- 

  / daniel.haxx.se

---

From: Achim Hoffmann <[email-masked]>
Date: Sun, 28 Feb 2010 14:10:04 +0100

Daniel Stenberg wrote on 09.02.2010 10:16:
> As this topic twists on, we all see that no server can rely on any
> specific order on cookies that are sent by clients, so thus we can in
> fact discourage the use of multiple cookies with the same name as
> there's basically no way for a server to know in which order it'll get
> the cookies. We can _perhaps_ find a sort order we can say clients
> should use, but as seen here that wouldn't change how a large amount of
> existing clients behave.
> 
> Also, allow me to re-iterate that the 5 major browsers are not even
> close to 100% of the cookie using HTTP clients and we can find several
> other widely used implementations that will send the cookies in a
> different sort order than the big browsers.

Just to sumarize the facts:
  1) multiple cookies (key-value pairs) are allowed currently
  2) browsers make use of 1)
  3) there is no conformity in which order cookies are send by the browser
  4) it's even allowed to have more than one Cookie request header
     (I'm not aware of a browser which does it this way)

As 1) is used in the wild, it cannot be discouraged.
Up to now no browser vendor (except Opera:) gave a clear and undoubtful
description in which order cookies are send.
Even I don't have seen it in the wild, a proxy may change the order of
multiple Cookie headers.
All this renders the attempt to make a suggestion about the sequence of
cookies uncertain.

Hence I'll change my mind about cookie ordering, and suggest a probably
heretic definition:

  The order of multiple cookies, wether in one or multiple Cookie headers,
  is random and up to the client implementation.
  Servers/applications MUST NOT rely on any sequence.

Such a definition keeps the logical order and other such details out of
the cookie protocol specification. It also forces the server/application
to make precautionary measures how to identify the proper cookie.
This is then also in sync with 7. Security Considerations which makes the
application responsible for that.

Achim