Subject: TICKET 259: 'treat as invalid' not defined

From: Adam Barth <[email-masked]>
Date: Mon, 1 Nov 2010 19:56:15 -0700

On Mon, Nov 1, 2010 at 5:43 PM, Mark Nottingham <[email-masked]> wrote:
>> => a header field value with multiple instances of the same parameter
>> SHOULD be treated as invalid.
>>
>> Similarly, this requirement probably should read "user agents SHOULD
>> treat a header field value with multiple instances of the same
>> paramater as invalid."  Furthermore, the document should define what
>> treating a header field value as invalid means.  Presumably the author
>> intends that user agents ought to ignore such header field values.
>> I'm skeptical that is the optimum behavior for user agents.  I would
>> have expected user agents to either use the first or the last instance
>> of each paramater.
>
> Ticket:
>  http://trac.tools.ietf.org/wg/httpbis/trac/ticket/259
>
> Note that it may be resolved by indicating that 'treat as invalid' is specific to the application at hand. As such, I'd like initial discussion of this in the WG to focus on:
>  a) use cases: how different implementations / applications may want to have different notions of 'invalid' (or not), and

The browser use case proceeds from the following premises.

1) Many servers send invalid messages to user agents.
2) Many existing user agents have some behavior for these invalid messages.
3) If a new user agent wishes to compete in the market, that user
agent needs to handle the invalid messages in the same way as the
existing user agents.

Use case: Users benefit when there is competition among browser
vendors.  Without specifying how to handle invalid messages, new user
agents need to reverse engineer the behavior of existing user agents,
making it more difficult to compete in the marketplace.

>  b) security: what the security impact of having different notions of 'invalid' here may be, and
>  c) interoperability: likewise, the interop impact.

The largest interoperability problems here are the different user
agents handle invalid messages differently.  For example, today, it's
unpredictable how user agents will interpret a filename parameter with
a % character.  That part of the protocol is less useful to servers
because the results are not predictable.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Sun, 07 Nov 2010 17:24:35 +0100

OK, although everything that can be said about this *has* been said, 
I'll repeat my p.o.v. here because this is the only mailing list thread 
specific to issue 259.

On 02.11.2010 03:56, Adam Barth wrote:
> ...

No data was provided that this is indeed the case for C-D.

> 2) Many existing user agents have some behavior for these invalid messages.

Yes.

> agent needs to handle the invalid messages in the same way as the

As my tests show, there is no "same" behavior right now. It's totally 
not clear that this is a problem in practice.

> making it more difficult to compete in the marketplace.

My tests show that no, they do not need to compete.

> agents handle invalid messages differently.  For example, today, it's

No, the largest interop problem is different handling of *valid* header 
fields, and lack of agreement how to handle I18N.

> because the results are not predictable.

Now that's a different issue as filename parameters containing "%" *are* 
valid.

Two UAs currently do not treat them per spec, and also do it in a 
currently not well-documented way, and may even differ on the details.

This is a problem tracked in 
<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/245>; I added a 
warning notice to the spec:

"Note: Many user agents do not properly handle escape characters when 
using the quoted-string form. Furthermore, some user agents erroneously 
try to perform unescaping of "percent" escapes (see Appendix C.2), and 
thus might misinterpret filenames containing the percent character 
followed by two hex digits."

We have heard from you that those UAs that currently percent-unescape 
are unlikely to stop doing that, so all we can do is point out that 
there's a problem. When I wrote the note mentioned above there was also 
no workaround to recommend because those UAs did not implement the RFC 
5987 which would provide a workaround.

Maybe we should expand the note though to say that RFC 5987 may be used 
as workaround, as long as it's used in addition to "filename=".

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Sun, 7 Nov 2010 12:32:14 -0800

On Sun, Nov 7, 2010 at 8:24 AM, Julian Reschke <[email-masked]> wrote:
> OK, although everything that can be said about this *has* been said, I'll
> repeat my p.o.v. here because this is the only mailing list thread specific
> to issue 259.
>

No data was provided that this isn't the case.  Given that we see
invalid message everywhere else, common sense tells us that we will
see invalid messages here too.

> As my tests show, there is no "same" behavior right now. It's totally not
> clear that this is a problem in practice.

It is a problem in practice.

> My tests show that no, they do not need to compete.

Of course user agents need to compete with each other for market
share.  Your statement doesn't make any sense.

> Two UAs currently do not treat them per spec, and also do it in a currently
> not well-documented way, and may even differ on the details.
>
> <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/245>; I added a warning
> notice to the spec:
>
> "Note: Many user agents do not properly handle escape characters when using
> the quoted-string form. Furthermore, some user agents erroneously try to
> perform unescaping of "percent" escapes (see Appendix C.2), and thus might
> misinterpret filenames containing the percent character followed by two hex
> digits."
>
> We have heard from you that those UAs that currently percent-unescape are
> unlikely to stop doing that, so all we can do is point out that there's a
> problem. When I wrote the note mentioned above there was also no workaround
> to recommend because those UAs did not implement the RFC 5987 which would
> provide a workaround.
>
> Maybe we should expand the note though to say that RFC 5987 may be used as
> workaround, as long as it's used in addition to "filename=".

In that case, perhaps we should make including the % character invalid
as the results are not predictable from the point of view of servers.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Sun, 07 Nov 2010 21:50:24 +0100

> see invalid messages here too.

In the absence of data telling me something else, I'll assume that 
servers do sane things. I may be wrong. I just don't see why a server 
would *ever* send two disposition types, given it's a waste of bytes and 
that it doesn't cause the same thing to happen in different UAs.

> It is a problem in practice.

Why?

> share.  Your statement doesn't make any sense.

So you claim that user agents need to process invalid headers for market 
share. But they do not do the same thing. In the absence of a better 
explanation, my conclusion is that it may not matter.

> as the results are not predictable from the point of view of servers.

We currently have a warning. I would *love* to tell server implementors 
about a way to workaround this problem, but there isn't (for released 
versions of Chrome and IE). So what do you propose to tell senders? 
"%xx" is not allowed in filenames?

 From a spec writing point of view, I very much prefer to state that 
it's valid, but risky due to UA bugs.

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Sun, 7 Nov 2010 13:59:33 -0800

On Sun, Nov 7, 2010 at 12:50 PM, Julian Reschke <[email-masked]> wrote:
> In the absence of data telling me something else, I'll assume that servers
> do sane things. I may be wrong. I just don't see why a server would *ever*
> send two disposition types, given it's a waste of bytes and that it doesn't
> cause the same thing to happen in different UAs.

Why would a server ever send two Content-Types headers?  Why an HTML
document ever mis-nest tags?  Why would a server ever send nonsense
characters instead of an HTTP header?  All these things happen in
practice because not everyone who operates servers is perfect.

>> It is a problem in practice.

Lack of interoperability usually a problem.  Here's an example that
has been causing me a lot of pain recently.  Consider this syntax in
an HTML document:

<script src="..." />

Should we interpret that as a self-closing script tag or as an open
script tag?  Turns out, historically, WebKit was the only major HTML
parser that treated it as a self-closing tag.  Consequently, content
that was authored primarily for WebKit assumed it was a self-closing
tag and content that was authored more generally assumed it was a open
tag.

This put WebKit in a bind.  It couldn't change its behavior to be
compatible with general web content because that would break
WebKit-specific content.  Now, when we implemented the HTML5 parsing
algorithm in WebKit, we changed WebKit's behavior to treat this syntax
as an open tag, but the pain this has caused is, and continues to be,
very real.

Notice that the lack of interoperability in this area was a problem
even before there were compatibility constraints.  The compatibility
constraints come later and make it harder / more painful to fix the
lack of interoperability.  Browser vendors have seen this story play
out many, many times in different aspects of HTML, JavaScript APIs,
and even HTTP.  That's why we're so passionate about nailing down
error handle behavior.  We want to avoid this pain in the future.

> explanation, my conclusion is that it may not matter.

It does matter.  Please see above.

> versions of Chrome and IE). So what do you propose to tell senders? "%xx" is
> not allowed in filenames?

Yes, just like we tell them that U+2665 isn't allowed.

Adam

---

From: "Eric J. Bowman" <[email-masked]>
Date: Sun, 7 Nov 2010 15:10:56 -0700

Adam Barth wrote:
> Julian Reschke wrote:

Plus, invalid instances of headers which aren't even defined by HTTP.
Instead of spreading this concern out and imposing a new requirement on
authors of such headers (first, seek browser-vendor approval of error-
correction language, which as Julian mentioned may result in decreased
participation if such authors simply ignore IETF standardization), it
would seem that common sense tells us that defining error correction
behavior for user-agents will always exceed the scope of HTTP.

> as the results are not predictable from the point of view of servers.

The API I'm developing (which I've mentioned on rest-discuss, this isn't
a hypothetical) works as follows:  user-agent dereferences a resource,
the response representation contains form code which instructs the user-
agent how to PUT a local file to a server-defined URI, preserving the
filename of the local resource in Content-Disposition.  If the local
filesystem allows spaces, for example, they're percent-encoded because
the target filesystem does not allow spaces.  Expressing the sender
intent requires a protocol mechanism which allows '%' without dictating
that it must be decoded.  From the POV of my server, the results are
completely predictable, because it isn't the sender in this case; and
has exactly zero interoperability issues.  I would prefer my API not to
be deprecated based on an issue that doesn't even affect it.

-Eric

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 08 Nov 2010 12:02:16 +1300

On 8/11/2010 10:59 a.m., Adam Barth wrote:
> On Sun, Nov 7, 2010 at 12:50 PM, Julian Reschke<[email-masked]>  wrote:

I think we are getting way off base here.  There are probably at least 
trillions of ways in which UAs and Servers can send non-compliant messages.

I wouldn't propose modifying HTTP to cover those cases.  It doesn't make 
sense IMO for http to define a response for non-compliant behaviour 
except for it to be rejected.

Whether some software decides to second-guess intent and "recover" from 
such errors is in the end up to the developer of the software.  History 
has shown us such cures are often worse than the disease.

That's an area where security vulnerabilities have commonly crept in.  
It also provides zero incentive for broken UAs / servers / sites to 
clean up their act.  This then perpetuates the problems and makes it 
worse for everyone else for the inter-op reasons you mention.

If market share is your issue, maybe you should talk to PR about how you 
can spin breaking broken sites into a positive (which IMO it is 
generally - for security reasons).

In fact I thought we'd already thrashed out issues like this (specifying 
behaviour for behaviour that falls outside the spec) and decided we 
weren't doing that.

My 2c

---

From: Adam Barth <[email-masked]>
Date: Sun, 7 Nov 2010 15:17:31 -0800

On Sun, Nov 7, 2010 at 3:02 PM, Adrien de Croy <[email-masked]> wrote:
>>> servers

Thankfully defining how to handle all those trillion ways isn't
actually that difficult.

> sense IMO for http to define a response for non-compliant behaviour except
> for it to be rejected.

Rejecting invalid message is not implementable by browser user agents.
 If we'd like our specs to be implemented, that's not an option.

> Whether some software decides to second-guess intent and "recover" from such
> errors is in the end up to the developer of the software.  History has shown
> us such cures are often worse than the disease.

You and I appear to have a different interpretation of history.

> That's an area where security vulnerabilities have commonly crept in.  It
> also provides zero incentive for broken UAs / servers / sites to clean up
> their act.  This then perpetuates the problems and makes it worse for
> everyone else for the inter-op reasons you mention.

We disagree about whether or not that is a problem.

> If market share is your issue, maybe you should talk to PR about how you can
> spin breaking broken sites into a positive (which IMO it is generally - for
> security reasons).

This isn't really even worth responding to.

> behaviour for behaviour that falls outside the spec) and decided we weren't
> doing that.

It appears not everyone shares that opinion.

Adam

---

From: "Eric J. Bowman" <[email-masked]>
Date: Sun, 7 Nov 2010 16:19:54 -0700

Adam Barth wrote:
>
> Julian Reschke wrote:
> >
> > So you claim that user agents need to process invalid headers for
> > market share. But they do not do the same thing. In the absence of
> > a better explanation, my conclusion is that it may not matter.
> >
>

I believe Julian meant that it may not matter to market share.  You've
pointed out that there's a market-share difference between browsers,
which varies by market, but haven't provided evidence that differences
in Content-Disposition handling accounts for any of it.

-Eric

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 08 Nov 2010 12:27:34 +1300

On 8/11/2010 12:17 p.m., Adam Barth wrote:
> On Sun, Nov 7, 2010 at 3:02 PM, Adrien de Croy<[email-masked]>  wrote:

the day the first major browser chooses to do it one of 2 things will 
happen.  If they do it properly that is.  If they don't they are hosed.

If they post a big red page and say

"this website responded in an improper manner, which may have security 
implications.  We are protecting you from this..." or words to that 
effect, then who will the user blame?  The browser or the website?

Then users are more likely to push the website to be fixed, and in any 
case the website wants to be seen, and not be seen as a potential 
security risk.  In fact Google could do this unilaterally with its 
existing infrastructure (the one that tells us a page is unsafe to 
visit).  Then the other browser vendors can breathe a sigh of relief and 
undo all the hacks they were forced (so they believed) to put in to 
compete with IE.

> You and I appear to have a different interpretation of history.

Funny, I thought you shared that view, and that's why you wrote your I-D 
on content sniffing.

> We disagree about whether or not that is a problem.

see above.

Regards

Adrien

---

From: Adam Barth <[email-masked]>
Date: Sun, 7 Nov 2010 15:43:26 -0800

On Sun, Nov 7, 2010 at 3:27 PM, Adrien de Croy <[email-masked]> wrote:
> implications.  We are protecting you from this..." or words to that effect,
> then who will the user blame?  The browser or the website?

Honestly, the browser.  We've tried this experiment with mixed content
(HTTP resources inside HTTPS pages).  That's pretty much exactly what
the messaging is in the user interface.  We get feedback from users
that they've switched to other browsers that don't show them these
scary warnings.

Crying wolf doesn't help.  What does help is to show warnings when
there's a real risk of something bad happening (e.g., we've detected
that this web site is actively exploiting users that visit it).

> Then users are more likely to push the website to be fixed, and in any case
> the website wants to be seen, and not be seen as a potential security risk.
>  In fact Google could do this unilaterally with its existing infrastructure
> (the one that tells us a page is unsafe to visit).  Then the other browser
> vendors can breathe a sigh of relief and undo all the hacks they were forced
> (so they believed) to put in to compete with IE.

I even tried calling Chase customer service about the mixed content
warnings on the login page to their banking web site.  They told me to
use Firefox because the warning didn't show in that browser (this was
a case where Chrome's mixed content detector was more accurate than
Firefox's in detecting the vulnerability).

In any case, you should feel free to bring a browser to market that
makes different decisions here.  I'm happy for the market to prove me
wrong, but so far every successful entrant into the market has used
the same strategy here.

Adam

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 08 Nov 2010 01:05:07 +0100

>see invalid messages here too.

Common sense tells us many things, for instance, that it is improper to
respond to an argument about "not many" with "maybe some".

What you are saying is untrue. Julian has gathered and shared data on
how inconsistent error handling is across implementations; we can draw
conclusions from that due to the obvious causal relationship between
error rates and failure rates.

And I have gathed and analyzed a data set and shared my findings about
http://lists.w3.org/Archives/Public/ietf-http-wg/2010OctDec/0029.html
actual use of the Content-Disposition header on web sites, with a clear
result as far as I am concerned.

You are the one demanding change, so the burden of proof is yours to
bear. Seriously, I get my fair share of entertainment reading about the
differences between 99.99% and 99.999% compatibility and how Mozilla
have decided to not compete for market share in Asia, in the end we'll
resolve this issue through rough consensus and running code, not based
on how many times you say you really like to standardize error handling.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Adam Barth <[email-masked]>
Date: Sun, 7 Nov 2010 16:14:15 -0800

> error rates and failure rates.

This is not a correct inference.

> result as far as I am concerned.

I don't see any data in that message.

> on how many times you say you really like to standardize error handling.

Indeed.  We've been shipping running code years now.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Mon, 08 Nov 2010 09:58:11 +0100

> Why would a server ever send two Content-Types headers?  Why an HTML

Because the code is setting it twice, maybe in different components.

> document ever mis-nest tags?  Why would a server ever send nonsense

Because UAs accept it, and authors have almost zero reason to test it?

Yes.

But it doesn't happen for all areas with the same frequency. Thus I 
think you conclusion that the answer to this must be the same across all 
layers is wrong.

> has been causing me a lot of pain recently.  Consider this syntax in

I was talking about Content-Disposition specifically.

> Yes, just like we tell them that U+2665 isn't allowed.

The inability to express a literal %xx sequence in a filename IMHO would 
be very strange. Right now the restriction is there de factor, but so is 
the restriction to ISO-8859-1. This spec tries to improve that.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 08 Nov 2010 10:02:49 +0100

> that they've switched to other browsers that don't show them these

Can we please stop mixing up things?

Ignoring a header field does *not* require displaying an error message 
to the user. Nor does it break any kind of contract, as C-D is advisory 
only anyway.

What we *can* do is to say what a good way of "ignoring" broken stuff is.

For instance:

- if the value doesn't parse per the ABNF, it's invalid, so ignore the 
whole header

- if a parsed parameter value is invalid (percent-escaped sequence does 
not match the specified charset, for instance), ignore the *parameter* 
and go on

...but I'm not even sure that having the 2nd case makes any measurable 
difference in practice.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 9 Nov 2010 13:13:19 +1100

My inclination here is to (as with the others) punt the issue of actually defining 'invalid' in the 'core' specification; it can be either an appendix or a separate document.

However, I'm wondering if we can at least identify those places where errors may occur consistently; e.g., by saying things like:

"Header values with multiple instances are considered invalid."
"Header values that do not match the ABNF are considered invalid."
"..."

and then something like

"A header value that is considered invalid MAY be ignored, or MAY have error handling algorithms applied to attempt to recover the intent of the message."

Thoughts?

--
Mark Nottingham   http://www.mnot.net/

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 09 Nov 2010 17:03:05 +1300

> "A header value that is considered invalid MAY be ignored, or MAY have error handling algorithms applied to attempt to recover the intent of the message."

On the topic about suggesting invalid headers be ignored.

Typically ignoring invalid headers is actually a recovery strategy, 
since the goal is to attempt to proceed to process the message.  Not 
ignoring the error would be the case where the message is rejected somehow.

Some invalid headers are not safe to ignore.  E.g. Content-Length, 
Transfer-Encoding, Content-Encoding and maybe some others.

So, maybe we need 2 classes of header.  One where we feel it's safe to 
ignore errors (act as if the header were absent), and others where we 
feel it's not safe.  I don't think one blanket rule for all really works.

Maybe even differentiate between errors in say parameters (e.g. q 
values) vs the rest of the header.

Or did you use ignored in the sense of "treat as if the header were 
omitted"?

In which case in the case of Content-Length, there's defined processes 
for how to deal with lack of that.  My gut feel is this particular one 
would possibly lead to smuggling issues, so we would be back to maybe 2 
classes of header (ignorable and not).

For intermediaries it might help to have some language about headers 
believed to be in error (in respect of forwarding).

E.g. should a proxy:

* strip
* forward as is (leave it up to the next agent)
* attempt to correct (I wouldn't recommend this)
* reject

Cheers

Adrien

>> On Mon, Nov 1, 2010 at 5:43 PM, Mark Nottingham<[email-masked]>  wrote:

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 9 Nov 2010 15:06:10 +1100

Hi Adrien,

That's good background thinking, but for the moment I'm trying to restrict us to just Content-Disposition, to simplify the discussion. Can we come back to this once we see how that goes?

Regards,

On 09/11/2010, at 3:03 PM, Adrien de Croy wrote:

> Typically ignoring invalid headers is actually a recovery strategy, since the goal is to attempt to proceed to process the message.  Not ignoring the error would be the case where the message is rejected somehow.
> 
> Some invalid headers are not safe to ignore.  E.g. Content-Length, Transfer-Encoding, Content-Encoding and maybe some others.
> 
> So, maybe we need 2 classes of header.  One where we feel it's safe to ignore errors (act as if the header were absent), and others where we feel it's not safe.  I don't think one blanket rule for all really works.
> 
> Maybe even differentiate between errors in say parameters (e.g. q values) vs the rest of the header.
> 
> Or did you use ignored in the sense of "treat as if the header were omitted"?
> 
> In which case in the case of Content-Length, there's defined processes for how to deal with lack of that.  My gut feel is this particular one would possibly lead to smuggling issues, so we would be back to maybe 2 classes of header (ignorable and not).
> 
> For intermediaries it might help to have some language about headers believed to be in error (in respect of forwarding).
> 
> E.g. should a proxy:
> 
> * strip

---

From: Julian Reschke <[email-masked]>
Date: Tue, 09 Nov 2010 15:34:06 +0100

> "A header value that is considered invalid MAY be ignored, or MAY have error handling algorithms applied to attempt to recover the intent of the message."

Here's a rough proposal 
(<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/259/i259.diff>) 
how this could look like:

-- snip --
5.  Parsing

    This document does not require any specific handling of invalid
    header field values.  With this in mind, the text below describes a
    simple strategy for parsing the header field and detecting problems
    in general, or in specific parameters.

5.1.  Combine Multiple Instances of Content-Disposition

    If the HTTP message contains multiple instances of the Content-
    Disposition header field, combine all field values into a single one
    as specified in Section 4.2 of [RFC2616].

5.2.  Parsing for Disposition Type and Parameters

    Using the simplified grammar below:

      field-value = disp-type *( ";" param )
      disp-type   = token
      param       = token "=" value

    ...parse the field value into a disp-type (disposition type) and a
    sequence of parameters (pairs of name (token) and value).

    If the field value does not conform to the grammar (such as when not
    exactly one disposition type is specified), ignore the whole header
    field.

5.3.  Checking Cardinality Constraints

    If the parameter sequence contains multiple instances of the same
    parameter name, ignore the whole header field.

5.4.  Post-Process Parameter Values

    For each parameter, post-process the associated value part according
    to the grammar:

    o  According to Section 3.2.1 of [RFC5987] for parameters using the
       RFC 5987 syntax (such as "filename*").  If this fails, just ignore
       this parameter.

    o  According to the grammar for quoted-string (Section 2.2 of
       [RFC2616]) for values starting with a double quote character (").

    o  Verbatim otherwise.

    Note that this step starts with an octet sequence obtained from the
    HTTP message, and results in a sequence of Unicode characters.
-- snip --

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 12 Nov 2010 15:58:39 +1100

I'm confused. I thought that we were going to talk about error handling in an appendix, but it appears you're starting to talk about it here.

On 10/11/2010, at 1:34 AM, Julian Reschke wrote:

> Here's a rough proposal (<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/259/i259.diff>) how this could look like:
> 
> -- snip --
> 5.  Parsing
>

---

From: Julian Reschke <[email-masked]>
Date: Fri, 12 Nov 2010 08:53:15 +0100

> I'm confused. I thought that we were going to talk about error handling in an appendix, but it appears you're starting to talk about it here.

1) Yes, it should be an appendix.

2) Well, it's parsing advice. It appears that some readers have trouble 
understanding how to derive a parsing strategy from the way how we 
currently write specs, so this is an attempt to describe just that.

Best regards, Julian