Subject: [#259] Handling invalid Content-Dispostion headers

From: Mark Nottingham <[email-masked]>
Date: Thu, 4 Nov 2010 13:42:40 +1100

In looking at Adam's feedback and the resulting discussion, I'm considering whether we should try to define an OPTIONAL error-handling profile that defines a method of handling invalid Content-Disposition field-values.

E.g., this could be an appendix to the specification that lists a set of behaviours, such as:
  - when there are multiple parameters with the same value, choose the first one
  - when there are characters that aren't allowed in the BNF, transform them to...
  - etc.
that implementers could follow if they wish to. Other specifications could, at their option, require conformance to such a profile.

By necessity, such a thing would not duplicate requirements from the authoritative text, would not override those requirements, and it would not provide alternate algorithms.

To help figure out if this is a productive way to go, I'd like to hear:
 a) thoughts from folks about this approach, and 
 b) some indication of who'd be willing to contribute text and/or time for testing, and 
 c) expressions of interest from implementers in such an optional profile (i.e., an indication that it's something they'd find useful).

</chair hat>

Regards,

--
Mark Nottingham   http://www.mnot.net/

---

From: "Thomson, Martin" <[email-masked]>
Date: Thu, 4 Nov 2010 10:54:13 +0800

On 2010-11-04 at 13:42:40, Mark Nottingham wrote:
> In looking at Adam's feedback and the resulting discussion, I'm
> considering whether we should try to define an OPTIONAL error-handling
> profile that defines a method of handling invalid Content-Disposition
> field-values.

You preface this with a very specific case, but the description you provide could apply to many other headers.

It seems likely that Adam's complaints are applicable to other headers in other (httpbis) specifications.  Do you want to address this question more generally as well?  Or did you want to keep a lid on that for now?

--Martin

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 4 Nov 2010 13:55:09 +1100

I'm viewing this as something of a test case.

Regards,

---

From: Julian Reschke <[email-masked]>
Date: Thu, 04 Nov 2010 16:16:59 +0100

>   c) expressions of interest from implementers in such an optional profile (i.e., an indication that it's something they'd find useful).

I believe it would be a waste of energy to specify a specific strategy 
*unless* we have evidence that it's needed in practice. For this 
particular case (Content-Disposition in HTTP) I added several tests for 
invalid headers just to observe whether implementations agree on what to 
do, and in most cases they do not.

In addition, micro-managing this may even be harmful if we end up 
defining different strategies for different headers (for instance for 
Content-Type as compared to Content-Disposition). It seems it would be a 
better use of our time to work on test cases, identify common syntax 
patterns, and help UAs to get their code fixed.

That being said: a potential security risk changes things (see 
Content-Length).

Finally, I'm not opposed to people working on things like Informational 
RFCs on this topic, but we should be careful that these documents then 
really represent cross-UA agreement, and not just what somebody learned 
from several browser's source code. (For instance, HTML5 has 
requirements on Content-Type parsing that disagree with what IE does, 
but the spec claims it's needed for "compatibility with existing content").

Best regards, Julian

---

From: Maciej Stachowiak <[email-masked]>
Date: Thu, 04 Nov 2010 17:48:17 +0100

I think such a profile would be useful, both for Content-Disposition and potentially for other HTTP header values.

Regards,
Maciej

---

From: Dan Winship <[email-masked]>
Date: Thu, 04 Nov 2010 15:42:11 -0400

> do, and in most cases they do not.

So if there's no consistent behavior now, then rather than specifying
how to interpret incorrect headers, can browsers just agree that the
new-and-improved consistent behavior will be "draconian"? ("If the
header doesn't exactly match the grammar, then you MUST ignore it in its
entirety".)

-- Dan

---

From: Adam Barth <[email-masked]>
Date: Thu, 4 Nov 2010 12:45:43 -0700

> header doesn't exactly match the grammar, then you MUST ignore it in its

That seems unlikely to be deployable.  What's more likely is that the
new-and-improved consistent behavior will be permissive rather than
draconian.  For example, in the multiple disposition-type case, we'd
take the first disposition-type (or whatever) rather than ignoring the
header field entirely.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Thu, 04 Nov 2010 23:15:08 +0100

> header doesn't exactly match the grammar, then you MUST ignore it in its

The Konqueror developers are actually doing this.

Best regards, Julian

---

From: "Eric J. Bowman" <[email-masked]>
Date: Thu, 4 Nov 2010 16:54:38 -0600

Julian Reschke wrote:
> Content-Type as compared to Content-Disposition). It seems it would
> be a better use of our time to work on test cases, identify common
> syntax patterns, and help UAs to get their code fixed.
> 

+1.  Also, +1 to comments by Mike, Simon, Shelley, Dimitre, Alain and
Gerrit about browser-driven standardization:

http://saxonica.blogharbor.com/blog/_archives/2010/11/4/4671786.html

That discussion isn't about HTTP, but all the sentiments apply.  Is
there one standardized way to parse what are essentially MIME headers,
or 1+n algorithms because that's how browsers are (unaccountably to
anyone) written?

-Eric

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 05 Nov 2010 02:32:54 +0100

* Mark Nottingham wrote:
> a) thoughts from folks about this approach,

Well, to offer a very simple perspective on this, and to pick just some
random examples, several years ago I had to use a web service where I'd
to upload a file, make changes to the file locally and upload it again.
The browser I was using allowed re-uploading the file using "reload",
but when the file size changed it specified the old size of the file,
and sent the new content. That's a clear violation of the HTTP standard,
there is no need to conduct research on the real-world impact of fixing
the bug, there should be nothing remotely difficult about fixing it,
this is an actual end-user affecting bug, one where a user bothered to
both file a bug about it and talk to staff of said vendor about it, and
several years later that bug is still not fixed.

As another example, I was experimenting with fully conforming but some-
what long domain name labels. One browser exhibited utterly strange be-
havior, it truncated the label according to non-obvious rules. This may
have security implications, is a clear violation of the specifications
in question, fixing it has zero compatibility impact, it's a real-world
problem considering that long domain names aren't that uncommon, it was
reported, and several years later the problem is still not fixed.

What we have here with the Content-Disposition header, well, there is
zero confusion about how to handle escapes in quoted strings, there are
test cases, bugs filed for in some cases a long time, affected browser
vendors have yet to fix their code. Similarily, while there may have
been confusion about the somewhat arcane escaping syntax implemented by
some, that browsers differ in which syntax they support has been known
for quite some time, again there are test cases and bugs and so on, the
ability to specify non-ascii filenames reliably is actually in important
feature, yet browser developers aren't exactly jumping on the chances to
address this problem.

And I could go on with any number of browser bugs that have been known
for years, that do not require research into how compatiblity with ex-
isting content may be impacted, do not require forming consensus around
some issue, do not require writing specification text, that clearly do
affect real-world use cases, that are not being fixed, apparently for a
lack of botherones and round-tuits.

We have problems suggesting file names with non-ascii characters for
HTTP resources. The draft attempts to address that, by specifing a
profile that already works across a number of implementations, so that
the other implementations can catch up and so that people wishing to
suggest such file names have a reference on how to do that. It does
that reasonably well, and there is reason to believe that the profile
will actually be implemented where it isn't already.

There is no reason to believe that browser vendors will invest rather
significant resources to come up with a specification and a test suite
and making sure the specification gets implemented in their products
only to make their implementations behave more alike. This is low-level
parsing code, which is where the next unanticipated compatibility pro-
blem and the next security advisory lie. Much less is there evidence
that non-browser implementations will follow suit. And most certainly
there are more cost-effective things browser vendors could invest in.

Any effort that will be spent on specifying how to handle malformed
Content-Disposition headers will without a doubt negatively affect
what effort is spent on fixing other bugs, implementing new features,
creating test suites, and so on. The moment, say, Maciej says Apple
is committed to spend $10,000 USD in engineering resources on this, I
will be all supportive. Until then I will consider anyone arguing we
should specify how to handle content that does not exist, for all
that I can tell, a candidate for an "Enemy of Interoperability" award.

An even simpler perspective is this:

  Don't start a business if you can't explain what pain it solves,
  for whom, and why your product will eliminate this pain, and how
  the customer will pay to solve this pain. -- Joel Spolsky.

Here there is no evidence of an actual pain, there is no evidence
that there is anything we could reasonably do to eliminate it, and
much less evidence that anyone would actually pay to have it solved.
I can understand that this discussion makes sense from an engineering
perspective, but it does not make sense from a business perspective.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Maciej Stachowiak <[email-masked]>
Date: Fri, 05 Nov 2010 09:17:09 +0100

> http://saxonica.blogharbor.com/blog/_archives/2010/11/4/4671786.html

Posting links to divisive flamebait doesn't seem like a good way to move the conversation forward.

Regards,
Maciej

---

From: "Eric J. Bowman" <[email-masked]>
Date: Fri, 5 Nov 2010 10:57:52 -0600

> move the conversation forward.

Meekly holding my tongue in the face of an effort to make browser
concerns of paramount importance here, doesn't seem like a good way to
move HTTPbis forward.  There's a reason the HTML 5 effort has led to
such out-of-character posts from the likes of Michael Kay.  Should I be
pleased with Adam's dismissive, arrogant, passive-aggressive responses
to my concerns here, regarding his insistence that C-D is "useless"?

Now that I've experienced first-hand exactly the attitude which others
have been complaining about for some time, not only do I understand the
prevailing frustration, I find myself in complete agreement with it.
Treating others professionally, and with the respect they deserve, is
what moves the conversation forward -- and prevents the need to post
rants/links like that.

-Eric

---

From: Maciej Stachowiak <[email-masked]>
Date: Sat, 06 Nov 2010 07:31:27 +0100

Hi Eric,

> what moves the conversation forward -- and prevents the need to post

For a long time, browser implementors have not been a significant part of the conversation for standards like HTTP. Perhaps as a result of this, HTTP underserves the needs of Web browsers in some ways. Now, browsers are not the only HTTP clients in the world, but they are fairly important ones - a whole lot of people in the world run HTTP servers with the primary goal of delivering content to users browsing the Web with a Web browser. It's not the only use case, but it's one important use case, and it often has specialized requirements. To have a good standard, it's important to have a wide range of the important stakeholders at the table.

Now, at least a few browser implementors are trying to join the conversation and express our requirements. The topic of this particular thread is a fairly trivial request - an optional specification for error recovery when parsing the Content-Disposition header. Browser implementors like to agree on error recovery, because we find that in general it makes things better for our users. It seems like experimenting with this approach using an *optional* spec for *one* header field is a low-stakes experiment that won't hurt anyone who doesn't care to participate.

For you to post angry tirades in this thread, whether your own or someone else's, seems out of all proportion to the point at stake here. Further, it makes browser vendors feel attacked and unwelcome. That's not going to draw people into the conversation. People with different opinions from you generally have their own good reasons, perhaps due to different experiences or culture.

Please consider the consequences of your attitude. We are all much better off if browser vendors can participate effectively in this group. It's ok to disagree on technical points, but please, try to keep it polite and professional. Stick to technical points and not personal attacks. And consider giving a little extra leeway to some people who are important stakeholders, but often feel unwelcome in this group. I find it especially ironic that you call for professionalism and respect while not really granting the same here. I hope you will reconsider.

Regards,
Maciej

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 6 Nov 2010 17:41:55 +1100

Thanks, Maciej.

For the record, I've asked the browser vendors to engage here, am happy that they're showing up, and I expect the group to work with them professionally. 

I've already warned a few people privately about the tone and content of their messages here; if it continues, those warnings will be repeated publicly, which means we'll be walking down the road of RFC3934 -- steps that I'd rather not take, but won't hesitate to if necessary.

Regards,

---

From: "Eric J. Bowman" <[email-masked]>
Date: Sat, 6 Nov 2010 02:51:33 -0600

Maciej Stachowiak wrote:
> 
> For a long time, browser implementors have not been a significant
> part of the conversation for standards like HTTP. Perhaps as a result
> of this, HTTP underserves the needs of Web browsers in some ways.
> Now, browsers are not the only HTTP clients in the world, but they
> are fairly important ones - a whole lot of people in the world run
> HTTP servers with the primary goal of delivering content to users
> browsing the Web with a Web browser. It's not the only use case, but
> it's one important use case, and it often has specialized
> requirements. To have a good standard, it's important to have a wide
> range of the important stakeholders at the table.
> 

Nevertheless, browser vendors have managed to generate significant ill
will in the developer community by ignoring process and consensus.  In
the case of microdata, refusing to even reveal who the stakeholders
*are*, and ignoring the request of the TAG to remove that document in
favor of the consensus reached around RDFa, is just the sort of thing
which might result in my questioning the motives of browser vendors'
participation in this process.  Desirable as that participation may be
in the general sense, my posts reflect a suspicion that there is no
intent to respect established consensus on the scope of HTTP.  I'll
tone it down, as I've been warned, but my suspicion remains and IMHO,
is justified.

>
> Now, at least a few browser implementors are trying to join the
> conversation and express our requirements. The topic of this
> particular thread is a fairly trivial request - an optional
> specification for error recovery when parsing the Content-Disposition
> header. Browser implementors like to agree on error recovery, because
> we find that in general it makes things better for our users. It
> seems like experimenting with this approach using an *optional* spec
> for *one* header field is a low-stakes experiment that won't hurt
> anyone who doesn't care to participate.
> 

That is one opinion.  My opinion is that this is not a trivial request,
it's a request for fundamental change.  My reasoning is that there's an
architectural concern which precludes HTTP from standardizing error
recovery, and that this request is best handled as part of an optional
extension to HTTP -- a set of best practices which is free to be
updated _independently_ of the protocol specification, as such
practices will most likely change more frequently than should the
underlying protocol.  I'm more than willing for browser vendors to
engage on this technical issue which I find important, but no
explanation as to why HTTP must be extended to account for error
correction has been forthcoming, to justify even experimenting with it.

> 
> Please consider the consequences of your attitude. We are all much
> better off if browser vendors can participate effectively in this
> group.
>

Please consider the consequences of others' attitudes.  We're better
off working together to standardize an open platform.  Browser vendors
are demanding a one-way street, from where I sit as mostly a keenly-
interested observer.  Time and again, I've watched the input from this
group to the HTML WG be cavalierly dismissed, not to mention the input
from the TAG, or professionals like Dr. Kay.  This makes me sensitive
to things like Julian's draft being called "useless" when it's just
like any of the specs I read to teach myself HTTP -- specs which Julian
played a large role in authoring, and which have resulted in my being
able to create working, interoperable code.  Including from this latest
C-D draft, as both sender and receiver.

>
> And consider giving a little extra leeway to some people who are
> important stakeholders, but often feel unwelcome in this group.
>

Those stakeholders need to recognize the importance of others, right
down to lowly developers like myself who aren't standards wonks.  My
participation in this group only recently began, and my own welcome is
up in the air, so I'd appreciate if my concerns weren't ignored by
those they're directed at -- more like a request for common courtesy,
than leeway.

-Eric

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 6 Nov 2010 20:53:09 +1100

Eric,

Saying that you'll tone things down isn't an opportunity to continue to rail against browser vendors. It's off-topic and counter-productive, and as such this is a public warning; if you continue in this vein, you'll be temporarily banned from posting to the list.

Let's get back to work, please.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 06 Nov 2010 15:36:47 +0100

* Maciej Stachowiak wrote:
>conversation and express our requirements. The topic of this particular
>thread is a fairly trivial request - an optional specification for error
>recovery when parsing the Content-Disposition header. Browser
>implementors like to agree on error recovery, because we find that in
>general it makes things better for our users. It seems like
>experimenting with this approach using an *optional* spec for *one*
>header field is a low-stakes experiment that won't hurt anyone who
>doesn't care to participate.

The Content-Disposition header is now 15 years old. There has never been
any confusion about how to handle backslash characters in quoted-strings
yet in those 15 years none of Microsoft, Mozilla, Apple, and Google have
bothered to handle them correctly, despite people filing bugs about that
years ago, https://bugs.webkit.org/show_bug.cgi?id=22381 for instance.

When Julian filed a bug for this in Chromium the implementer commented
on his tests saying they appear to be "tests for the sake of test rather
than solving real-life problems". That Apple does implment the RFC 2231
encoding has been https://bugs.webkit.org/show_bug.cgi?id=15287 known
for over three years, https://bugs.webkit.org/show_bug.cgi?id=20407 with
duplicate reports even. These are valid things, you need them in order
to suggest file names containing "special" characters, which actually is
an important feature.

We have been running an experiment, if you will, testing whether browser
implementors will implement the Content-Disposition consistently based
on a specification for fifteen years (and similar experiments are on-
going for any number of specifications, and the IETF httpstate working
group is about to launch another large scale experiment in this area).

So far nobody has volunteered to conduct the research required to come
up with a specification for error-tolerant Content-Disposition headers
where we could be reasonably sure that, in the unlikely event that the
browser implementers care to align their implementations with the spe-
cification, implementers will not run into compatibility problems that
they feel they need to handle in a manner contrary to the specification.

So we do not have the resources to conduct such an experiment, if we
did, it would be redundant with other experiments, and past experience
suggests we would not care much for the outcome. So I suggest we thank
the browser implementers telling us they like it when their code behaves
in a manner consistent with the code of their competitors; and revisit
this issue when browser implementers have shown through running code
that they not only like that, but actively invest resources into that,
something they haven't done much in the past couple of years as far as
the Content-Disposition header is concerned.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Sat, 06 Nov 2010 17:22:38 +0100

On 04.11.2010 20:45, Adam Barth wrote:
> ...
> ...

Making IE non-compliant (or even more-compliant than before).

Can you remind me what the gain is? Do you have data on content of this 
type relying on one specific behavior?

I'll say it again: there are many things that can be done to improve 
things (for the whole HTTP space):

- clarify the spec for conforming headers, add examples

- write test cases

- identity bugs, lobby for fixing and/or supply patches

- identify common header micro syntaxes

- write reference implementations for the above

- encourage re-use of those patterns

- think about standardizing error recovery (or rule it out)

The last point in *my* point of view has the lowest priority, in 
particular as long as the implementations do not even treat *valid* 
header field values correctly.

Claiming we need to do all of this at the same time isn't constructive 
as long as the pool of people doing actual work doesn't grow.

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Sat, 6 Nov 2010 14:21:42 -0700

On Sat, Nov 6, 2010 at 9:22 AM, Julian Reschke <[email-masked]> wrote:
> Can you remind me what the gain is?

The gain is that in the future user agent implements can read this
document and converge their behavior be the same.  Writing down
desirable, implementable error handling leads to interoperability.

> Do you have data on content of this type relying on one specific behavior?

That's not relevant to this discussion.  We're focused on the future.
Existing content is a constraint on what type of error handling is
implementable, not one whether we should recommend a specific way of
handling errors.

> I'll say it again: there are many things that can be done to improve things
> (for the whole HTTP space):

I'd encourage folks who are interested in doing those things to do those things.

[...]

> The last point in *my* point of view has the lowest priority, in particular
> as long as the implementations do not even treat *valid* header field values
> correctly.

I think this discussion is evidence that different folks in this
working group have different priorities.  In particular, my point of
view is that defining precise error handling semantics is quite
important.

> Claiming we need to do all of this at the same time isn't constructive as
> long as the pool of people doing actual work doesn't grow.

We have many problems in life, but a lack of engineering resources
isn't one of them, thankfully.

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 7 Nov 2010 10:02:01 +1100

My personal .05:

> Making IE non-compliant (or even more-compliant than before).

How so? If it's an optional error-handling profile, whether they implement it or not doesn't affect conformance to Content-Disposition. It's true that they won't conform to the error handling profile, but I'd hope that the folks coming up with that profile would do everything they can to make it at least mostly backwards-compaitlbe with IE.

> I'll say it again: there are many things that can be done to improve things (for the whole HTTP space):
> 
> The last point in *my* point of view has the lowest priority, in particular as long as the implementations do not even treat *valid* header field values correctly.

I agree that there are many things that we need to do, and that the valid cases are the low-hanging fruit. Based on recent bug traffic I've seen, it appears that the browser vendors *are* paying attention to it. 

That doesn't mean we can't do the last point, however, if we have the attention of the right people, as well as the appropriate resources. 

> Claiming we need to do all of this at the same time isn't constructive as long as the pool of people doing actual work doesn't grow.

Agreed. We cannot and <chair hat>will not</chair hat> do this work without the involvement of browser folks.

---

From: Adam Barth <[email-masked]>
Date: Sat, 6 Nov 2010 16:07:31 -0700

> up with a specification for error-tolerant Content-Disposition headers

Thankfully Julian has already done most of the legwork in this area.
Should have time in the next couple of weeks to sift through the data
and write up a proposal for some specific text for the appendix.

Adam

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sun, 07 Nov 2010 01:24:06 +0100

>and write up a proposal for some specific text for the appendix.

Very well, I think we can wait a week or two. Do note that things like,
say, using non-ASCII characters in a quoted-string filename value, or
using percent-encoding there, do not qualify as error handling, since
the protocol does not disallow those there. If you do intend to propose
text in that area, I would expect you to detail exactly what Internet
Explorer does and, if your proposal diverges from that, show us actual
real-world live web site data, including information on how to reproduce
your findings, that would justify that. Microsoft is exceedingly unlike-
ly to change its code there, and other browsers are very likely to
attempt to match Internet Explorer there, as we can see by looking at
Chromium; it would be disingenuous to recommend anything else.

(In other areas compatibility with Internet Explorer is less important,
for instance `Content-Disposition: inline; filename=attachment` can be
treated as "inline", even though IE treats it as "attachment". That of
course is no error case either, there are in fact almost none, so it
should not take you long to submit your proposal.)
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Adam Barth <[email-masked]>
Date: Sat, 6 Nov 2010 17:29:23 -0700

> should not take you long to submit your proposal.)

Thanks for your advice.  I look forward to your comments on whatever
text I propose.

Adam