Subject: [dnsext] Not quite OT: on format=flowed

From: Nicholas Weaver <[email-masked]>
Date: Thu, 3 Jun 2010 09:51:25 -0700

There was one complaint recently on a thread about how some mail was no longer being sent with format=flowed.

	With, of course, no info on who was sending such "offending" messages!  Hardly helpful...

	Well, here's the thing:  Mac Mail USED to use format=flowed and delsp=yes (RFC 3676), up until 10.6.2.  Then Apple dropped them both (because actually, you need BOTH).

	So what does these options do?  Well, it basically allows the text/plain format to NOT be text/plain.  

	Instead, it indicates that the mail reader can undo pseudo-linebreaks and word wrap however it desires, but in order to provide compatibility with old (*cough* Unix *cough*) tools focused around the notion of an 80 character, fixed-width font, the message was sent with a 72 character linewrapping using a convention of "space CRLF" as a linefeed that can be removed.

	So why did Apple drop this?

	Why?  Simply put, because it seems some mail clients never did this right:

From the 10.6.2 release notes:  "improved line wrapping compatibility when sending to certain mail clients and Microsoft Exchange Server"

	So if you have a mail client that uses a variable width font but never redid the line breaks, you'd get ugly stuff if its format=flowed, delsp=yes.

	So they decided to drop that and instead just send plain text with no additional linebreaks.

	So why is this discussion relevant?

a:  In developing software, you can't please everybody.  

	format=flowed, delsp=yes was supposed to make it so that old stuff could stay, well, broken.  But it relied on new stuff (non-fixed-width font using stuff) to respect the option.  Instead there developed a situation where you have TWO different classes of brokenness:

1)  Fixed-width stuff that doesn't linebreak non-line-broken text.
2)  Variable-width stuff that doesn't relinebreak already line-broken text.

	And the result is you CAN NOT please both:  One or the other has to get ugly.  For the longest time, the decision was obviously to favor class 1.  Now its to favor class 2.  

	This is perfectly fine IMO as long as things are explicit about it.  Pick-your-poison moments happen all the time.

b:	In deciding which brokenness to respect and which to say "go F-yourself", you need to decide whether the amount of brokenness matters.  Obviously, Apple decided that the old Unix side is now less important than variable-width mail readers.

	This in particular makes me worry about fragments, EDNS, and DNSSEC.  Its probably OK to say "go F-yourself" (or at least hit performance seriously) on networks that assume DNS<=512B or EDNS doesn't exist.  This is actually relatively rare in measurements on the resolver side.  

	But it is probably NOT OK to assume UDP fragments work.

c: 	Established practice can trump standards.

	format=flowed, delsp=yes is a 6+ year old standard.  But because there were significant modernish (variable-width) mail readers not accepting the convention, Apple dropped it.

	Similarly, RFC 2181 says that the authority should return all records for the associated RRSet.  Nobody who's hosting an RRSet of >100+ entries (eg, www.google.com) in their right mind actually does this.  The DNS tricks underlying CDN distribution are now established practice, no matter what the standard says.

d:	Don't expect multisided client change to fix things.

	Why did apple drop format=flowed, delsp=yes?  Because the client changes to support it simply weren't happening.

	This is why I really find AMAZINGLY silly the arguments that "Because the CDN DNS tricks mostly affect just HTTP service, HTTP should change to support the necessary redirect semantics".  

	If you can't get a few mail-readers to linewrap text right! what makes you think you can get every web browser to change to support a different mechanism for redirection (that is largely unnecessary because the CDN DNS tricks largely DO work today!).

---

From: Paul Vixie <[email-masked]>
Date: Thu, 03 Jun 2010 17:12:42 +0000

incidentally, but perhaps illuminatingly, i find it bizarre that folks will
just buy a new computer and use whatever Mail.app it comes with, and change
their personal practices as nec'y to conform to the limitations of the tool.

i wear out my computers every few years and i pick new ones based in part
on their ability to support my existing personal practices and preferred
tools. i say "illuminatingly" because this may underscore other positions.

> From: Nicholas Weaver <[email-masked]>
> Date: Thu, 3 Jun 2010 09:51:25 -0700
> ...
> 	If you can't get a few mail-readers to linewrap text right! what
> 	makes you think you can get every web browser to change to support
> 	a different mechanism for redirection (that is largely unnecessary
> 	because the CDN DNS tricks largely DO work today!).

because in http, there are dominant open source implementations of both the
browser (firefox) and server (apache), and the rest of the industry has to
play ketchup with those, no matter how commercially successful they are or
how many eyeballs or how much content they control. "sweet."

---

From: Nicholas Weaver <[email-masked]>
Date: Thu, 3 Jun 2010 11:35:24 -0700

> tools. i say "illuminatingly" because this may underscore other positions.

Uh, that has nothing to do with my choice of mail reader.  I actually stuck with mutt for a long time after switching platforms....

And it actually isn't a limitation on MY tool, but OTHER tools: there is a pick-your-poison design decision that the authors of my mail client had to make, which in either case ONLY affect stupid tools: old Unix tools that mistakenly assume that DNS is 512B, err, that all mail is 72 character lines of fixed-width text, or new gui tools that do not understand that you can have plain text with a trivial amount of additional, in-band formatting and still call it "plain text" as a mimetype.

> how many eyeballs or how much content they control. "sweet."

I didn't know you were a comedian Paul.

Look at HTTP pipelining:  Only Opera dares support it on by default, and thats in the HTTP 1.1 standard right from the get-go.  Apache does it just fine.  And that is a huge performance win, in a business which cares a lot about speed.  

And Firefox has supported it for YEARS, but doesn't enable it.  Why? because there are random broken proxies and old IIS servers that barf on it, and Firefox's pipelining code even includes special-casing of old IIS servers. [1]

Effectively, HTTP pipelining is a "universal deployment" problem, you need ALL servers and proxies not to barf on it before you turn it on, or you need to have intelligent fallback detection that causes things to fail gracefully when you DO have something that barfs on it.

Similarly, a "HTTP site redirect" option would require both new server and new client options, you need it to STILL work with old clients, and you'd still be burning 3 more RTTs when compared with just sending the client to the right host in the first place which you can do today with DNS tricks.

And as far as clients, IE is probably the most important despite what open-source evangelists think: what IE does is the standard in many cases and OTHERS have to play catch-up.  In fact, we have code in the Netalyzr web page to deal specifically with an IE6 rendering bug.

If Apple, after for YEARS pushing out mail which is standard-compliant with format=flowed, stop and gave up because of a universal deployment problem and that shoving standard-compliant mail down the throat of incompliant tools didn't get the tools to change, what hope do you really think there would be for an HTTP-level redirect, especially when the problem is already largely solved by DNS authority behavior, except in the rather perverted cases (eg, third party resolvers) where DNS goes on a widely divergent path compared with the rest of the user's traffic.

[1] And its also far less important if that nice DNS authority sends you to a web server thats 20ms away instead of 100ms away...  Amdahl's law and all that good stuff.