Subject: Re: Canonical RFC formatting

From: Yaron Sheffer <[email-masked]>
Date: Wed, 02 Feb 2011 08:56:38 +0200

Hi Joe, can you clarify what do you mean by "does not support UTF-8? I am composing this mail on an Android phone that supports Hebrew just fine, I.e. UTF-8 with LTR rendering . And so do all current platforms I am familiar with.

Thanks,
      Yaron

Joe Touch <[email-masked]> wrote:

>Hi, Jeff,
>
>On 2/1/2011 9:21 PM, Jeffrey Hutzelman wrote:
>> --On Wednesday, January 26, 2011 10:29:54 AM -0800 Joe Touch
>> <[email-masked]> wrote:
>>
>>>
>>>
>>> On 1/26/2011 10:21 AM, Carsten Bormann wrote:
>>>> On Jan 26, 2011, at 18:38, Joe Touch wrote:
>>>>
>>>>> The tool could probably be modified to accept .xml and turn it into
>>>>> .txt, but that seems like a tool issue, not a "canonical formatting"
>>>>> issue.
>>>>
>>>> I don't think so, because then you wouldn't have submitted the legal
>>>> boilerplate.
>>>
>>> FWIW, that can be fixed on the confirmation page, where the text output
>>> including the legal boilerplate could appear and require positive
>>> confirmation.
>>
>> That seems OK for a document like
>> draft-arkko-dual-stack-extra-lite-03.txt, which one recent review
>> described as "a very brief (8-page) document", or for a document like
>> RFC347.
>>
>> I worry, though, that the submission process might time out while the
>> author verified correct compilation on a document like RFC5661.
>
>We have a two phase process right now. That can be part of it, i.e., you 
>submit, it compiles off-line, then emails you only when it's ready to 
>confirm.
>
>> More seriously, I also worry that the author is more likely to just
>> click through such a confirmation, rather than actually verifying the
>> correctness of the generated document.
>
>They do that now. People cut/paste without reading things. They generate 
>the TXT without reading it either.
>
> > Legal questions aside, this is a
>> problem when the generated document is the one that will be considered
>> "authoritative" if there is any discrepancy between versions. Better
>> IMHO to insist that the author submit the authoritative form, along with
>> whatever alternates are desired.
>
>I'm not sure what "authoritative" means for drafts. They're drafts, 
>after all ;-) IMO, integrating the on-line version with the submission 
>process just makes it easier for the author - they still can and do use 
>automated tools - some online -- and they can and do submit txt that 
>they haven't checked.
>
>IMO, providing a one-stop submission process just makes it less obscure 
>as a process. I see no reason we require the knowledge of magical 
>incantations to submit a doc.
>
>(I'd be all for .doc->.txt too, if I could figure out how to automate 
>it, i.e., .doc->.prn->.txt, where the last step is the tool I wrote and 
>*is* already online)
>
>> FWIW, I completely agree with Joe Hildebrand on the question of moving
>> to UTF-8 (which seems like a no-brainer), and on elimination of fixed
>> page breaks. Not so much on allowing 10000-character lines or generated
>> HTML with 10x as much tags as content.
>
>I was involved in some of the UTF8 discussions of the past. My challenge is:
>
>	- try to read the txt on a variety of currently common devices
>	if most render UTF8 fine, then let's change things
>
>The last time I checked, UTF8 didn't work in Windows; I just checked 
>again with Win7: the default text reader - Wordpad - changes fonts 
>mid-stream when it hits such chars, and Emacs doesn't work either.
>
>I also just checked on an iPhone4 and an iPad, and it doesn't work there 
>either.
>
>I'm all for whatever new format works in most of the most common places, 
>but knocking out Windows' default reader, an iPhone, and an iPad, and my 
>conclusion remains that it's not ready for prime time.
>
>We can continue that on rfc-interest. This isn't a new issue; I just 
>wanted to give the current state of things.
>
>Joe
>

---

From: Joe Touch <[email-masked]>
Date: Tue, 01 Feb 2011 23:21:19 -0800

On 2/1/2011 10:56 PM, Yaron Sheffer wrote:
> Hi Joe, can you clarify what do you mean by "does not support UTF-8?
> I am composing this mail on an Android phone that supports Hebrew
> just fine, I.e. UTF-8 with LTR rendering . And so do all current
> platforms I am familiar with.

I opened a UTF-8 file as an email attachment, and it displayed the UTF-8 
chars as garbage.

My metric is: open a .txt file, encoded in UTF-8, and see what happens.

Joe

> Joe Touch<[email-masked]>  wrote:
>
>> Hi, Jeff,
>>

---

From: Lachlan Andrew <[email-masked]>
Date: Wed, 2 Feb 2011 18:29:56 +1100

Greetings Joe,

Could you circulate an example problematic attachment, to see how many
people have the problem?  May it depend on whether a particular
Windows installation is set up for UTF-8?

For example, if you choose not to install fonts with accents, then
those characters can't be displayed.  The same applies to reading
ASCII if you choose to install an OS without any ASCII fonts.

Just a thought...

Lachlan

-- 
Lachlan Andrew  Centre for Advanced Internet Architectures (CAIA)
Swinburne University of Technology, Melbourne, Australia
<http://caia.swin.edu.au/cv/landrew>
Ph +61 3 9214 4837

---

From: Joel Jaeggli <[email-masked]>
Date: Tue, 01 Feb 2011 23:35:59 -0800

> My metric is: open a .txt file, encoded in UTF-8, and see what happens.

I opened several utf-8 text samples I have on the ipad and they rendered
reasonably well (except for amharic), including although of questionable
utility the braile sample. this was via the browsers rendering engine.

---

From: Joe Touch <[email-masked]>
Date: Tue, 01 Feb 2011 23:35:26 -0800

Sure - it was the test that Paul Hoffman wrote a while back. Attached.

My claim is that, ****out of the box, the following systems should be 
able to read the attached file with no additional configuration or software:

- Windows
- Linux
- MacOS
- FreeBSD
- IOS (iPhone/iPad)
- Android
- Blackberry

I'm sure there are a few others that people would like to throw in, 
notably things like:
- Kindle
- Sony e-reader

Maybe some others too. Not sure. But that's the basic test as a start.

I can read .txt on all these (excepting the e-book readers, which 
require a txt to something converter) *right now*.

Again, we can take this over to rfc-i...

Joe

On 2/1/2011 11:29 PM, Lachlan Andrew wrote:
> On 2 February 2011 18:21, Joe Touch<[email-masked]>  wrote:
>>
>>

---

From: Joel Jaeggli <[email-masked]>
Date: Wed, 02 Feb 2011 00:22:42 -0800

> able to read the attached file with no additional configuration or

	right, so this opened in textedit by default

which rendered:

	晓东 Fältström

> - IOS (iPhone/iPad)

this rendered in preview by default, which did not render that properly
however the browser and the email client both did when the text was
rendered in them.

>>>>> I was involved in some of the UTF8 discussions of the past. My
>>>>> challenge
>>>>> is:
>>>>>

---

From: Yaron Sheffer <[email-masked]>
Date: Wed, 02 Feb 2011 12:25:58 +0200

Add Linux: gedit and Thunderbird on Ubuntu.

But not the Android mail reader, or the default Android preview apps (I 
suspect a more general mail-related problem there).

But I have a problem with the attitude. Some systems are broken. That's 
no reason to have people's name mangled on the documents that they write.

I don't have the statistics at hand, but I suppose *many more* people 
are using UTF-8 today, compared to plain ASCII.

Thanks,
	Yaron

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 02 Feb 2011 20:00:31 +0900

A few comments:

- The attached file was in UTF-8, but the Content-Type didn't say so.
   (I'm not sure if it had, that would have fixed the problem, but at
    least I think we should follow our own standards.)
- Browsers and email readers (for the main part of an email) should work
   without problems these days because they actually handle Content-Type
   charset information, which the file systems in general are very bad
   at doing.
- A file like this one is a tough job for software that tries to
   deal with both legacy encoding (e.g. ISO-8859-1 in Western Europe)
   and UTF-8. There's very little actual UTF-8 in there, and it's
   towards the end. (That could be fixed with a BOM, but then that
   might create other problems.)

> able to read the attached file with no additional configuration or

"with no additional configuration or software" can definitely be done. 
What may be needed is a choice of software (not just a double-click) and 
maybe even a choice of encoding (from a menu).

What is impossible to do is "with whatever software I always used to 
read Drafts/RFCs", because some such software may be older than UTF-8.

The question of what characters should be allowed to reduce the "missing 
font problem" is something that can be addressed even on a case-by-case 
base by the RFC specialists.

Regards,   Martin.

P.S.: To Yaron: I think you mean RTL when you talk about Hebrew,
not LTR.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 02 Feb 2011 20:18:29 +0900

> are using UTF-8 today, compared to plain ASCII.

For the Web, that seems to be very much so indeed:

http://trends.builtwith.com/encoding/UTF-8
http://w3techs.com/technologies/details/en-utf8/all/all
http://googleblog.blogspot.com/2008/05/moving-to-unicode-51.html

I think the statistics for documents used locally would trail behind 
(UTF-8 vs. legacy encodings in general, not UTF-8 vs. US-ASCII), but the 
direction would be the same.

I wonder what percentage US-ASCII had when it was decided that that 
would be used for RFCs. In the old days, EBCDIC was still going strong, 
so my assumption would be that it wouldn't have been that much more than 
UTF-8 now :-).

Regards,   Martin.

---

From: Yaron Sheffer <[email-masked]>
Date: Wed, 02 Feb 2011 16:33:40 +0200

RTL, seY :-)

---

From: Joe Touch <[email-masked]>
Date: Wed, 02 Feb 2011 09:30:25 -0800

Hi, Martin,

> least I think we should follow our own standards.)

I dragged and dropped it into Thunderbird, FWIW.

> charset information, which the file systems in general are very bad

Maybe they should, but the evidence is that they do not. The 
drag-and-drop into Tbird didn't set the content type as needed, and as a 
result the iPhone/iPad didn't render it correctly.

> might create other problems.)

That might be very typical, e.g., if only an example or two and the 
authors' names actually were outside the existing RFC ASCII subset.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 02 Feb 2011 09:34:47 -0800

> no reason to have people's name mangled on the documents that they write.

Yes, actually, it is.

The two alternatives are:

	- honor the specific requirements of a subset of people
	- make text readable to everyone

That's the same trade-off that led us to IP, which "works everywhere", 
but, AFAICT, is best nearly nowhere.

IMO, sticking to the RFC ASCII subset is no different. We *are* eating 
our own dogfood, or at least living by our own standards.

Yes, this is a good reason to fix these apps/systems. But it's also 
evidence that shifting to UTF-8 will make the RFCs less accessible than 
they are now.

> are using UTF-8 today, compared to plain ASCII.

By that logic we should write RFCs in Chinese ;-)

Joe

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 02 Feb 2011 09:36:59 -0800

Again: why is this thread on the wg-chairs mailing list?

---

From: John Levine <[email-masked]>
Date: 3 Feb 2011 02:23:48 -0000

Here's what I saw when I tried to read it:

In article <[email-masked]> you write:
>SGkgSm9lLCBjYW4geW91IGNsYXJpZnkgd2hhdCBkbyB5b3UgbWVhbiBieSAiZG9lcyBub3Qgc3Vw
>cG9ydCBVVEYtOD8gSSBhbSBjb21wb3NpbmcgdGhpcyBtYWlsIG9uIGFuIEFuZHJvaWQgcGhvbmUg

 etc.

"It works fine for me" is rarely a useful metric.

R's,
John

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 04 Feb 2011 16:37:24 +0900

Hello Joe,

> I dragged and dropped it into Thunderbird, FWIW.

Yes. You dragged it from the file system, which doesn't know the 
encoding. Thunderbird implementers had a few options for this case:
1) Ask the user (doesn't want to be bothered)
2) Assume it's the system encoding (may be wrong)
3) Leave the encoding information out (not wrong, but not right either)

The implementers choose 3).

> Maybe they should, but the evidence is that they do not.

They do as far as the text is under their control (e.g. you type it in).

Btw, it gets quite a bit easier if HTML is used, because HTML can carry 
the encoding information internally. So it might be better to try an 
experiment with UTF-8 + HTML (clean, of course).

> The

Yes indeed. One way to fix this would be to replace some regular spaces 
with UTF-8 non-breaking spaces or something along these lines.

 >> no reason to have people's name mangled on the documents that they 
write.

It's not only people's names. It's also examples, where in some cases it 
gets quite a bit more complicated to understand an example if it is 
reduced to ASCII that if it uses some actual non-ASCII characters.

 > - make text readable to everyone

This is not an either/or tradeoff. First, in most cases, the ASCII part 
of the text is still readeable in such software even if the encoding is 
not detected correctly. Second, on the level of human readability, 
examples such as the Art of Computer Programming books by Knuth show 
that you can use the authors' original name (even if that's Arabic or 
Chinese) without affecting readability.

 > but, AFAICT, is best nearly nowhere.

If we had stopped at IP, the Internet would be a very boring place. 
Fortunately, we have built a lot of very useful applications on top of it.

 > our own dogfood, or at least living by our own standards.

Distributing RFCs in electronic form was a big, forward-looking step 
when it was started decades ago. *Paper* was the lowest common 
denominator at that time. Compared with that, the current discussions 
about UTF-8 or HTML (clean and simple) or whatever are really, really 
small fish.

 > evidence that shifting to UTF-8 will make the RFCs less accessible than

It's difficult to beat US-ASCII (except on IBM mainframes), so even in 
10 years, I guess you will be able to find a hair in the UTF-8 soup. But 
I don't think requiring *absolutely perfect UTF-8 support everywhere* 
makes sense at all.

Regards,   Martin.