Subject: Re: Getting RFC 2047 encoding right
From: Keith Moore <[email-masked]>
To: "Michael Bell" <[email-masked]>
Cc: Keith Moore <[email-masked]>, "Arnt Gulbrandsen" <[email-masked]>, "IETF RFC-822 list" <[email-masked]>
Date: Sat, 13 Dec 2003 07:37:27 -0500
List-Id: <ietf-822.imc.org>

> It could be argued (or shot down <g>) that the truly conservative 
> approach would be then to add a header
>
> X-OriginalSubject: <raworiginalsubject> here
>
> so the user has a HOPE of recovering it if the MUA munged it 
> accidentally.

surely it's easier to say that MUAs shouldn't mung subjects 
accidentally?

(actually I've thought for a long time that it would be useful to have 
a Subject-Was: field, that would automatically include the Subject of 
the message being replied-to if the person composing the reply changed 
the Subject from that of the message being replied to.)

From: "Michael Bell" <[email-masked]>
To: "Keith Moore" <[email-masked]>, "Arnt Gulbrandsen" <[email-masked]>
Cc: "IETF RFC-822 list" <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Date: Sat, 13 Dec 2003 01:03:35 -0800
Sender: [email-masked]

It could be argued (or shot down <g>) that the truly conservative approach would be then to add a header

X-OriginalSubject: <raworiginalsubject> here

so the user has a HOPE of recovering it if the MUA munged it accidentally. Of course that only helps for a first hop between confused and angry MUAs. And not at all for the average user.

A truly creative but criminally insane MUA would simply always change the subject to

Subject: =?q?iso-8859-1?Read the damn message?=

Can we have an RFC on that <g,d,r>

  
----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "Arnt Gulbrandsen" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>; "IETF RFC-822 list" <[email-masked]>; "Charles Lindsey" <[email-masked]>
Sent: Friday, December 12, 2003 9:50 AM
Subject: Re: Getting RFC 2047 encoding right

> 
> >>> The MUA has a choice. Either is can be conservative in what it
> >>> generates and liberal in what it accepts, or it can blindly generate
> >>> whatever it accepts, or it can make a smart judgment.
> >>
> >> To me, being conservative in what it generates means not decoding and
> >> reencoding things it doesn't understand from a message being
> >> replied-to - it means keeping things as they are.
> >
> > I agree. But what "are" things when a MUA sends a reply? IMO, things
> > "are" whatever the user sees on-sceen when issuing the "send" command,
> > and that is what MUA should keep.
> 
> I see your point.  I suppose I would say that if the MUA does reliably 
> understand the charsets in the original message, it should use the same 
> encoding for the reply if the subject isn't changed.  But the receiving 
> MUA is having to use heuristics to guess what the encoding is, maybe 
> it's reasonable to re-encode the result to say "this is what the 
> responder thought the subject was".
> 

Cc: Keith Moore <[email-masked]>, IETF RFC-822 list <[email-masked]>, Charles Lindsey <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Date: Fri, 12 Dec 2003 12:50:53 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

>>> The MUA has a choice. Either is can be conservative in what it
>>> generates and liberal in what it accepts, or it can blindly generate
>>> whatever it accepts, or it can make a smart judgment.
>>
>> To me, being conservative in what it generates means not decoding and
>> reencoding things it doesn't understand from a message being
>> replied-to - it means keeping things as they are.
>
> I agree. But what "are" things when a MUA sends a reply? IMO, things
> "are" whatever the user sees on-sceen when issuing the "send" command,
> and that is what MUA should keep.

I see your point.  I suppose I would say that if the MUA does reliably 
understand the charsets in the original message, it should use the same 
encoding for the reply if the subject isn't changed.  But the receiving 
MUA is having to use heuristics to guess what the encoding is, maybe 
it's reasonable to re-encode the result to say "this is what the 
responder thought the subject was".

From: Arnt Gulbrandsen <[email-masked]>
To: Keith Moore <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Cc: IETF RFC-822 list <[email-masked]>, Charles Lindsey <[email-masked]>
Date: Fri, 12 Dec 2003 14:21:07 +0100
Sender: [email-masked]

Keith Moore writes:
> On Dec 12, 2003, at 5:42 AM, Arnt Gulbrandsen wrote:
>> The MUA has a choice. Either is can be conservative in what it 
>> generates and liberal in what it accepts, or it can blindly generate 
>> whatever it accepts, or it can make a smart judgment.
>
> To me, being conservative in what it generates means not decoding and 
> reencoding things it doesn't understand from a message being 
> replied-to - it means keeping things as they are.

I agree. But what "are" things when a MUA sends a reply? IMO, things 
"are" whatever the user sees on-sceen when issuing the "send" command, 
and that is what MUA should keep.

Suppose the original message had "Subject: =?latin_1?q?The price is 
=80216". The MUA fuzzily matches latin_1 to the IANA-defined alias 
latin1, knows about the Microsoft breakage, and presents the user with 
"Subject: The price is ¤216".

The user types a reply indicating acceptance and hits the send button. 
What should the MUA use as encoded subject?

1. "Subject: Re: The price is =?latin_1?q?=80216", ie. something 
matching what the original sender meant, but not necessarily matching 
what the user saw and agreed to.

2. "Subject: Re: The price is =?iso-8859-15?q?=A4216", ie. something 
that unambiguously describes what user chose to send.

> If you don't know what latin_1 is, then you can't make any sense of 
> the 0x80 anyway.  You might as well keep it in the reply.

I disagree. The reply is a message from the user to the recipient(s), 
and should faithfully encode whatever the user saw and typed. The MUA 
SHOULD NOT substitute some other text of unknown meaning for its user's 
text.

> There are almost certainly still some unregistered charsets in use in 
> some communities, and some new charsets are being added from time to 
> time.  You can't really expect your software to be aware of all of 
> them.  It makes sense to make your software tolerant of charsets it 
> doesn't understand yet.

Sure. But misrepresenting the user does not make sense to me.

--Arnt

Cc: Keith Moore <[email-masked]>, IETF RFC-822 list <[email-masked]>, Charles Lindsey <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Date: Fri, 12 Dec 2003 07:13:03 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

On Dec 12, 2003, at 5:42 AM, Arnt Gulbrandsen wrote:

> Keith Moore writes:
>> I don't think I understand what you are saying.  Would you really use
>> "latin_1" as a charset name?  Given that it's nonstandard, how could
>> that be conservative?
>
> I'd never do that. But code that simply copies the received subject 
> would.

I see.  Well, I don't think you're expected to fix that.  Furthermore, 
if your code doesn't know what "latin_1" is, I don't see how you could 
translate it into anything better anyway.  Leaving it as-is at least 
allows for the possibility that it's valid, and your software just 
hasn't learned about it yet.

> The MUA has a choice. Either is can be conservative in what it 
> generates
> and liberal in what it accepts, or it can blindly generate whatever it
> accepts, or it can make a smart judgment.

To me, being conservative in what it generates means not decoding and 
reencoding things it doesn't understand from a message being replied-to 
- it means keeping things as they are.

>> if (strcmp (savemsg->subject, newmsg->subject) == 0)
>>        newmsg->subject = prepend_Re (origmsg->subject);
>> else
>>          newmsg->subject = encode_2047 (newmsg->subject);
>>
>>        what am I missing?
>
> If origmsg->subject is "=?latin_1?q?=80?=" and the user doesn't change
> the subject, newmsg->subject is "Re: =?latin_1?q?=80?=". If the decoder
> knows whether the string could have been generated by a reasonably
> conservative generator, that case can be avoided. A very tricky
> decision.

If you don't know what latin_1 is, then you can't make any sense of the 
0x80 anyway.  You might as well keep it in the reply.

There are almost certainly still some unregistered charsets in use in 
some communities, and some new charsets are being added from time to 
time.  You can't really expect your software to be aware of all of 
them.  It makes sense to make your software tolerant of charsets it 
doesn't understand yet.

From: Arnt Gulbrandsen <[email-masked]>
To: Keith Moore <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Cc: IETF RFC-822 list <[email-masked]>, Charles Lindsey <[email-masked]>
Date: Fri, 12 Dec 2003 11:42:46 +0100
Sender: [email-masked]

Keith Moore writes:
> I don't think I understand what you are saying.  Would you really use 
> "latin_1" as a charset name?  Given that it's nonstandard, how could 
> that be conservative?

I'd never do that. But code that simply copies the received subject would.

> I assume the 0x80 is a non-break-space?  Other than using a 
> nonstandard charset, what is it that makes =?latin_1?q?=80?= a 
> monstrosity?

Neither latin_1 nor 0x80 are defined, yet I've seen both in real life. 
0x80 is not allocated in ISO 8859 character sets, it's a Microsoft 
extension for the euro sign. (Non-break-space is 0xA0.)

> And how does this relate to the problem of not changing encoded-words 
> from the subject message?

The MUA has a choice. Either is can be conservative in what it generates 
and liberal in what it accepts, or it can blindly generate whatever it 
accepts, or it can make a smart judgment.

The first is easiest to program. The second is only slightly harder and 
is what Charles said is the "obvious way", to which I took exception. 
The third is a great deal of work.

> why would the decoder need to care?

See below.

> to me this seems fairly simple:
...
> if (strcmp (savemsg->subject, newmsg->subject) == 0)
>        newmsg->subject = prepend_Re (origmsg->subject);
> else
>          newmsg->subject = encode_2047 (newmsg->subject);
>
>        what am I missing?

If origmsg->subject is "=?latin_1?q?=80?=" and the user doesn't change 
the subject, newmsg->subject is "Re: =?latin_1?q?=80?=". If the decoder 
knows whether the string could have been generated by a reasonably 
conservative generator, that case can be avoided. A very tricky 
decision.

(And if I've injured the English language again, I apologize. I suppose 
I'm too old to learn new languages without forgetting bits of the ones 
I know, or mixing them up. Sad.)

--Arnt

Cc: Keith Moore <[email-masked]>, IETF RFC-822 list <[email-masked]>, Charles Lindsey <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Date: Thu, 11 Dec 2003 13:09:47 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

> My decode2047() was liberal, and my encode2047() was conservative, and 
> I found myself in a position where the code would nevertheless 
> generate monstrosities like "Re: =?latin_1?q?=80?=' (which should be 
> "Re: =?iso-8859-15?q?=A4?=" but can be decoded).

I don't think I understand what you are saying.  Would you really use 
"latin_1" as a charset name?  Given that it's nonstandard, how could 
that be conservative?

I assume the 0x80 is a non-break-space?  Other than using a nonstandard 
charset, what is it that makes
=?latin_1?q?=80?= a monstrosity?

And how does this relate to the problem of not changing encoded-words 
from the subject message?

> After spending a few hours trying to rewrite the decoder such that I 
> could know whether the raw text was something a conservative encoder 
> could make, I decided that the result wasn't worth the complexity.

why would the decoder need to care?

to me this seems fairly simple:

struct msg *newmsg, *savmsg;

/* initialize two message structures; one for the message to be edited;
    another to save decoded fields from the message being replied-to */

newmsg = new_message();
savmsg = new_message ();

savemsg->subject = newmsg->subject = prepend_Re (decode_2047 
(origmsg->subject));
...
edit_message (newmsg);
...
/* now that the message has been edited,
    encode any header fields from the edited message that contain non 
ASCII chars
    if the user didn't change the field, use the original encoding */

if (strcmp (savemsg->subject, newmsg->subject) == 0)
	newmsg->subject = prepend_Re (origmsg->subject);
else
          newmsg->subject = encode_2047 (newmsg->subject);

what am I missing?

To: [email-masked]
Xref: clerew local.mime:2818
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Date: Thu, 11 Dec 2003 11:19:41 GMT
Lines: 34
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>I guess it depends on what you mean by "fire up an editor".  I wasn't 
>aware that "most" MUAs these days used external programs to do message 
>editing.

Yes, I think "fire up editors" got lost in the rush for "plug and play
software" that forces you to use the built-in (and usually woefully
inadequate) editor.

>In that case things would work better if the MUA could detect whether 
>the user chose to change any of these fields that were initialized from 
>header fields that contained encoded-words, and if no changes were 
>made, to use the original (encoded) field text.  For address fields, 
>you'd want to do this on a per-address, rather than per-field basis.

Indeed, but I doubt that many currently take that trouble. They have to
decode the field in order to display it. The easy way out is to reencode
it before sending. If it is a Subject starting with "Re: ", they might
well try to include the "Re: " inside the encoding. Even (especially) if
it was a "Re: " they had added themselves. Somebody needs to tell them
that would be a Bad Thing.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Arnt Gulbrandsen <[email-masked]>
To: Keith Moore <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Cc: IETF RFC-822 list <[email-masked]>, Charles Lindsey <[email-masked]>
Date: Thu, 11 Dec 2003 18:22:04 +0100
Sender: [email-masked]

Keith Moore writes:
> if there's no good way to save state (like the original header fields) 
> unchanged across that edit, so that the encoder has the ability to 
> tell whether the human user changed those fields from those 
> originally specified, then I can understand the difficulty.

Of course there are such ways.

> If I were writing an MUA that used such an arrangement for editing 
> header fields I'd try to find some way to save that state and make it 
> available to the encoder - perhaps by using hidden fields that the 
> human can't change, or by storing the original state in a parent 
> process and having the editor run in a subprocess, or something.

Sure. I've been down that road myself. Not once, but twice. My 
decode2047() was liberal, and my encode2047() was conservative, and I 
found myself in a position where the code would nevertheless generate 
monstrosities like "Re: =?latin_1?q?=80?=' (which should be "Re: 
=?iso-8859-15?q?=A4?=" but can be decoded).

After spending a few hours trying to rewrite the decoder such that I 
could know whether the raw text was something a conservative encoder 
could make, I decided that the result wasn't worth the complexity.

I can understand why a lot of people don't even start down that road.

--Arnt

Cc: Keith Moore <[email-masked]>, IETF RFC-822 list <[email-masked]>, Charles Lindsey <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Date: Thu, 11 Dec 2003 10:11:23 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

> If the editor were internal the MUA could do 2047-decoding for display 
> purposes and keep the raw data as its basic storage. But since the 
> editor is external, the MUA must do 2047-decoding and hand the result 
> to the editor. Later, when the editor hands it back, the "obvious" way 
> is to 2047-encode the editor's result use it. Then there's only one 
> encoder to write and test, and it's used for original messages, for 
> forwarding and for replies. Less to write, less to test, fewer bugs.

if there's no good way to save state (like the original header fields) 
unchanged across that edit, so that the encoder has the ability to tell 
whether the human user changed those fields from those originally 
specified, then I can understand the difficulty.  If I were writing an 
MUA that used such an arrangement for editing header fields I'd try to 
find some way to save that state and make it available to the encoder - 
perhaps by using hidden fields that the human can't change, or by 
storing the original state in a parent process and having the editor 
run in a subprocess, or something.

From: Arnt Gulbrandsen <[email-masked]>
To: Keith Moore <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Cc: IETF RFC-822 list <[email-masked]>, Charles Lindsey <[email-masked]>
Date: Thu, 11 Dec 2003 10:41:22 +0100
Sender: [email-masked]

Keith Moore writes:
>> The "obvious" way for most MUAs is to fire up an editor and use the 
>> subject line it gets back from the editor. And that's what most of 
>> them do.
>
> I guess it depends on what you mean by "fire up an editor".

Something like this, usually:

a = new QLineEdit(...); // http://doc.trolltech.com/3.0/qlineedit.html
if ( reply )
     a->setText( "Re: " + orig->subject() );
else if ( forwarding )
     a->setText( "Fwd: " + orig->subject() );
a->show();

> I wasn't aware that "most" MUAs these days used external programs to 
> do message editing.

I didn't say program. An editor in a third-party DLL/shared library is 
external, just like a program, because in both cases the MUA author has 
little or no control over how it works.

If the editor were internal the MUA could do 2047-decoding for display 
purposes and keep the raw data as its basic storage. But since the 
editor is external, the MUA must do 2047-decoding and hand the result 
to the editor. Later, when the editor hands it back, the "obvious" way 
is to 2047-encode the editor's result use it. Then there's only one 
encoder to write and test, and it's used for original messages, for 
forwarding and for replies. Less to write, less to test, fewer bugs.

> I do suspect that "most" MUAs these days initialize a buffer with the 
> (decoded) subject field and give the user a chance to edit it, along 
> with to/cc/etc. fields and the message body.

Yes.

--Arnt

Date: Thu, 11 Dec 2003 09:03:31 +0000 (GMT)
From: Philip Hazel <[email-masked]>
To: Keith Moore <[email-masked]>
cc: Arnt Gulbrandsen <[email-masked]>, Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: Getting RFC 2047 encoding right
Sender: [email-masked]

On Wed, 10 Dec 2003, Keith Moore wrote:

> I wasn't aware that "most" MUAs these days used external programs to
> do message editing.

I've been using an external editor from my MUA for more than 10 years,
but not, as it happens, for the Subject: line; just for the body.

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Cc: Keith Moore <[email-masked]>, Charles Lindsey <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Date: Wed, 10 Dec 2003 16:50:46 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

> The "obvious" way for most MUAs is to fire up an editor and use the 
> subject line it gets back from the editor. And that's what most of 
> them do.

I guess it depends on what you mean by "fire up an editor".  I wasn't 
aware that "most" MUAs these days used external programs to do message 
editing.

I do suspect that "most" MUAs these days initialize a buffer with the 
(decoded) subject field and give the user a chance to edit it, along 
with to/cc/etc. fields and the message body.

In that case things would work better if the MUA could detect whether 
the user chose to change any of these fields that were initialized from 
header fields that contained encoded-words, and if no changes were 
made, to use the original (encoded) field text.  For address fields, 
you'd want to do this on a per-address, rather than per-field basis.

From: Arnt Gulbrandsen <[email-masked]>
To: Charles Lindsey <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Cc: [email-masked]
Date: Wed, 10 Dec 2003 18:30:46 +0100
Sender: [email-masked]

Charles Lindsey writes:
> In <[email-masked]> Keith Moore 
> <[email-masked]> writes:
>
>> The correct way to reply to a message that has an RFC 2047-encoded 
>> subject is to use the same RFC 2047-encoded subject that appeared in 
>> the message being replied-to. That's very easy to do - just prepend 
>> "Re: " to that subject line, and rewrap the header (leaving 
>> encoded-words intact) if necessary to accomodate RFC 2047 
>> line-length restrictions.
>
> Sure. That's the "obvious" way to do it, but since when have 
> implementors of User Agents invariably done the "obvious" thing?

The "obvious" way for most MUAs is to fire up an editor and use the 
subject line it gets back from the editor. And that's what most of them 
do.

--Arnt

To: [email-masked]
Xref: clerew local.mime:2812
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Date: Tue, 9 Dec 2003 18:59:50 GMT
Lines: 23
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>The correct way to reply to a message that has an RFC 2047-encoded subject
>is to use the same RFC 2047-encoded subject that appeared in the message
>being replied-to.  That's very easy to do - just prepend "Re: " to that
>subject line, and rewrap the header (leaving encoded-words intact) if 
>necessary to accomodate RFC 2047 line-length restrictions.

Sure. That's the "obvious" way to do it, but since when have implementors
of User Agents invariably done the "obvious" thing?  So it needs to be
said somewhere. Usefor has simply taken the view that "somewhere" should
be BCP rather than standards-track.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Cc: Keith Moore <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Getting RFC2047 encoding "right"
Date: Tue, 9 Dec 2003 17:24:56 -0500
To: [email-masked]
Sender: [email-masked]

I've been talking about doing the "right" thing wrt RFC 2047.  It would 
have been more accurate for me to say "this is how 2047 was intended to 
be used" or even "this is what I had in mind".

I think I can speak with reasonable authority on how 2047 was intended 
to be used.  Whether or not those intentions were naive or shortsighted 
or less scalable than we'd like - those are of course separate 
questions.  Still, when a protocol is used in a way that is different 
than was intended it does seem likely that some problems will result.  
One thing that seems obvious in hindsight (and with ~12 years more 
experience) is that protocols are quite frequently used in ways that 
are different than their designers intended.

Cc: Keith Moore <[email-masked]>, [email-masked], [email-masked], [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Date: Tue, 9 Dec 2003 17:07:20 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

>> The basic answer is that what you do with illegal input is generally 
>> not specified - but clearly you aren't expected to make the subject 
>> of the reply match the subject of the message being replied to in 
>> that case.
>
> Right.
>
> What I cannot see is how to make something reasonable, correct and 
> fairly simple.
>
> In most cases I have code that is right when the input is good, and 
> not wrong when the input is bad. RFC 2047 just doesn't seem to make 
> that simple.

Well for untagged text basically you just have to guess the charset.  
ISO-2022-* and UTF-8 can be distinguished from other charsets simply 
and fairly reliably, and you can make guesses at some of the others 
using heuristics.  It's difficult to tune the heuristics, and subject 
lines are too brief for them to work really well.   But I really don't 
see how RFC 2047 makes determining the charset label of untagged text 
any worse than it inherently is.

>>>  Suppose I want to answer with "subject: re: <original> <ticket 
>>> id>",  then I risk having two encoded-words separated only by 
>>> whitespace,  and must do magic in order to preserve that space.
>>
>> why not just use an ASCII ticket id?
>
> Why should I make "always ASCII" a requirement for that case, in code 
> that otherwise allows all of Unicode?

For the same reason that you should probably avoid using some forms of 
email addresses even though they are perfectly valid - such as "Keith 
\"Mr. Cynic\" Moore"@cs.utk.edu - corner cases that are seldom seen 
often fail in practice.

If you want to be entirely reliable your code to detect ticket-ids has 
to be able to find them whether or not they're embedded in 
encoded-words.  And it's not as if you can't put a ticket-id into an 
encoded-word, though (as you point out) you might have to encode %20 as 
the first character of that encoded-word.

From: Arnt Gulbrandsen <[email-masked]>
To: Keith Moore <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Cc: [email-masked], [email-masked], [email-masked]
Date: Tue, 9 Dec 2003 18:58:15 +0100
Sender: [email-masked]

Keith Moore writes:
> For that matter, suppose you want to answer a message that has some 
> other kind of invalid header field - maybe one that isn't encoded at 
> all, and has illegal characters.

Right.

> The basic answer is that what you do with illegal input is generally 
> not specified - but clearly you aren't expected to make the subject 
> of the reply match the subject of the message being replied to in 
> that case.

Right.

What I cannot see is how to make something reasonable, correct and 
fairly simple.

In most cases I have code that is right when the input is good, and not 
wrong when the input is bad. RFC 2047 just doesn't seem to make that 
simple.

>>  Suppose I want to answer with "subject: re: <original> <ticket id>", 
>>  then I risk having two encoded-words separated only by whitespace, 
>>  and must do magic in order to preserve that space.
>
> why not just use an ASCII ticket id?

Why should I make "always ASCII" a requirement for that case, in code 
that otherwise allows all of Unicode?

--Arnt

Date: Mon, 8 Dec 2003 13:44:50 -0500
From: Keith Moore <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Getting RFC 2047 encoding right
Sender: [email-masked]

> Well, I do search, and it's extremely expensive to decode all the 
> messages in order to do a search. To search, I must decode when the 
> message goes into the data store.
> 
> Storing two instances of the same header is also not a good answer. 
> First of all, it's bad practice. Never keep two separate variables
> that are supposed to stay in sync.

Every fast search engine I know of creates indices out of the data
being searched.  I don't see why searching mail should be different.
You do have to rebuild the index for that message if the message 
is changed, but that should be a rare case.

> Second, it only solves the common case - if I go down that path I have
> problems again soon enough. Suppose I want to answer a message which 
> had an overlong encoded-word. Should I blithely emit overlong 
> encoded-words? 

For that matter, suppose you want to answer a message that has some
other kind of invalid header field - maybe one that isn't encoded at
all, and has illegal characters.  The basic answer is that what you do
with illegal input is generally not specified - but clearly you aren't
expected to make the subject of the reply match the subject of the
message being replied to in that case.

> Suppose I want to answer with "subject: re: <original> 
> <ticket id>", then I risk having two encoded-words separated only by 
> whitespace, and must do magic in order to preserve that space.

why not just use an ASCII ticket id?

From: Arnt Gulbrandsen <[email-masked]>
To: Keith Moore <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Cc: Pete Resnick <[email-masked]>, [email-masked], [email-masked]
Date: Mon, 8 Dec 2003 19:32:56 +0100
Sender: [email-masked]

Keith Moore writes:
>> Do you consider sorting a "display purpose"? How about searching 
>> given some user input?
>
> seems fine to me.

Well, I do search, and it's extremely expensive to decode all the 
messages in order to do a search. To search, I must decode when the 
message goes into the data store.

Storing two instances of the same header is also not a good answer. 
First of all, it's bad practice. Never keep two separate variables that 
are supposed to stay in sync.

Second, it only solves the common case - if I go down that path I have 
problems again soon enough. Suppose I want to answer a message which 
had an overlong encoded-word. Should I blithely emit overlong 
encoded-words? Suppose I want to answer with "subject: re: <original> 
<ticket id>", then I risk having two encoded-words separated only by 
whitespace, and must do magic in order to preserve that space.

I don't like adding hacks like the "store twice" if I have the same 
problem again five minutes later.

--Arnt

Date: Mon, 8 Dec 2003 13:08:00 -0500
From: Keith Moore <[email-masked]>
To: Pete Resnick <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Getting RFC 2047 encoding right
Sender: [email-masked]

> >>Do you consider sorting a "display purpose"? How about searching 
> >>given some user input?
> >
> >seems fine to me.
> 
> Well, in that case I think you're being a bit naive when you say 
> earlier, "Decoding is only supposed to happen prior to display or 
> printing, not prior to storage."

point taken.

> Requiring a search to do 2047 (and 
> therefore likely 822) parsing on each and every searched message in a 
> large message store is much too processor intensive in some 
> environments.

it's hardly unusual for search engines to construct indices to speed
searching. I don't see any problem with the search indices being built 
with decoded header fields, but it's cleaner if the actual messages 
are not altered.

> In 
> either case, allowing the user to edit the subject of a reply 
> probably means de-coding and re-encoding the 2047, which runs into 
> exactly the kind of problem that Arnt was getting at in his original 
> message.

yes, but once the subject is edited, it's not the same subject any
longer, and you don't expect it to match up with previous subjects in
the thread.

> It would be nice to think that "decode only for display purposes" 
> would be a complete answer. It's not.

no, but it's close.  the point is that if you try to decode these things
too early then you break things.

Keith

Date: Mon, 8 Dec 2003 12:01:04 -0600
To: Keith Moore <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Cc: [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

On 12/8/03 at 12:40 PM -0500, Keith Moore wrote:

>>On 12/8/03 at 10:43 AM -0500, Keith Moore wrote:
>>
>>>Nope, user agents SHOULD NOT decode RFC 2047 except for display purposes.
>>
>>Do you consider sorting a "display purpose"? How about searching 
>>given some user input?
>
>seems fine to me.

Well, in that case I think you're being a bit naive when you say 
earlier, "Decoding is only supposed to happen prior to display or 
printing, not prior to storage." Requiring a search to do 2047 (and 
therefore likely 822) parsing on each and every searched message in a 
large message store is much too processor intensive in some 
environments. Now, maybe that would require a shadow database of 
pre-parsed messages along with the un-parsed 2047 fields to use in 
replies, but that starts to become a significant burden too. In 
either case, allowing the user to edit the subject of a reply 
probably means de-coding and re-encoding the 2047, which runs into 
exactly the kind of problem that Arnt was getting at in his original 
message.

It would be nice to think that "decode only for display purposes" 
would be a complete answer. It's not.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Mon, 8 Dec 2003 12:40:24 -0500
From: Keith Moore <[email-masked]>
To: Pete Resnick <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Getting RFC 2047 encoding right
Sender: [email-masked]

> On 12/8/03 at 10:43 AM -0500, Keith Moore wrote:
> 
> >Nope, user agents SHOULD NOT decode RFC 2047 except for display purposes.
> 
> Do you consider sorting a "display purpose"? How about searching 
> given some user input?

seems fine to me. 

Date: Mon, 8 Dec 2003 11:25:45 -0600
To: Keith Moore <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Cc: "Charles Lindsey" <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

On 12/8/03 at 10:43 AM -0500, Keith Moore wrote:

>Nope, user agents SHOULD NOT decode RFC 2047 except for display purposes.

Do you consider sorting a "display purpose"? How about searching 
given some user input?

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Mon, 8 Dec 2003 10:43:27 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Getting RFC 2047 encoding right
Sender: [email-masked]

> >Now, to be kind and courteous that program should use the same subject 
> >field, perhaps prefaced by "Re: " (or "Auto: "), such that if the 
> >recipient threads based on subject, everything work, no matter whether 
> >the recipient supports RFC 2047 or not.
> 
> By and large, user agents should (or SHOULD) decode the RFC2047 stuff
> before doing anything "clever" with the header.

Nope, user agents SHOULD NOT decode RFC 2047 except for display purposes.

> However, one of the tasks the USEFOR WG is undertaking is a companion "Best
> Current Practice" document, and in the initial draft for that I have
> written that if you add a "Re: " to a Subject-header, then it SHOULD be in
> the clear _before_ the start of any RFC 2047-encoded stuff. 

Yes, but this happens naturally if you just prepend "Re: " to whatever
subject happens to be in the message being replied to (encoded or not).

> Best practice is propably to store the original, and decode it on the fly
> whenever it is needed for humans.

right.

Keith

Date: Mon, 8 Dec 2003 10:41:32 -0500
From: Keith Moore <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Getting RFC 2047 encoding right
Sender: [email-masked]

The correct way to reply to a message that has an RFC 2047-encoded subject
is to use the same RFC 2047-encoded subject that appeared in the message
being replied-to.  That's very easy to do - just prepend "Re: " to that
subject line, and rewrap the header (leaving encoded-words intact) if 
necessary to accomodate RFC 2047 line-length restrictions.

> Suppose that a mail client receives a message whose subject is encoded 
> according to RFC 2047. Suppose further that the message is decoded and 
> stored in a database somewhere - in RAM, on disk, on a server 
> somewhere.

RFC 2047 encoded words were not intended to be used in this way.  Decoding
is only supposed to happen prior to display or printing, not prior to
storage.

Keith

To: [email-masked]
Xref: clerew local.mime:2802
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Date: Mon, 8 Dec 2003 10:16:37 GMT
Lines: 38
Sender: [email-masked]

In <[email-masked]> Arnt Gulbrandsen <[email-masked]> writes:

>Now, to be kind and courteous that program should use the same subject 
>field, perhaps prefaced by "Re: " (or "Auto: "), such that if the 
>recipient threads based on subject, everything work, no matter whether 
>the recipient supports RFC 2047 or not.

By and large, user agents should (or SHOULD) decode the RFC2047 stuff
before doing anything "clever" with the header. Note that all the encoded
stuff is really for human consumption, so it should not matter (but of
course it does). The only thing that is even mentioned in any
standards-like document regarding interpreting the Subject-header is
"Re: " (it has a mention in RFC 2822, for example, and is likely to have
a slightly firmer mention in USEFOR).

However, one of the tasks the USEFOR WG is undertaking is a companion "Best
Current Practice" document, and in the initial draft for that I have
written that if you add a "Re: " to a Subject-header, then it SHOULD be in
the clear _before_ the start of any RFC 2047-encoded stuff. That seems
like a good practice to me, since it means that agents that want to strip
it off (before producing sorted lists of headers, for example) can at least
do that correctly.

>That implies using the same character set, q/b encoding etc. as the original.

Best practice is propably to store the original, and decode it on the fly
whenever it is needed for humans.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 8 Dec 2003 07:42:21 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-04)
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Simon Josefsson <[email-masked]> wrote:

> > The lines consisting of just a greater-than sign and a space are fixed,
> > right?  (Because the space is a stuffing space.)
> 
> I dunno.  -04 also says:
> 
>     If the line ends in a space, the line is flowed.  Otherwise it is
>     fixed.
> 
> This suggest the line is flowed.

The sentences immediately preceeding that one are:

    If the first character of a line is a space, the line has been
    space-stuffed (see section 5.4).  Logically, this leading space is
    deleted before examining the line further (that is, before checking
    for flowed).

AMC

To: IETF RFC-822 list <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-04)
From: Simon Josefsson <[email-masked]>
Date: Mon, 08 Dec 2003 02:35:58 +0100
Sender: [email-masked]

"Adam M. Costello" <[email-masked]> writes:

> Charles Lindsey <[email-masked]> wrote:
>
>> Opera rendered it...
>
> The lines consisting of just a greater-than sign and a space are fixed,
> right?  (Because the space is a stuffing space.)  So there should be two
> separate quoted "foo foo foo..." paragraphs separated by quoted blank
> lines, right?

I dunno.  -04 also says:

    If the line ends in a space, the line is flowed.  Otherwise it is
    fixed.  The exception to this rule is a signature separator line,
    described in Section 5.3.  Such lines end in a space but are neither
    flowed nor fixed.

This suggest the line is flowed.

Also of relevance may be that the document considers

> foo

and

>foo

to be logically the same during encoding.  I'm not sure if the lines
are still considered equivalent if the 'foo' is removed from both of
them.

It is also not clear if the DELSP parameter should modify the output
for this example.  I.e., is the SPC a stuffing or trailing SPC?  The
Opera output quoted by Charles Lindsey did not appear to remove the
trailing SPC when flowing lines together, but presumably it implement
RFC 2646 and not draft-gellens-format-bis-04.txt.  Having different
behaviour in 2646 and 2646bis might not be ideal either.

Pseudo code for a correct decoder would make matters simpler...

Thanks,
Simon

From: Arnt Gulbrandsen <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Getting RFC 2047 encoding right
Date: Sun, 7 Dec 2003 17:42:30 +0100
Sender: [email-masked]

Hi,

I have a tiny little problem that you chaps may be able to help me with.

Suppose that a mail client receives a message whose subject is encoded 
according to RFC 2047. Suppose further that the message is decoded and 
stored in a database somewhere - in RAM, on disk, on a server 
somewhere. Some time later, the same or a different program sends a 
reply.

Now, to be kind and courteous that program should use the same subject 
field, perhaps prefaced by "Re: " (or "Auto: "), such that if the 
recipient threads based on subject, everything work, no matter whether 
the recipient supports RFC 2047 or not.

That implies using the same character set, q/b encoding etc. as the original.

To be kind and courteous, the program should use a widely supported 
character set when 2047-encoding (whether it's composing original 
messages or replies), and use as few 2047-encoded words as possible.

To be kind and courteous, the program should observe all other rules set 
out in RFC 2047 and 2822 (including ones which were broken by an 
earlier message).

To be reliable and bug-free, the algorithm that does all this should be 
simple and straightforward.

There's a conflict here. How do you all address this?

--Arnt

Date: Sat, 6 Dec 2003 05:57:14 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-04)
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Charles Lindsey <[email-masked]> wrote:

> Opera rendered it...

The lines consisting of just a greater-than sign and a space are fixed,
right?  (Because the space is a stuffing space.)  So there should be two
separate quoted "foo foo foo..." paragraphs separated by quoted blank
lines, right?

AMC

To: [email-masked]
Xref: clerew local.mime:2798
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: ***SPAM-3*** Re: Format=Flowed/RFC 2646 Bis (-04)
Date: Fri, 5 Dec 2003 11:47:45 GMT
Lines: 43
Sender: [email-masked]

In <[email-masked]> Simon Josefsson <[email-masked]> writes:

>Has anyone collected test vectors for format=flowed?  Considering the
>amount of buggy implementations, I believe it would be useful to have
>these available.  I'm writing a test harness for my implementation, so
>I'd appreciate any pointers to existing test vectors I could adopt.

>To give one (to me) problematic example, how to render this:

>> foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo 
>> foo 
>> 
>> 
>> 
>> foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo
>> bar 

Opera rendered it (no extra '>'s added) as

--------------------------

>foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo 
>   foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo
>bar 
 
Thanks,
Simon

--------------------------

I have left the trailing spaces in, because this message is not
format=flowed, so you SHOULD NOT be reflowing it.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: IETF RFC-822 list <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-04)
From: Simon Josefsson <[email-masked]>
Date: Thu, 04 Dec 2003 16:57:07 +0100
Sender: [email-masked]

Has anyone collected test vectors for format=flowed?  Considering the
amount of buggy implementations, I believe it would be useful to have
these available.  I'm writing a test harness for my implementation, so
I'd appreciate any pointers to existing test vectors I could adopt.

To give one (to me) problematic example, how to render this:

> foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo 
> foo 
> 
> 
> 
> foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo
> bar 

Thanks,
Simon