Subject: [EAI] Discussion of draft-ietf-eai-downgrade-03.txt

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 27 Mar 2007 20:28:19 +0100

As compared to the Utf8headers and Smtpext drafts, this one still seems a  
long way from completion.

1.  Introduction

    Downgrading in UTF8SMTP consists from following two parts:
                                     ^^^^
                                     of the

    This document assumes that MIME headers are not extended to use UTF-8
    characters because it is not clearly defined in
    [I-D.ietf-eai-utf8headers].

On the contrary, [I-D.ietf-eai-utf8headers] now makes it very clear that  
those MIME headers are to be extended, and the downgrading of those  
headers now needs to be addressed.

3.1.  Timing and conditions of downgrading

    Conditions:  SMTP client detects that non-ASCII characters are
       included in the SMTP envelope or mail header fields in the SMTP
       DATA.

That needs to include mail header fields at any level within the MINE  
structure.

3.2.  Requirements

    4.  Downgrading and upgrading should be easy and lightweight as it is
        possible to do with MTA like 8BITMIME encapsulation.

But even downgrading 8BITMIME is not "easy and lightweight" if down  
properly (e.g. as sendmail does it). Granted that the machinery for doing  
8BITMIME downgrading can easily be adapted to do header downgrading.

4.  Downgraded header

    downgraded  = "Downgraded:" [CFWS] field-name ":" dvalue CRLF
    field-name = 1*ftext
    ftext      = %d33-57 / %d59-126 ; printable ASCII except ":"
    dvalue     = 1*any-ascii
    any-ascii  = %d32-126

No, the proper syntax there is

    downgraded  = "Downgraded:" [CFWS] field-name ":" unstructured CRLF

Since <unstructured> is essentially what RFC 2047 is defined to work upon.  
And I am not sure whether you want that "[CFWS]" there. Shouldn't it be  
"[FWS]", because this header is always automatically generated, so there  
will never be any comments in it? Or do you want downgrading  
implementations to be able to add comments in there to cover unanticipated  
situations (just like Received headers have acquired comments in such  
ways).

There are then 3 ways to proceed:

1. You can say that <unstructured> is as defined in Utf8headers (using the  
extended <utext> defined there), in which case you then say that it MUST  
then be encoded according to RFC 2047.

2. You use the syntax for <unstructured> strictly as defined in RFC 2822  
(with the RFC 2822 <utext>), since that already includes what RFC 2047  
emcoding produces.

3. You change the syntax to

    downgraded  = "Downgraded:" [CFWS] field-name ":" encoded CRLF
    encoded     = *([FWS] (utext / encoded-word) [FWS]

where <utext> is the strict RFC 2822 version, and <encoded-word> comes  
 from RFC 2047.

But note that your <dvalue> is wrong in any case, because an unstructured  
header such as Subject is entitled to include <NO-WS-CTL>s, and there is  
no requirement for these to be 2047-encoded.

Next, you have to read Section 5(1) of RFC 2047 carefully to see whether  
it already allows you to use <encoded-word>s in your new Downgraded  
header. And if it is not clear, then it is probably safer to write some  
explicit wording to extend Section 5 of RFC 2047. At which point you have  
to realise that RFC 2047 is written in terms of RFC 822, and RFC 822 does  
not make any distinction between <text> and <utext>. And there are rules  
about requiring an explicit FWS between each <encoded-word> amd any  
adjacent <utext> (and I leave it to you to decide whether that requires an  
explicit FWS if the 2nd ":" in the Downgraded header is immediately  
followed by an <encoded-word>).

Note that the present text in RFC 2047 section 5(2) and 5(3) as it stands  
is probably enough to cover the downgrading of <comment>s and <phrase>s;  
and also that those have their own requirements for when FWS is required  
next to an <encoded-word>.

    To preserve SMTP envelope downgraded information, another new header
    is required, and it is specified as "Envelope-Downgraded".  Details
    are described in Section 5.  The header field syntax is specified as
    follows:

    fields =/ edowngraded
    edowngraded = "Envelope-Downgraded:" [CFWS] field-name ":" value CRLF
    field-name = "From" / "To"
    value = CFWS "<" uPath ">" CFWS "<" Mailbox ">" CFWS

Several problems here:

1. Same question about whether CFWS rather than FWS is appropriate in a  
machine-generated header.

2. Please don't use <value> as an ABNF name. Too much risk of confusion  
with <value> as used in MIME parameters

3. Would it be better to have a notation (e.g. using some 'arrow' like  
"->") which immediately suggested which of the two addresses was the  
original and which was the downgraded?

4. Since this header will itself inevitably have to be downgraded,  
wouldn't it be simpler to do it all in one step by providing for the  
<uPath> to be encoded (e.g. by RFC 2047). All you would need would be a  
syntax like

    value = CFWS "<" encoded-word ">" CFWS "<" Mailbox ">" CFWS

5.  SMTP Downgrading

    One SMTP session may contain multiple recipients.  Downgrading SHOULD
    be performed for each SMTP recipient address individually.  First,
    split a multiple recipients session to each sessions.  If the
    recipient address is downgradable, the SMTP session to the recipient
    is downgradable.

It is not clear what you mean by the term "session" here. Is it defined  
anywhere?

    Also note that by appending "Envelope-Downgraded:" headers, MUA/MTA
    MUST perform Email header Downgrading described in Section 6.

And that is the unnecessary step I want to avoid by defining the  
Envelope-Downgraded header to be already pure ASCII.

    ESMTP ORCPT parameter is used for Delivery Status Notifications
    (DSNs) [RFC3461].  ORCPT parameters contain mail addresses.  After
    extending ORCPT parameter to support Non-ASCII mail addresses, ORCPT
    parameter downgrading should be defined here.

Then please do it, or agree with Chris that it is to be defined in the DSN  
document, and cross-reference it from here (but I think it would be better  
to do it here, and I think we are agreed that it has to be done by %hex  
encoding of the UTF-8).

6.  Email header Downgrading

    This section defines conversion method to US-ASCII for each header
    which may contain Non-ASCII characters.  If the whole mail header
    does not contain Non-ASCII characters, email header downgrading is
    not required.

You need to add that headers in MIME body parts and in contained  
message/rfc822 objects also need to be examined.

   Otherwise, email header downgrading checks each header
    whether it contains UTF-8 characters or not.
                        ^^^^^^^^^^^^^^^^
                        <UTF8-xtra-char>s
                 {or whatever they are eventually called}

    .......... If the header contains
    UTF-8 characters, convert the header in a suitable method for of each
    header.  Each header's downgrading method is described below:

    This section defines conversion method to US-ASCII for each header
    which may contain Non-ASCII characters.  If the whole mail header
    does not contain Non-ASCII characters, email header downgrading is
    not required.  Otherwise, email header downgrading checks each header
    whether it contains UTF-8 characters or not.  If the header contains
    UTF-8 characters, convert the header in a suitable method for of each
    header.  Each header's downgrading method is described below:

    From, To, CC, Reply-To, Return-Path:

You need to add the Sender header, and also say that it applies to any  
other header that might be defined to use a <mailbox>.

And please s/header/header field/ in all places where it is not already  
done.

       The header field body is composed of single or multiple angle-
       addr/addr-spec fields defined in [I-D.ietf-eai-utf8headers].
            ^^^^^^^^^
          utf8-addr-spec
       If the header has no 'angle-addr' or 'utf8-addr-spec' which
       contains UTF-8 characters, only "display-name" part contains UTF-8
                ^^^^^^^^^^^^^^^^                                    ^^^^^
                <UTF8-xtra-char>s                              
<UTF8-xtra-char>s
       characters, encode the header by [RFC2047] with UTF-8 tag and
       ^^^^^^^^^^         ^^^^^^^^^^
                       that <display-name>
       replace it.

Generally speaking, when refering to objects defined by ABNF, it is better  
to enclose them within <.....> as recommended in RFC 4234, than to enclose  
them within '.........'.

       ... When
       this address header downgrading fails, this downgrading fails and
       the mail MUST be bounced.

For some reason, we decided to use "rejected", rather than "bounced".

    Subject:
       Encode the header by [RFC2047] with UTF-8 tag and replace it.

There are other headers where this would be the proper treatment. E.g.  
Keywords, and header defined as <unstructured> by other standards (e.g.  
the Summary header in Netnews), and all X-headers.

You also need to mention Envelope-Downgraded in this list, unless you have  
adopted my suggestion to make it already all-ASCII when it is generated.

    Downgraded headers should be inserted or replaced at the same
    position of the original header.

Yes, I think that is correct, although it might be argued that they are  
"Trace" headers.

6.1.  Downgrading address headers

    This procedure targets "From:", "To:", "CC:", "Reply-To:", "Return-
    Path:" headers which contains Originator/Destination address(es).

You need to add Sender: to that list. Even Bcc:, though it should not  
really be there. But it would also apply to any other header that might be  
defined in future with a <mailbox> in it (it should be permitted for  
agents that recognixe it, though we cannot REQUIRE it for headers newly  
invented after our document is published).

And somewhere around here (section 6.2 maybe) you have to describe the  
downgrading of MIME <value>s, which are allowed to contain UTF-8 according  
to Utf8headers. The downgrade should, of course, use RFC 2231, and a  
Downgraded header should be provided as well.

You also need at least a placeholder for downgrading List-* headers,  
although discussion of these in ongoing in a separate thread. And maybe a  
placeholder for any downgrading that may sries from the DSN draft.

7.  Upgrading downgraded header

    Upgrading downgraded header is described below:
    o  Check if the mail header contains 'Downgraded:' headers.
       Otherwise, upgrading is not required.

You actually need to check through the message body in order to find body  
part headers and heders within message/rfc822 objects that may have been  
downgraded.

       *  If the decoded header is an address header described in
          Section 6.1,
          +  Generate ASCII only header described in Section 6.1 from t!
             decoded header.
          +  Remove the header line which is the same with the generated
             ASCII only header.  (REQUIRE HEADER NORMALIZATION)

You need to explain that further. Presumably you mean that any refolding  
that took place during the downgrade process may need to be ignored.

       *  If the decoded header is a "Received:" header,
          +  Removing the 'FOR' clause from the decoded header generates
             ASCII only header.
Make it clear that this is a temporary removal to aid in identifying the  
sibstituted header in the next step. It reads a little oddly as you have  
written it.

          +  Remove the header line which is the same with the generated
             header.  (REQUIRE HEADER NORMALIZATION)
    o  If each mail header has [RFC2047] encoded part and which encoding
       is "UTF-8", it may be a downgraded header, so decode it.

That brings a risk that you may upgrade something that was already in RFC  
2047 form when it was composed. In fact, you need some discussion here (or  
maybe in Security Comsiderations) of the extent to which upgrading may  
fails to restore exactly what was in the original message. This could be  
important when it comes to considering signature schemes such as DKIM.

8.  Implementation consideration

    MUA MAY encode UTF-8 in Subject header with the same encoding of body
    part while downgrading.

That also applies to other unstructured headers, and to display-names,  
etc. It is essentially the point I made earlier about upgrading possibly  
undoing more than the earlier downgrading did.

    UTF8SMTP compliant MUA MUST upgrade downgraded mail and MUST show
    Non-ASCII mail addresses on display.

s/MUST/SHOULD/ (twice). There is no interoperability - just confusion :-( .

10.  IANA Considerations

    IANA is requested to add the "Envelope-Downgraded:", and
    "Downgraded:" new headers to the registry with the entries pointing
    to this specification for its definition.

You need to provede proper templates for these, as required by RFC 3864.

Appendix A.  Examples

A.1.  Downgrading example 1

    In this example, there are two sessions, one is To:, the other is
    CC:.  Both sessions are downgradable.  Figure 4 shows To: session
    downgrading.

I don't think "sessions" is the right word here. There is only one SMTP  
"session", which lasts from the EHLO until the connection is broken. What  
you are talking about here is just a part of the processing of one  
message, namely that which applies to a single RCPT. Perhaps someone else  
can suggest a better word here, but "session" is definitely wrong.

    Result of the header downgrading.

    Downgraded: Envelope-Downgraded: From:
                MIME(<NON-ASCII-FROM>) <ASCII-FROM>
    Downgraded: Envelope-Downgraded: To:
                MIME(<NON-ASCII-TO>) <ASCII-TO>

s/MIME/RFC2047/ (or some other word, e.g. "DECODE")

And at the end of this downgraded example, please show how a Return-Path  
(derived from the MAIL FROM) would look. And how would you show an  
<alt-address> in it? Is it accompanied somehow by a Downgraded:  
Return-Path: ... . I don't think we ever discussed how that would work.

A.2.  Upgrading example

And somehow you have to show how a Return-Path got upgraded (if at all)  
here.

                Figure 10: Downgraded header upgraded message

    As a result, in this simple example, all headers are preserved.
                                            ^
                                         the original

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 27 Mar 2007 23:11:47 +0200

Charles Lindsey wrote:

> [I-D.ietf-eai-utf8headers] now makes it very clear that those
> MIME headers are to be extended, and the downgrading of those

Disputed, the downgrade I-D only has to get UTF8SMTP to 8BITMIME
right, not UTF8SMTP to 7bit.  In another article you said that
you want to get rid of the 7bit legacy...

> That needs to include mail header fields at any level within
> the MINE structure.

...for UTF8SMTP to 7bit.

>   downgraded  = "Downgraded:" [CFWS] field-name ":" unstructured CRLF

Not yet, ... [FWS] CRLF is an 2822 erratum.  The <dvalue> is the
original header field 2047-encoding anything that's neither FWS
nor an FWS delimited VCHAR-string.

In other words it 2047-encodes all FWS delimited words containing
UTF8-non-ASCII or NO-WS-CTL.  While it's at it encoding obs-FWS
should be also possible by a definition of FWS excluding obs-FWS
(i.e. bare CR or LF), because obs-FWS might be no proper delimiter
for 2047-encoded words (I'm not sure about that).

A tricky case are 2231 or 2047-encoded words in the input next to
a word needing encoding, example:

subject: =?UTF-8?Q?test?= ä =?UTF-8?Q?this?=

If that's downgraded to something like (I'm cheating here):

subject: =?UTF-8?Q?test?= =?UTF-8?Q?auml?= =?UTF-8?Q?this?=

Then the original subject would be displayed as "test ä this", and
the upgraded subject would be shown as "testäthis".  Something is
missing.  We can solve it, but it might be tricky.

> And I am not sure whether you want that "[CFWS]" there.

Nor me.  Your explanation for its potential purpose is plausible,
but I don't see why a downgrade algorithm would need to announce
its version or whatever in any (?) downgraded header field.

> 1. You can say that <unstructured> is as defined in Utf8headers
> (using the extended <utext> defined there), in which case you then
> say that it MUST then be encoded according to RFC 2047.

I don't like that, implementors need to know how to do this, and
there are some traps and pitfalls like 2231 encodings in the input.

> 2. You use the syntax for <unstructured> strictly as defined in
> RFC 2822 (with the RFC 2822 <utext>), since that already includes
> what RFC 2047 emcoding produces.

No, I don't like to preserve obs-FWS or NO-WS-CTL as is, better
"protect" such crap by encoding it.  And the 2822-<unstructured>
has this bogus folding opportunity before the CRLF.

> where <utext> is the strict RFC 2822 version, and <encoded-word>
> comes from RFC 2047.

Yes, that's a good direction.  But we need FWS instead of [FWS] as
word delimiter, and it doesn't address the 2231-input issue yet.
The second [FWS] at the end should be only *WSP.

> But note that your <dvalue> is wrong in any case, because an
> unstructured header such as Subject is entitled to include
> <NO-WS-CTL>s, and there is no requirement for these to be 
> 2047-encoded.

If <dvalue> doesn't allow NO-WS-CTL there's an implicit requirement
to encode NO-WS-CTL, that's IMO a feature, not a bug.

> there are rules about requiring an explicit FWS between each
> <encoded-word> amd any adjacent <utext>

Yes, and it's ignored between encoded words in the decode step, see
my example above.  I've no time to look into the rest of your article
now, but the downgrade I-D is as you say not yet ready for prime time.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 28 Mar 2007 20:19:10 +0100

> you want to get rid of the 7bit legacy...

No, that doesn't work. You start with:

Yadda-yadda-headers: ...
Content-Type: multipart/mixed; boundary=foobar

--foobar
Content-Disposition: attachment; filename="mañana"

yadda-yadda
--foobar

more multiparts
--foobar--

Please tell mw what you want to do with it when the next hop supports  
8BITMIME but not UTF8SMTP. You cannot leave that "mañana" alone, because  
8BITMIME does not permit that (because there is then no way to downgrade  
it if the next-hop-but-one does not even support 8BITMIME).

So you MUST downgrade is NOW, before it leaves the UFT8SMTP world (and RFC  
2231 provides a suitable downgrade mechanism).
>> That needs to include mail header fields at any level within

No, for UFF8SMTP to 8BITMIME.

> nor an FWS delimited VCHAR-string.

Well, actually, I want to replace <dvalue> by 1*(utext / encoded-word),  
and you seem to agree, so let us skip to that for the detailed discussion.

> missing.  We can solve it, but it might be tricky.

Yes, MIME contains three cans of worms which are, in decreasing order of  
worminess, RFC 2231, RFC 2047 and message/types. I think the answer is  
that if you want to encode something in RFC 2047, you don't start with  
something that already has encoded-words in it. So for that case you undo  
all the existing encoded words (that gives you some sensiblke UTF-8) and  
then re-encode it properly from scratch.

Exercise for students who have understood it so far:

Encode the following:

subject: =?UGLY?Q?asjfdkjf?= á =?UGLY?Q?asklsdfld?=

where "UGLY" is some arbitrary charset containing characters not  
expressible in UTF-8 and "á" is a character in UTF-8 not expressible in  
UGLY.

>> 1. You can say that <unstructured> is as defined in Utf8headers...
>
> I don't like that,

Nor me

>> RFC 2822 (with the RFC 2822 <utext>), ...
>
> "protect" such crap by encoding it.

Agreed that NO-WS-CTL is ugly, it is already allowable in such contexts in  
RFC 2822, and RFC 2047 does not require it to be encoded (though you MAY).  
I don't think this WG is in the business of fixing bugs in RFC 2822 or RFC  
2047.

> The second [FWS] at the end should be only *WSP.

No, the [FWS] I have shown is correct. A <utext> is just a single  
character, so you can have lots of them side-by-side to make a "word".  
According to RFC 2047, an <encoded-word> can stand where any single  
<utext> (or rather <text>, because RFC 822 doesn't have <utext>) can stand  
(did I mention that RFC 2047 is a can of worms?).

As to 2231-input, what is your problem there? I see no reason why  
something that has gone through RFC 2231 ever needs to go through RFC 2047  
as well.

> to encode NO-WS-CTL, that's IMO a feature, not a bug.

Yes, but as I said, we are not trying to fix the misfeatures of RFC 2822.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 29 Mar 2007 19:22:31 +0200

Charles Lindsey wrote:
 
> that doesn't work. You start with:

> Please tell mw what you want to do with it when the
> next hop supports 8BITMIME but not UTF8SMTP.

For starters the unknown 8bit in the filename parameter
isn't allowed in MIME 1.0 part headers.  If the charset
of a non-ASCII is unknown as it's the case here it's not
possible to downgrade it.

If you insist on it maybe something like this works:

| Content-Disposition: inline; filename*=unknown-8bit''ma%F1ana

> You cannot leave that "mañana" alone, because 8BITMIME
> does not permit that

Doesn't it ?  My 2231-attempt was intended for 7BIT, why
does 8BITMIME care about 8bit crap in MIME part headers ?

> So you MUST downgrade is NOW, before it leaves the UFT8SMTP
> world (and RFC 2231 provides a suitable downgrade mechanism).

"Suitable" isn't the adjective I'd use for this kludge, but
yes, filename*=unknown-8bit''ma%F1ana is pure ASCII gibberish.

===
> actually, I want to replace <dvalue> by 1*(utext / encoded-word),

Yes, and two days later I think we're deep in the woodwork:  It's
strictly impossible to 2047-encode something, that might already
contain 2231 encoded-words, without knowing its structure.

(=?utf-8?Q?test?= X =?utf-8?Q?this?=) contains no encoded word
at all if used in an unstructured header field like "subject".  But
it contains a comment with two encoded words in some other contexts.

If the "X" in this example is something that needs downgrading,
then the proper 2231 way to get this right depends on the structure:

In a subject you can simply encode X, because there are no adjacent
encoded word.  If the complete beast is a comment simply encoding
"X" would join it with its neighbours when it's decoded again.

> I think the answer is that if you want to encode something in
> RFC 2047, you don't start with something that already has
> encoded-words in it.

Hard.  First you have to know if it really is encoded.  And then
you say that it has to be decoded, and that means supporting _all_
registered MIME compatible charsets.  Finally you lose any 2231
language tags in those "pre-encoded" words when you decode them.

> expressible in UTF-8 and "á" is a character in UTF-8 not expressible
> in UGLY.

+1.  Anybody passing that test can try this one:

subject: =?utf-8*de?Qtest?= ü =?utf-8*en?Q?this?=

The info "de" for test and "en" for this is supposed to survive ;-)

Won't fly, we lose.  For downgrade we need a "new" mechanism that
is not bound by any 2047 or 2231 rules, it could be as simple as:

"Take whatever it is with folding etc., and B64 encode it, ready."
User agents supporting "upgrading" will have to learn a new trick.

> Agreed that NO-WS-CTL is ugly, it is already allowable in such
> contexts in RFC 2822, and RFC 2047 does not require it to be
> encoded (though you MAY). I don't think this WG is in the
> business of fixing bugs in RFC 2822 or RFC 2047.

We're in the business of designing something that works with real
software.  I'm not interested in bug reports where the cause of
the alleged "EAI problem" is some unencoded obs-FWS or NO-WS-CTL.

Or a 822 NUL... <shudder> My MUA crashes for some NULs </shudder>
Let "downgrade" hide it, with a "B64 anything" approach it's gone.

>>>     encoded     = *([FWS] (utext / encoded-word) [FWS]
[...]
>> Yes, that's a good direction.  But we need FWS instead of [FWS]
>> as word delimiter
[...]

> character, so you can have lots of them side-by-side to make a

ACK, but what you don't want are adjacent encoded-words separated
by [FWS], i.e. nothing if the [FWS] is omitted.  So the ABNF should
use *utext for an utext-string of any length, and FWS as delimiter:

        encoded     = [*utext / encoded-word] *(FWS
                     (1*utext / encoded-word)      ) *WSP

But as explained above I think that approach doesn't fly, because
it can't solve the problem of 2331-words in the "downgrade" input.

> (did I mention that RFC 2047 is a can of worms?).

Not yet.  IMO 2047 is fine, we just can't use it.  It's a one time
encoder, applying it on its own output causes havoc.

> As to 2231-input, what is your problem there?

Lost language tags.  EAI is an I18N effort, losing language tags
(now also containing script subtags like Hans or Hant) is no option.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 30 Mar 2007 17:49:56 +0100

On Thu, 29 Mar 2007 18:22:31 +0100, Frank Ellermann  
<[email-masked]> wrote:

> isn't allowed in MIME 1.0 part headers.

It is allowed by UTF8headers, as that document is currently written. So it  
needs a downgrade mechanism to go with it.

> | Content-Disposition: inline; filename*=unknown-8bit''ma%F1ana

Is that supposed to be what the writer of the message puts there, or is  
that supposed to be what it becomes after the downgrade? If so, then that  
is almost exactly how you would downgrade it using RFC 2231, except it  
should be
     Content-Disposition: inline; filename*=utf-8''ma%C3%B1ana

> does 8BITMIME care about 8bit crap in MIME part headers ?

Because there is no way to downgrade them to 7bit for a further MTA which  
does not advertise 8BITMIME. So you need to downgrade it to  
filename*=utf-8''ma%C3%B1ana before it leaves the UTF8SMTP universe.

> contain 2231 encoded-words, without knowing its structure.

Then don't put yourself in a position where that is necessary. If the  
writer of the message is foolish enough to put some of his Subject line  
encoded in RFC 2047 and some of it as raw UTF-8, then he deserves all he  
gets (and should consider himself lucky if he gets a bounce rather than an  
incorrdct RFC 2047 encoding sent on).

> The info "de" for test and "en" for this is supposed to survive ;-)

How about
   subject: =?utf-8*de?Qtest_?= ?utf-8?Q?%C3%BC?= =?utf-8*en?Q?_this?=

>                      (1*utext / encoded-word)      ) *WSP

Yes, that seems fine.
> it can't solve the problem of 2331-words in the "downgrade" input.

What problem? Normally, only Content-* headers will need the RFC 2231  
treatment, and there is no need for those to be accompanied by a  
Downgraded header.

But even if there is, and you start with

    Injection-Info: news.example.com; posting-account="mañana"

that would simply downgrade to

    Injection-Info: news.example.com; posting-account*=utf-8''ma%C3%B1ana
    Downgraded: Injection-Info: news.example.com; posting-account=
      =?utf-8?Q?"ma=C3=B1ana"?=

> (now also containing script subtags like Hans or Hant) is no option.

Yes, there was a time when the header language could be specified as a  
<parameter> to the Header-Type header. Now that is gone, authors may have  
to use the RFC 2231 or RFC 2047 machinery (whichever is appropriate for  
the particular context). If they do that with RFC 2231, then no further  
downgrade will be necessry. With RFC 2047 it gets messy, though not  
impossible.

Actuslly, what would be nice would be a new Identity encoding mechanism  
for RFC 2047, so you could write

    Subject: =?utf-8*es?I?mañana=?

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 31 Mar 2007 13:34:04 +0200

[...]

> It is allowed by UTF8headers, as that document is
> currently written. So it needs a downgrade mechanism to
> go with it.

Are you talking about what I see, a Latin-1 &ntilde; ?
Or was it meant to be an UTF-8 &ntilde; ?

> Is that supposed to be what the writer of the message puts there, or is> that supposed to be what it becomes after the downgrade? If so, then that

I tried to 2231 encode the undeclared F1 octet.  Your example
didn't state that it's a new MIME version allowing raw UTF-8
in Content-* header fields.

> Because there is no way to downgrade them to 7bit for a further MTA
> which does not advertise 8BITMIME. So you need to downgrade it to

Maybe start with a new example where it's clear what you're talking
about.  MIME 1.0 with an unknown %xF1 garbage is very different
from "MIME 2.0" with UTF-8 %xC3.B1.  In both cases I don't see
why 8BITMIME should care about 8bit stuff in the body, it has a
parameter BODY=8BITMIME.  The definition in RFC 1652 says:

| The value associated with the BODY parameter indicates whether the
| content body which will be passed using the DATA command consists of
| a MIME message containing some arbitrary octet-aligned material
| ("8BITMIME") or is encoded entirely in accordance with [1] ("7BIT").

IMO "some arbitrary octet-aligned material" does _not_ imply that
servers are supposed to check the MIME syntax, let alone version.

> Then don't put yourself in a position where that is necessary. If
> the writer of the message is foolish enough to put some of his
> Subject line encoded in RFC 2047 and some of it as raw UTF-8, then
> he deserves all he gets

That's not foolish.  IMO using UTF-8 without language tags, word by
word if necessary, can be foolish.  For that using 2231 is required,
unless you intend to talk about the raw + deprecated plane 14 tags.

>    subject: =?utf-8*de?Qtest_?= ?utf-8?Q?%C3%BC?= =?utf-8*en?Q?_this?=

I think that's perfect.  You could also add the spaces to the second
word, leaving the adjacent words alone -- for cases where you have no
clue what charset / language / encoding it might be - but there can't
be any encoding different from "Q" or "B", or can it ?.

> Yes, there was a time when the header language could be specified as
> a <parameter> to the Header-Type header.

That allowed only "global" language definitions for a complete header.
Header fields are created by different agents, for that you need 2231.
And as soon as you have 2231 anyway using different languages within
a header field is trivial (e.g. for Keywords: in different languages.)

The BCP 47 concept of "language tags" now also covers scripts.  Bruce
didn't continue to work on his Content-Script I-D.  And that covered
again only the content (body), not the header.

> what would be nice would be a new Identity encoding mechanism

>     Subject: =?utf-8*es?I?mañana=?

Really nice.  A missing piece for a future 2231bis integrating 2047.
Or at least a point in any future "MIME version 2.0" requirements...

Frank