Subject: [EAI] Downgrade question

From: Frank Ellermann <[email-masked]>
Date: Sat, 03 Feb 2007 19:26:38 +0100

Hi, IMO the following table describes all "downgrade" scenarios for
a message arriving via UTF8SMTP.

If the header is not UTF-8, then "downgrade" is a NOOP, resulting
in a valid or invalid message/rfc822 with a 7bit or 8bit body.  If
the header is UTF-8 a successful "downgrade" transforms it into a
valid message/rfc822 with a 7bit or 8bit body.

The assumption is that the conversion of any 8bit message/rfc822
to a 7bit message/rfc822 is a solved problem for all 8BITMIME MTAs
trying to talk with 7bit MTAs.  An 8bit message/rfc822 can contain
nested message parts with arbitrary subtypes, that's already the
case today before the introduction of message/utf-8 (and family).

Therefore I think he complete problem of getting 8bit to 7bit
right is simply not the job of EAI, we're not doing "2045bis" or
"MIME version: 2.0" (unless we must if anything else fails).  

Is that correct, or am I missing something very critical ?

Frank

        Table: Downgrade results for a (wannabe) message/utf-8

Message header |  8bit message body  |  7bit message body  | Comment
---------------+---------------------+---------------------+--------
UNKNOWN-8BIT   | (actually invalid)  | (actually invalid)  | NOOP
US-ASCII       | message/rfc822 8bit | message/rfc822 7bit | NOOP
UTF-8          | message/rfc822 8bit | message/rfc822 7bit |

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Kari Hurtta <[email-masked]>
Date: 03 Feb 2007 20:57:44 +0200

> Message header |  8bit message body  |  7bit message body  | Comment

Is 'header' there in RFC 2822 sense?   Then there is also case where
whom header fields are UTF-8 and some UNKNOWN-8BIT. That makes

mixed          |                     |                     |
 UNKNOWN-8BIT  | (actually invalid)  | (actually invalid)  |
 UTF-8         | message/rfc822 8bit | message/rfc822 7bit |

That also occurs when top level header of mail is  UTF-8
and header of nested message/rfc822 uses UNKNOWN-8BIT.

UTF-8 with      | (actually invalid)  |    (not exists)    |
  nested        | message/rfc822 8bit |                    |
   UNKNOWN-8BIT |                     |                    |

If on nested datatypes have 8-bit header fields, it actually
makes top level 8bit (as content-transfer-encoding)

On some cases there is possible

UTF-8          | message/rfc822 7bit  |                     |

that is when on original 8bit body is just because UTF-8 header
fields on mime structure. Downgrade changes them 7bit.

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 03 Feb 2007 20:42:08 +0100

> Is 'header' there in RFC 2822 sense?

Actually in a message/utf-8 sense, extending some 2822 constructs
to allow UTF-8.  Some years ago Charles already did this for news,
there's some "prior art" in the USEFOR archive... :-)

> Then there is also case where whom header fields are UTF-8 and some
> UNKNOWN-8BIT.

IMO anything that's neither valid UTF-8 nor pure ASCII is just wrong,
an invalid message/rfc822.  

> and header of nested message/rfc822 uses UNKNOWN-8BIT.

My table is only for the top level message as transmitted by UTF8SMTP.

Nested mesage parts belong to the body, we can ignore them for the
purpose of "downgrade UTF8SMTP to 8BITMIME".  Body problems are not
our problems.  

If we'd offer application/message-munged it's a voluntary solution
of a RFC 1652 exercise, not really a part of EAI.  Only then do we
ever need to look into the body, deep in 1652 / MIME territory.

> makes top level 8bit (as content-transfer-encoding)

Yes, but that's again an 8BITMIME issue, not an EAI issue.  Nothing
is wrong with an 8bit message/rfc822 (top level) containing anything
it likes in its body.  

 
> fields on mime structure. Downgrade changes them 7bit.

A 7bit message/utf-8 is perfectly okay.  It's an UTF-8 header with
any 7bit content type (including QP and B64).  It's in the lower
right cell of my table (UTF-8 row, 7bit column).

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 03 Feb 2007 22:19:18 +0200

> purpose of "downgrade UTF8SMTP to 8BITMIME".  Body problems are not

I disagree strongly!
MIME header fields on nested mime structure IS our problem.

Top level header fields can be all US-ASCII only, but we have

draft-ietf-eai-utf8headers-02.txt:

-----------------------------------------------------------------------
7.2.  MIME headers

   The syntax of <value&gt, as defined in RFC 2045, is

   value   =       token / quoted-string

   In all those headers, such as Content-Type and Content-Dispoaition
   [and maybe our own Header-Type, plus lots of others being defined in
   various other documents], which make use of <value&gts within
   <parameter&gts as defined in [RFC2045] as modified by [RFC2231], it
   will now be allowed to use <quoted-string>s containing UTF-8
   characters (see the revised syntax of <qtext> in Section 6.2 of this
   document).

-------------------------------------------------------------------------

Therefore body problems are our problems when they are on MIME headers!

That means also that you must parse whole MIME structure, before you know
is message UTF8SMTP.

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 03 Feb 2007 23:12:44 +0100

> MIME header fields on nested mime structure IS our problem.
[...]
> draft-ietf-eai-utf8headers-02.txt:
[...]
> 7.2.  MIME headers
[...]
>    it will now be allowed to use <quoted-string>s containing UTF-8

Well, I never supported to screw with MIME header fields or comments,
including 2822 comments.  I never will support it, MIME is sacrosanct.

So from my POV the problem doesn't exist.  At some point in time when
legacy charsets are ancient history, and UTF-8 is the only charset
used "on the wire", somebody else can tackle this.  RFC 2277 says that
this will be 50 years after its publication *or* *later*.  Maybe SMTP
is also ancient history at this time.  I don't care about it now.  

Getting message/utf-8 (and family) right is interesting enough, adding
nonsense like UTF-8 in comments and Content-* only risks a complete
failure.  You've told me that popular MTAs don't support RFC 1652 yet.
That's not the point where adding _unnecessary_ features to UTF8SMTP
sounds like an attractive idea.

> That means also that you must parse whole MIME structure, before you
> know is message UTF8SMTP.

You only need to do that if you want to know if it's Mime-Version: 2.0.
But the Mime-Version: 2.0 is already noted in the "top-level" header.

Besides EAI isn't MIME 2.0:  We're not doing 2045bis, 2046bis, 2047bis,
2049bis, and 2231bis, gibbous or not.  You mentioned that twice today:
The message/utf-8 draft must not touch 2045.

We only (hopefully) lay the fundament for somebody else to create a
MIME 2.0 on top of UTF8SMTP, later.  When the last 7bit MTAs are in a
museum.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 04 Feb 2007 09:35:16 +0200

> The message/utf-8 draft must not touch 2045.

I wrote: ----------------------------------------------------------------
> Therefore that needs update RFC 2045    (and that may be difficult,
> when documents of that working group are Experimental and  RFC 2045
> is Standards track. )

Specially when that tries update something which affect these, 
which do not participate experiment defined on these Experimental
RFCes.
------------------------------------------------------------------------

These Content-* affect only these which participate experiment defined 
on these Experimental RFCes.

That is not different that these Experimental RFCes update RFC 2821
which is Standards track.

If message/utf-8  was used after 7-bit downgrade, it was affect these,
which do not participate experiment.

( That is irrelevent, is it using
    Mime-Version: 1.0
  or
    Mime-Version: 1.0
  when these changes affect only these which participate experiment. 
)

If I guess correctly, it is just Last Call which says is something 
allowed or not.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 04 Feb 2007 12:57:34 +0200

> The message/utf-8 draft must not touch 2045.

Actually draft-ietf-eai-utf8headers-02.txt updates RFC 2822 (or RFC 822).
It just indirectly affects MIME.  That is what quoted text says.

( Of course RFC 2822 and RFC 822 are on Standards track, but these are
  assumed to affect only when UTF8SMTP is negotiated. )

| 6.2.  Syntax extend from RFC 2822
|
|   The following rules are intended to extend the corresponding rules in
|   RFC 2822 to allow UTF8 characters.

These naturally affects to all RFCes which uses these RFC 2822 syntax
elements. There is probbaly also other than just MIME. What limits
it is following:

|   SMTP client can send header fields in UTF-8 format, if the UTF8SMTP
|   extension advertised by SMTP server or as permitted by other
|   transport mechanisms.

That was not redefining anything:

| 7.2.  MIME headers
| 
|    The syntax of <value>, as defined in RFC 2045, is
| 
|   value   =       token / quoted-string

( And I say:

  If message/utf-8 uses base64 / quoted-printable, then it must
  also update (that wording of) RFC 2045, so that base64 / quoted-printable 
  is allowed.  
) 

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 04 Feb 2007 15:06:12 +0100

>| Therefore that needs update RFC 2045    (and that may be difficult,
>| when documents of that working group are Experimental and  RFC 2045
>| is Standards track. )
>

> on these Experimental RFCes.

It's allowed to put complete message/utf-8 (and family) objects as MIME
parts into normal 2822 Internet Messages (or FWIW s-o-1036 NetNews), and
then it will affect folks not participating in the experiment.

As soon as message/utf-8 is registered it can be used and will pop up
everywhere (as part of ordinary mesages), and nobody will ask if that's
an "experiment" if UTF-8 in Content-* confuses their anti virus software.

That's a major security issue, claiming that it's "only" experimental
doesn't help.  Don't screw with MIME.  For a sneak preview of the MIME
security considerations elsewhere (limited to RFC 2231 in NetNews) see
<http://tools.ietf.org/html/draft-ietf-usefor-usefor-12#section-5>

> If I guess correctly, it is just Last Call which says is something

Yes, and the IESG evaluation.  I've a vivid imagination of what they'd
do if Ned or Keith ask "now what's that?" wrt RFC 2045 in a Last Call.

Any "screw with MIME" might be also interesting enough for the IAB -
if not I'd wonder what they're good for.

Try a Google search for "MIME vulnerability", IMO it's madness to join
this club for the obscure and unnecessary feature "UTF-8 in Content-*".

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 04 Feb 2007 16:05:20 +0100

>> 2049bis, and 2231bis, gibbous or not.
[...]

>   assumed to affect only when UTF8SMTP is negotiated. )

EAI is experimental.  2822 is a PS.  MIME is a DS, and it's based on 822,
not 2822.  Whatever EAI or a 2822bis say, it has zero consequences for
the minimal MIME conformance specified in RFC 2049.

The future message/utf-8 when registered has to work with 2049 MIME, not
only with "proposals" like 2231 or 2822, let alone any EAI "experiments".

We can't B64 any message subtypes, and we can't allow UTF-8 in Content-*.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 04 Feb 2007 18:13:16 +0200

> <http://tools.ietf.org/html/draft-ietf-usefor-usefor-12#section-5>

Is that same issue already  on top level mail header fields which
have UTF-8 inside of message/utf-8 ?  Anti virus software often
picks Subject: -header field to report. And sometimes From: -header 
field.

I do not see, how that is limited to Content-* header fields.

Either anti virus software knows what  message/utf-8 is 
or treat it as unknown type. There is lot of media types.

You are these problems anyway, if you can not limit where
message/utf-8 is allowed. 

/ Kari Hurtta

---

From: John C Klensin <[email-masked]>
Date: Sun, 04 Feb 2007 11:24:20 -0500

--On Sunday, 04 February, 2007 16:05 +0100 Frank Ellermann
<[email-masked]> wrote:

> EAI is experimental.  2822 is a PS.  MIME is a DS, and it's
> based on 822, not 2822.  Whatever EAI or a 2822bis say, it has
> zero consequences for the minimal MIME conformance specified
> in RFC 2049.
> 
> The future message/utf-8 when registered has to work with 2049
> MIME, not only with "proposals" like 2231 or 2822, let alone
> any EAI "experiments".
> 

Frank,

This type of argument is sometimes known as behaving like a
"procedural lawyer", an extension of being a "protocol lawyer".
It rarely works well in the IETF and is, I would suggest,
inappropriate here.

I think there is some merit in what you are advocating and that
it needs to be considered carefully on the basis of
interoperability with existing (non-upgraded) implementations
that conform to existing standards.  However, let's base those
considerations on the technical facts and figuring out what the
right thing to do is.  Once we establish that, we can then
figure out how to make it happen, rather than assuming that
existing rules seriously constrain our choices.

Just my opinion.

    john

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 04 Feb 2007 19:04:40 +0100

> "procedural lawyer", an extension of being a "protocol lawyer".

The more substantial security concern was in another message.

The procedural stuff was triggered by Kari's argument, that if 
EAI updates 2822, then MIME would automatically inherit this,
because MIME uses 2822.  I think that's wrong, procedurally.

In practice it means that existing MIME software might be less
robust than we hope and choke on UTF-8 in Content-*.  Kari's
argument also hit my "FUSSP" (here for worldwide MIME upgrade)
nerve, and that made me angry, sorry.  I like MIME up to 2049.

Frank

---

From: John C Klensin <[email-masked]>
Date: Sun, 04 Feb 2007 13:25:59 -0500

> The more substantial security concern was in another message.

Yes, I know.   But when you, or Kari, or Soobok, or Charles, or
all of you in combination, generate a small storm of messages on
closely-related subjects, many of us won't read carefully all
the way to the end of all of them.  The consequence is that the
noise drowns out the signal.

> because MIME uses 2822.  I think that's wrong, procedurally.

It is also wrong factually.  What is important in that regard is
what actual implementations, especially those that have made a
serious effort to be conforming, are doing, not what the
standards happen to say.  So, even if Kari (or your
interpretation of what he is saying) were correct about impact
on the standards, it would still be the case that conforming
implementations of 2822 would not inherently know how to handle
these new constructions.

For the record, you just got unlucky -- I could as easily have
complained about one of Kari's notes or one of Soobok's or one
of Charles's, since I think all of you have fallen into the
problem at one time or another.

> nerve, and that made me angry, sorry.  I like MIME up to 2049.

Everything we do here that could hit a non-upgraded server or
client is dangerous.  The hardest part of our work, IMO, is to
analyze and evaluate those risks to legacy conforming (and even
nearly-conforming) implementations and, following something
Randy Gellens has been saying repeatedly recently, make sure
that we don't introduce problems for the longer term (in which
most of the Internet has upgraded) in order to make things a
little easier for short-term transitions.  

If we can focus on that case identification and analysis, we
will make progress.  If we get distracted by discussions about
procedural models, we are less likely to be successful, IMO.

       john

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 04 Feb 2007 20:48:25 +0100

John C Klensin wrote:

 [analyze and evaluate]
> risks to legacy conforming (and even nearly-conforming)
> implementations and
[...]
> make sure that we don't introduce problems for the longer
> term (in which most of the Internet has upgraded) in order
> to make things a little easier for short-term transitions.

> If we can focus on that case identification and analysis,
> we will make progress.

Okay, I try a summary of Kari's and my "brain-storming" here
wrt UTF-8 in Content-*:

A message transported by SMTP (or NNTP or UUCP) is today
implicitly a message/rfc822.  It might be even "valid" as
defined in the relevant RFCs like 2822.

After the introduction of UTF8SMTP that's not more always
the case, it can be a message/rfc822 or a message/utf-8.

A valid message/rfc822 is also a valid message/utf-8, but
the opposite isn't always true, they're not equivalent.

Therefore we need a way to identify cases, where a message
is a "real" message/utf-8.

One way to do this is to look at its header, if it's pure
ASCII it's a message/rfc822, if it's valid UTF-8 it's a
message/utf-8.  In both cases not necessarily syntactically
valid, but it could be valid.

If it's anything else ("non-ASCII") it can't be valid, and
we can simply put it in the "invalid message/rfc822" bin.

Another way to identify it is to parse its complete MIME
structure, its "top" header, and headers of all contained
MIME parts in its body.

The "top header only" approach can't work if somewhere in
the nested multipart MIME headers raw UTF-8 is used in a
Content-* including comments (as the first used UTF-8).

[end of summary]

The "parse nested MIME parts" might be difficult, it has
to descend into multipart/* parts, but not into message/*,
and it's apparently an update of MIME.

Kari, please correct me if I got it wrong.  IMO we've to
decide what we want, allowing UTF-8 in MIME part headers
has various consequences, it also affects downgrading.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 05 Feb 2007 21:27:59 -0000

> valid, but it could be valid.

But that simply will not fly.
> structure, its "top" header, and headers of all contained

And I don't see any alternative to that, given that we no longer (it  
seems) have Header-Type:.
> Content-* including comments (as the first used UTF-8).

We have carefully given the impression, in [utf8headers], that (certainly  
at the top level) you can freely use Utf-8 in comments and unstructured  
headers (places where hitherto you would have had to use 2047). I think it  
is clear that this is not restricted to the set of headers defined in RFC  
2822. Surely it applies everywhere? Which means that, for those header  
still defined in terms of RFC 822 (and 822 also allowed comments) you  
either assume that those comments can now be in UTF-8, or you mentally  
rewrite the ABNF of those headers into the RFC 2822 style and proceed from  
there. If you do not do that, then people are going to be utterly confused  
as to where they can use UTF-8 and where they can't. And for sure, all the  
Content-* headers come under that. Additionally, some of those |Content-*  
headers can make use of UTF-8 within parameters, in lieu of the ghastly  
RFC 2231.

So if I can write
    Content-Description: some utf-8 text
in the top level headers, then users are going to find it totally bizarre  
that they cannot also write it in the headers of individual (multi)parts  
(where it is arguably much more useful, since you do not have a Subject  
header available down there, and it would be nice to explain what the  
immage/jpeg that follows is actually a picture of the "Space Shuttle" , or  
"Le shuttle d'éspace" -feel free to mend my French).

But even if you do not allow it in those body-part headers, you certainly  
have to allow it within an embedded message/rfc822 or message/utf-8  
(whichever it turns out to be), and so you simply cannot avoid that  
recursive descent scan.
> and it's apparently an update of MIME.

But yes, it DOES have to descend into message/* (or at least some of them,  
though it is not yet clear which).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 09 Feb 2007 15:28:58 -0000

On Tue, 06 Feb 2007 15:01:44 -0000, Kari Hurtta  
<[email-masked]> wrote:

But apparently not sent to the list, so it is repeated in full here.

> "Charles Lindsey" <[email-masked]> writes in gmane.ietf.ima:
>
> <...>
>> But yes, it DOES have to descend into message/* (or at least some of
>> them,  though it is not yet clear which).
>>
>
> Avoiding descend to message/rfc822 requires that  message/utf-8
> attachments are not allowed on RFC 822 mail.
>
> I'm advocating that, but Frank Ellermann was claiming on some mail
> that message/utf-8 is allowed if it is registered.
>
> We must keep that message/utf-8-anything types out regular mail
> and message/rfc822 attachments or types.

You can't state that without stating very clearly exactly how a  
message/utf-8 is constructed, otherwise we shall all be arguing at  
cross-purposes.

If its internal structure is identical to that of message/rfc822 (e.g. it  
can contain message/rfc822s inside it, which might have to be downgraded  
when leaving the 8BITMIME world), then MTAs will have to be able to  
descend through its structure. In which case, would it not be simpler just  
to use message/rfc822 in the first place? I have not yet seen a single  
technical argument to show that we need a separate message type here.

Introducing new message types unnecessarily is just going to cause yet  
more trouble with assorted agents that are not aware of them (we have  
already seem that sendmail gets some message types wrong).

---

From: Kari Hurtta <[email-masked]>
Date: 11 Feb 2007 19:23:26 +0200

> message/utf-8 is constructed, otherwise we shall all be arguing at

message/utf-8  exists only on draft-ietf-eai-dsn-00.txt

|  The second type, used for content return, is message/utf-8 is similar
|  to message/rfc822, except it contains a message with UTF-8 headers.

( That whole thread started from comments about draft-ietf-eai-dsn-00.txt )

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 12 Feb 2007 02:47:00 +0100

> message/utf-8  exists only on draft-ietf-eai-dsn-00.txt

Anything like a message/rfc822 with UTF-8 in its header is a
message/utf-8, because it obviously can't be a message/rfc822.

| NOTE:  In certain transport enclaves, RFC 822 restrictions such as
| the one that limits bodies to printable US-ASCII characters may not
| be in force. (That is, the transport domains may exist that resemble
| standard Internet mail transport as specified in RFC 821 and assumed
| by RFC 822, but without certain restrictions.) The relaxation of
| these restrictions should be construed as locally extending the
| definition of bodies, for example to include octets outside of the
| US-ASCII range, as long as these extensions are supported by the
| transport and adequately documented in the Content- Transfer-Encoding
| header field.  However, in no event are headers (either message
| headers or body part headers) allowed to contain anything other than
| US-ASCII characters.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 13 Feb 2007 21:35:42 -0000

> message/utf-8, because it obviously can't be a message/rfc822.

Yes it can, if we extend the definition (within UTF8SMTP messages only, of  
course) of message/rfc822 accordingly. That implies, of course, that the  
downghrade process must upgrade such 'extended' message/rfc822s, but then  
we already agree that it would have to dowbgrade each message/utf-8, so  
the requirement is exactly the same whatever name we call it by.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 13 Feb 2007 21:57:51 -0000

On Fri, 09 Feb 2007 18:27:55 -0000, Kari Hurtta  
<[email-masked]> wrote:

Sigh! Another message to me from Kari not posted to the full list.

>> If its internal structure is identical to that of message/rfc822
>> (e.g. it  can contain message/rfc822s inside it, which might have to
>> be downgraded  when leaving the 8BITMIME world), then MTAs will have
>> to be able to  descend through its structure. In which case, would it
>> not be simpler just  to use message/rfc822 in the first place? I have
>> not yet seen a single  technical argument to show that we need a
>> separate message type here.
>
> To me it makes sense to use  message/rfc822

Good! That makes two of us.
>
> draft-ietf-eai-dsn-00.txt gives following reason why message/rfc822
> is not used:
>
>    The second type, used for content return, is message/utf-8 is similar
>    to message/rfc822, except it contains a message with UTF-8 headers.
>    This type has profound implications on the email infrastructure.
>    First, Internet Message Access Protocol [RFC3501] servers MUST NOT
>    descend a message/utf-8 when generating the message BODYSTRUCTURE, it
>    is likely a new variant on BODYSTRUCTURE will be necessary that does
>    descend message/utf-8 body parts.

Can someone please explain the nature of this problem? I have looked at  
RFC 3501, and it says NOTHING about BODYSTRUCTURE, except that such a  
thjing exists.

But, in any case, this argument is bogus. A UTF8SMTP message is meant to  
be dongraded before it ever reaches any non-extended agent, and that  
includes IMAP stores. So any message/rfc822 that succeeds in reaching an  
old IMAP store should already be 100% safe.

OTOH, new IMAP systems will presumably be aware of the extensions, and  
will do the "right thing", whatever that is (and our IMAP draft needs to  
cover whatever BODYSTRUCTURE problem there might be).
>
> It gives also another reason, which is bogus:
>
>                                      Second, if this type is sent to a
>    7-bit-only system, it could be encoded in base64 or quoted-printable
>    [RFC2045].  As a result, SMTP servers and other systems which
>    transfer a message/utf-8 body part MAY choose to down-convert it to a
>    message/rfc822 body part using the rules described in Downgrading
>    mechanism for Email Address Internationalization [I-D.ietf-eai-
>    downgrade].
>
>
> That second reason do not take account that base64 and quoted-printable
> are fobidded for message/*

+1
>
>
> I do not know is that first rwason valid.

See above.

---

From: Kari Hurtta <[email-masked]>
Date: 14 Feb 2007 08:03:03 +0200

"Charles Lindsey" <[email-masked]> writes in gmane.ietf.ima:

> Yes it can, if we extend the definition (within UTF8SMTP messages
> only, of  course) of message/rfc822 accordingly. That implies, of
> course, that the  downghrade process must upgrade such 'extended'
> message/rfc822s, but then  we already agree that it would have to
> dowbgrade each message/utf-8, so  the requirement is exactly the same
> whatever name we call it by.

+1    ( I agree )

draft-ietf-eai-dsn-00.txt gives IMAP BODYSTRUCTURE as reason for
message/utf-8.

I can be wrong, but

There are also on IMAP extensions for UTF-8 which protects UTF8SMTP
messages.  Therefore UTF8SMTP unaware imap clients do not see  UTF8SMTP
messages, but instead they are downgraded.

Therefore they do not see UTF-8 fields on BODYSTRUCTURE, because also these 
message/rfc822 objects are downgraded.

/ Kari Hurtta

---

From: Chris Newman <[email-masked]>
Date: Thu, 15 Feb 2007 15:12:39 -0800

MIME today has three types of body parts:

1. unstructured body parts.
2. structured body parts: multipart/*, message/rfc822
3. quasi-structured body parts: application/zip, etc

All compliant MIME agents are required to process types 1 and 2 consistently.

For case 3, which has never been formalized, MIME agents are not required to 
understand or descend the quasi-structured body part (further, the nested 
encoding restriction has already been ignored by the real-world).  But certain 
MIME agents are fundamentally broken if they don't descent the quasi-structured 
part.  In the case of application/zip, any virus scanner which fails to descend 
the structure is just plain broken.

The intention of message/utf-8 is to be another quasi-structured body part like 
application/zip.  In this case, we expect UTF-8 header aware agents to descend 
the structure and we expect non-UTF-8-header agents to not descend the 
structure.  There is no requirement for a UTF8SMTP agent to downgrade a 
message/utf-8 body part (and in the case of a DSN, I'd say it SHOULD NOT 
downgrade message/utf-8).  An argument could be made that message/utf-8 should 
be "application/utf-8-message" -- I'd consider that discussion helpful and 
would like to see information about what implementations do with message/utf-8 
vs. application/utf-8-message so we can make an intelligent choice.

I'm strongly opposed to putting UTF-8 header content into a MIME body part with 
a message/rfc822 label.  Having a label which has different meanings in 
different contexts is dangerous and having a data-object label with that 
property is even more dangerous.  MIME agents allow users to save MIME bodies 
to files which don't have any typing context beyond the filename extension 
derived from the media type label.  I'm much more comfortable that we're 
avoiding installed base breakage if "stuff which traditional agent might crash 
or mangle when processing" has a different label so the traditional agent won't 
even attempt to process it.

                - Chris

Charles Lindsey wrote on 2/13/07 21:35 +0000:

> downghrade process must upgrade such 'extended' message/rfc822s, but then  we
> already agree that it would have to dowbgrade each message/utf-8, so  the
> requirement is exactly the same whatever name we call it by.
>
> --

---

From: Kari Hurtta <[email-masked]>
Date: 16 Feb 2007 07:01:01 +0200

Chris Newman <[email-masked]> writes in gmane.ietf.ima:

> For case 3, which has never been formalized, MIME agents are not
> required to understand or descend the quasi-structured body part
> (further, the nested encoding restriction has already been ignored by
> the real-world).  But certain MIME agents are fundamentally broken if
> they don't descent the quasi-structured part.  In the case of
> application/zip, any virus scanner which fails to descend the
> structure is just plain broken.
> 
> The intention of message/utf-8 is to be another quasi-structured body
> part like application/zip.  In this case, we expect UTF-8 header aware
> agents to descend the structure and we expect non-UTF-8-header agents
> to not descend the structure.  There is no requirement for a UTF8SMTP
> agent to downgrade a message/utf-8 body part (and in the case of a
> DSN, I'd say it SHOULD NOT downgrade message/utf-8).  An argument
> could be made that message/utf-8 should be "application/utf-8-message"
> -- I'd consider that discussion helpful and would like to see
> information about what implementations do with message/utf-8

Effectively 8BITMIME downgraders are required to bounce messages if
they see unknown message/* with 8-bit content -- this is only available
possibility on specifications which they know.

Sendmail's documentation says that it strip unknown message/* to 7-bit
(instead of base64 ou quoted-printable encoding).

When I tested it (see below), sendmail passed 8-bit content as it to
next host although next host not announced 8BITMIME (other parts of
mail sendmail base64 encoded.)

When message/utf-8 is used for MUAs for attached messages, on that
case downgrading message/utf-8 to message/rfc822 is required on
UTF8SMTP downgraders.  MUAs on general can not know is recipient
of message UTF8SMTP capable, so to them it makes sense to use 
message/utf-8 when user attached some UTF8SMTP message to new mail.
And if recipient is not UTF8SMTP capable, it is better that recipient
still get message and also attached message on downgraded form.
Ie. on same form than new recipient would get attached message if
it was sent to him/her on first place.

> I'm strongly opposed to putting UTF-8 header content into a MIME body
> part with a message/rfc822 label.  Having a label which has different
> meanings in different contexts is dangerous and having a data-object
> label with that property is even more dangerous.  MIME agents allow
> users to save MIME bodies to files which don't have any typing context
> beyond the filename extension derived from the media type label.  I'm
> much more comfortable that we're avoiding installed base breakage if
> "stuff which traditional agent might crash or mangle when processing"
> has a different label so the traditional agent won't even attempt to
> process it.
> 
>                 - Chris

That work group already decided once that marker is not required
(on removal of Header-Type vote).  Using message/utf-8 versus message/rfc822
is just another kind label. Many of same considerations apply to here. 

( I also suggested HDR=UTF8  marker of ESTMP. )

/ Kari Hurtta

--------------------------------------------------------------------------
From: Kari Hurtta <[email-masked]>
Subject: sendmail 8.13.5 tested 
	(Re: 8to7 downgrade and message/* (Re: [EAI] Re: Registeration forms))
Newsgroups: gmane.ietf.ima
To: [email-masked]
Date: 03 Feb 2007 21:39:52 +0200

Kari Hurtta <[email-masked]> writes in gmane.ietf.ima:

> Frank Ellermann <[email-masked]> writes gmane.ietf.ima:
> 
> > Kari Hurtta wrote:
> >  
> > > parts on these registrations forms do not fly.
> > 
> > Yes, it's impossible to send a "real" message/utf-8 via 7bit routes,
> > it has to be "downgraded" and after that fed into some 8bit to 7bit
> > gateway.
> > 
> > > See RFC 2045 restrtrion of encodings for multipart and message 
> > > types. ( I quoted it already twice. )
> > 
> > What do existing 8to7 gateways with nested and unknown 8bit message
> > subtypes ?
> > 
> > We discussed that already, but I forgot the details - at that time
> > we were still far into the woods with a Header-Type: UTF-8 trying to
> > twist a message/rfc822 into something it isn't and can't be.
> 
> Sendmail strips message/* (other than  message/rfc822) to 7-bit.     
> I do not know about others. Except that some MTAs announce 8BITMIME 
> and always sends 8-bit even when next host do not announce 8BITMIME.

Seems that actually sendmail 8.13.5 code just send 8-bit as is (instead of
downgrading).

------------------------
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=AA

--AA
Content-Type: text/plain

Tässä 8-bit body

--AA
Content-Type: message/utf-8

Subject: Tässä 8-bit header

Tässä 8-bit body

--AA--
-------------------------

results after 8BITMIME downgrade:

-------------------------
....
Date: Sat, 3 Feb 2007 21:32:24 +0200
From: Kari Hurtta <hurtta@CENSORED>
Message-ID: <200702031932.l13JWO0P005559@CENSORED>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=AA
To: undisclosed-recipients:;

--AA
Content-Type: text/plain
Content-Transfer-Encoding: base64
X-MIME-Autoconverted: from 8bit to base64 by CENSORED id l13JWa5K005560

VMOkc3PDpCA4LWJpdCBib2R5DQo=
--AA
Content-Type: message/utf-8

Subject: Tässä 8-bit header

Tässä 8-bit body

--AA--
-------------------------

It it does not strip unknown message/* to 7bit as KNOWNBUGS claim.
They are actually  passed as it.

On code there is

       if (sm_strcasecmp(type, "message") == 0)
        {
                if (!wordinclass(subtype, 's'))
                {
                        flags |= M87F_NO8BIT;
                }
                else

But M87F_NO8BIT have not actually that effect.

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 16 Feb 2007 22:20:44 -0000

> All compliant MIME agents are required to process types 1 and 2  

Ah! I see where you are coming from now, and your explanations clarify  
what the possibilities are (which is not to say that I agree with your  
conclusion).

I had regarded cases #3 and #1 as equivalent for the puspose of most  
unspecialized agents, since their content can be regarded as opaque unless  
you really need to understand what is inside it (which probably means you  
are at one of the end points of the communication). In particular, you are  
allowed to apply any C-T-E that suits you, and then it is applied over the  
whole of the object. The difference with case #2 is that you are not  
allowed to do that.

> agents to descend the structure and we expect non-UTF-8-header agents to  
> not descend the structure.  There is no requirement for a UTF8SMTP agent  
> to downgrade a message/utf-8 body part (and in the case of a DSN, I'd  
> say it SHOULD NOT downgrade message/utf-8).  An argument could be made  
> that message/utf-8 should be "application/utf-8-message" -- I'd consider  
> that discussion helpful and would like to see information about what  
> implementations do with message/utf-8 vs. application/utf-8-message so  
> we can make an intelligent choice.

Yes, it would definitely have to be application/utf-8-message, but then  
you have just invented an encapsulation mechanism (rather like  
application/new-transmission which is intended to enable arbitrary news  
articles to be tunneled through 7bit only channels, with a view to having  
them reconstituted at the far end in exactly their original form, then to  
be injected into Usenet, as when an article is emailed to a moderator, for  
example).

But we have been strenuously trying to avoid the encapsulation method as a  
routine means of sending I18N mail. Moreover, we have been trying to make  
the UTF8SMTP world appear and behave just like the ASCII world within its  
own confines (leaving all the messiness of downgrading etc to cases where  
messages pass between those two worlds). In which case, people will  
naturally expect to be able to attach UTF8SMTP messages as message/rfc822  
objects within other UTF8SMTP messages in the normal manner, in the  
expectatation that appropriate downgrading (or bouncing) will take place  
in the few cases where such messages are passed into the other world. In  
fact, you won't be able to stop them using message/rfc822 in that way, so  
it had better work correctly (and in fact it is not hard to do, because  
8BITMIME downgraders have already invented the necessary basic machinery).
> label with that property is even more dangerous.

It is no different from allowing complete RFC 2822 messages to have  
different properties in the two worlds, which is in fact the fundamental  
basis of our whole approach. But it only works if you eliminate unintended  
leaks between the two worlds (which is what we are strenuously trying to  
do). And ensuring that things do not leak through being contained with  
some message/rfc822 is no different (except in detail) from ensuring that  
things do not leak through being contained within some multipart, and yet  
I hear no clamour from people wanting to invent multipart/utf-8.

Or is there something special about DSN that means that message/rfc822  
will not work within DSN, even though it is quite capable of being made to  
work in normal message passing?

---

From: Chris Newman <[email-masked]>
Date: Fri, 16 Feb 2007 18:49:58 -0800

Kari Hurtta wrote on 2/16/07 7:01 +0200:
> UTF8SMTP downgraders.

I've been convinced it should be "application/utf-8-message" (due to the 
ambiguity in the MIME specification with respect to unknown message subtypes). 
In that case, there is no requirement to downgrade.

> MUAs on general can not know is recipient

Sometimes this may be true and sometimes this may not be true.  I'd leave the 
decision of whether to pass or downgrade application/utf-8-message to site 
policy, although for DSNs, I'd say "SHOULD NOT downgrade" since an undamaged 
encapsulated message is more likely useful for that scenario.

> is just another kind label. Many of same considerations apply to here.

A robust UTF-8 aware MIME agent should treat message/rfc822 and 
application/utf-8-message as identical types (except the latter might be QP or 
base64 encoded).  However, the label distinction is necessary so that legacy 
MIME agents will not attempt to descend the structure of 
application/utf-8-message and thus will continue to function properly.  The 
difference between this case and Header-Type is that the media type label 
already exists so we have to make sure the label doesn't lie in order to avoid 
breaking deployed software.  After all, a UTF-8 header message is not an RFC 
822/2822 message.  They are incompatible types.

                - Chris

---

From: Chris Newman <[email-masked]>
Date: Fri, 16 Feb 2007 19:19:06 -0800

Charles Lindsey wrote on 2/16/07 22:20 +0000:
> Yes, it would definitely have to be application/utf-8-message, but then  you
> have just invented an encapsulation mechanism

Ok, I've been convinced application/utf-8-message is a safer label for the type.
And DSNs require an encapsulation mechanism for return-of-content, so it was 
absolutely my intention to create an encapsulation mechanism.

> routine means of sending I18N mail.

Agreed.

> Moreover, we have been trying to make
> expectatation that appropriate downgrading (or bouncing) will take place  in
> the few cases where such messages are passed into the other world.

I disagree with all your assertions here.

> In  fact,
> you won't be able to stop them using message/rfc822 in that way, so  it had
> better work correctly

I agree with this statement.  I would expect a robust UTF-8-aware MIME agent to 
do the following things:

1. Treat message/rfc822 and application/utf-8-message as semantically identical
   except the latter might be QP or base-64 encoded at the top-level.
2. Correct the label in the event content which is not, in fact, RFC 822/2822
   complaint happens to be mislabeled as such.
3. When passing to a non-UTF-8 system, perform downgrading on an
   application/utf-8-message according to site policy.  I expect the initial
   policy to be "don't downgrade application/utf-8-message in a DSN/MDN", but
   do downgrade it otherwise.  However, over time I expect downgrading of
   application/utf-8-message to become less and less desirable/useful.  It
   really depends on how well utf-8-message support deploys in clients
   at any given site, so writing strict rules would be highly inappropriate.

The goal here is to minimize risk of breaking legacy systems by protecting them 
from mislabeled content, but also to avoid burdening the future with today's 
limitations.  The former has always been a goal of the IETF.  What's new about 
this WG is we're attempting the latter at the same time.  Please keep both of 
these goals in mind as I suspect it's why you're having trouble syncing up with 
the rest of the working group.

> basis of our whole approach.

I could not disagree more.  An SMTP server which advertises UTF8SMTP is 
advertising that it accepts not only RFC 2822 messages, but that it also 
accepts UTF-8 header messages.  However, such systems also must assume all 
other systems support _only_ RFC 2822 messages until they learn otherwise 
through a negotiation mechanism.  Otherwise we risk breaking the installed base.

We're not redefining RFC 2822, we're creating a new infrastructure which 
accepts a new message format that is incompatible with RFC 2822.

> But it only works if you eliminate unintended
> do). And ensuring that things do not leak through being contained with  some
> message/rfc822 is no different (except in detail) from ensuring that  things
> do not leak through being contained within some multipart, and yet  I hear no
> clamour from people wanting to invent multipart/utf-8.

That's because multipart/utf-8 (if it does what I think it does) is a terrible 
idea.  Turning an atomic media type into a compound type with syntax 
incompatible with the canonical format causes all sorts of problems.

                - Chris

---

From: Kari Hurtta <[email-masked]>
Date: 17 Feb 2007 10:24:40 +0200

<...>
> That's because multipart/utf-8 (if it does what I think it does) is a
> terrible idea.  Turning an atomic media type into a compound type with
> syntax incompatible with the canonical format causes all sorts of
> problems.
> 
>                 - Chris

You are refering to multipart/utf8-encapsulated on my 
draft-hurtta-eai-encapsulation-00  draft?

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 18 Feb 2007 16:58:26 +0100

> For case 3, which has never been formalized, MIME agents are
> not required to understand or descend the quasi-structured
> body part

I don't understand this classification.  Is (1) the same as
text/plain and only text/plain ?

If yes (3) could be "all other MIME types excl. multipart/*
and message/rfc822", and a simple example would be text/html.

(3) would then also cover all message/* excl. message/rfc822,
message/partial, and message/external-body.  

A message/partial or message/external-body would then belong
to (2), but these message subtypes are anyway irrelevant for
our purposes, they use ASCII headers like messsage/rfc822.

> But certain MIME agents are fundamentally broken if they 
> don't descent the quasi-structured part.  In the case of 
> application/zip, any virus scanner which fails to descend

It would look into any part, remove the CTE and check it, no
matter what the content-type claims to be.  And as long as 
nobody invents a new MIME version, e.g. here "allowing" UTF-8
in body part headers, this continues to work (or not) as is.

The one situation where this might fail is a message/utf-8, a
"legacy" scanner would consider it as "unknown garbage", and
if it then verifies that it's no B64 starting with TV or UE
it will never consider the body of the message/utf-8.  Tough,
but unavoidable, the new message/utf-8 needs clear security
considerations.   After all message/partial is still worse.

> The intention of message/utf-8 is to be another 
> quasi-structured body part like application/zip.

By definition (as above, in an RFC 2049 sense).  Ideally it 
will belong to (2) in the future, obsoleting messsage/rfc822.

A gentle case of "embrace, extend, and extinguish".  If the
UTF8SMTP experiment works we'll have to update 2049, adding
message/utf-8 to "minimal MIME conformance for UTF8SMTP", or
similar.

> There is no requirement for a UTF8SMTP agent to downgrade a
> message/utf-8 body part (and in the case of a DSN, I'd say
> it SHOULD NOT downgrade message/utf-8).

Yes.  But of course the 8bit DSN like any other 8bit message
(rfc822 and utf-8 alike) is in trouble when it hits a 7bit
relay.

> An argument could be made that message/utf-8 should be
> "application/utf-8-message" -- I'd consider that discussion
> helpful

That argument is stated in chapter 5.2.4 of RFC 2046, and it's
the very idea of UTF8SMTP to ignore this advice.  In the "real
world" (8BITMIME) application/utf-8-message is an unnecessary
and unintuitive kludge.  It's only required to tunnel 7bit hops
with base64 (or QP (?)).

IMO "8to7" gateways have a problem with _any_ 8bit message part,
not limited to message/utf-8.  Therefore we could offer a more
general kludge, application/8bit-message or similar, preserving
the original subtype as parameter.

All these "8to7" headaches are not directly a part of UTF8SMTP,
we should extract the kludge into a separate "8to7" document.

And while we're at it let's promote RFC 1652 to STD, no erratum
in 13 years sounds like "ready" for me.

Frank