Subject: Re: [Filtered!] [EAI] I-D ACTION:draft-ietf-eai-dsn-00.txt

From: Tony Finch <[email-masked]>
Date: Thu, 1 Feb 2007 11:23:40 +0000

On Wed, 31 Jan 2007, Chris Newman wrote:
>
> MIME only forbids message encodings for the message subtypes it defines, no
> normative restrictions are placed on other message subtypes.  If one of the
> functions of this experiment is to make email UTF-8 clean, then the advisory
> text in section 5.2.4 of RFC 2046 does not apply to the documents in this
> experiment.

That seems sensible. The draft should say so explicitly.

> Downgrading message/utf-8-delivery-status to message/delivery-status is a bad
> idea, IMHO.  It's not going to break anything if you leave it alone and it
> will lose information if downgraded. [...] Encoding with QP or base64 is
> fine if necessary.

You're probably right. However these new types must be encoded when they
leave the utf8 email system. The draft says "if this type is sent to a
7-bit-only system, it could be encoded in base64 or quoted-printable"
which is not sufficient. If the DSN is sent from a utf8 mailer to a
8bitmime mailer without encoding, and the 8bitmime mailer then tries to
send it to a 7bit mailer, the 8bitmime mailer will not know that it may
encode the new message/* types so delivery will fail. This is catastrophic
for a DSN.

> FYI, Alexey is working on an individual submission adding the SMTP LANG
> extension so clients can request localized SMTP error text.

I know :-)

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOUTH UTSIRE FORTIES: WEST 5 TO 7, OCCASIONALLY GALE 8 AT FIRST IN SOUTH
UTSIRE, DECREASING 4 OR 5. ROUGH OR VERY ROUGH, DECREASING MODERATE LATER.
SHOWERS THEN OCCASIONAL RAIN. GOOD, OCCASIONALLY MODERATE.

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Chris Newman <[email-masked]>
Date: Thu, 01 Feb 2007 12:41:40 -0800

> That seems sensible. The draft should say so explicitly.

Good idea.  I'll add the following text to the end of section 5 in the next 
revision:

   All three new types will typically use the "8bit" Content-Transfer-
   Encoding (in the event all content is 7-bit, the equivalent
   traditional types for delivery status notifications are advised for
   greater backwards compatibility).  While MIME [RFC2046] advises
   against the use of 8-bit in new message subtypes intended for the
   email infrastructure, that advice does not apply to these new types
   which are intended primarily for use by newer systems with full
   support for 8-bit MIME and UTF-8 headers.

Alternative text proposals are welcome.

> encode the new message/* types so delivery will fail. This is catastrophic

The MIME specification directs MIME agents to treat unknown message/* subtypes 
as if they were application/octet-stream.  That means complaint MIME agents 
will know to base64 encode or quoted-printable encode 8bit body parts 
automatically as needed.  I really don't see any chance of problems with 
compliant systems.  If there are enough non-compliant systems talking to 
7-bit-only systems deployed to cause real problems, then I'm sure the 
experiment will discover that.

                - Chris

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 01 Feb 2007 21:23:33 -0000

On Wed, 31 Jan 2007 23:37:59 -0000, Chris Newman <[email-masked]>  
wrote:

> Tony Finch wrote on 1/31/07 2:20 +0000:

>> MIME forbids the encoding of message/* parts, so a message/utf-8 part
>> would have to be downgraded instead. Note that new message/* parts  
>> inside
>> 8bit messages will cause problems for existing 8BITMIME downgraders.
>
> MIME only forbids message encodings for the message subtypes it defines,  
> no normative restrictions are placed on other message subtypes.  If one  
> of the functions of this experiment is to make email UTF-8 clean, then  
> the advisory text in section 5.2.4 of RFC 2046 does not apply to the  
> documents in this experiment.

Hmmmm! That wording uses "should" rather than "SHOULD", so you are  
technically correct. However, much existing software may have assumed that  
advice would be followed, and so I do not think is necessarily wise to go  
against it.

The problem we are faced with is this:

If there is to be no Header-Type header (and it seems I am about to be  
outvoted on that), then there is a necessity to walk through the whole of  
the MIME tree looking for things that might need to be downgraded. Lots of  
software will be written that does that (and hopefully correctly, though  
our drafts need to stress the point so that nobody overlooks it).

BUT if somebody then goes and invents a new message/* subtype that  
invalidates that carefully crafted software, then we are going to be in  
serious trouble, because we cannot expect writers of MTAs etc to keep up  
with all the latest MIME fads, and even less can we expect mailservers  
throughout the world to routinely upgrade to the latest release. It will  
be quite an achievement if we can get the bulk of the world's mailservers  
to upgrade to UTF8SMTP once, but to expect them to keep on upgrading for  
ever after is too much.

Fortunately, treating all message subtypes as it they were message/rfc822  
just happens to work with all the existing subtypes (and that might  
hopefully remain the case).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Chris Newman <[email-masked]>
Date: Thu, 01 Feb 2007 15:14:39 -0800

> hopefully remain the case).

Software which did that would not be following this advice (RFC 2046, section 
5.2.4):

 MIME implementations must in general treat unrecognized subtypes of "message"
 as being equivalent to "application/octet-stream".

And would also incorrectly process message/external-body and 
message/delivery-status among others.

Now if there turns out to be a lot of deployed software which doesn't follow 
that advice, then I presume we'll discover that as part of the experiment and 
we can move the new 8bit types under application.  But shouldn't we at least 
try to put them under the semantically correct top-level type?

                - Chris

---

From: Tony Finch <[email-masked]>
Date: Thu, 1 Feb 2007 23:56:42 +0000

On Thu, 1 Feb 2007, Charles Lindsey wrote:
>
> That wording uses "should" rather than "SHOULD"

It's pre-2119.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
HUMBER THAMES: WEST OR SOUTHWEST 2 OR 3, INCREASING 4, OCCASIONALLY 5 IN
HUMBER LATER. SMOOTH OR SLIGHT, OCCASIONALLY MODERATE. OCCASIONAL RAIN LATER.
MODERATE OR GOOD.

---

From: Tony Finch <[email-masked]>
Date: Fri, 2 Feb 2007 00:12:46 +0000

>   support for 8-bit MIME and UTF-8 headers.

I like that. I'd suggest referring specifically to section 5.2.4 of RFC
2046 and pointing out that the "equivalent to application/octet-stream"
clause implies that existing 8bitmime mailers should already be happy to
encode message/* if necessary.

> The MIME specification directs MIME agents to treat unknown message/*
> subtypes as if they were application/octet-stream.  That means complaint
> MIME agents will know to base64 encode or quoted-printable encode 8bit
> body parts automatically as needed.

Hmm, yes. I think I've been more conservative than necessary because I've
been mixing up the message/* requirements with the stricter multipart/*
requirements. I still have some residual worries about the message/utf-8
type, which you have mostly addressed in section 5 of the draft. I think
the text about IMAP is too specific: it's just an example of possible
interop problems (but still rather vague about the IMAP implications). In
particular you are introducing a type that would normally be multipart-ish
like message/rfc822 is, such that MUAs can extract attachments from
enclosed messages, but unlike other multipart types it can be encoded.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
HUMBER THAMES: WEST OR SOUTHWEST 2 OR 3, INCREASING 4, OCCASIONALLY 5 LATER.
SMOOTH OR SLIGHT, OCCASIONALLY MODERATE. OCCASIONAL RAIN LATER. MODERATE OR
GOOD.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 02 Feb 2007 10:13:08 -0000

>> That wording uses "should" rather than "SHOULD"

But RFC 2046 uses MUST and SHOULD elsewhere in the customary manner. That  
convention was well-established long before 2119 formalized it.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 02 Feb 2007 12:18:05 +0100

Chris Newman wrote:

> shouldn't we at least try to put them under the semantically
> correct top-level type?

Yes.  And state that minimal MIME conforming software (RFC 2049)
wishing to participate in the UTF8SMTP experiment MUST support
message/utf-8.

A better name might be message/utf822, a mix of utf-8 + rfc822.
In your draft you say:

-   This media type contains Internationalized Email Headers
-   [I-D.ietf-eai-utf8headers] and MIME message body content.

How about this:

+   This media type supports Internationalized Email Headers
+   [I-D.ietf-eai-utf8headers], otherwise it's identical to
+   message/rfc822.

Later you have:

   Applications that use this media type: SMTP servers and email
      clients that support multipart/report generation or parsing.
      Email clients which forward messages with international
      headers as attachments.

The "attachments" is IMO wrong, it should be "MIME part or body".
A message/rfc822 does not necessarily use a Content-Disposition:
attachment, it can as well use Content-Disposition: inline.

I think you need a note somewhere stating that multipart/digest
continues to have a default message/rfc822, and multipart/digest
parts deriving from that default need an explicit Content-Type,
like Content-Type: message/utf-8 (or similar) for messages with
an UTF-8 header.

I don't like the proposed file extension ".u8msg", but I have no
better idea, or is ".ums" still available ?  What's the point of
a different extension for the header, could we get away with the
same file extension ?

I think you need a chapter about complete MIME message objects
which are not a part of a container:  Complete messages used to
be message/rfc822 by definition.  With UTF8SMTP complete messages
can be message/rfc822 or message/utfxxx, and the former is a
proper subset of the latter.  Any valid message/rfc822 is always
a valid message/utfxxx.  The opposite is not true.  Applications
trying to embed a complete valid message as MIME part or content
have two options:

1 - Don't care, just say message/utfxxx.  The draft should say
    that this lazy approach is NOT RECOMMENDED at this time.
2 - Figure it out, i.e. parse the header (only the header), if
    it's ASCII they can use message/rfc822, if it's valid UTF-8
    they MUST use message/utfxxx, and if it's another variant
    of non-ASCII they MUST NOT use message/utfxxx.

What I have in mind are "message/news" articles using non-ASCII
in the header (especially in From: and Subject: fields) where
non-ASCII is certainly not valid UTF-8 - maybe it's Latin-1 or
even windows-1252.  Of course that's no valid message/news, but
folks try it anyway.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 02 Feb 2007 12:46:45 +0100

Chris Newman wrote:
 
> I'll add the following text to the end of section 5 in the next revision:
 
>    traditional types for delivery status notifications are advised for

Please s/advised/RECOMMENDED/  Let the future standard say "advised" if
the experiment showed that there were no serious problems.  But I think
there will be serious problems for the remaining fourty "RFC
2277-years".

>    support for 8-bit MIME and UTF-8 headers.

+1  (Ignoring that advice is actually the point of UTF8SMTP experiment)

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 02 Feb 2007 14:46:54 -0000

On Thu, 01 Feb 2007 23:14:39 -0000, Chris Newman <[email-masked]>  
wrote:

> Software which did that would not be following this advice (RFC 2046,  
> section 5.2.4):
>

Not so. I am suggesting you treat any message/* as follows:

If the C-T-E is Q-P or Base64, then you treat it as such, so effectively  
the same as octet string. No current message subtype actually allows that  
AFAIK.

Otherwise (it is 7bit, 8bit or binary) you proceed as it the next thing  
(up to the first empty CRLF-only line) was a header. As indeed it is for  
message/rfc82 and message/partial.

A message/rfc822 can, of course, contain multiparts within it, so the  
recursive tree walk must follow those as it did in the Perl script I wrote.

The case of message/partial is more problematic. The headers within the  
message/partial cannot contain any 8bit stuff, because the C-T-E of a  
message/partial MUST be 7bit, so the downgrading would have nothing to do.  
But I can see problems if the message that was split into partial its was  
itself a multipart, because it is not safe to do a tree walk on that until  
the partial bits have been reassembled. Is that correct? It is clear that  
splitting a UTF8SMTP message into partial bits cannot be done at present,  
though we might make an extension to allow it.

Message/external-body also starts with its own header, though RFC 2046 is  
exceedingly vague as to what it is allowed to contain, except that a  
Content-ID must be included. The examples show Content-Type headers, and a  
Content-Type: multipart/* here could be troublesome as for  
message/partial. But I see no reason why UTF8SMTP headers should not be  
present there, and be downgraded as normal.

I see no particular problem with message/partial. Its C-T-E must be 7bit  
(unless we choose to extend that for use withn UTF8SMTP). So one does not  
expect any 8bit stuff within it, so there can be nothing to downgrade. The  
first thing within it is a set of "per-message" fields which are,  
syntacticvally, of the form of email header fields, and would therefore be  
parsed as such by the treewalk program. But none of those fields can take  
the forn of a Content-Type or Content-Transfer-Encoding, so there is  
nothing there to set the tree-walk off on further descents. If the whole  
thing is parsed the same way as a message/rfc822, tho "per-message" fields  
would look like the rfc822 headers, and the "per-recipient" fields would  
look like its body. All that would be absorbed and cause no action  and no  
harm.

If we wanted to extend message/delivery-status to permit 8bit stuff in any  
of those fields (as an alternative to inventing a new message subtype for  
the opurpose), then care would have to be taken to ensure that it was done  
in a way which permitted a downgrade.

Most of the other IANA-registered message-subtypes I have looked at seem  
to follow a similar structure, and treating them the same as  
message/rfc822 would have a similar effect - namely nothing untoward would  
happen.

So the troublesome cases would appear to be partial and external-body  
where we do NOT want to trigger exploration of further subtrees. OTOH, if  
we don't treat them as message/rfc822, then some newly-invented types  
might appear which DID require such exploration. So it is hard to see how  
any tree-walk can be made future-proof if there is no Header-Type header  
which could be used to guide it.

---

From: Kari Hurtta <[email-masked]>
Date: 03 Feb 2007 16:20:32 +0200

> > body parts automatically as needed.

No. 8BITMIME downgraders do not base64  encode or quoted-printable encode
unknown message/* subtypes.  When these downgraders are written, base64
and quoted-printable was not allowed on message/* subtypes. They are expected 
to bounce these mails instead.

( And all of them do not bounce mail, but instead do nasty things. I have
  said that already. )

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 03 Feb 2007 16:54:50 +0200

Chapter "5.  UTF-8 Delivery Status Notifications"

|   descend message/utf-8 body parts.  Second, if this type is sent to a
|   7-bit-only system, it could be encoded in base64 or quoted-printable
|   [RFC2045].  As a result, SMTP servers and other systems which

Should be:

                                       Second, if this type is sent to a
    system which do not supports 8BITMIME, it could be encoded in base64 
    or quoted-printable [RFC2045].  

( You must not assume that 8BITMIME downgraders like unknown message/*
  subtypes, which have 8-bit data. This is discussed elsewhere. )

Suggested addition was (by  Chris Newman):

This do not address RFC 2045 nested encoding rule. Which need be also
addressed when message/utf-8 is base64 or quoted-printable encoded:

   Certain Content-Transfer-Encoding values may only be used on certain
   media types.  In particular, it is EXPRESSLY FORBIDDEN to use any
   encodings other than "7bit", "8bit", or "binary" with any composite
   media type, i.e. one that recursively includes other Content-Type
   fields.  Currently the only composite media types are "multipart" and
   "message".  All encodings that are desired for bodies of type
   multipart or message must be done at the innermost level, by encoding
   the actual body that needs to be encoded.

Therefore that needs update RFC 2045    (and that may be difficult,
when documents of that working group are Experimental and  RFC 2045
is Standards track. )

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 03 Feb 2007 17:04:10 +0200

> when documents of that working group are Experimental and  RFC 2045

Specially when that tries update something which affect these, 
which do not participate experiment defined on these Experimental
RFCes.

Is that allowed?

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 03 Feb 2007 17:24:37 +0200

|   The third type, used for header return, is message/utf-8-headers and
|   contains only the UTF-8 headers of a message (all lines prior to the
|   first blank line in a UTF8SMTP message).  Unlike message/utf-8, this
|   body part provides no difficulties for present infrastructure.

This is not completely true, because RFC 2045 forbids base64 and 
quoted-printable encoding here:

|   Certain Content-Transfer-Encoding values may only be used on certain
|   media types.  In particular, it is EXPRESSLY FORBIDDEN to use any
|   encodings other than "7bit", "8bit", or "binary" with any composite
|   media type, i.e. one that recursively includes other Content-Type
|   fields.  Currently the only composite media types are "multipart" and
|   "message".  All encodings that are desired for bodies of type

Therefore text/utf-8-headers works better than message/utf-8-headers.

Also for text/utf-8-headers there helps following rule of RFC 2046:

| 4.1.4.  Unrecognized Subtypes
|
|   Unrecognized subtypes of "text" should be treated as subtype "plain"
|   as long as the MIME implementation knows how to handle the charset.
|   Unrecognized subtypes which also specify an unrecognized charset
|    should be treated as "application/octet- stream".

My draft-hurtta-eai-encapsulation-00  was using text/utf8-header
on here.

( That is  "header" versus "header field" on terminology. )

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 03 Feb 2007 17:41:59 +0200

Encoding considerations: and Restrictions on usage:
parts on these registrations forms do not fly.

See RFC 2045 restrtrion of encodings for multipart and
message types. ( I quoted it already twice. )

If this updates RFC 2045, it need to be noted on here I 
think.  

It may be that these types are OK only on UTF8SMTP
environment. On that case RFC 2045 need not be updated.

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 03 Feb 2007 17:08:46 +0100

>     or quoted-printable [RFC2045].
[...]
> addressed when message/utf-8 is base64 or quoted-printable encoded
[...]
> Therefore that needs update RFC 2045

That's IMO impossible in Mime-Version: 1.0.  A message has a header and a
body, its header defines the Content-Type details of its body.  But the
message (header) itself has no Content-Type and no CTE, it can be 7bit or
8bit.  If it's some kind of binary different from 8bit it's anyway invalid
(for our purposes).

By definition a message/rfc822 (header) is 7bit US-ASCII.  In the wild it
sometimes degenerates into unknown 8bit "extended ASCII".  A message/utf-8
(header) is by definition 8bit UTF-8.  In rare cases, explicitly noted as
NOT RECOMMENDED somewhere, it can degenerate into 7bit US-ASCII if there
happens to be no code point above U+007F in the header.

We can put a message/utf-8 as is into the various multipart Content-Types,
including multipart/report DSNs, as long as it's transmitted on 8BITMIME
routes.

We can also put it as is in the body of a message/rfc822 or other message
types, using Content-Type: message/utf-8 + Content-Transfer-Encoding: 8bit.

It's not possible to send a message/utf-8 (header) to a 7bit system, it's
also impossible to send it via a 7bit relay to a 8bit system.  We could
invent some crude appplication/utf-8-message, but in essence that would be
a variation of "ZIP it and send it as B64 application/zip" (or similar),
an application/utf-8-message type is pointless (*1)

The next thing we could try is to downgrade the message/utf-8 (header) to
message/rfc822.  After that its body might be still 8bit, among others it
could contain text/plain Latin-1, text/html UTF-8, or nested message/utf-8
parts.  But from our POV it's an ordinary message/rfc822 with 8bit body if
the downgrade worked, and ordinary 8BITMIME to 7bit gateways can deal with
that.

Or is the problem that these gateways choke on a *nested* message/utf-8 ?
They shouldn't, they're supposed to treat it as application/octect-stream.

Frank

1: application/utf-8-message could be interesting for a transport via
   non-UTF8SMTP, if receivers know to "unpack" it into a message/utf-8.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 03 Feb 2007 17:58:24 +0100

> parts on these registrations forms do not fly.

Yes, it's impossible to send a "real" message/utf-8 via 7bit routes,
it has to be "downgraded" and after that fed into some 8bit to 7bit
gateway.

> See RFC 2045 restrtrion of encodings for multipart and message 
> types. ( I quoted it already twice. )

What do existing 8to7 gateways with nested and unknown 8bit message
subtypes ?

We discussed that already, but I forgot the details - at that time
we were still far into the woods with a Header-Type: UTF-8 trying to
twist a message/rfc822 into something it isn't and can't be.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 03 Feb 2007 20:29:48 +0200

> twist a message/rfc822 into something it isn't and can't be.

Sendmail strips message/* (other than  message/rfc822) to 7-bit.     
I do not know about others. Except that some MTAs announce 8BITMIME 
and always sends 8-bit even when next host do not announce 8BITMIME.

Effectively gateways are required bounce unknown message/* types
if they include 8-bit.

Some probably encode these parts with quoted-printable or base64.

----------------------------------------------------------------------
From: Kari Hurtta <[email-masked]>
Subject: Sendmail message/TBD handling 
	(Re: [EAI] text for eai-framework re: 2047+S/MIME)
Newsgroups: gmane.ietf.ima
Date: 14 Oct 2006 11:10:27 +0300

Frank Ellermann <[email-masked]> writes in gmane.ietf.ima:

> [email-masked] wrote:
> 

> > IMHO it will _not_ work  as I stated earlier  -- for
> > encapsulation is needed something like application/utf8smtp
> 
> A message/TBD 8bit won't work with a 7bit-MTA at the next hop,
> but it's okay for an 8BITMIME-MTA.  For 8bit to 7bit RFC 1652
> apparently says that this is left as an exercise for gateway
> operators.

Yes, but in practise there is problems.  Sendmail will strip
message/TBD to 7-bit if next host do not announce 8BITMIME.
( I have already mentioned that. )

That hapens with sendmail 8.7 (1995/09/16) or any newer version.

(Yes.  I'm perhaps somewhat quilty about that. (*) )

/ Kari Hurtta

(*) Well, I suggested that mail should be bounced, but it is not 
    implemented. However, I may have some difficultions to found
    my over 10 year old mail archives... So on some details my
    memory may fail.

--------------------------------------------------------------------
From: Kari Hurtta <[email-masked]>
Subject: Re: [EAI] Re: Sendmail message/TBD handling
Newsgroups: gmane.ietf.ima
Date: 14 Oct 2006 18:58:04 +0300

Frank Ellermann <[email-masked]> writes in gmane.ietf.ima:

> Kari Hurtta wrote:
>  
> > Sendmail will strip message/TBD to 7-bit if next host do not
> > announce 8BITMIME.
> 
> Odd, does it try that stunt with any 8bit content ?  1652 says
> "it must cause no loss of information; MIME transport encodings
> must be employed as needed to insure this is the case".  Losing
> one of eight bits is FUBAR.
> 
> > I suggested that mail should be bounced, but it is not
> > implemented. 
> 
> We can't freeze this WG until sendmail is upgraded to support 
> RFC 1652 published more than 12 years ago.  Maybe all poor old 
> pre-1652 MTAs (are forced to) duck behind non-sendmail MXs.

I just warn :-)

> Are you really saying that sendmail announces 8BITMIME support,
> and then forwards 8bit mails to non-8BITMIME hops by stripping
> this bit ?

Only for types where quoted-printable or base64 is not allowed
and type can not handled recursively.   

That means message/* other that message/rfc822

( 8bit is also stripped from multipart preamble, I think.
  But that 8bit on multipart preamble is more or less undefined
  area. )

quoted-printable or base64 are applied to leaf mime types,
where these encodings are allowed.

> [2821 2.4] An originating SMTP client which has not successfully
> | negotiated an appropriate extension with a particular server
> | MUST NOT transmit messages with information in the high-order
> | bit of octets.
> 
> That certainly doesn't mean "just strip the 8th bit on the side
> of the client if the server doesn't offer 8BITMIME".  
> 
> Frank

/ Kari Hurtta

KNOWNBUGS file from sendmail distribution:

* 8->7 bit MIME conversion

  When sendmail is doing 8->7 bit MIME conversions, and the message
  contains certain MIME body types that cannot be converted to 7-bit,
  sendmail will strip the message to 7-bit.

doc/op/op.me from sendmail distribution:

      $=s  contains the set of subtypes of message that  can
           be  treated  recursively.  By default it contains
           only "rfc822".  Other "message/*" types cannot be
           8->7  bit encoded.  If a message containing eight
           bit data is sent to a seven bit  host,  and  that
           message  cannot  be  encoded  into seven bits, it
           will be stripped to 7 bits.

So at least documentation say so.

8BITMIME downgrade happens on sendmail/mime.c but it
is difficult to give meaningfull quote.

---------------------------------------------------------------------------

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 03 Feb 2007 20:05:54 +0100

> Sendmail strips message/* (other than  message/rfc822) to 7-bit.

Let's say that's a bug and move on, these sendmail versions won't
claim to support UTF8SMTP.

> some MTAs announce 8BITMIME and always sends 8-bit even when
> next host do not announce 8BITMIME.

Another bug, explicitly forbidden in RFC 1652.  Not our problem.

We should mention that this never worked, it also won't work for
message/utf-8, and implementors had already 13 years to figure
this out.

> Effectively gateways are required bounce unknown message/* types

Okay, so we should offer a better solution for message/utf-8 and
family for the purpose of 8BITMIME to 7bit gateways - nobody else
needs this.  While we're at it let's support any message subtype,
including invalid message/rfc822 headers with UNKNOWN-8BIT octets.

How about an application/message-munged with a mandatory parameter
indicating the original message subtype ?  Where "munged" could be
defined as a very radical procedure:  Take the complete unknown
message part (or a complete invalid message/rfc822 part), gzip it,
B64 it, ready.  Maybe gzip is overkill.

> Some probably encode these parts with quoted-printable or base64.

My first idea was QP, but it might be too easy to get this wrong or
to misinterpret its (wannabe) user friendly output, if the consumer
is a stupid script, not a human user or "unmunge" agent.

Thanks for the copies of our thread in October.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 03 Feb 2007 21:39:52 +0200

> and always sends 8-bit even when next host do not announce 8BITMIME.

Seems that actually sendmail 8.13.5 code just send 8-bit as is (instead of
downgrading).

------------------------
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=AA

--AA
Content-Type: text/plain

Tässä 8-bit body

--AA
Content-Type: message/utf-8

Subject: Tässä 8-bit header

Tässä 8-bit body

--AA--
-------------------------

results after 8BITMIME downgrade:

-------------------------
....
Date: Sat, 3 Feb 2007 21:32:24 +0200
From: Kari Hurtta <hurtta@CENSORED>
Message-ID: <200702031932.l13JWO0P005559@CENSORED>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=AA
To: undisclosed-recipients:;

--AA
Content-Type: text/plain
Content-Transfer-Encoding: base64
X-MIME-Autoconverted: from 8bit to base64 by CENSORED id l13JWa5K005560

VMOkc3PDpCA4LWJpdCBib2R5DQo=
--AA
Content-Type: message/utf-8

Subject: Tässä 8-bit header

Tässä 8-bit body

--AA--
-------------------------

It it does not strip unknown message/* to 7bit as KNOWNBUGS claim.
They are actually  passed as it.

On code there is

       if (sm_strcasecmp(type, "message") == 0)
        {
                if (!wordinclass(subtype, 's'))
                {
                        flags |= M87F_NO8BIT;
                }
                else

But M87F_NO8BIT have not actually that effect.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 03 Feb 2007 23:42:01 +0200

> including invalid message/rfc822 headers with UNKNOWN-8BIT octets.

Yes.

Note that providers of that solution are UTF8SMTP to rfc(2)822 gateways.

They must do that munging so that there is not these message/utf-8,
message/utf-8-headers and so on types.   

> B64 it, ready.  Maybe gzip is overkill.

message/utf-8   can be downgraded to message/rfc822 or converted to
multipart/utf8-encapsulated.

message/utf-8-headers is better replaced with text/utf8-header 

Then there is left   message/utf-8-delivery-status and 
message/utf-8-disposition-notification

There is some details which need to be figured out.

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 04 Feb 2007 00:18:32 +0100

> Note that providers of that solution are UTF8SMTP to rfc(2)822

We're not exactly on the same track.  I think it's conceptually
cleaner to consider UTF8SMTP to 8BITMIME as one step "downgrade",
and then 8BITMIME to 7bit as _optional_ second step.

These two steps aren't necessarily on one relay, an UTF8SMTP MTA
finding "only" 8BITMIME MTAs as next hops can downgrade UTF8SMTP
to 8BITMIME, forward it, end of story from its POV.  An 8BITMIME
MTA finding a 7bit MTA needs some "8to7" RFC 1652 magic, but it
doesn't need to know what UTF8SMTP or message/utf-8 are.

It's also possible that the UTF8SMTP MTA finds only a 7bit MTA,
and then it either has to do both steps, or bounce.  It really
can bounce, the framework RFC guarantees that, no MTA is forced
to support both UTF8SMTP to 7bis steps.

> message/utf-8   can be downgraded to message/rfc822
>                 or converted to multipart/utf8-encapsulated.

Yes.  The latter is tricky, you tried that already, and while I
didn't look at the details my vague impression was that it might
be not simple and intuitive enough.

> message/utf-8-headers is better replaced with text/utf8-header

ACK, corresponding to text/rfc822-headers in RFC 3462.  I guess
that's a consistent typo in I-D.ietf-eai-dsn-00.

> message/utf-8-disposition-notification

The message/delivery-status isn't the same kind of MIME message
like message/rfc822, message/news, or message/utf-8.  They all
have a header and a body, but for rfc822 etc. that body can be
anything, a general container, zero or more parts.

For delivery-status the "body" is apparently a single part with
a rather strict structure (maybe I missed RFC 3464 details, and
what I say is nonsense).  IFF that's the case we could use a
different MIME type, application/* or text/*, for delivery-status.

And then we'd be free to use B64 or QP as CTE for its complete
structure, including the "body".

> some details which need to be figured out.

It's not hopeless if we stay away from UTF-8 in Content-* ;-)

Frank

---

From: Chris Newman <[email-masked]>
Date: Mon, 12 Feb 2007 15:08:02 -0800

> which do not participate experiment defined on these Experimental

We're certainly free to require systems that support message/utf-8 to support 
nested encodings, as that does not place any constraints on systems that don't 
support message/utf-8 and treat it as a single opaque object.

However, you are correct the specification needs text to address this issue.

                - Chris

---

From: Kari Hurtta <[email-masked]>
Date: Tue, 13 Feb 2007 08:27:38 +0200 (EET)

>>> Chris Newman

> However, you are correct the specification needs text to address this issue.

OK.

Only issue then 8BITMIME downgraders, which do not know about this 
specification and see 8-bit message/utf-8 types. 

( If 8BITMIME downgrade is also UTF8SMTP downgrader, then we can assume that
  it follows this specification. )

If we assume that they follow RFC 2045, that means that they will bounce 
message which includes message/utf-8  (*). 

And bacause that message/utf-8 occurs on DSN this means that envelope sender 
is <>.  Therefore these DSNes is just dropped (or forwarded to postmaster of  
8BITMIME downgrader.)

However DSN which includes message/utf-8 (ot other these message/utf-8-* 
types), implicates that original bounced mail was using UTF8SMTP.  So either 
return route which mail travels is somewhat strange or envelope sender address 
of original message was pointing some other system than from where that 
original UTF8SMTP message was sent.

If danger of that is high enough, then it must be required that UTF8SMTP 
downgrader also base64 or quote-printable encodes message/utf-8 and 
message/utf-8-* types.

/ Kari Hurtta

(*)  In reality some 8BITMIME downgraders does strange things for
     8-bit message/* types.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 13 Feb 2007 21:31:33 -0000

>> Is that allowed?

Of course it is allowed. We (an experimental draft) have already updated  
RFC 2822, which is Standards Track. So it we can do it for 2822, we can  
also do it for 2045. However, in both cases, the extensions are done on  
the strict understanding that rules will be in place to prevent extended  
messages from being seen by unextended software (hence downgrading and/or  
bouncing^H^H^H^H^H^H^H^Hrejecting).
>
> We're certainly free to require systems that support message/utf-8 to  
> support nested encodings, as that does not place any constraints on  
> systems that don't support message/utf-8 and treat it as a single opaque  
> object.

But if some systems are going to treat message/utf-8 and some are going to  
treat it as application/octet stream, there is going to be chaos. Which  
suggests that it would be far better to use message/rfc822 for all cases,  
since existing software knows what structure is present there.

---

From: [email-masked]
Date: Tue, 20 Feb 2007 20:43:55 +0900 (JST)

draft-ietf-eai-dsn-00.txt section 3.  UTF-8 Address Type is defined:

> utf-8-type-addr     = "utf-8;" utf-8-address
> utf-8-address       = "<" Mailbox [ *WSP "<" Mailbox ">" ] ">"

But mailbox is defined in RFC 2822 as:

mailbox         =       name-addr / addr-spec
name-addr       =       [display-name] angle-addr
angle-addr      =       [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr
addr-spec       =       local-part "@" domain

As a result, this utf-8-address may be parsed as
  "<display name<UTF8@UTF8> <display name <ASCII@ASCII>>".

The 'utf-8-address' should be 'angle-addr' defined in [utf8smtp].

Or
	utf-8-type-addr     = "utf-8;" angle-addr
                   ; angle-addr is defined in [utf8smtp]

This specification cannot hold '[route]' parameter defined in RFC 3461
section 9.1 rfc822 address-type.

--
Kazunori Fujiwara, JPRS

---

From: John C Klensin <[email-masked]>
Date: Tue, 20 Feb 2007 13:14:27 -0500

--On Tuesday, 20 February, 2007 20:43 +0900 [email-masked]
wrote:

> angle-addr      =       [CFWS] "<" addr-spec ">" [CFWS] /
> obs-angle-addr addr-spec       =       local-part "@" domain
> 

Arggh.  Yes.  Good catch.
    john

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 21 Feb 2007 16:36:40 +0100

> Arggh.  Yes.  Good catch.

Sure ?  I thought it was another (*1) case of confusing 2822 <mailbox>
with 2821 <Mailbox>  The eai-dsn draft says:

   utf-8-address       = "<" Mailbox [ *WSP "<" Mailbox ">" ] ">"
          ; The first  occurrence of 'Mailbox' is defined in [utf8smtp]
          ; The second occurrence of 'Mailbox' is defined in RFC 2821
.............................................................^^^^^^^^

RFC 2821 has no display-name etc., it's  (in 4.1.2):

   Mailbox = Local-part "@" Domain

There's no [utf8smtp] in the references.  I guess it's supposed to be
[I-D.ietf-eai-smtpext] chapter 2.3 <uMailbox>:

   uMailbox = uLocal-part "@" uDomain
            ; Replace Mailbox in RFC 2821, section 4.1.2

Frank

*1:  One other person confusing <mailbox> and <Mailbox> was me... :-)

---

From: [email-masked]
Date: Thu, 22 Feb 2007 20:48:22 +0900 (JST)

sorry my confusion about <Mailbox>.

I have two questions about ORCPT UTF8 address type.

1. UTF-8 address type is used in UTF8SMTP environment.
   When and where UTF-8 encoded address type used ?

2. Is ORCPT UTF-8 address downgrading required 
   when SMTP downgrading occures.

--
Kazunori Fujiwara, JPRS

> From: Frank Ellermann <[email-masked]>
> John C Klensin wrote:
>  
> > Arggh.  Yes.  Good catch.
>

---

From: Chris Newman <[email-masked]>
Date: Thu, 22 Feb 2007 14:19:06 -0800

>    When and where UTF-8 encoded address type used ?

The quick overview would be:
* The utf-8 address type is completely un-encoded.
* The utf-8-enc address type has two types of encoding: mandatory (SP, +, =),
  and optional (Unicode characters).

The utf-8-enc address type is always used in the ORCPT parameter, but the 
Unicode encoding is only required when an SMTP server advertises DSN without 
UTF8SMTP.

The utf-8-enc address type is used with Unicode encoding in a 7-bit 
message/delivery-status.

The utf-8 address type is recommend for utf-8-delivery-status.

>    when SMTP downgrading occures.

Yes, the Unicode-encoded version of utf-8-enc address type must be used in that 
case.

                - Chris

---

From: [email-masked]
Date: Fri, 23 Feb 2007 15:22:11 +0900 (JST)

> From: Chris Newman <[email-masked]>

utf-8 address type is not used in SMTP ORCPT parameter ?
(and always utf-8-enc address type is used?)

> Yes, the Unicode-encoded version of utf-8-enc address type must be used in that 

before downgrading, (it's equal to first question.)
 which is right? both?
  ORCPT=utf8;<UTF8@UTF8<ascii@ascii>>
  ORCPT=utf-8-enc;<encodedUTF8@encodedUTF8<ascii@ascii>>

after downgrading, which is right?
  ORCPT=rfc822;ascii@ascii
  ORCPT=utf-8-enc;<encodedUTF8@encodedUTF8<ascii@ascii>>

I'm updating draft-ietf-eai-downgrading.
My current text is written as
'ORCPT=utf8;<UTF8@UTF8<ascii@ascii>>' to be downgraded as
'ORCPT=rfc822;ascii@ascii'.

--
Kazunori Fujiwara, JPRS

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 23 Feb 2007 09:29:38 +0100

>   ORCPT=utf-8-enc;<encodedUTF8@encodedUTF8<ascii@ascii>>
As far as I understood Chris, if the address is <[email-masked]>:

Before downgrading (UTF8SMTP environment)

ORCPT=utf-8-enc;<[email-masked]>

After downgrading (DSN supported, but UTF8SMTP not supported):

ORCPT=utf-8-enc:<h%u0073vard@d%u00F5rum.com>

You never, ever, ever throw away information in an ORCPT as long as you 
can carry it.

Clearer?

(the codepoints are wrong. Sorry 'bout that.)

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 23 Feb 2007 14:11:51 +0100

Harald Alvestrand wrote:

 [raw UTF-8 not fixed, what you see is what I get]
> if the address is <[email-masked]>:

> ORCPT=utf-8-enc;<[email-masked]>

So here it's a "normal" percent-encoded "+", %2B

> ORCPT=utf-8-enc:<h%u0073vard@d%u00F5rum.com>

But here it's some obscure %u1234 or %U12345678
style.  IMO that's ugly.  We're not in a position
where we have to escape Unicode as in John's
I-D.klensin-unicode-escapes-02.

The input is already valid (one hopes) raw UTF-8,
why not simply use RFC 3987 style for it ?  I.e.
"percent-encoded" everywhere, not only for some
forbidden ASCII octets (CTL, SP, %, +, and =).

With 3987 style we'd get a decent chance that the
result will be compatible (to some degree) with
IRIs in a future 2368ter.

The %u plus %U invention is very near to \u and \U
in chapter 2.1 of John's draft, but not exactly
the same, see http://www.w3.org/TR/charmod/#C042

Frank

---

From: Chris Newman <[email-masked]>
Date: Fri, 23 Feb 2007 21:14:21 -0800

> forbidden ASCII octets (CTL, SP, %, +, and =).

I slightly favor HEX-UTF-8 encoding for this particular application, but I felt 
using the unicode-escapes approach in the first draft was more likely to result 
in appropriate consideration of the issue.  This is an interesting design 
tradeoff with no clear right or wrong answer.

> IRIs in a future 2368ter.

A relevant and substantive benefit.

> the same, see http://www.w3.org/TR/charmod/#C042

The current proposal arguably violates <http://www.w3.org/TR/charmod/#C042> but 
HEX-UTF-8 arguably violates <http://www.w3.org/TR/charmod/#C045> and is a 
double encoding.

So we have:

HEX-UTF-8 Benefits:
* Simpler for most software using this format
* Potential benefits from IRI mailto compatibility
* Familiar encoding
Unicode-escape benefits:
* Prevents a double-encoding
* Simpler for humans to lookup and interpret in transcripts
* Encoded form is often shorter than HEX-UTF-8

While I don't feel strongly about the outcome, I do feel strongly that this 
topic is worthy of debate.

                - Chris

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 24 Feb 2007 15:16:21 +0100

Chris Newman wrote:

> HEX-UTF-8 arguably violates <http://www.w3.org/TR/charmod/#C045>

ACK.  Maybe we can say that we should use one style consistently,
either 3987 or %u + %U,  No exceptions for CTL, SP, %, +, and =.

> * Encoded form is often shorter than HEX-UTF-8

%23%56%89 vs. %u3456 for u+0800 up to u+FFFF or
%23%56%89%BC vs. %U3456789A for u+10000 up to u+10FFFF
(of course excluding non-characters and surrogates).

> I don't feel strongly about the outcom

Nor me, but using three styles, %23, %u3456, and %U3456789A
is against my KISS instincts.

The original <xtext> used only "+" for its own purposes,
i.e. introducing 2HEXDIG to escape CTL, SP, +, or =.

You've replaced "+" by "%" for <QLOWCHAR>.  I like that
better, but why did you keep "+" in the list of forbidden
characters requiring %2B ?  Is it necessary to escape "+"
in the context of utf-8-enc ?

Frank

---

From: John C Klensin <[email-masked]>
Date: Sat, 24 Feb 2007 14:07:03 -0500

Chris,

A few observations...

--On Friday, February 23, 2007 9:14 PM -0800 Chris Newman 
<[email-masked]> wrote:

> I slightly favor HEX-UTF-8 encoding for this particular
> application, but I felt using the unicode-escapes approach in
> the first draft was more likely to result in appropriate
> consideration of the issue.  This is an interesting design

Note that the strong recommendation for \u \U (or %u %U) has 
been taken out of unicode-escapes after a good deal of fairly 
aggressive input and no defenders.  If there is a recommendation 
left in that document, it is for delimited strings rather than 
length cues built into the introducer-syntax (especially not 
cues based on case-sensitive interpretations).

I agree that there is a tradeoff with no clear answer.  I do 
believe that there is a very strong case to be made for a single 
encoding in a given string, not a mixture.  If that forces us to 
Hex-UTF-8, that would sadden me a bit, but it seems more 
attractive than mixtures.

> A relevant and substantive benefit.

This opinion may not make me very popular, but I think it needs 
to be expressed.  With all due respect to Martin, Michel, and 
the review process, some of our work, the thinking that led to 
the Unicode-Escapes work, and the RFC 4690 and IDNAbis efforts 
have left me wondering (at least) whether it is possible that we 
completely botched IRIs.  As an intermediate translation format, 
it may be fine.  But, for general-purpose, end-user use, it 
appears insufficiently ambitious.  It can generate some very 
strange-looking strings when right-to-left elements are 
included, it doesn't provide a clean way to map between 
characters that seem natural in the user's localized environment 
and protocol (scheme) names, and so on.  I know that 3987-style 
IRIs are deployed in a few important products, but it seems to 
me that there is still an opportunity to review some of the 
decisions and, if appropriate, define a different and more 
user-friendly abstraction that rests above URIs and perhaps 
about IRIs.

To that end, I think we should figure out what we really need 
for UTF8SMTP and the associated protocols, create an appropriate 
abstraction and syntax for a mail-address reference, and then 
figure what that means for mailto (2368ter or otherwise) and 
IRIs, not the other way around.    As one handy example, there 
are several reasons, including both R to L scripts and countries 
in which a "mail host, then mailbox" syntax seems more natural 
than "mailbox at host", why an internationalized email address 
format, expressed in abstract reference form, might benefit from 
avoiding use of the "@" entirely.  I don't have good ideas that 
don't take us back a few steps toward the dreaded X.400 syntax, 
but I wonder whether, at an object-naming abstraction level, we 
wouldn't be better off with
   mailbox="Chris.Newman" org="Sun.COM" altmailbox=...
or its XML-ish equivalent
   <mailaddr mailbox="Chris.Newman" org="Sun.COM" ... />

with the name-value pairs being order-independent and maybe a 
bunch of synonyms or mappings for the keywords, rather than 
seeing whether we can preserve "mailto:" and what IRIs turned 
out to be given the constraints that were assumed.   And, in 
_that_ format, I go back to contending that we are better off 
with code point offsets than Hex-encoded UTF-8.

> The current proposal arguably violates
> <http://www.w3.org/TR/charmod/#C042> but HEX-UTF-8 arguably
> violates <http://www.w3.org/TR/charmod/#C045> and is a double
> encoding.
>
> So we have:
>
> HEX-UTF-8 Benefits:

About which I'm dubious... see above.

> * Familiar encoding

Depends on who one thinks it is being familiar with.  I contend 
that it is pathological for the casual user, which is exactly 
the concern that started my on what became the uncode-escapes 
exercise.  Certainly if is more geek- and URI-friendly, but that 
may be a symptom of the problem, not a benefit.

> * Encoded form is often shorter than HEX-UTF-8

And, of course, often much shorter with East Asian languages 
whose users are assumed to be a major beneficiary of i18n email.

> While I don't feel strongly about the outcome, I do feel
> strongly that this topic is worthy of debate.

Yes.

    john

---

From: "Charles Lindsey" <[email-masked]>
Date: Sun, 25 Feb 2007 14:59:03 -0000

>> forbidden ASCII octets (CTL, SP, %, +, and =).

Or, for that matter, why not just use <xtext>? That would be no more of a  
mess that RFC 3461 already made of it. There is nothing in RFC 3461 thst  
prevents the use of <xtext> for octets >=128, except that they didn't  
anticipate it. Existing software (and humans) that understand the  
addr-type rfc822 would then immediately understand addr-type utf-8-enc.  
<xtext> ain't (that) broken. Why fix it? There are already too many ways  
of encoding 8bits into 7. Why invent yet another one?
>
> I slightly favor HEX-UTF-8 encoding for this particular application,
+1

> <http://www.w3.org/TR/charmod/#C042> but HEX-UTF-8 arguably violates  
> <http://www.w3.org/TR/charmod/#C045> and is a double encoding.

I think your average user will be vaguely aware that he is using utf-8,  
but will never have heard of Unicode. Moreover, our whole EAI endeavour is  
based on the use of UTF-8 throughout (some software may internally  
translate it into Unicode - probably in the form of UCS-16 - but that  
will, and should remain, entirely invisible from the outside). Hence he  
will not regard it as a "double encoding", and hence IMO it is better to  
violate #CO45 than to violate #CO42 (and #CO42 would support xtext in  
these contexts also).
> * Potential benefits from IRI mailto compatibility

and even more familiar, in this context, as xtext.

> * Simpler for humans to lookup and interpret in transcripts

Not true for a human who thinks in terms of Utf-8.

> * Encoded form is often shorter than HEX-UTF-8

For something to be used only in ORCPT/etc parameters and in DSNs, that is  
a minor consideration, especially as raw UTF-8 is also to be permitted in  
UTF8SMTP environments which is where this stuff will mostly be seen.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 25 Feb 2007 16:59:32 +0100

Charles Lindsey wrote:

> why not just use <xtext>?

Good question, it's not nice, but as you say it's also not broken.

What you get is in essence the same as "3987 style" swapping "+"
and "%" everywhere.  John argued it's too long for the relevant
range u+0800 up to u+FFFF.

Frank

---

From: Martin Duerst <[email-masked]>
Date: Sun, 25 Feb 2007 18:46:29 +0900

At 04:07 07/02/25, John C Klensin wrote:
>Chris,
>
>A few observations...
>
>--On Friday, February 23, 2007 9:14 PM -0800 Chris Newman <[email-masked]> wrote:
>
>Note that the strong recommendation for \u \U (or %u %U) has been taken out of unicode-escapes after a good deal of fairly aggressive input and no defenders.  If there is a recommendation left in that document, it is for delimited strings rather than length cues built into the introducer-syntax (especially not cues based on case-sensitive interpretations).
>
>I agree that there is a tradeoff with no clear answer.  I do believe that there is a very strong case to be made for a single encoding in a given string, not a mixture.  If that forces us to Hex-UTF-8, that would sadden me a bit, but it seems more attractive than mixtures.
>
>>> With 3987 style

%hh escaping didn't in any way originate in RFC 3987.
The only thing that RFC 3987 did with respect to escaping
was to make it possible to go from characters to %hh,
rather than only from arbitrary octets to %hh.

>>> we'd get a decent chance that the

As far as I understand, this escaping discussion only
concerns DSN, right? I'm not an expert, but I think that
while such 'to some degree compatibility' might not hurt
too much, it's also not terribly important.

>This opinion may not make me very popular, but I think it needs to be expressed.  With all due respect to Martin, Michel, and the review process, some of our work, the thinking that led to the Unicode-Escapes work, and the RFC 4690 and IDNAbis efforts have left me wondering (at least) whether it is possible that we completely botched IRIs.  As an intermediate translation format, it may be fine.  But, for general-purpose, end-user use, it appears insufficiently ambitious.

There was never a claim that IRIs would address user's needs around the
world better than URIs do this job for English-speaking, basic-Latin-letter-
using people.

Google and other search services are way better for "give me
the Web page I want that I don't know about, but I know must
be out there". IRIs just close an important gap to make sure
references to character strings not in US-ASCII work reliably
in IRIs and via URIs.

>It can generate some very strange-looking strings when right-to-left elements are included,

Yes. That has virtually nothing to do with URIs or IRIs, and
quite a lot with the fundamental problems of bidi. Strange-looking
things can easily be produced for your syntax proposals below, too.

>it doesn't provide a clean way to map between characters that seem natural in the user's localized environment and protocol (scheme) names,

Yes, we didn't provide a way to write http:, ftp:, mailto:, and so on
in different scripts. For some work in this direction, please check
out http://www.ietf.org/rfc/rfc2324.txt :-). Given that to most people,
strings such as http and ftp are opaque, providing lists of correspondences
for each script wouldn't have been too difficult (choices can essentially
be almost arbitrary, and there are only around 100 scripts, and probably
way less really relevant ones). But still, there are scalability
and deployment problems.

>and so on.  I know that 3987-style IRIs are deployed in a few important products, but it seems to me that there is still an opportunity to review some of the decisions and, if appropriate, define a different and more user-friendly abstraction that rests above URIs and perhaps about IRIs.

If we had any reasonably good idea of how such an abstration would look
and work (as opposed to what services and benefits it should provide),
that might be a good way to spend our time.

>To that end, I think we should figure out what we really need for UTF8SMTP and the associated protocols, create an appropriate abstraction and syntax for a mail-address reference, and then figure what that means for mailto (2368ter or otherwise) and IRIs, not the other way around.    As one handy example, there are several reasons, including both R to L scripts and countries in which a "mail host, then mailbox" syntax seems more natural than "mailbox at host",

There are definitely a lot of countries where postal addresses are
written outside-in, rather than inside-out as in the US and Western
Europe. But once we go there, we would have to turn around domain
name hierarchies, too, e.g. COM.Sun in the examples below
(which is probably why X.400 had things chopped apart much more
finely).

>why an internationalized email address format, expressed in abstract reference form, might benefit from avoiding use of the "@" entirely.  I don't have good ideas that don't take us back a few steps toward the dreaded X.400 syntax, but I wonder whether, at an object-naming abstraction level, we wouldn't be better off with
>with the name-value pairs being order-independent and maybe a bunch of synonyms or mappings for the keywords, rather than seeing whether we can preserve "mailto:" and what IRIs turned out to be given the constraints that were assumed.   And, in _that_ format, I go back to contending that we are better off with code point offsets than Hex-encoded UTF-8.

Of course, except that once you're using XML, real UTF-8 comes
for free.

>About which I'm dubious... see above.

Same for me, but with somewhat different reasons.

>> * Familiar encoding
>
>Depends on who one thinks it is being familiar with.  I contend that it is pathological for the casual user, which is exactly the concern that started my on what became the uncode-escapes exercise.  Certainly if is more geek- and URI-friendly, but that may be a symptom of the problem, not a benefit.
>
>> Unicode-escape benefits:
>And, of course, often much shorter with East Asian languages whose users are assumed to be a major beneficiary of i18n email.

It's actually not East Asian languages that benefit most.
Because they use a large character repertoire and correspondingly
short actual data (in number of characters), length restrictions
are much less of an issue for them. Those who get hit hardest
are languages that use alphabetic scrits, or scripts close in
size to alphabetic ones, but don't get the benefit of short
UTF-8 sequences.

>> strongly that this topic is worthy of debate.

Well, I guess I have written too much already to contradict here :-(.

Regards,     Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp      mailto:[email-masked]

---

From: Chris Newman <[email-masked]>
Date: Mon, 26 Feb 2007 15:27:29 -0800

Charles Lindsey wrote on 2/25/07 14:59 +0000:
> anticipate it. Existing software (and humans) that understand the  addr-type
> rfc822 would then immediately understand addr-type utf-8-enc.  <xtext> ain't
> (that) broken. Why fix it? There are already too many ways  of encoding 8bits
> into 7. Why invent yet another one?

Good question.  I would have preferred to use xtext, but this text from RFC 
3461 makes that not possible without too much risk of breakage:

   Due to limitations in the Delivery Status Notification format, the
   value of the original recipient address prior to encoding as "xtext"
   MUST consist entirely of printable (graphic and white space)
   characters from the US-ASCII [4] repertoire.  If an addr-type is
   defined for addresses which use characters outside of this
   repertoire, the specification for that addr-type MUST define the
   means of encoding those addresses in printable US-ASCII characters
   when are then encoded as xtext.

The problem is that xtext is a transfer encoding which is removed when a 
traditional message/delivery-status part is generated and there's a hard 
requirement in today's deployed MTAs that the result of xtext removal be 7-bit 
ASCII.  So the choice is to produce a non-xtext encoding for all the 8-bit 
characters and have two encodings present, or to design an encoding that 
obviates the need for xtext.  The latter seemed cleaner to me, although an 
approach that used xtext for encoding 7-bit and something else for non-ASCII is 
an approach the working group might want to consider.

So xtext was badly botched.

                - Chris

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 27 Feb 2007 14:25:20 -0000

On Mon, 26 Feb 2007 23:27:29 -0000, Chris Newman <[email-masked]>  
wrote:

> Good question.  I would have preferred to use xtext, but this text from  
> RFC 3461 makes that not possible without too much risk of breakage:
>
> requirement in today's deployed MTAs that the result of xtext removal be  
> 7-bit ASCII.

Ah! I had not appreciated that the xtext had to be unscrambled when it was  
put into a message/delivery-status. So they force us into a double  
encoding.

All right, suppose we use %HEX for the utf-8, and we have a local-part
     <esc>Aa+å
then it becomes
     %1BAa%2B%C3%B8
which then has to be encoded into <xtext>, which turns out to be a Null  
transformation. I suppose that is as good as we are going to get.

The problem arises, of course, because delivery status was made (quite  
wrongly) to be a message type, and you are not allowed to C-T-encode  
message types. Really, one or other of those decisions needs to be  
reversed, but that is not so easy.

> So xtext was badly botched.

Indeed so.