Subject: [EAI] I-D ACTION:draft-ietf-eai-imap-utf8-01.txt

From: [email-masked]
Date: Wed, 07 Mar 2007 10:50:02 -0500

A New Internet-Draft is available from the on-line Internet-Drafts 
directories.
This draft is a work item of the Email Address Internationalization Working Group of the IETF.

	Title		: IMAP Support for UTF-8
	Author(s)	: P. Resnick, C. Newman
	Filename	: draft-ietf-eai-imap-utf8-01.txt
	Pages		: 15
	Date		: 2007-3-7
	
This specification extends the Internet Message Access Protocol
   version 4rev1 (IMAP4rev1) to support unencoded international
   characters in user names, mail addresses and message headers.  This
   is an early draft and intended as a framework for discussion.  Please
   do not deploy implementations of this draft.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-eai-imap-utf8-01.txt

To remove yourself from the I-D Announcement list, send a message to 
[email-masked] with the word unsubscribe in the body of 
the message. 
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce 
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the 
username "anonymous" and a password of your e-mail address. After 
logging in, type "cd internet-drafts" and then 
"get draft-ietf-eai-imap-utf8-01.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-eai-imap-utf8-01.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.
_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Kari Hurtta <[email-masked]>
Date: 07 Mar 2007 19:41:00 +0200

> http://www.ietf.org/internet-drafts/draft-ietf-eai-imap-utf8-01.txt

Just note. I'm still concernes about these things what I noted on thread,
which I started with message:

      From: Kari Hurtta <[email-masked]>
      Subject: [EAI] Messages on original form (Re: I-D
	ACTION:draft-ietf-eai-imap-utf8-00.txt)
      Newsgroups: gmane.ietf.ima
      To: [email-masked]
      Date: 06 Feb 2007 21:29:49 +0200

In general that is concern that UTF8SMTP capable mailstore should
store messages on that form what it is received them. And
that form should be accessible.

I see that someones disagree with me.

That is not just draft-ietf-eai-imap-utf8, but also 
draft-ietf-eai-smtpext and  draft-ietf-eai-pop.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 20 Mar 2007 20:14:01 +0200

> that form should be accessible.

Specially question is:

        How to prevent up-conversion of non-UTF8SMTP messages 
        without to triggering downgrading of UTF8SMTP messages ?

        ( Up-conversion of downgraded UTF8SMTP messages is OK. )

draft-ietf-eai-imap-utf8-01.txt: --------------------------------------
8.  Up-Conversion Server Requirements

   When an IMAP4 server uses a traditional mailbox format that includes
   7-bit headers and it chooses to permit access to that mailbox with
   the UTF8 parameter, it MUST support minimal up-conversion as
   described in this section.  Minimal up-conversion is described in
   this section.

   The server MUST support up-conversion of the following address
   header-fields in the message header: From, Sender, To, CC, Bcc,
   Resent-From, Resent-Sender, Resent-To, Resent-CC, Resent-Bcc, and
   Reply-To.  This up-conversion MUST include address local-parts
   encoded according to [TBD], address domains encoded according to IDNA
   [RFC3490], and MIME header encoding [RFC2047] of display-names and
   any RFC 2822 comments.

   The following charsets MUST be supported for up-conversion of MIME
   header encoding [RFC2047]: UTF-8, US-ASCII, ISO-8859-1, ISO-8859-2,
   ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7,
   ISO-8859-8, ISO-8859-9, ISO-8859-10, ISO-8859-14, and ISO-8859-15.
   Other widely deployed MIME charsets SHOULD be supported.

   Up-conversion of MIME header encoding of the following headers MUST
   also be implemented: Subject, Date (RFC 2822 comments only),
   Comments, Keywords, Content-Description.

   Server implementations also SHOULD up-convert all MIME body headers,
   SHOULD up-convert or remove the deprecated (and misused) name
------------------------------------------------------------------------

/ Kari Hurtta

( This is somewhat sidestepped on

  http://www.elmme-mailer.org/ID/draft-hurtta-eai-messagestore-00.txt
  http://www.elmme-mailer.org/ID/draft-hurtta-eai-messagestore-00.xml

  These are posted to RFC Editor some days ago, but seems that there is
  some backlog.

  There is also 

  http://www.elmme-mailer.org/ID/draft-hurtta-eai-encapsulation-01.txt
  http://www.elmme-mailer.org/ID/draft-hurtta-eai-encapsulation-01.xml
)

---

From: Chris Newman <[email-masked]>
Date: Wed, 21 Mar 2007 07:49:22 +0000

There are three formatting options:
1. A format compatible with requirements of RFC 3501.
2. A format with minimal RFC 2047/2231 turds.
3. The format that is in the mailstore (initially this will be identical to 1,
   but as EAI-aware mailstores are deployed it should become indistinguishable
   from 2).

The protocol must offer 1 to preserve interoperability.  The current proposal 
offers options 1 and 2.

And argument could be made the protocol should offer option 3 instead of option 
2.  I would be interested in that debate.

If you are are suggesting the protocol offer both options 2 and 3 in addition 
to 1, then I would question if the benefit merits the additional complexity.

                - Chris

---

From: Kari Hurtta <[email-masked]>
Date: Wed, 21 Mar 2007 11:40:28 +0200 (EET)

>>> Chris Newman

My point is that there must be possibility to get

     The format that is in the mailstore

( I accept upgrade only for messages, which are downgraded
  UTF8SMTP messages. Do not mess with other messages. )

To me it does not matter, is there optionally also possible get

      A format with minimal RFC 2047/2231 turds.

whatever that means.

/ Kari Hurtta

---

From: John C Klensin <[email-masked]>
Date: Thu, 22 Mar 2007 05:44:39 -0400

--On Wednesday, March 21, 2007 07:49 +0000 Chris Newman 
<[email-masked]> wrote:

> The protocol must offer 1 to preserve interoperability.  The
> current proposal offers options 1 and 2.
>
> And argument could be made the protocol should offer option 3
> instead of option 2.  I would be interested in that debate.

To start the debate, I will pose my usual question: if we were 
developing this with no constraints from the legacy (ASCII, in 
this case) environment, what would we do?  I think, in this 
case, that points to option 3.  And that, in turn, argues for 
precisely the 1/3 case rather than the 1/2 case, with 1 
supported primarily for legacy compatibility purposes.

My only concern about that reasoning is that there are some 
advantages of maintaining an abstraction between "what is in the 
mailstore" and "the image/view of the mailstore content that is 
seen by a client".  I don't think the above prevents that, but 
we need to be careful about reading.  Access to the mailstore 
version on a "by reference" basis is pretty close to a 
showstopper for me: we should not be specifying those formats at 
all, only what must be able to be produced from them.

> If you are are suggesting the protocol offer both options 2
> and 3 in addition to 1, then I would question if the benefit
> merits the additional complexity.

Concur.

     john

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 22 Mar 2007 14:39:34 -0000

On Thu, 22 Mar 2007 09:44:39 -0000, John C Klensin <[email-masked]> wrote:
>
> My only concern about that reasoning is that there are some advantages  
> of maintaining an abstraction between "what is in the mailstore" and  
> "the image/view of the mailstore content that is seen by a client".  I  
> don't think the above prevents that, but we need to be careful about  
> reading.  Access to the mailstore version on a "by reference" basis is  
> pretty close to a showstopper for me: we should not be specifying those  
> formats at all, only what must be able to be produced from them.
>
I think one of the primary requirements is that the user should be able to  
demand, and be given, a copy of the message as actually received by the  
MDA (plus maybe a few tracing headers added by the storage agent).

Which strongly suggests (but does not mandate) that that be what is kept  
in the mailstore.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 22 Mar 2007 15:52:35 +0100

--On 22. mars 2007 14:39 +0000 Charles Lindsey <[email-masked]> wrote:

> I think one of the primary requirements is that the user should be able
> to demand, and be given, a copy of the message as actually received by
> the MDA (plus maybe a few tracing headers added by the storage agent).

Why?

(I can see a couple of possible reasons why you would want this. But I 
can't tell from your messages which ones you are thinking of.)

                Harald

---

From: Kari Hurtta <[email-masked]>
Date: 22 Mar 2007 18:46:08 +0200

> can't tell from your messages which ones you are thinking of.)

Well, I have needed these kind functionality at least

         - For debugging   
              (as postmaster I usually require to see original message
               -- some time I answer -- sorry, that was not original
               message as received -- before I can help, I want original
               message -- not forwarded text or reformatted message. 

               This is usually related to junk filtering)

         - For learn junk filters

Also I can see that MUAs needs to see messages on original form
when checking signatures of messages. Most of signature algotihms
sign messages on form where no further encoding is not needed.
That means that all signed data is encoded to 7-bit.  

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 22 Mar 2007 18:18:24 +0100

Chris Newman wrote:
 
> 3. The format that is in the mailstore (initially this
>    will be identical to 1, but as EAI-aware mailstores
>    are deployed it should become indistinguishable
>    from 2).
 
> The protocol must offer 1 to preserve interoperability.
> The current proposal offers options 1 and 2.
 
> And argument could be made the protocol should offer
> option 3 instead of option 2.  I would be interested in
> that debate.
 
> If you are are suggesting the protocol offer both options
> 2 and 3 in addition to 1, then I would question if the
> benefit merits the additional complexity.

I'm lost with anything related to IMAP.  Which options are
needed for RFC 4468 compatibility ?  Maybe the draft should
discuss this "somehow" (and that could be difficult because
IMAP URLs are a moving target at the moment (?)).

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 22 Mar 2007 20:14:26 +0200

>       A format with minimal RFC 2047/2231 turds.

Note that I'm talking other FETCH items than BODYSTRUCTURE 
or ENVELOPE. 

Specially that means RFC822 and BODY[] items.
(specially items like BODY[], BODY[HEADER], BODY[TEXT], RFC822,
 RFC822.HEADER, RFC822.TEXT )

( not BODY which is like BODYSTRUCTURE )

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 22 Mar 2007 20:35:12 +0200

John C Klensin <[email-masked]> writes in gmane.ietf.ima:

> developing this with no constraints from the legacy (ASCII, in this
> case) environment, what would we do?  I think, in this case, that
> points to option 3.  And that, in turn, argues for precisely the 1/3
> case rather than the 1/2 case, with 1 supported primarily for legacy
> compatibility purposes.

Yes. I agree. 

If we suppose that legacy formats vanish, we should not
require conversion support for them.   

> pretty close to a showstopper for me: we should not be specifying
> those formats at all, only what must be able to be produced from them.

Replace "format that is in the mailstore" with "format on what local
system received messages".

/ Kari Hurtta

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 23 Mar 2007 11:09:28 +0100

So, to paraphrase, you need to make sure you can:

- verify digital signatures on messages (exactly what's needed here may 
vary per signing technology)
- extract pieces of the message in a format that is the same as that seen 
by the MTA path, because that's useful for configuring stuff (spam filters, 
configuration)

Fair summary?

---

From: Kari Hurtta <[email-masked]>
Date: 23 Mar 2007 15:04:56 +0200

Harald Tveit Alvestrand <[email-masked]> writes in gmane.ietf.ima:

> - verify digital signatures on messages (exactly what's needed here
> may vary per signing technology)
> - extract pieces of the message in a format that is the same as that
> seen by the MTA path, because that's useful for configuring stuff
> (spam filters, configuration)
> 
> Fair summary?

Yes.

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 26 Mar 2007 16:50:30 +0100

On Sat, 24 Mar 2007 11:06:37 -0000, Harald Alvestrand  
<[email-masked]> wrote:

Sorry, my original message did not go to the whole list.

> Charles Lindsey wrote:
>>
>> Generally speaking I just regard it as a fundamental requirement of any  
>> internet protocol to be able to examine what was actually on the wire,  
>> before other agents started to "improve" it.
> If taken to mean what the words mean in their dictionary definitions,  
> what you are asking for is packet trace.
> The most reasonable tools for that are called "tcpdump" and "ethereal"  
> (provided that your "wire" is an Ethernet, of course).

Yes, I am not concerned about the inner structure of TCP packets. But what  
I DO want to see is that anyone who receives an email in his MUA (whether  
via an IMAP storage server of otherwise) can have access to the original  
RFC 2822 object as it arrived on the wire at the final MTA. The ability to  
provide that service constrains somewhat what IMAP implementations can do  
internally.

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Mar 2007 21:58:12 -0400

(combined answer)

--On Monday, 26 March, 2007 16:50 +0100 Charles Lindsey
<[email-masked]> wrote:

>> Charles Lindsey wrote:
>>> 
>>> Generally speaking I just regard it as a fundamental
>>> requirement of any   internet protocol to be able to examine
>>> what was actually on the wire,   before other agents started
>>> to "improve" it.
>> If taken to mean what the words mean in their dictionary
>> definitions,   what you are asking for is packet trace.
>> The most reasonable tools for that are called "tcpdump" and
>> "ethereal"   (provided that your "wire" is an Ethernet, of
>> course).
> 
> Yes, I am not concerned about the inner structure of TCP
> packets. But what I DO want to see is that anyone who receives
> an email in his MUA (whether via an IMAP storage server of
> otherwise) can have access to the original RFC 2822 object as
> it arrived on the wire at the final MTA. The ability to
> provide that service constrains somewhat what IMAP
> implementations can do internally.

I do not believe this is a requirement for IMAP implementations
today (for all-ASCII messages).   Retrieving the various pieces
as pieces and reassembling them doesn't count, since that may or
may not exactly restore the original, received-by-MTA, version.
If it is not, I suggest that it is out of scope for this WG
until and unless IMAPext gets around to making the change for
the base specification.

--On Thursday, 22 March, 2007 18:46 +0200 Kari Hurtta
<[email-masked]> wrote:

>> (I can see a couple of possible reasons why you would want
>> this. But I can't tell from your messages which ones you are
>> thinking of.)
>> 
>>                 Harald
> 
>               (as postmaster I usually require to see original
> message                -- some time I answer -- sorry, that
> was not original                message as received -- before
> I can help, I want original                message -- not
> forwarded text or reformatted message. 
> 

For debugging, you need a direct interface to the message store.
You may even need to specify what your delivery MTA puts in the
message store, or that it do a write-aside on messages as well
as logging them and you need to access that.  But you don't need
to impose extra requirements on either EAI formats or IMAP.
Such requirements might make your life easier, but they come at
the price of more complexity for everyone.
 
>          - For learn junk filters

Maybe, but, as you suggest above, this may be more similar to
debugging and may need special interfaces for other reasons.

> when checking signatures of messages. Most of signature
> algotihms sign messages on form where no further encoding is
> not needed. That means that all signed data is encoded to
> 7-bit.

That is a reason to not up-convert before getting to the final
MUA (or user portion thereof).  If I put a signature on a
message as an integrity check, I really don't want an
intermediate system to turn the message inside out, or even
convert it to something else, and then have another intermediate
system try to restore it sufficiently to make the signature
work.  That is really scary to me.  

We shall see what happens in practice, but I think we will see
pressure from user/ consumers of mail systems to be able to
configure "no downgrade with signed messages", "no downgrade
with DKIM headers", "strip signatures and/or dkim headers on
downgrade" and probably some similar options.

    john

---

From: Kari Hurtta <[email-masked]>
Date: 27 Mar 2007 06:49:42 +0300

> until and unless IMAPext gets around to making the change for

IMAP is cabable to return full rfc822 message.
There is several queries which assumes existance of rfc822 message.

RFC 3501:

      BODY[<section>]<<partial>>
         The text of a particular body section.  The section
         specification is a set of zero or more part specifiers
         delimited by periods.  A part specifier is either a part number
         or one of the following: HEADER, HEADER.FIELDS,
         HEADER.FIELDS.NOT, MIME, and TEXT.  An empty section
         specification refers to the entire message, including the
         header.
<...>

     RFC822
         Functionally equivalent to BODY[], differing in the syntax of
         the resulting untagged FETCH data (RFC822 is returned).

      RFC822.HEADER
         Functionally equivalent to BODY.PEEK[HEADER], differing in the
         syntax of the resulting untagged FETCH data (RFC822.HEADER is
         returned).

      RFC822.SIZE
         The [RFC-2822] size of the message.

      RFC822.TEXT
         Functionally equivalent to BODY[TEXT], differing in the syntax
         of the resulting untagged FETCH data (RFC822.TEXT is returned).

My MUA uses commands RFC822.HEADER and RFC822.TEXT for retrieval.
Also getting rfc822 message in one querie is possible. 

There is nothing what require asking message in parts.

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 27 Mar 2007 14:49:03 +0100

> until and unless IMAPext gets around to making the change for

I think it is more a question of what current IMAP implementations  
actually DO, rather than what they are required to do.

Let us take an example. A message body was in Q-P on the wire. Is the IMAP  
implementation entitled to store it in only its decoded (8bit) form?  
Suppose the message was in fact signed by DKIM or PGP-mime (both of which  
are supposed to sign the Q-P form - a bad design decision, but there it  
is). The recipient (or his MUA) finds that the signature is broken. So the  
recipient requests to see the original form of the message so that he can  
check for himself exactly why the signature did not work (maybe some  
trailing empty line had got lost, and by manually restoring it he can make  
the signature work).

Now, if all that is required to work in current IMAP (or in practice does  
work), then I want corresponding scenarios in UTF8SMTP to work as well.

The corresponding situation in UTF8SMTP is where the message arrived at  
the final MTA in downgraded form. Is the IMAP implementation allowed to  
upgrade it, and store only the upgraded form (thus depriving the recipient  
of any possibility of seeing the downgraded form and thus possibly being  
able to debug some obscure side effect of the upgrading process)?