Subject: [EAI] Re: From Jari: Re: Your DISCUSS on draft-ietf-eai-framework-05 (fwd)

From: John C Klensin <[email-masked]>
Date: Wed, 07 Mar 2007 22:18:26 -0500

Jari,

As I assume you know by now, your message to the WG of 14
February got stuck somewhere and was apparently never circulated
to the WG until now.

Again with the understanding that this is a personal note/
explanation that may or may not represent a WG position...

--On Wednesday, 07 March, 2007 23:46 +0100 Harald Tveit
Alvestrand <[email-masked]> wrote:

> ------------ Forwarded Message ------------
> Date: 14. februar 2007 14:33 +0200
> From: Jari Arkko <[email-masked]>
> To: Harald Alvestrand <[email-masked]>
> Cc: [email-masked], EAI WG <[email-masked]>
> Subject: Re: Your DISCUSS on draft-ietf-eai-framework-05
> 
>>> > Since the final delivery SMTP server (or, to be more
>>> > specific, its corresponding mail storage agent) cannot
>>> > safely assume that agents accessing email storage will
>>> > always be capable of handling the extensions proposed
>>> > here, it MAY either downgrade internationalized emails or
>>> > specially identify messages that utilize these extensions,
>>> > or both. If this done, the final delivery SMTP server
>>> > SHOULD include a mechanism to preserve or recover the
>>> > original internationalized forms without information loss
>>> > to support access by UTF8SMTP-aware agents.
>>> 
>>> Does this suggest that the final server downgrades mails
>>> without knowing that the client is uncapable of reading
>>> them in the internationalized form? This seems surprising.
>>> Can you elaborate why? And what is the identification
>>> mechanism, and how does it affect POP/IMAP access to the
>>> messages?
>> Since this is an issue with a bit of substance, I'm CCing the
>> WG on this note.
>> The issue with POP and IMAP servers is that there is no
>> knowing what kind of client software the user will use to
>> connect to the mail store the *next* time it connects.
>> 
>> Especially in the case of IMAP, it is not uncommon to connect
>> to a single mailbox using multiple clients (often including a
>> Web client) - at the same time, serially, or switching
>> randomly between them.
>> 
> 
> Right.
> 
>> So not only is the final delvery agent incapable of knowing
>> the client's capabilities, it wouldn't help if it knew at the
>> time of delivery, because the capabilities might change at
>> any time. And messages in mailstores last for years, unline
>> messages in transit, which generally expire after days at
>> most, so it is very possible that a client's capabilities
>> will change over the lifetime of the message in the mailstore.
 
> Exactly. That is why it is important that we do not downgrade
> mail permanently if we can avoid it. I reacted to the text
> because it seemed to be saying that. Specifically, it says the
> storage agent may downgrade i18n e-mail. Does that mean that
> it would actually downgrade it in the storage, or just for the
> purpose of serving it to this particular client at this one
> time?

Actually, it doesn't say a thing about the storage agent
because, as far as 2821 is concerned, there is no such thing as
a storage agent.  More specifically, while there might be one,
its properties and interfaces are not defined at all.
Certainly, if one can avoid loss of information, one should do
that.  We can say that in the document, but it seems too
self-evident to be worth much discussion.   But suppose that (i)
the interface to the mail store, or the store itself, cannot
support messages with any non-ASCII content and (ii) at the time
the message is received, the POP and IMAP servers associate with
the mail store are known to the person implementing or
configuring the delivery SMTP server to have no mechanisms or
conventions for converting from mail-store format back to
UTF8SMTP and header format.   Storing the information in UTF-8
form is simply not a possibility here, nor is storing it in some
form from which the UTF-8 form can be reconstructed.

That leaves two options/theories:

	(a) The delivery MTA has no business offering the
	UTF8SMTP option if the situation downstream is that
	weak.  There is an additional complication with this,
	involving different mail stores for different addresses:
	see below.
	
	(b) It is desirable to deliver the mail if at all
	possible, even if it is somewhat damaged.

That language and the "MAY" provision were intended to allow for
the second case since "deliver something if at all possible" is
often chosen in practice.

>> What is knowable is whether or not the *mailbox server* is
>> capable of supporting UTF8SMTP; in the case of delivery via
>> SMTP or LMTP, the "UTF8SMTP" extension is a good way of
>> signalling such support.
>> 
>> The issues in the specific context of IMAP and POP, including
>> identification mechanisms, are addressed in the drafts
>> specific to these protocols; however, I think it would not be
>> appropriate to go into details on those protocols in an
>> overview document - also because those proposals are still
>> under discussion by the WG, and might change considerably
>> before they are finished.
> 
> Fair enough.

>> I don't know if this is enough clarification of why the text
>> is the way it is - do you need more information to clear your
>> DISCUSS?
> 
> Let me suggest a small reformulation:

This really doesn't work any better than the original and it
ignores some important cases.  Dissection below.

> Since the final delivery SMTP server (or, to be more specific,
> its corresponding mail storage agent) 

As mentioned above, "mail storage agent" is not well-defined or
formally part of the model

> cannot safely assume that agents accessing email storage

It is a little worse than that.  It may use some protocol --
either LMTP or something that is not standardized in the IETF --
to deliver to the mail store.  If the mail store, or the path to
it, doesn't have UTF8SMTP capability, the delivery SMTP server
is stuck.   Worse, it may deliver messages to different
addresses (mailboxes) into different message stores using
different protocols.  If some of them are fully capable of
handling UTF-8 headers and message content, and some are not, it
should probably advertise the options but it is not clear what
it should do in the cases in which, e.g., someone has assigned a
non-ASCII address to a UTF-8-incapable mail store.  That case is
arguably a configuration error, but some others, including those
involving backward-pointing addresses might not be.

> will always be capable of handling the extensions proposed
> here, it MAY either provide a downgraded version of the
> internationalized email to these agents, or specially identify
> messages that utilize these extensions, or both. In any case,
> the final delivery SMTP server SHOULD allow the original
> internationalized forms to be accessed by UTF8SMTP-aware
> agents without information loss, even if they are also
> accessed in downgraded form by other agents.

And, of course, the final delivery SMTP server isn't doing any
allowing because, in general, it doesn't talk to the servers or
access methods that support the "other agents" you posit.

I am sure the vocabulary and concepts her could be straightened
out but strongly suspect that the result would not be
substantively different from what is in the text now.  The
problems here reflect both the history and context of email work
in the IETF (e.g., POP and IMAP assume the existence of a mail
store, but SMTP does not) and some implementation options and
flexibilities that are 2821/2822-conforming.

best,
     john

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 08 Mar 2007 12:50:12 -0000

On Thu, 08 Mar 2007 03:18:26 -0000, John C Klensin <[email-masked]> wrote:

> should probably advertise the options ...

If a server advertises UTF8SMTP, then it is ofering a guarantee:

    "Either I will ensure that this messages is handed off, in exactly the  
form received, to some user/mailbox/store/system/whatever that is willing  
to offer essentially the same guarantee;
    OR I will downgrade it before passing it to any  
user/mailbox/store/system/whatever that cannot give that guarantee;
    OR I will send it back up the Return-Path with an explanation that it  
could not be delivered."

This implies, to me, that even if there are internal communications (LMPT,  
porocmail scripts, whatever) for routeing mail for different local-parts  
through different channels, the MTA itself needs to be configured with  
basic information as to which local-parts/channels can offer the UTF8SMTP  
capability. Sounds like a case for a bit of sendmail.cf hacking :-( .

> but it is not clear what
> arguably a configuration error,...

agreed.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Kari Hurtta <[email-masked]>
Date: 08 Mar 2007 17:59:54 +0200

"Charles Lindsey" <[email-masked]> writes
in gmane.ietf.ima:

>     "Either I will ensure that this messages is handed off, in exactly
> the  form received, to some user/mailbox/store/system/whatever that is
> willing  to offer essentially the same guarantee;
>     OR I will send it back up the Return-Path with an explanation that
> it  could not be delivered."
> 
> This implies, to me, that even if there are internal communications
> (LMPT,  porocmail scripts, whatever) for routeing mail for different
> local-parts  through different channels, the MTA itself needs to be
> configured with  basic information as to which local-parts/channels
> can offer the UTF8SMTP  capability. Sounds like a case for a bit of
> sendmail.cf hacking :-( .

On stronger form that  user/mailbox/store/system/whatever  system
is arranged that way that non UTF8SMTP agents no not see UTF8SMTP
messages.

For local delivery agent (LDA) that means that UTF8SMTP and ASCII messages
are stored to different mailbox (files or directories or whatever).

If communigation between MTA and LDA use LMTP no "sendmail.cf hacking"
should be needed -- UTF8SMTP negation should work also with LMTP.

/ Kari Hurtta

---

From: John C Klensin <[email-masked]>
Date: Thu, 08 Mar 2007 11:49:56 -0500

--On Thursday, 08 March, 2007 17:59 +0200 Kari Hurtta
<[email-masked]> wrote:

>> > As mentioned above, "mail storage agent" is not
>> > well-defined or formally part of the model
>> > 
>> > either LMTP or something that is not standardized in the
>> >...

>>     "Either I will ensure that this messages is handed off,
>>     in exactly the  form received, to some
>> user/mailbox/store/system/whatever that is willing  to offer
>> essentially the same guarantee; OR I will downgrade it before
>>     passing it to any
>> user/mailbox/store/system/whatever that cannot give that
>> guarantee; OR I will send it back up the Return-Path with an
>>     explanation that it  could not be delivered."

ok.  But, for the case that Jari was concerned about, the second
option is eliminated since it cannot be guaranteed that
downgrading and subsequent reading by a legacy system will not
lose any information.  So the statement above is equivalent to
	"Unless I can pass the message into a mechanism or
	repository that guarantees full UTF8SMTP support, I must
	reject or return (bounce) it".  
For those of us who don't think highly of returning (bouncing)
messages, that is in turn almost equivalent to:
		"If a delivery MTA cannot guarantee that it can deliver
		the message to an UTF8SMTP-capable system, it SHOULD NOT
		advertise the SMTP extension at all"

I don't think one wants to go there, but they are certainly
options that, if I were implementing such an MTA, I would make
configurable.   They are, after all, just another set of points
in the tradeoffs between "deliver a message, even damaged,
whenever possible" and "deliver only those messages whose
integrity can be completely guaranteed".  

>> This implies, to me, that even if there are internal
>> communications (LMPT,  porocmail scripts, whatever) for
>> routeing mail for different local-parts  through different
>> channels, the MTA itself needs to be configured with  basic
>> information as to which local-parts/channels can offer the
>> UTF8SMTP  capability. Sounds like a case for a bit of

In the most general cases, the MTA can't possibly know enough.
Worse, things can happen after "final delivery" over which the
MTA has no control.  For example, I would expect that, over
time, mail stores would migrate from 7-bit-only to
8-bit-capable, not only in content and headers, but in keys,
etc.  I would not expect them to migrate the other way, but see
no way to ban that.   So, just as a given message in a mail
store might be accessed by clients and user agents with a wide
variety of different capabilities, there is no way for the MTA
to know the behavior and capabilities, over time, of the message
destination.

> On stronger form that  user/mailbox/store/system/whatever
> system is arranged that way that non UTF8SMTP agents no not
> see UTF8SMTP messages.

By "not see" you mean, perhaps, "be told that they aren't
there?", "be told that they are not in a readable format?".
Note that we've already got provisions for the cases that do
work in the IMAP and POP documents, but, if a message is
delivered to a mail store or mechanism that is, itself,
UTF8SMTP-capable, but that store is accessed with a legacy POP
server, things may happen that are not predictable (since they
depend a lot on how the mail store and POP server are
implemented) or standardizable by this WG.

> For local delivery agent (LDA) that means that UTF8SMTP and
> ASCII messages are stored to different mailbox (files or
> directories or whatever).

One could implement it that way.  One could also implement it in
a variety of other ways.   Note that, if I have a mailbox for
which I have established several names (by aliasing or
otherwise), some of which involve non-ASCII addresses, and that
mailbox is fully UTF8SMTP-capable, I would consider the
restriction implied by the above completely unacceptable.

So I think you are talking about a whole series of
implementation or configuration choices above.  They are choices
I think our specifications should permit (even when I think they
would be dumb).  But, unless one of you is actually suggesting
standardizing something, I don't see where this discussion takes
us.  And, if you are suggesting standardizing something, let's
see the I-D.

     john

---

From: Kari Hurtta <[email-masked]>
Date: 08 Mar 2007 20:16:37 +0200

> restriction implied by the above completely unacceptable.

That means that when MUA is accesing directly (*) mailbox (files
or directories) and do not known about UTF8SMTP, it does not
not see UTF8SMTP messages, because it does not know from where 
to look them.

UTF8SMTP cabable MAU can be programmed to look also from
place where UTF8SMTP ares stored.  For _user_ this is still
one incoming mailbox. 

(*) "directly == via filesystem, not with POP or IMAP protocol"

> us.  And, if you are suggesting standardizing something, let's

Well, seems that there need something to be said about mailstores.
Not actual implementation, but some requirements for them when
interacting with UTF8SMTP.   But I do not know yet, what these are.

"Mailstore requirements for Email Address Internationalization (EAI)" ?

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 08 Mar 2007 19:32:44 +0100

>     OR I will send it back up the Return-Path with an explanation that it

Before picking the last exit it could also reject the mail at RCPT TO or
DATA time.  Bouncing to unverified Return-Paths is considered as net abuse
today, the receiver would be blacklisted.

Actually a tricky issue if MAIL FROM and RCPT TO are all-ASCII, and only
the header indicates that the DATA is a message/utf-8.

> the MTA itself needs to be configured with basic information as to which
> local-parts/channels can offer the UTF8SMTP capability. Sounds like a
> case for a bit of sendmail.cf hacking :-( .

Really tricky.  Minimally we've to explain the issue somewhere, probably
in the SMTP draft.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 09 Mar 2007 09:52:20 +0200

Kari Hurtta <[email-masked]> writes in gmane.ietf.ima:

<...>
> should be needed -- UTF8SMTP negation should work also with LMTP.

I assume following model:

                  +----------+              +----------+
                  | final    |              |          |
    ----SMTP--->  |  MTA     | ---LMTP----> |   LDA    |
                  |          |              |          |
                  +----------+              +----------+
                                                | |
                                                | |   write
                                                \ /
                                                 |

                                            (----------)
                                            (  mail-   )
                                            (  store   )
                                            (----------)
                                                 
                                                 |    read
       MTA = Mail Transport Agent                |
       LDA = Local Delivery Agent           +----------+
       MUA = Mail User Agent                |          |
                                            |   MUA    |
                                            |          |
                                            +----------+

Actual there is one problem. There is no HDR=UTF8 flag on SMTP.

Therefore LDA needs parse mail header fields, including 
mail header fields from mime parts.

LDA need know which messages are UTF8SMTP, so that it can
arrange them that way, that UTF8SMTP ignorant MUA do not see them.

Normally LDA do not parse mail messages or header fields.
Parsing of mail message on here is too much burder, I think.

Therefore at least LMTP  needs HDR=UTF8   paramater.

( I do not remember, have I posted message message, where I analyze
  when HDR=UTF8 is needed also on ESMTP. )

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 09 Mar 2007 17:12:55 -0000

> lose any information.

I don't see that. We have carefully defined the downgrading process so  
that downgraded headers get replaced by a Downgraded: <header-name> : <RFC  
2047 stuff>. So we now have a valid ASCII email, but no information has  
actually been lost (though it might be inconvenient to extract it).

So, so long as downgraders between the MDA and the actual mailboxes stick  
to our 'downgrade' draft, everything should be OK.

---

From: Jari Arkko <[email-masked]>
Date: Sat, 17 Mar 2007 14:19:38 +0200

John, all,

Sorry for not getting back to this topic sooner. In any case, I have
looked at John's earlier message and decided that with the
additional information I can clear my Discuss.

Jari

---

From: John C Klensin <[email-masked]>
Date: Sat, 17 Mar 2007 08:33:32 -0400

--On Saturday, 17 March, 2007 14:19 +0200 Jari Arkko
<[email-masked]> wrote:

> John, all,
> 
> Sorry for not getting back to this topic sooner. In any case,
> I have looked at John's earlier message and decided that with
> the additional information I can clear my Discuss.

thanks.  good news.
   john

---

From: Kari Hurtta <[email-masked]>
Date: 22 Mar 2007 18:53:16 +0200

<...>

> us.  And, if you are suggesting standardizing something, let's

You asked I-D.

See: -----------------------------------------------------------------
From: [email-masked]
Subject: I-D ACTION:draft-hurtta-eai-messagestore-00.txt
Newsgroups: gmane.ietf.announce
To: [email-masked]
Date: Thu, 22 Mar 2007 10:50:02 -0400
Reply-To: [email-masked]

A New Internet-Draft is available from the on-line Internet-Drafts 
directories.

	Title		: Message Store requirements for Internationalized Email
	Author(s)	: K. Hurtta
	Filename	: draft-hurtta-eai-messagestore-00.txt
	Pages		: 13
	Date		: 2007-3-22
	
   The Email Address Internationalization (EAI) is implemented by
   allowing UTF-8 characters in SMTP envelope and mail headers.
   UTF8SMTP extension of ESMTP takes care that mails with UTF-8
   characters in SMTP envelope and mail headers are not delivered to EAI
   non-compliant SMTP servers.  This document describes mechanism how to
   keep messages with UTF-8 characters on mail headers separated from
   EAI non-compliant Mail User Agents.  This document also describes
   general requirements for UTF8SMTP Message Store.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-hurtta-eai-messagestore-00.txt

To remove yourself from the I-D Announcement list, send a message to 
[email-masked] with the word unsubscribe in the body of 
the message. 
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce 
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the 
username "anonymous" and a password of your e-mail address. After 
logging in, type "cd internet-drafts" and then 
"get draft-hurtta-eai-messagestore-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-hurtta-eai-messagestore-00.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.
_______________________________________________
I-D-Announce mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/i-d-announce

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 22 Mar 2007 19:54:29 +0100

> http://www.ietf.org/internet-drafts/draft-hurtta-eai-messagestore-00.txt

Some quick notes from reading it once:

|  (g)  WWW browser access mail server with HTTP [RFC2616].

I think you can resolve (2g) by a note stating that Web mail
is typically based on IMAP or POP3, and therefore covered by
your (2e) or (2f).  If that's true, my POV is limited to a
few ISPs and my guesses what they might do, behind what I see.

| There is two mail reason why UTF8SMTP ignorant MUAs must
| not see UTF8SMTP messages:

What about users using different MUAs and different access
methods depending on where they are ?  E.g. users sometimes
need Web mail, and that should be an "UTF8SMTP aware" access
method, but generally use whatever their box offers (= some
legacy software).  You can't let the message/utf-8 rot in a
message store until the user happens to try Web mail again.

| This implies that UTF8SMTP ignorant MUA does not see any
| message (if downgrading is not provided).

Well, that case is kind of obvious.  And maybe better than
splitting the message store into a part for message/rfc822
working always, and another part for message/utf-8 working
only "sometimes" (with an "UTF8SMTP aware" MUA).

| To discover message is UTF8SMTP message may require that
| all message header fields (including header fields from
| MIME body parts) are parsed.

As always that violates anything I think to know about data
abstraction and modularization.  MIME part headers are not
the business of agents transporting MIME 1.0 messages.  You
are not designing message stores for 7bit MUAs.  If "old"
8bit MUAs can't handle a message/utf-8 part within an 8bit
message/rfc822 it's no problem to be solved by the MDA:

There are tons of MIME types not supported by any given MUA,
and maybe the user has helper applications to deal with
some of these "unknown" (from the MUA's POV) MIME types.

To read message/utf-8 parts within a message/rfc822 behind
a "UTF8SMTP ignorant" MUA a decent text editor with macros
to decode QP or B64 should be enough.

And if it's not enough the user can ask what these parts
are supposed to be, the message/rfc822 has an US ASCII
header working with his "UTF8SMTP ignorant" MUA.

| If communication between final delivery MTA and MDA use
| LMTP, UTF8SMTP response to LHLO command tells that MDA
| is UTF8SMTP

That's rather late to decide this.  Is your model limited
to verified (SPF PASS or similar) return paths ?  It won't
fly if you drop undeliverable message/utf-8 at the final
delivery MTA.

| UTF8SMTP messages are stored to /var/mail/{username}:UTF8
| file by MDA (for UTF8SMTP messages ':UTF8' is appended
| to name of file.)

Implementation details.  Please note that this chapter is
only an example (e.g. not working on file systems where a
colon isn't allowed within the path).

| 8.  Security Considerations

Oh, there you have my issue noted above, the message/utf-8
rotting in a place rarely accessed by the user.  I think
that case has to be avoided as good as possible.  For the
file system case you've proposed a way how that _could_ be
done (presence of user:UTF8 mbox file), but I think you
need some MUSTard to cover it.

Otherwise the I-D is clear, and I think the WG should add
it to its agenda.  You've mentioned changes needed for
LMTP, does one of the many EAI I-Ds discuss this already,
and I missed it, or should it be added somewhere ?

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 23 Mar 2007 06:56:44 +0200

Frank Ellermann <[email-masked]> writes in gmane.ietf.ima:

> abstraction and modularization.  

Sorry. That just stated fact what WG decided.

Parsing everything is required when "Header-Type: UTF8SMTP" header field 
is dropped and HDR=UTF8SMTP does not exists on draft-ietf-eai-smtpext.

( draft-ietf-eai-smtpext also either needs similar language, or it need 
  add HDR=UTF8SMTP parameter... )

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 23 Mar 2007 07:18:13 +0200

> message/rfc822 it's no problem to be solved by the MDA:

You are saying that.

Was that discussed on meeting? You was suggested to ask
from MIME experts that

       Is UTF-8 possible in MIME version 1.0 header fields ?

What was result?

My feel is that
   - If UTF-8 can be added to address and subject
     header fields without label, then UTF-8 can be added also
     to MIME header fields without label.

   - And if HDR=UTF8SMTP option is added ESMTP, it can cover also
     MIME header fields. MIME header field syntax is already updated 
     without changing MIME-Version header field. So changing MIME header 
     field syntax yet one more time is not different.

   - Label
        MIME-Version: 2.0
     may be usefull to indicate that downgrade is required also
     for header fields on MIME structure. 

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 23 Mar 2007 07:53:04 +0200

> fly if you drop undeliverable message/utf-8 at the final

That was on part of "Notes:" items. Previous item was:

|   o  If final delivery MTA is UTF8SMTP aware, it is recommended that
|      MDA is arranged that way that it is UTF8SMTP aware.

You was taking it out from context.

Yes, if next hop (this time it is MDA) is not UTF8SMTP aware
there is following possibilities on final MTA:

        - MTA rejects final dot on DATA on SMTP level, if there is 
          at least one RCPT TO which was local
          and MTA discovers that message is UTF8SMTP (by parsing it).
        - MTA downgrades message if there is 
          at least was at least one RCPT TO which was local
          and MTA discovers that message is UTF8SMTP (by parsing it).
        - MTA sends undeliverable DSN for local recipients
          when  MTA discovers that message is UTF8SMTP (by parsing it).

If HDR=UTF8SMTP parameter is added protocol, then there is 
also following possibility on table:

       - MTA rejects RCPT TO command which was local if
         HDR=UTF8SMTP parameter was given on MAIL FROm -command.

Actually it is often late in also when final MTA rejects message
on SMTP level.  Usually picture is:

                             +-------------+                +-------------+
                             |  "border"   |                |    final    |
 Internet     -----ESMTP---> |    MTA      |  --- ESMTP---> |     MTA     |
                             |             |                |             |
                             +-------------+                +-------------+

Anything what was rejected by final MTA tend to cause backscatter
on case of junk mail with forged return path. Often that is full mailbox.
That information is quite difficult to transfer to "border" MTA.
( On general case you even do not know is mailbox "full" before
  you are seen final dot after DATA -- before that you do not really
  know size of mail.)

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 23 Mar 2007 13:48:44 +0100

Kari Hurtta wrote:

 ["what is MIME"] 
> Was that discussed on meeting?

Don't think so, I only know the jabber log:
http://www3.ietf.org/meetings/ietf-logs/eai/2007-03-20.html

> You was suggested to ask from MIME experts that

Nothing, unless "[10:39:54] --- nsb has joined" 
indicates something that's not visible in the log.

>      to MIME header fields without label.

Yes, that's where we disagree, because it makes the
identification of a message/utf-8 too difficult:

Looking into the header is one thing, but parsing
the body for MIME part headers is another thing:

Looking for somebody else's trouble, none of our
business, what if the MIME structure is broken,
should that affect message/utf-8 transport, etc.

> MIME header field syntax is already updated
> without changing MIME-Version header field.

In an IETF theory styling itself as "RFC 2231".

It's IMO possible to implement 2049 ignoring
2231, and treat 2231 "features" as garbage.

Of course that software then couldn't claim to 
support the (approved) NetNews message format,
but that RFC won't be published before another
RFC is ready... <sigh />

> So changing MIME header field syntax yet one
> more time is not different.

2231 is in a certain sense "compatible" with
MIME 1.0, while raw UTF-8 in Content-* header
fields directly _violates_ a requirement in 
2045 (or whereever it was, we quoted it here
often enough).

Show me that Keith, Ned, and "nsb" (or maybe
Bruce overruling them all :-) say "no problem",
and I might consider an "it's only MIME, we're
free to screw with it" approach... <shudder />

[[ I'd still claim that it's unnecessary, and
   that it should be done in a proper 2045bis
   after EAI turned out to work as expected ]]
  
> Label
>   MIME-Version: 2.0
> may be usefull to indicate that downgrade is
> required also for header fields on MIME
> structure.

Yes, and it would warn old MIME parsers that
they might stumble over some surprises (in the
form of raw UTF-8) if they try to parse "2.0"
as "1.0".  We could also formally require 2231
support for MIME-Version 2.0, just to be sure.

Maybe the result would be more like a 2049bis
than a 2045bis.  

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 23 Mar 2007 17:18:49 +0200

Frank Ellermann <[email-masked]> writes

> few ISPs and my guesses what they might do, behind what I see.

Adding note seems guite good idea.

That is not however universal truth :-)

For good measurement I disabled POP and IMAP
listeners on testing server.  WWW interface
still works :-)

hurtta@leija:~$ telnet testiXXXX.fmi.fi 110
Trying 193.166.XXX.XXX...
telnet: Unable to connect to remote host: Connection refused
hurtta@leija:~$ telnet testiXXXX.fmi.fi 143
Trying 193.166.XXX.XXX...
telnet: Unable to connect to remote host: Connection refused
hurtta@leija:~$

That is Communigate Pro.

(  Communigate Pro   is quite strange product. 
    All listeners seems just to be one process. 
    Whole server is one process ( SMTP, HTTP, IMAP, 
    POP, SIP, Radius, LDAP and whatever)
   there is is just big number of threads ...
)

Also if I read documentation of Courier-MTA correctly,
it's webinterface read directly Maildirs -- it is not 
using IMAP   ( http://www.courier-mta.org/sqwebmail/ )

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 23 Mar 2007 19:12:33 +0200

<...>
> support for MIME-Version 2.0, just to be sure.

It is unlikely that MIME-Version header field is checked.

RFC 2049: Multipurpose Internet Mail Extensions
                  (MIME) Part Five:
             Conformance Criteria and Examples

--------clip---------------clap---------------------------------
    (1)   Always generate a "MIME-Version: 1.0" header field in
          any message it creates.

    (2)   Recognize the Content-Transfer-Encoding header field
          and decode all received data encoded by either quoted-
          printable or base64 implementations.  The identity
          transformations 7bit, 8bit, and binary must also be
          recognized.

<...>

    (13)  The following sentence has been removed from the
          discussion of the MIME-Version header: "However,
          conformant software is encouraged to check the version
          number and at least warn the user if an unrecognized
          MIME-version is encountered."

--------clip---------------clap---------------------------------

It is quite clear, that MIME-Version: header field is 
expeted to be ignored !

(My MUA checks MIME-Version: header field, and perhaps
 do not be MIME-conformant ... :-) )

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 23 Mar 2007 19:11:14 +0100

>    there is is just big number of threads ...

For some time I thought that "Communigate Pro" is a
product for professsional spammers.  That was before
some weapons of mass destruction aka Barracuda used
2821-processing literally, and were promptly blocked
from the public Internet until they fixed the buggy
default.  From time to time old 2821-Barracudas still
show up, and get their owners on any decent DNSBL.  

I've not seen "Communigate Pro trial version" spam
for some years now, maybe they also fixed something.

> if I read documentation of Courier-MTA correctly,
> it's webinterface read directly Maildirs -- it is
> not using IMAP

For squirrelmail I know that it can use IMAP and POP,
but I can't tell if it also supports direct access.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 23 Mar 2007 20:29:28 -0000

> message/rfc822 it's no problem to be solved by the MDA:

On the contrary, MTAs that support 8BITMIME regularly take note of both  
Content-Type and Content-Transfer-Encoding in MIME part headers of MIME  
1.0 messages, in order to downgrade them for the benefit of  
non-8BITMIME-compliant MTAs. They could just as easily do the same for  
UTF8SMTP (in fact, the same piece of code could be adapted to do both  
jobs).

That was all eecognized and, AIUI, agreed at the time we abolished the  
Header-Type header, and the main argument for abolishing it was that you  
would still have to check through the whole MIME structure because you  
could not be sure that such a Header-Type header had not been left out or  
deleted by some bad implementation.
> some of these "unknown" (from the MUA's POV) MIME types.

But an MTA does not need to know about those "tons of MIME types" to do  
the job we are talking about. It is really only interested in multiparts  
and a few message types.
> to decode QP or B64 should be enough.

But even a "decent text editor" is no use unless it knows the rules for  
finding all the C-T-E headers hidden within the struture.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 23 Mar 2007 20:44:45 -0000

> should that affect message/utf-8 transport, etc.

If the MIME structure of broken, then Garbage-in Garbage-out. That is  
already true with out UTF8SMTP.
> In an IETF theory styling itself as "RFC 2231".

No, it is not a "theory", it is a "proposed standard", just like RFC 2045  
and RFC 2822.
> 2045 (or whereever it was, we quoted it here

And raw UTF-8 in any header directly violates a requirement in RFC 2822,  
but that has not stopped us,

We are constructing a new kind of mail object - call it a RFC2822-u object  
if you like. And we are designing a protocol which is supposed to ensure  
that RFC2822-u objects are converted to RFC2822 objects before they are  
allowed to enter the old non-EAI universe.

And we are also inventing a new kind of MIME-Version-1.0 object, let us  
call it a MIME-Version-1.0-u object, and we are designing a protocol which  
is supposed to ensure thar MIME-Version-1.0-u objects are converted to  
MIME-Version-1.0 objects before they enter toe non-EAI iniverse.

There is NO difference in principle between the two cases.

Yes, we might decide to call them MIME-Version-1.1 objects instead of  
MIME-Version-1.0-u objects, and require a new MIME-Version: 1.1 header.  
But having just got rid of a Header-Type header to distinguish RFC2822  
objects from RCC2822-u objects, it would be rather odd (and confusing  
IMHO) to take a different route as regards MIME.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 24 Mar 2007 01:25:26 +0100

Charles Lindsey wrote:

> If the MIME structure of broken, then Garbage-in
> Garbage-out. That is already true with out UTF8SMTP.

Kari's proposal would bounce anything misidentified
as message/utf-8 at the final delivery MTA, if the
LMTP-MDA says "no, thanks, legacy mailbox".  That's
not GiGo, it's more like "garbage in, garbage back".

> No, it is not a "theory", it is a "proposed
> standard", just like RFC 2045 and RFC 2822.

http://purl.net/net/rfc/2045       => DRAFT STANDARD
http://tools.ietf.org/html/rfc2045 => DRAFT STANDARD

Ditto 2046, 2047, and 2049,  The registry stuff in
2048 is BCP 13 (now consisting of 4288 and 4289).

We discussed the implicit side-effects of 2231 wrt
multiple parameter name=value pairs using the same
name for some months in USEFOR with Bruce, Ned, and
Alexey.

Fortunately Bruce caught this bug before we pulled
a stunt like introducing multiple "ComplaintsTo"
parameters into a single header field claiming to
use MIME syntax.  Ned saying that he always had an
"environment" model in mind for MIME 1.0 on a list
isn't the same as saying it explicitly in RFC 204x.

Let's hope that nobody else tried it "successfully",
it is rather tempting to simplify the syntax of an
enumeration.  And it breaks miserably with the 2231
concept of multi-line name=value paramaters.

> And raw UTF-8 in any header directly violates a
> requirement in RFC 2822, but that has not stopped us,

2822 is the Internet Message format, we "adapted" it
as necessary in USEFOR, because it didn't fit several
reqirements for NetNews (in essence creating a proper
subset of 2822 + plus adding header fields only used
in NetNews).

We didn't modify MIME 1.0 for NetNews, in my parallel
universe that would be blasphemy.  I'd really prefer
it if EAI also stays away from "extending" MIME 1.0 -
above adding a bunch of new MIME types as outlined in
the DSN draft.

> We are constructing a new kind of mail object - call
> it a RFC2822-u object if you like.

<sarcasm> I like to call it message/utf-8 </sarcasm>

> we are designing a protocol which is supposed to
> ensure that RFC2822-u objects are converted to RFC2822
> objects before they are allowed to enter the old
> non-EAI universe.

You've skipped an important step:  We know how to convert
message/utf-8 into message/rfc822 for 8BITMIME.  For this
8bit part of the universe anything is straight forward.

We're less sure about the 7bit part of the universe, it
could make sense to invent a general mechanism working
for any 8 bit message/foo (not only message/utf-8) if a
message/foo is forced to be transported by 7bit relays.

But such "8to7" considerations are out of scope for EAI,
getting this right without any application/message hack
is a major headache (see Kari's encapsulation I-D) and
limited to message/utf-8.

> we are also inventing a new kind of MIME-Version-1.0
> object, let us call it a MIME-Version-1.0-u object

Any MIME 1.0 message/rfc822 can be an 8bit multipart/*
with message/utf-8 parts, and vice versa.  So far not
"new", that's how MIME and message/utf-8 are designed.

The _new_ concept would be MIME Version-1.0-u (or 2.0)
allowing raw UTF-8 in its Content-* header fields and
MIME part headers.  It changes the meta language used
to talk about MIME parts and their structure.  It's
not absolutely necessary for EAI to introduce MIME 2.0
now.

> we are designing a protocol which is supposed to
> ensure thar MIME-Version-1.0-u objects are converted
> to MIME-Version-1.0 objects before they enter to a
> non-EAI iniverse.

That's shaky.  IMO you want too much in one giant leap,
the transition from MIME 1.0 to 2.0 will be a rough
ride, what with implementations checking the version
and treating anything that's not MIME 1.0 as garbage ?

And why would you wish to bind the success or failure
of EAI to this MIME 2.0 feature ?  It's tricky enough
without it.

It reminds me of Martin's mailto-bis I-D where he wanted
to jump from 1738 to 3987+EAI without a pit stop at 3986.

Or your news-and-nntp proposals.  Or the Sender-ID RFCs
trying to change the email architecture fundamentally
based on observations working for c. 80% of all emails.

I don't believe in "wordwide upgrade" scenarios, it's
very hard to get folks to upgrade, just because an RFC
says so has nothing to do with it.

> There is NO difference in principle between the two

Yes, in principle you can do MIME 2.0.  What you should
NOT do is claim that 2.0 is still 1.0.  I hate anything
in the direction of "embrace and extend", where that's
actually "embrace, extend, and extinguish".

MIME is a text book example of a standard guaranteeing
backwards compatibility as far as possible and beyond.
The authors should get a Turing-award or something for
this masterpiece.

> having just got rid of a Header-Type header to
> distinguish RFC2822 objects from RCC2822-u objects,
> it would be rather odd (and confusing IMHO) to take
> a different route as regards MIME.

Well, there's no message/rfc822 header field in a top
level mail or news message, but message/rfc822 is used
as MIME Content-Type for message bodies or MIME parts.

Like a message/utf-8 will be used as Content-Type for
message bodies or parts with raw UTF-8 in the "header
field bodies" (i.e. headers, we know the terminology).

MIME part headers are a different issue, it's not a
part of 2822 with the Message format, it's a part of
MIME.  The MIME design guarantees that MIME ignorants
can transport MIME messages, they can even read MIME
to some degre, and reply to or forward MIME messages
without knowing what =?utf-8*en-DE?Q?this?= means.

MIME is completely transparent, if you don't know it
you can savely ignore it, and it still works.  Okay,
admittedly I never bothered to really understand what
message/partial is, let's say s/completely/almost/

Only Bruce knows what that is, and only Bruce tried
to join 2822 and MIME.  I still have his four I-Ds,
it's a shame that nobody (including me) tried to use
his work.

Ignoring Bruce's I-Ds MIME works on top of 2822, you
can use 2822 without knowing what MIME is.  And you
can use MIME on top of 822 ignoring anything in 2822
that you don't like (but I guess there's only one
Resent-* monstrosity in RFC 2822 where it could make
sense to stick to 822).  2882 is also a masterpiece,
minor nits not withstanding, splitting the ABNF in
obs-* and non-obs-* mast have been hell.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 24 Mar 2007 03:16:03 +0100

> You was taking it out from context.

Yes, but I read it top down once, and then used any cheap
shots I could find.  I like your ASCII art figures, they
make it much easier to understand what the prose is about.

> If HDR=UTF8SMTP parameter is added protocol, then there
> is also following possibility on table:

>   HDR=UTF8SMTP parameter was given on MAIL FROm -command.

How about joining BODY=8BITMIME with HDR=UTF8SMTP ?  What
I've in mind is this:

After the client said EHLO, indicating that it's willing
to hear which SMTP extensions are supported by the server,
the server may say that it can do UTF8SMTP plus 8BITMIME.

If the client wants 8BITMIME it will say BODY=8BITMIME.
In theory it can also say BODY=7BIT (I'm not sure how
that could be interesting for the server).

For what you have as HDR=UTF8SMTP a client would always
use BODY=8BITMIME.  This includes cases where the DATA
doesn't contain a single non-ASCII, but the MAIL FROM
(ending up as Return-Path) or a RCTP TO (ending up as
a from-clause or some X-EnvelopeSender header field at
least in theory) is an address with <UTF8-non-ASCII>.

That's quite a lot of parameters, ALT-ADDRESS, HDR, and
BODY.  If we're sure that HDR=UTF8SMTP _always_ implies
BODY=8BITMIME we can define a new BODY=UTF8SMTP joining
these parameters.

IFF.  Of course the name of the parameter is somewhat
misleading when it's used to talk about the header, but
the definition in RFC 1652 apparently fits.

IFF.  Otherwise we'd get a mouthful BODY=8BITMIME plus
HDR=UTF8SMTP plus (optional) ALT-ADRESS=[email-masked]
plus (likely) raw UTF-8 in the MAIL FROM address, which
would render the two BODY + HDR parameters as redundant.

After all the client is talking to a server claiming to
support UTF8SMTP + 8BITMIME.

> it is often late in also when final MTA rejects message
[trimmed by me]
|            +-------------+             +-------------+
|            |  "border"   |             |    final    |
| --ESMTP--> |    MTA      |  --ESMTP--> |     MTA     |
|            +-------------+             +-------------+

Yeah, a picture I've used on the SMTP list where those
"ADMDs" in the email-arch I-D try to hide the existence
of this border behind fog was this:

          +-----+   +-----+ : +-----+   +-----+
         /| MSA |-->| MO  |-->| MX  |-->| MDA |\
        / +-----+   +-----+ : +-----+   +-----+ \
+-----+/     |       ^ ^    :    |       ^ ^     \+-----+
| MUA |      | +-----+ |    :    | +-----+ |      | MUA |
+-----+\     V V       |    :    V V       |     /+-----+
        \ +-----+   +-----+ : +-----+   +-----+ /
         \| MDA |<--| MX  |<--| MO  |<--| MSA |/
          +-----+   +-----+ : +-----+   +-----+

For a mail flow mainly from left to right the left half
is an "MON" (mail originiating network, without counting
"ADMDs"), and the right half is an "MRN" (mail receiving
network) in Keith's terminology.

That picture is too simple to cover mediators, or details
of various MXs.  But it covers the important point that
the border MTA (MX) and the final delivery MTA (MDA) are
different.

In your I-D you've made it more interesting by adding an
LMTP hop between MDA and final delivery MTA.  One or two
hops isn't that important for "reject" vs. "bounce", at
least one is enough to get into serious trouble - either
forced to drop mails, or to send bounces to unverified
addresses.

> Anything what was rejected by final MTA tend to cause
> backscatter on case of junk mail with forged return path.
> Often that is full mailbox.  That information is quite
> difficult to transfer to "border" MTA.

So far all my attempts to convince the Spamcop folks that
this case can legitimately happen (for SPF NONE or NEUTRAL)
failed miserably.

But the info "legacy mailbox" vs. "EAI mailbox" isn't as
volatile as "mailbox full" (or not), it's more like the
info "mailbox exists" (or not).  We're talking about a
single bit, I don't care how it's tranferred to the MX.

And for an SPF PASS I don't care if it's transferred at
all, actually "PASS" stands for "legit bounces welcome".

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 24 Mar 2007 07:00:10 +0200

> BODY=8BITMIME we can define a new BODY=UTF8SMTP joining

I asked that earlier.  Problem is then that BODY=BINARYMIME
does not fit to picture.

Answer was that UTF8SMTP does not forbid BODY=BINARYMIME.

Therefore BODY=UTF8SMTP does not fly.

/ Kari Hurtta

draft-ietf-eai-smtpext-04.txt: 

|                                                         The UTF8SMTP
|   extension MAY be used with the BODY=8BITMIME parameter if that is
|   appropriate given the body content or, if the server advertises it
|   and it is appropriate, with the BODY=BINARYMIME parameter specified
|   in [RFC3030].
|
|   Assuming that the server advertises UTF8SMTP and 8BITMIME, and at
|   least one non-ASCII address, with or without ALT-ADDRESS, the precise
|   interpretation of these parameters on the MAIL command is:
|
|   1.  Headers are in UTF-8, body parts are in ASCII.
|   2.  Headers are in UTF-8, some or all body parts contain 8-bit line-
|       oriented data.
|   3.  Headers are in UTF-8, some or all body parts contain binary data
|       without restriction as to line lengths or delimiters.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 24 Mar 2007 11:27:37 +0100

Kari Hurtta wrote:
 
> BODY=BINARYMIME does not fit to picture.

Sorry, I only checked 1652 and forgot this. 
 
> Therefore BODY=UTF8SMTP does not fly.

A pair of parameters like UTF88BITMIME and
UTF8BINARYMIME could cover it.  With nicer
names, maybe.

Anyway, some parameter like HDR=UTF8SMTP is
required, otherwise MTAs would be forced to
receive DATA that they have to reject at the
end of DATA.

With that parameter they can reject a.s.a.p.
per receiver (for a legacy mailbox).  We get
a new reply code "erroneous BODY parameter"
for a missing HDR=UTF8SMTP (after DATA).

An unnecessary HDR=UTF8SMTP is also ugly if
a wannabe message/utf-8 was rejected for a
legacy mailbox, and later turned out to be
no message/utf-8.  But what went wrong in
this case should be obvious for the sender,
it's not the job of the server to outsmart
the client.

###

One day later after some sleep:  My idea to
use HDR=UTF8SMTP based on the header and
info that _could_ be added by the server in
a timestamp line or Return-Path or some
non-standard "envelope to" header field was 
*_horribly_* wrong.

HDR=UTF8SMTP has to reflect what really is
in the header, not what might end up there.

This gets us the interesting case, where a
mail arriving without HDR=UTF8SMTP has to be
forwarded with it.  Depending on what the 
MTA adds to the header (only ASCII or UTF-8).

###

I think HDR=UTF8SMTP is not some premature
optimization, it's required for all cases
where some mailboxes accept message/utf-8
and others don't.

Otherwise the mail is in essence sent (N+1)
times:  1st attempt, receivers 1...N, no 
UTF-8 in the envelope, but in the header,
server says "5xx some users hate UTF-8"
afte DATA.

2nd attempt (receiver 1), etc. up to (N+1),
each with a single RCPT TO trying to find
out which of the N receivers accept UTF-8.

OTOH with HDR=UTF8SMTP the whole business
for N receivers can be handled at once.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 24 Mar 2007 13:23:43 +0200

> UTF8BINARYMIME could cover it.  With nicer

I do not think that MAIL FROM command line
need to be short. It is anyway required that
SMTP extensions specifies how much they increases 
the maximum length of the MAIL and RCPT command.

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 24 Mar 2007 12:49:11 +0100

> It is unlikely that MIME-Version header field is checked.

 [2049]
>| (13)  The following sentence has been removed from the
>|       discussion of the MIME-Version header: "However,
>|       conformant software is encouraged to check the version
>|       number and at least warn the user if an unrecognized
>|       MIME-version is encountered."

> It is quite clear, that MIME-Version: header field is

Wait a moment, 2049 is (only) the _MINIMAL_ conformance.  The
full discussion is in 2045 chapter 4:

| It is not possible to fully specify how a mail reader that
| conforms with MIME as defined in this document should treat
| a message that might arrive in the future with some value
| of MIME-Version other than "1.0".

And what you've quoted was apparently a modification (1996),
are you sure that older MIME software isn't used today ?
  
Here's what I see when I click on "About - about netscape":
| Copyright © 1994-1997 Netscape Communications Corporation

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 24 Mar 2007 17:08:11 +0100

> the maximum length of the MAIL and RCPT command.

Okay, joining BODY= and HDR= is unnecessary.  

You wrote it elsewhere, the minimally required
additional length for ALT-ADDRESS= is 268, for
HDR=UTF8SMTP it's in theory 12, together that's
281 for MAIL.  

Maybe we could use this worst case also for RCPT
introducing only one magic number 281 (?) 

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 25 Mar 2007 11:23:20 +0300

> colon isn't allowed within the path).

Yes. That chapter is example. It is perhaps better to place 
on appendix (-- need figure out how this is done with xml source.)

However, I think that one example how to implement this is needed.
Implementators of MUAs are different people than implementors
of Mail Delivery Agent (MDA) or Local Delivery Agnet (LDA).

On one sysmtem there is several MUAs, but only one MDA.

/ Kari Hurtta

---

From: "abel" <[email-masked]>
Date: Mon, 26 Mar 2007 10:16:00 +0800

----- Original Message ----- 
From: "Kari Hurtta" <[email-masked]>
To: <[email-masked]>
Sent: Saturday, March 24, 2007 7:23 PM
Subject: [EAI] MAIL command length (Re: HDR=UTF8SMTP)

> the maximum length of the MAIL and RCPT command.

+1,
Reference to other extension, they have a small section to talk about LENGTH
effect in MAIL/RCPT commands, inclue an extreme example.

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 26 Mar 2007 03:24:30 +0200

> On one sysmtem there is several MUAs, but only one MDA.

After you convinced me that MDA can be what I knew as Mlocal,
yes, there can be only one mailer known as "local".  But it
won't surprise me if sendmail.cf can be tweaked into having
additional Mlocal1, Mlocal2, ... used depending on the phase
of the moon... <gd&r>

---

From: Kari Hurtta <[email-masked]>
Date: 26 Mar 2007 07:13:46 +0300

> additional Mlocal1, Mlocal2, ... used depending on the phase

Yes, of course. Basically sendmail routes different domains
to mailers defined on sendmail.cf. Mailer defination start with
'M' on sendmail.cf. 

Normally mail domain equivalent of hostname on where sendmail 
is running is routed to mailer defination named with 'local' on 
sendmail.cf. 

Internally on sendmail there is also '*file*' mailer. That
can be considered also be MDA (it is used when mail is directed 
to file on .forward.)

So it is easy to setup (via mailertable) that 'domain1' goes
to 'local1' and so on ....

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 26 Mar 2007 16:32:59 +0100

> name for some months in USEFOR with Bruce, Ned, and

Yes, using RFC 2231 for wrappng long lines of parameters is a "ghastly  
mess", but all we are concerned with here is using it for i18n values, and  
for that is is merely an "ordinary mess", and all that is available. For  
sure, we do not want to invent "yet another 8-t0-7 bit conversion", and  
yet Utf8headers requires us to deal with MIME <value>s with UTF-8 in them,  
and RFC2231 provides the only downgrade mechanism already available for  
our use.

> Let's hope that nobody else tried it "successfully",

For sure, RFC 2231 HAS been implemented, though possibly not widely.

> <sarcasm> I like to call it message/utf-8 </sarcasm>

Please do not use the term "message/utf-8". If used, it would apply only  
to a certain MIME mechanism, and not as a generic term for any message  
that happens to have UTF8 in its headers somewhere.

And even as a MIME machanism it will not fly in that form, because it  
violates RFC 2045.
> 8bit part of the universe anything is straight forward.

But we don't know how to convert amessage/utf-8 for an MTA that does not  
do 8BITMIME.

> MIME part headers.

Exactly so. And that feature has been a solid part of Utf8headers for  
quite some time, and Utf8headers is now thought to be almost ready for WG  
Last Call. You are the only person who seems not to accept this feature.

> That's shaky.

It is no shakier than converting 8BITMIME stuff to non-8BITMIME.

> you can savely ignore it, and it still works.

No it's NOT transparent. You cannot downgrade 8BOITMIME unbless you  
understand it.

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 26 Mar 2007 19:45:49 +0200

> For sure, RFC 2231 HAS been implemented, though possibly

Bruce and Kari said they did (maybe they could join
forces for a 2231-to-DS interoperability report. :-)

But what I meant above ("it") was about attempts to 
implement value sets x := {y1, y2, y3} with multiple
MIME parameters x=y1; x=y2; x=y3 because it's tricky
to use a say comma separated list in a quoted string,
if the values can be itself quoted strings.

If anybody did that (and we nearly did it in USEFOR)
in an RFC it's strictly incompatible with RFC 2231.

> Please do not use the term "message/utf-8". If used,
> it would apply only to a certain MIME mechanism

It's the message/utf-8 as outlined in the dsn-00 I-D.
It's the name of a structure almost identical with a
message/rfc822, "only" allowing UTF-8 in the header
(or more precisely as specified in the header draft.)

> as a MIME machanism it will not fly in that form,
> because it violates RFC 2045.

Nothing's wrong with the message/utf-8 MIME type as
outlined in the DSN I-D.  What's that business about
a "MIME mechanism" ?  Do you mean "MIME type" ?

> we don't know how to convert a message/utf-8 for an
> MTA that does not do 8BITMIME.

So let them put an 8BITMIME to 7bit step behind their
UTF8SMTP to 8BITMIME step.  Where 8BITMIME to 7bit is
impossible any "direct" UTF8SMTP to 7bit approach is
also impossible.  

UTF8SMTP can completely ignore 7bit because it builds
on 8BITMIME.

> Utf8headers is now thought to be almost ready for WG
> Last Call.

Utf8headers is to 2822, what UTF8SMTP is to 2821.  But
Utf8headers also updating MIME is IMO too much.  We
don't have Bruce in this WG (and his proposal to join
MIME and 2822 consisted of four I-Ds).

Mailers, even if it's a gateway, forced to look into
the body of a mail (apart from finding CRLF and dots)
are in trouble.  Looking into the header is -- for a
gateway -- aceptable, but the body ?!?  It will find
crap and more crap, broken UTF-8, broken boundaries,
broken Content-* header fields, NULs, ... <shudder />

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 26 Mar 2007 21:02:56 +0300

> not GiGo, it's more like "garbage in, garbage back".

Lets look one possibility when downgrade fails.

        C   is SMTP clent
        F   is is final delivery MTA
        D   is MDA

C -> F      is SMTP
F -> D      is LMTP

         Lets assume that there is two new SMTP reply codes (used
             after final dot)

         ABC             Some recipeints do not support UTF8SMTP,
                         downgrade failed, retry with one RCPT TO
                         on transaction

         XYZ             All recipeints do not support, downgrade failed

         XYZ             LMTP:  This recipeint does not support
                                 UTF8SMTP, downgrade failed

C -> F             [ Smtp clients open connection ]
C <- F             220 mail.example.com ready
C -> F             EHLO mail.example.org
C <- F             250-mail.example.com
C <- F             250-8BITMIME
C <- F             250-SIZE
C <- F             250 UTF8SMTP
C -> F             MAIL FROM:<[email-masked]> BODY=7BIT SIZE=323
C <- F             250 Ok.
C -> F             RCPT TO:<[email-masked]>
C <- F             250 Ok.
C -> F             RCPT TO:<[email-masked]>
C <- F             250 Ok.
C -> F             DATA
C <- F             354 Start mail input; end with <CRLF>.<CRLF>
C -> F             Received: by mail.example.org with UTF8SMTP
C -> F               id JGR17356;
C -> F               Wed, 13 Sep 2007 22:27:25 +0300
C -> F             From: <[email-masked]>
C -> F             To: <[email-masked]>
C -> F             Subject: [#XCVXBX] Näppäimistö rikki
C -> F             Message-ID: <[email-masked]>
C -> F             Mime-Version: 1.0
C -> F             Content-Type: text/plain
C -> F
C -> F             Your case #XCVXBX is closed.
C -> F             .                  
     F -> D        [ final delivery MTA opens connection to MDA ]
     F <- D        220 mail.example.com ready
     F -> D        LHLO mail.example.com
     F <- D        250-mail.example.com       
     F <- D        250-8BITMIME
     F <- D        250-SIZE
     F <- D        250 UTF8SMTP
     F -> D        MAIL FROM:<[email-masked]> BODY=7BIT SIZE=437
     F <- D        250 Ok.
     F -> D        RCPT TO:<[email-masked]>
     F <- D        250 Ok.
     F -> D        RCPT TO:<[email-masked]>
     F <- D        250 Ok.
     F -> D        DATA
     F <- D        354 Start mail input; end with <CRLF>.<CRLF>
     F -> D        Received: from mail.example.org by mail.example.com with UTF8SMTP
     F -> D         id dfsdafsdf;
     F -> D         Wed, 13 Sep 2007 22:27:26 +0300
     F -> D        Received: by mail.example.org with UTF8SMTP
     F -> D          id JGR17356;
     F -> D          Wed, 13 Sep 2007 22:27:25 +0300
     F -> D        From: <[email-masked]>
     F -> D        To: <[email-masked]>
     F -> D        Subject: [#XCVXBX] Näppäimistö rikki
     F -> D        Message-ID: <[email-masked]>
     F -> D        Mime-Version: 1.0
     F -> D        Content-Type: text/plain
     F -> D
     F -> D        Your case #XCVXBX is closed.
     F -> D        .                  
     F <- D        250 Message parsed, UTF8SMTP detected
     F <- D        250 <[email-masked]> sttored
     F <- D        XYZ <[email-masked]> UTF8SMTP not accepted

     [ final delivery MTA adds triple
        ([email-masked],[email-masked],[email-masked]) = delivered
       to database

       final delivery MTA tries downgrade message, but fails

       final delivery MTA adds triple
        ([email-masked],[email-masked],[email-masked]) = downgrade failure
       to database
     ]

     F -> D        QUIT
     F <- D        221 mail.example.com closing connection

C <- F             ABC UTf8SMTP downgrade fails for some recipients
C -> F             MAIL FROM:<[email-masked]> BODY=7BIT SIZE=323
C <- F             250 Ok.
C -> F             RCPT TO:<[email-masked]>
C <- F             250 Ok.
C <- F             354 Start mail input; end with <CRLF>.<CRLF>
C -> F             Received: by mail.example.org with UTF8SMTP
C -> F               id JGR17356;
C -> F               Wed, 13 Sep 2007 22:27:25 +0300
C -> F             From: <[email-masked]>
C -> F             To: <[email-masked]>
C -> F             Subject: [#XCVXBX] Näppäimistö rikki
C -> F             Message-ID: <[email-masked]>
C -> F             Mime-Version: 1.0
C -> F             Content-Type: text/plain
C -> F
C -> F             Your case #XCVXBX is closed.
C -> F             .                  
C <- F             250 Message already delivered
C -> F             MAIL FROM:<[email-masked]> BODY=7BIT SIZE=323
C <- F             250 Ok.
C -> F             RCPT TO:<[email-masked]>
C <- F             250 Ok.
C -> F             DATA
C <- F             354 Start mail input; end with <CRLF>.<CRLF>
C -> F             Received: by mail.example.org with UTF8SMTP
C -> F               id JGR17356;
C -> F               Wed, 13 Sep 2007 22:27:25 +0300
C -> F             From: <[email-masked]>
C -> F             To: <[email-masked]>
C -> F             Subject: [#XCVXBX] Näppäimistö rikki
C -> F             Message-ID: <[email-masked]>
C -> F             Mime-Version: 1.0
C -> F             Content-Type: text/plain
C -> F
C -> F             Your case #XCVXBX is closed.
C -> F             .                  
C <- F             XYZ UTF8SMTP downgrade fails
C -> F             QUIT
C <- F             221 mail.example.com closing connection

In that example it was assumed that there is MIME parser
on MDA. That is very unlikely.  So actually it is very
usefull that there is HDR=UTF8SMTP parameter on LMTP
MAIL FROM. 

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 26 Mar 2007 21:09:32 +0200

> Lets look one possibility when downgrade fails.

Thanks for the example, I didn't know that LMTP behind ESMTP
works (or can work) "within" the SMTP session, a bit like a
"call forward verification", but here in your example after
DATA.

[...]
>      F <- D        XYZ <[email-masked]> UTF8SMTP not accepted
[...]
  C -> F             DATA           
> C <- F             354 Start mail input; end with <CRLF>.<CRLF>
[...2nd attempt for client@ ...]
> C -> F             RCPT TO:<[email-masked]>
[...3rd attempt for boss@, client doesn't try to be smart, good...]
 
> So actually it is very usefull that there is HDR=UTF8SMTP
> parameter on LMTP MAIL FROM.

Sorry, I know very near to nothing about LMTP, I can't judge
it.  If you think that HDR=UTF8SMTP helps with LMTP even if
it's not adopted by the UTF8SMTP folks here, go for it.

Frank

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Mar 2007 21:21:59 -0400

--On Monday, 26 March, 2007 21:02 +0300 Kari Hurtta
<[email-masked]> wrote:

>          Lets assume that there is two new SMTP reply codes
> (used              after final dot)
> 
>          ABC             Some recipeints do not support
> UTF8SMTP,                          downgrade failed, retry
> with one RCPT TO                          on transaction
> 

Note, fwiw, that the last three lines above violate 2821 (first
two lines must have continuation markers, at least) and that,
unless things have changed recently, many MTAs will assume that
the  return codes on all members of a continuation set will be
the same.

      john

---

From: Kari Hurtta <[email-masked]>
Date: 27 Mar 2007 06:39:14 +0300

> the  return codes on all members of a continuation set will be

You missed that this part is LMTP.   

It can not violate RFC 2821, because this is not SMTP :-)

Hovever, on that picture there was violation of RFC 2033. 
It should be

> >      F <- D        250 <[email-masked]> stored

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 27 Mar 2007 15:08:43 +0100

> outlined in the DSN I-D.

Yes it is, and Chris Newman had agreed that it would have to be  
application/message-utf-8. That was until a roomful of people in Prague  
forgot that message/utf-8 as they envisaged it was "EXPRESSLY FORBIDDEN"  
by RFC 2045.

> UTF8SMTP can completely ignore 7bit because it builds

UTF8SMTP requires 8BITMIME to work. If 8BITMIME cannot handle a  
message/utf-8, then neither can UTF8SMTP.
> MIME and 2822 consisted of four I-Ds).

Internet mail has been suffering from the 7bit mentality for the last 30  
years, and it has now got to the point where further patching is  
impossible. The whole purpose of this EAI effort it to finally get rid of  
this 7bit nonsense, and hopefully after not too many years all email  
messages will be in UTF8SMTP form and we can finally forget 7bit.

But there is only one chance to get it right, and it we leave little  
pockets of 7bit only stuff around, then it will be much harder to remove  
them later. So that means we have to fix MIME at the same time. But,  
fortunately, it is not too hard to do it, though it does require that  
downgraders must be able to descend through the whole MIME structure.  
Which is nothing new, because 8BITMIME downgraders routinely do that  
anyway.