Subject: [EAI] Your DISCUSS on draft-ietf-eai-framework-05

From: Harald Alvestrand <[email-masked]>
Date: Wed, 14 Feb 2007 10:30:21 +0100

(Apologies for my previous mail, which should have been addressed to 
Lars Eggert)

You raised an issue at IESG, with the following text:

> Jari Arkko:
>
> Discuss [2007-02-08]:
> I believe this is important work and long overdue.
> I do have a question abou the server downgrade
> behaviour, however:
>
> > Since the final delivery SMTP server (or, to be more specific, its
> > corresponding mail storage agent) cannot safely assume that agents
> > accessing email storage will always be capable of handling the
> > extensions proposed here, it MAY either downgrade internationalized
> > emails or specially identify messages that utilize these extensions,
> > or both. If this done, the final delivery SMTP server SHOULD include
> > a mechanism to preserve or recover the original internationalized
> > forms without information loss to support access by UTF8SMTP-aware
> > agents.
>
> Does this suggest that the final server downgrades mails
> without knowing that the client is uncapable of reading
> them in the internationalized form? This seems surprising.
> Can you elaborate why? And what is the identification mechanism,
> and how does it affect POP/IMAP access to the messages?
Since this is an issue with a bit of substance, I'm CCing the WG on this 
note.
The issue with POP and IMAP servers is that there is no knowing what 
kind of client software the user will use to connect to the mail store 
the *next* time it connects.

Especially in the case of IMAP, it is not uncommon to connect to a 
single mailbox using multiple clients (often including a Web client) - 
at the same time, serially, or switching randomly between them.

So not only is the final delvery agent incapable of knowing the client's 
capabilities, it wouldn't help if it knew at the time of delivery, 
because the capabilities might change at any time. And messages in 
mailstores last for years, unline messages in transit, which generally 
expire after days at most, so it is very possible that a client's 
capabilities will change over the lifetime of the message in the mailstore.

What is knowable is whether or not the *mailbox server* is capable of 
supporting UTF8SMTP; in the case of delivery via SMTP or LMTP, the 
"UTF8SMTP" extension is a good way of signalling such support.

The issues in the specific context of IMAP and POP, including 
identification mechanisms, are addressed in the drafts specific to these 
protocols; however, I think it would not be appropriate to go into 
details on those protocols in an overview document - also because those 
proposals are still under discussion by the WG, and might change 
considerably before they are finished.

I don't know if this is enough clarification of why the text is the way 
it is - do you need more information to clear your DISCUSS?

                       Harald, document shepherd

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Jari Arkko <[email-masked]>
Date: Wed, 14 Feb 2007 14:33:35 +0200

> Since this is an issue with a bit of substance, I'm CCing the WG on
> this note.

Right.

> So not only is the final delvery agent incapable of knowing the
> client's capabilities, it wouldn't help if it knew at the time of
> delivery, because the capabilities might change at any time. And
> messages in mailstores last for years, unline messages in transit,
> which generally expire after days at most, so it is very possible that
> a client's capabilities will change over the lifetime of the message
> in the mailstore.
>

Exactly. That is why it is important that we do not downgrade mail
permanently
if we can avoid it. I reacted to the text because it seemed to be saying
that. Specifically, it says the storage agent may downgrade i18n e-mail.
Does that mean that it would actually downgrade it in the storage,
or just for the purpose of serving it to this particular client at this
one time?

> identification mechanisms, are addressed in the drafts specific to
> these protocols; however, I think it would not be appropriate to go
> into details on those protocols in an overview document - also because
> those proposals are still under discussion by the WG, and might change

Fair enough.

>
> I don't know if this is enough clarification of why the text is the
> way it is - do you need more information to clear your DISCUSS?

Let me suggest a small reformulation:

Since the final delivery SMTP server (or, to be more specific, its
corresponding mail storage agent) cannot safely assume that agents
accessing email storage will always be capable of handling the
extensions proposed here, it MAY either provide a downgraded
version of the internationalized email to these agents, or
specially identify messages that utilize these extensions,
or both. In any case, the final delivery SMTP server SHOULD allow
the original internationalized forms to be accessed by UTF8SMTP-aware
agents without information loss, even if they are also accessed
in downgraded form by other agents.

Jari

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 14 Feb 2007 15:46:46 +0100

> or just for the purpose of serving it to this particular client at this
Actually there are two possible conceptual approaches, which are 
equivalent seen from the outside:

- Store the message in a downgraded format, but with enough information 
to reconstruct the non-downgraded format. This is faster for old 
clients, slower for new ones.
- Store the message in an UTF8SMTP compatible format, and downgrade at 
read time. This is faster for new clients, and slower for old ones.

Tradeoff.
> in downgraded form by other agents.
I'll have to think about whether that actually says what I want it to 
say.....

---

From: Jari Arkko <[email-masked]>
Date: Wed, 14 Feb 2007 16:56:55 +0200

Harald,
 
> - Store the message in a downgraded format, but with enough
> information to reconstruct the non-downgraded format. This is faster
> for old clients, slower for new ones.

Sure. But if these are equivalent from the outside, maybe
we should not be debating how to do it... the external
behavior matters. (See also the last sentence of my
proposed text.)

> I'll have to think about whether that actually says what I want it to

Ok.

Jari

---

From: John C Klensin <[email-masked]>
Date: Wed, 14 Feb 2007 10:23:04 -0500

Jari,

The IETF has never specified what happens between the point at
which the delivery MTA receives a message and the point at which
something that a mail reader connects to (a direct
mailbox-reader, POP or IMAP client, internal delivery mechanism
such as LMTP, etc.) might pick it up.  Our lack of
specifications in that area has been intentional and conscious,
not an accident.  But the absence of such specifications makes
it hard to specify how they should be extended for i18n email.  

We talk about "mailboxes" a lot  but they are an abstraction
without a precise definition (other than the 2821/2822
definition how they are named externally and the POP/IMAP
definitions of how they are named for access purposes).

In particular, suppose one assumes a model based on an
intermediate "mail store" (that model is _not_ required by
anything we now have standardized.  We lack protocols or
specifications for
   delivery-MTA -> mail store
   Format of mail store
   Format of data in mail store and what metadata are stored.
   API, primitives, or protocol for accessing the mail store
from (POP, IMAP, LMAP, and other protocols)

I suggest that assuming enough about those things, starting with
whether a mail store is required at all, to start specifying
i18n details is dangerous as well as hard and that putting in
further details would have us dancing on extremely thin ice.

More below.

--On Wednesday, 14 February, 2007 14:33 +0200 Jari Arkko
<[email-masked]> wrote:

>...
>> So not only is the final delvery agent incapable of knowing
>> the client's capabilities, it wouldn't help if it knew at the
>> time of delivery, because the capabilities might change at
>> any time. And messages in mailstores last for years, unline
>> messages in transit, which generally expire after days at
>> most, so it is very possible that a client's capabilities
>> will change over the lifetime of the message in the mailstore.
> 
> Exactly. That is why it is important that we do not downgrade
> mail permanently
> if we can avoid it. I reacted to the text because it seemed to
> be saying that. Specifically, it says the storage agent may
> downgrade i18n e-mail. Does that mean that it would actually
> downgrade it in the storage, or just for the purpose of
> serving it to this particular client at this one time?

If the interface between the delivery-MTA and the mailstore
(assuming there is one) is only capable of a legacy seven-bit
path, and the mailstore only offers legacy seven-bit interfaces
to things that read from it, then:

	(i) It is fairly stupid for the delivery-MTA to be
	offering these extensions at all because doing so will
	almost certainly cause a loss of information.  We can't
	prohibit that, because it is on the far side of a
	delivery boundary, but there is no question, with just
	about any protocol, that stupidity can be a source of
	problems.
	
	(ii) Because these interfaces are unspecified, all sorts
	of recoding options (not just downgrading in the sense
	specified by the WG)  are available to a mailstore that
	has some knowledge of the upgrades and intelligence
	about what is going on, even if its only storage options
	are 7bit.

But the bottom line is (i) yes, the message might be downgraded
before being stored if that seems to be the right thing to do
(ii) smart implementations won't get themselves into that
situation.

But (ii) is not unlike a ban on stupidity.

>> I don't know if this is enough clarification of why the text
>> is the way it is - do you need more information to clear your
>> DISCUSS?
> 
> Since the final delivery SMTP server (or, to be more specific,
> its corresponding mail storage agent) cannot safely assume
> that agents accessing email storage will always be capable of
> handling the extensions proposed here, it MAY either provide a
> downgraded version of the internationalized email to these
> agents, or specially identify messages that utilize these
> extensions, or both. In any case, the final delivery SMTP
> server SHOULD allow the original internationalized forms to be
> accessed by UTF8SMTP-aware agents without information loss,
> even if they are also accessed in downgraded form by other
> agents.

I am, personally, opposed to this change.  It assumes a model of
post-delivery processing that we have not standardized.  It
assumes  identification models have have been hotly debated in
the WG and concluded to be out of scope.  It imposes a
conformance requirement that is either so trivial and obvious as
to constitute a ban on stupidity or that requires the MTA
implementer to require the impossible (that the "mail storage
agent" be concurrently upgraded and in just the right way)... or
it imposes requirements on something whose existence we haven't
standardized.

At most, I think we could add a statement that says that it
would be ill-advised (and inconsistent with existing 2821
provisions prohibiting accepting mail that one expects to trash)
for a system to offer i18n extensions and mailbox names unless
it was capable of preserving the information in those messages
through to the message-receiving endpoint to the maximum limit
of that endpoints capabilities.   But, if we need to do that, I
propose, not entirely in jest, that we add a "Stupidity
Considerations" section and put this advice, and some related
points, into it.  :-(

Again, my opinion only.

regards,
    john

---

From: Tony Finch <[email-masked]>
Date: Wed, 14 Feb 2007 16:07:56 +0000

On Wed, 14 Feb 2007, John C Klensin wrote:
> something that a mail reader connects to [...] might pick it up.

I don't think this is entirely true. For example, (2)821 talks about
"final delivery" although I admit that it means little more than
appending a Return-Path: field.

> In particular, suppose one assumes a model based on an intermediate
> "mail store" (that model is _not_ required by anything we now have
> standardized.  We lack protocols or specifications for

LMTP is a counter example. It's experimental rather than standards-track,
and I understand that this is because it's supposedly just a private
arrangement within an organization. However there are multiple
interoperable implementations of LMTP, both clients and servers, and it's
normal for each end of an LMTP connection to be software from different
vendors. This is exactly what standards are supposed to be about. Also,
the "private arrangement" argument is bogus since it also applies to POP
and IMAP, and many other Internet standards.

ODMR also fits somewhere in this space - between the MX and the message
store - though it's closer to the MX than the message store. (LMTP is
closer to the message store than the MX.)

Fortunately the EAI SMTP extension should just work with both LMTP and
ODMR.

> from (POP, IMAP, LMAP, and other protocols)

I'm not sure what distinction you are getting at here. IMAP specifies in
some detail the mailbox metadata that MUAs must be able to store and
retrieve, and of course IMAP and POP are protocols for accessing mail
stores. I agree that message store formats and APIs are not topics for
IETF standardization, and this is a good thing - there's plenty of good
competition in the IMAP server space developing more efficient message
store formats.

> On Wednesday, 14 February, 2007 14:33 +0200 Jari Arkko <[email-masked]> wrote:
> >

No, this is a job for the message store (i.e. POP or IMAP server) not any
SMTP server. Message delivery into the message store (e.g. over LMTP) must
be internationalized if the message store wants to be able to serve UTF-8
email to MUAs.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
DOGGER: MAINLY SOUTHERLY 4 OR 5 INCREASING 6 OR 7. MODERATE, INCREASING ROUGH
LATER. RAIN AT FIRST. MODERATE OR GOOD.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 15 Feb 2007 12:06:16 -0000

> from (POP, IMAP, LMAP, and other protocols)

I think it is clear from the existence of RFCs defining the POP3 and IMAP  
protocols that "mail stores" undoubtedly exist (and that "mailboxes" must  
exist inside them). Moreover, those protocols pretty well constrain what  
is kept inside those stores - just that what we specify is the external  
view visible through the protocols, rather than the actual format inside  
them. And we do not expect to be able to use the IMAP protocol on a store  
designed for POP3 (though the converse might be possible).

So I see no harm in referring to "mail stores" in our documewnts.  
Moreover, since our drafts define UTF8SMTP extensions for both POP3 and  
IMAP, we should be able to use the term "UTF8SMTP extended mail store" to  
describe a mail store capable of being accessed by our extended POP3 or  
IMAP.

And then all we need to say is that if the "final delivery" MTA knows (and  
it needs to know) that it is being asked to store into an extended mail  
store, then it does not downgrade, and if it is being asked to store into  
an un-extended mail store then it MUST downgrade (or bounce). And that is  
all we should say.

So it is none of our business whether, in communicating with a supposedly  
extended mail store, the data is compressed into 7-bits in a manner which  
enables it subsequently to be restored to 8-bit (as you say, if  
implementors are determined to be stupid, then we can't stop them). So  
indeed the present wording says too much.

And likewise, it is none of the MTA's business how the mail store  
negotiates with the final user as to whether he sees an up- or down-graded  
nessage, though that IS a matter for our POP3 and IMAP drafts, and some  
mention of those drafts in the framework document might be useful at this  
point.

> post-delivery processing that we have not standardized.

It is an improvement on the present text, but I agree with you that it  
still goes too far. What we need is a text that says that it is the job of  
the mail storage agent (if it supports UTF8SMTP extensions at all) to  
arrange that UTF8SMTP-aware user agents see UTF8 and others don't. With  
reference to our POP3 and IMAP drafts.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5