Subject: [apps-discuss] apps-team review of draft-ietf-eai-rfc5336bis

From: Claudio Allocchio <[email-masked]>
Date: Wed, 22 Dec 2010 18:24:11 +0100 (CET)

I have been selected as the Applications Area Review Team reviewer for this
draft (for background on apps-review, please see
http://www.apps.ietf.org/content/applications-area-review-team).

Please resolve these comments along with any other Last Call comments you
may receive. Please wait for direction from your document shepherd or AD
before posting a new version of the draft.

Document: draft-ietf-eai-rfc5336bis
Reviewer: Claudio Allocchio (GARR, Italian Research and Academic Network)
Review Date: 2010-12-22
IETF Last Call Date:  N/A
IESG Telechat Date:  N/A

Summary:

The document specifies how to apply internationalization to some SMTP
elements, and to some Header elements which depends from the SMTP elements.
However there seem to be a discrepancy from what the abstract states, and
what is elaborated into the document details: the document also deals with
quite a number of elements which involve bodyparts, and affect the whole
email legacy system, and beyond, such as logs, traces, and wherever email
addresses are used (certificates, etc.). This seems thus to go beyond what
the summary states, even this fact is not explicitly discussed. It is
unclear thus if this "update" of RFC5321 and RFC5322 is only limited in
scope or not. My reading of the document say it goes beyond, and implicitly
affects also what is being transported into the email system. This is thus a
serious issue to be solved before the document can go to the Standard Track.
Another big issue is about blending the boundaries of servers in the
transport system (MTAs), clients (UAs) and their roles, when the document
specify that servers shall use guessing techniques from looking into
bodyparts to understand if what they're transporting is internationalized
according to this specification or not. This is IMHO a basic violation of
the role distiction in the legacy email system, and affects the MIME model
al well.

The document quite often repeats rules from other specifications, AND
re-describes these rules. This is dangerous (it makes very difficult to keep
aligned documents, and will lead to different interpretations) and not
needed.

The full specification probably needs a serious "english language" revision,
because the language issues which I specified in "Minor Issues" seems to be
a non neglegible number. I'll try to identify as many as possible and
suggest changes.

Major Issues:

This specification updates a number of SMTP and Headers fields from RFC5321
and RFC5322. However these fields are also used in Gatewaying to other email
systems, which are still existing. Section 3.7 in RFC5321 describes specific
cases. However this memo provides no clue on possible effect that
internationalized fields can have in case gatewaying occurs. Gatewaying
functions strongly rely on these values, and the introduction of these
modification should describe this scenario, too. I suggest to include a
specific section about these issues, and suggest at least a number of
possible actions which Gateways can take to survive this change.

The EAI WG charter states that in handling email message we shall be
"consistent with the long-term view that email with invalid addresses or
syntax should be rejected, rather than fixed up in transit between
submission servers and delivery servers.". However a number of sections in
the specification seem to contradict this, where they suggest actions, by
exammining message elements (in the SMTP and in the Headers and in the
Bodyparts) in order to try to deliver anyhow. This is a subtle line between
maximising the service chance to deliver, and "fixing" things during the
transaction, and it can encourage implementors to do many of these acions,
instead of stiching to the idea "not supported --> non delivery".

Major Issues by sections.

Abstract:

" This document specifies an SMTP extension for transport and delivery
    of email messages with internationalized email addresses or header
    information."

"internationalized" is not defined here in details. Sections 1 and 1.2 try 
to do this, but the language is not crystal clear: "to permit 
internationalized email addresses in envelopes, and UNICODE characters 
(encoded in UTF-8) [RFC3629] in headers." Are addresses in envelopes (SMTP 
envelopes!) also via UNICODE characters encoded in UTF-8? The reference to 
"the framework document" seem to come too late in the text to ensure the 
reader fully understand the relationship.

  3.2 The UTF8SMTPbis Extension

"An SMTP server that announces this extension MUST be prepared to
accept a UTF-8 string [RFC3629] in any position in which RFC 5321
specifies that a mailbox can appear."

s/mailbox/<mailbox>/  (it's a definition!).

The specification does NOT suggest a preferred order among the 3 possible
options when the SMTP server state it does not support UTF8SMTPbis. Having 3
"MAY" options at the same level can create a very uncertain email service
behaviour, and a very bad user's experience! This is consistent with what
happens in other cases for other "options" in the email system (like DSNs
and MDNs), but there should be at least an informative document describing
the various scenarios, and giving some guidance to implementors in order to
avoid totally incosistent behaviours.

If we do not specify a consistent procedure to try the 3 options, in many
cases we will just in practice break the concept of universal email common
service, creating islands which do not talk to each other. The 3.6.2 section
gives some suggestions about this, but this is limited to MX listed servers
for a single domain. My concern is more general. In 3.6.2 anyhow the
language should be more precise. It states "orgnisations", which something
totally undefined from a protocol point of view. MX records refer to
"domains", not to "organisations".

   3.3.  Extended Mailbox Address Syntax

The definition of

              uAtom = 1*ucharacter
                ; Replace Atom in RFC 5321, Section 4.1.2

              ucharacter = atext / UTF8-non-ascii

              atext = <See Section 3.2.3 of RFC 5322>

permits that uAtom is either an atext OR an UTF8-non-ascii... which in the
end allows a uMailbox to be a mixed contruction where some uAtom is in
atext, while some other uAtom is in UTF8-non-ascii AT THE SAME TIME, e.g. a
construct like

      [email-masked]

is allowed. However the distinction here seems to be unneeded, as ASCII is a
subset of UTF8.

Section 3.6 tries to clarify that this is not allowed; owever there is
normative language there apart on SHOULD for the domain name. I think this
shold be fixed with a clearer normative specfication.

This of course applies also to all other tokens where uAtom is used.

3.5 Body Parts and SMTP Extensions

I have problems in understanding why an SMTP server should peek extensively
into the message geing transfered to understand that it is handling an
interationalzed message. "There is no ESMTP parameter to assert that a
message is an internationalized message." in my opinion should be fixed in
another way, by ADDING an appropriate ESMTP parameter to state this! This
will solve already in the SMTP dialogue many isses which in this
specification are handeled by guessing techinques.

3.6.1 "the server normally sends..." --> "the server sends" (sorry... we do
not like ABnormal servers!) :-)

   Minor Issues:

1. Introduction

The sentence "This document use this mechanism to support an
internationalized email address." defintly needs a better form; if you do
not know already know what we are talking about, it can be misleading.

"This document uses this mechanism to allow internationalized email
addresses support in SMTP elements."

3.1 Framework for the Internationalization Extension

- bullet point 3. I do not see any need to state that clients MUST reject...
and MUST be fully compliant... etc. This is implicit in any specification.
If there is a specific need to add this detailed behaviour (to prevent wrong
implementions? why?) either we shall ensure the text is clear and
unambiguous, or we shall add an Implementers Note, as it was done in some
other RFCs.

3.2 The UTF8SMTPbis Extension

same general comment as per 3.1: in general the section states behaviours
which do not need to be specified again, because they're alredy described
seomewhere else. This might also become a Major issue if the specification
of such behaviours are changed elsewhere. Just poit to extrnal references.
(it seems editors are worried of implmentors which do not fully undesrtand
or know the other email specifications... )

3.3.  Extended Mailbox Address Syntax

What does the sentence "..., using the production for a mailbox and those
productions on which it depends." mean? I assume you meant <mailbox>, is
this correct?

5.  Security Considerations

there is one more issue: as internationalized items will also result into
logs and traces, their presence may affects trouble ticket systems used by
security operations teams (CERTs/CSIRTs). And they also can affect quick
handling of incidents, has it may require more time to correctly read and
understand logs and traces if they are not in current ascii form. This might
even be a Major Issue. I propose adding this text in the section, besides
pointing to the exteral reference:

---------------------- start of added text
Beyond the use inside the email global system (in SMTP envelopes and message
headers), internationalized email addresses will also show up inside other
scenarios, in particular:

  - the logging systems of SMTP transactions and other logs to monitor
    the email systems;

  - the trouble ticket systems used by Security Teams to manage security
    incidents, when an email address is involved

This will likely require extending support for full UTF8 also into these
systems, in order to avoid problems, which could cause also important loss
of data, or require to provide an adequate mechanism to map non ASCII
strings into them.

Another Security aspect to consider is related to the ability by Security
Team members to quickly understand/read and identify email addresses from
the logs, when they are tracking an incident. Mechanims to automatically and
quickly provide the origin or ownership of an internationalized email
address shall implemented for use also by log readers which cannot read
easily non-ASCII information.
--------------------------- end of added text

Also, in Security Consideration, there is no mention of the fact that we are
changing VRFY/EXPN commands to include internationalized addresses.
VRFY/EXPN commands can exist ALSO in an SMTP transaction which is not
supoused to transfer an email messages between MTAs. And their use has also
do to with Security. Here is an additional text for this:

---------------------- start of added text
The SMTP commands VRFY and EXPN are sometimes used in SMTP transactions
where there is no message to transfer (by tools used to take automated
actions in case potential spam messages are identified). RFC5321 section 3.5
and 7.3 give some detailed description of use and possible behaviours.
Implementation of internationalized addrsses can affect also logs and
actions by these tools.
--------------------------- end of added text

   Nits:

see
http://tools.ietf.org/idnits?url=http://tools.ietf.org/id/draft-ietf-eai-rfc5336bis-07.txt

Best Regards!

------------------------------------------------------------------------------
Claudio Allocchio             G   A   R   R          [email-masked]
                         Senior Technical Officer
tel: +39 040 3758523      Italian Academic and       G=Claudio; S=Allocchio;
fax: +39 040 3758565        Research Network         P=garr; A=garr; C=it;

            PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca

---

From: Ned Freed <[email-masked]>
Date: Wed, 22 Dec 2010 18:01:15 -0800 (PST)

I was asked to review draft-ietf-eai-rfc5336bis-07.txt. I was also provided
with a preliminary copy of the review Dave did, so these notes should be viewed
as an addendum to Dave's review, not as a standalone critique. In particular,
any failure to reiterate Dave's various points here should not be taken as
disagreement with him; on the contrary, I am in agreement with almost all of
his comments.

Now for the specifics:

(0) I cannot emphasize strongly enough how important it is to have a
    UTF8SMTP sort of parameter on the MAIL FROM to indicate that a message
    employing this extension is being submitted/relayed/delivered. This is
    especially important in the relay case - we do not want to require
    that relays grub through every message they receive looking for possible
    utf-8 so they can carry the negotiation of this extension forward, but
    that's exactly what the present proposal forces them to do.

(1) 3.1 item 4 and 3.6.4.2 add an optional UTF8REPLY parameter to VRFY and
    EXPN. They do this by redefining the syntax to be:

    vrfy = "VRFY" SP ( uLocal-part / uMailbox )
                     [ SP "UTF8REPLY" ] CRLF

    expn = "EXPN" SP ( uLocal-part / uMailbox )
                     [ SP "UTF8REPLY" ] CRLF

    The original syntax for VRFY and EXPN given in RFC 5321 is:

    vrfy = "VRFY" SP String CRLF
    expn = "EXPN" SP String CRLF
    String = Atom / Quoted-string
    Atom = 1*atext

    It's not immediately obvious, but the new syntax is in fact a superset of
    the old. (uLocal-part subsumes local-part, which in turn allows atoms
    and quoted strings.) However, this change does much more than hang a
    parameter on the command; for the first time an entire address is
    permitted as an EXPN/VRFY argument. But no semantics are given for
    this significant syntax expansion, nor does this expansion appear to be
    within scope for EAI.

    A more reasonable syntax would be:

    vrfy = "VRFY" SP uString [ SP "UTF8REPLY" ] CRLF

    expn = "EXPN" SP uString [ SP "UTF8REPLY" ] CRLF

    uString = uAtom / uQuoted-string

    But this now fails along a different dimension: When adding a parameter
    capability to a command, it should be done in a general way, so that any
    future extensions can build on a cleanly specified foundation. Like so:

    vrfy = "VRFY" SP uString [ SP vrfy-parameters ] CRLF
    vrfy-parameters = esmtp-param *(SP esmtp-param)

    expn = "EXPN" SP uString [ SP expn-parameters ] CRLF
    expn-parameters = esmtp-param *(SP esmtp-param)

    uString = uAtom / uQuoted-string

    Of course this will add 555 as a possible response to VRFY/EXPN, and that
    needs to be noted.

(2) In response to Dave's critique/query about why the parameter on VRFY and
    EXPN is needed, it's because these commands can appear outside of a
    transaction and thus there's not necessarily an indication that the client
    is prepared to accept UTF-8 responses. However, this brings up another
    issue, which is that a server that supports this extension needs a way
    to tell a client that doesn't that it cannot return a proper response
    to VRFY/EXPN without that parameter being specified. A new SMTP error
    status is needed, but the document only defines an extended status
    for this purpose; it doesn't specify a regular status code to use.

(3) Section 3.2 says:

   1.  If and only if the SMTP client (sender) is a Message Submission
       Server ("MSA") [RFC4409], it MAY, consistent with the general
       provisions for changes by such servers, rewrite the envelope,
       headers, or message material to make them entirely ASCII and
       consistent with the provisions of RFC 5321 [RFC5321] and RFC 5322
       [RFC5322].

    This is a bad idea no matter how you slice it. Such transformations
    depend on the availability of address mapping (from a UTF-8 address to
    an equivalent ASCII address) information; an agent being an MSA is neither
    necessary nor sufficient reason to have this information.

    But the bigger problem by far is defining what transformations are
    permissible. The present framework document is basically silent on
    what's allowed, so I for one have no idea what it means to be
    "consistent with the general provisions for changes by such servers"
    actually means. And unless you're willing to spell this out, I see this
    as practically guarnteeing a slew of interop problems.

(4) Also from 3.2:

  3.  It may find an alternate route to the destination that permits
       UTF8SMTPbis.  That route may be discovered by trying alternate
       Mail eXchanger (MX) hosts (using preference rules as specified in
       RFC 5321) or using other means available to the SMTP-sender.

    So you can use alternate MXes, but not alternate A records? I actually
    agree with this - an alternate A is presumably another address for
    the same host, so no point in trying it. But this needs to be spelled
    out.

    At an absolute minimum this needs to be accompanied by some
    security considerations text. The issue, I hope, is obvious: There
    are sites aplenty with large numbers of MX records and which won't
    support this extension any time soon. And all too often those MXes
    are *slow* to respond. So sending a message that causes them all to
    be tried is ... you get the idea.

    In fact I'm tempted to go one further and say that if an MX at a given
    priority is found not to support this extension, you SHOULD assume that
    MXes at lower priority don't either.

(5) Consider this text from 3.5:

   Assuming that the server advertises UTF8SMTPbis and 8BITMIME, and
   receives at least one non-ASCII address, the precise interpretation
   of "BODY=8BITMIME", and "BODY=BINARYMIME" in the MAIL command is:

   Since when are the addresses in the envelope a predictor of the addresses
   that appear in the outermost header, let alone any enclosed message/rfc822
   headers? Really, this could not be more fundamental: There's no requirement
   that envelope addresses appear in the outermost header or vice versa, and
   circumstances abound where the addresses in the header and envelope end up
   being partly or entirely disjoint. As such, while the use of UTF-8 in the
   envelope can be taken as an indication that this is an EAI message, the
   absence tells you nothing.

   Addrressing item (0) above and having a parameter on MAIL FROM addresses the
   technical side of this one, but the entire notion of coupling and
   confusion between envelope and header needs to be stamped out in this
   specification.

(6) A bunch of the references are to bis-versions, but the actual cite is to
    the current RFC.

One final comment. Several of the issues that have been raised are so
fundamental that once they are addressed (point (0) especially), this document
is almost certainly going to require another full review cycle.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Wed, 22 Dec 2010 18:03:33 -0800 (PST)

+1 to all of Claudio's points raised here.

				Ned

> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Sat, 25 Dec 2010 15:20:07 -0500

(writing as a personal opinion, not speaking for the WG.  Shawn
and others have already addressed parts of this, but, for
completeness...)

--On Wednesday, December 22, 2010 18:01 -0800 Ned Freed
<[email-masked]> wrote:

>...
> (1) 3.1 item 4 and 3.6.4.2 add an optional UTF8REPLY parameter
> to VRFY and     EXPN. They do this by redefining the syntax to
> be:
> 
>     It's not immediately obvious, but the new syntax is in
> fact a superset of     the old. (uLocal-part subsumes
> local-part, which in turn allows atoms     and quoted
> strings.) However, this change does much more than hang a
> parameter on the command; for the first time an entire address
> is     permitted as an EXPN/VRFY argument. But no semantics
> are given for     this significant syntax expansion, nor does
> this expansion appear to be     within scope for EAI.
> 

Yes.  I don't think the above mess was intentional.   In
fairness to the authors, the ABNF has been changed too many
times by too many people and I think the situation above is a
side-effect.   As you point out, the syntax changes may not be
adequate (see below).

>     But this now fails along a different dimension: When
> adding a parameter     capability to a command, it should be
> done in a general way, so that any     future extensions can
> build on a cleanly specified foundation. Like so:
> 
>     Of course this will add 555 as a possible response to
> VRFY/EXPN, and that needs to be noted.

Right. See below.

In addition, one must have a session for which mail transactions
are permitted (i.e., EHLO must have been sent) in order to use
any parameters at all.  RFC 5321, Section 4.1.4, explicitly
permits sending VRFY or EXPN without first having EHLO in the
session.  So there is another implicit requirement in all of
this that should almost certainly be made explicit.   (I think
this is the topic about handshaking that Dave and Shawn have
been circling around, but I'm not sure).

> (2) In response to Dave's critique/query about why the
> parameter on VRFY and     EXPN is needed, it's because these
> commands can appear outside of a     transaction and thus
> there's not necessarily an indication that the client     is
> prepared to accept UTF-8 responses. 

Yes, exactly.

> However, this brings up
> another     issue, which is that a server that supports this
> extension needs a way     to tell a client that doesn't that
> it cannot return a proper response     to VRFY/EXPN without
> that parameter being specified. A new SMTP error     status is
> needed, but the document only defines an extended status

If we go down this path, the WG clearly needs to reexamine
whether such a code is necessary.  I think you have made a
persuasive case, but I can't speak for the WG.

Indeed, this has all gotten sufficiently complex that I think
perhaps the WG should reexamine its decision to handle the "VRFY
and EXPN with possible non-ASCII responses" situation by
parameterizing those commands rather than introducing new
commands, say VRF8 and EXP8.  IIR, the WG did consider that
alternative (long ago) and conclude that adding a parameter was
the more straightforward option.  But, perhaps, given the
various complexities that have emerged, that was not the right
decision.  With other commands, such as MAIL and RCPT, we've
already got a history of parameters and the base (821/5321)
syntax is very clearly delimited.  By contrast, I certainly
remember seeing VRFY followed by unquoted strings with blanks in
them in the wild (e.g., arrangements like 
    VRFY Ned Freed
) and some servers tolerating them in the name of robustness or
in memory of 821, which had
    VRFY <SP> <string> <CRLF>
    <string> ::= <char> | <char> <string>
    <char> ::= <c> | "\" <x>
    <c> ::= any one of the 128 ASCII characters, 
        but not any <special> or <SP>
    <x> ::= any one of the 128 ASCII characters
       (no exceptions)
which makes
    VRFY Dave\ Crocker
perfectly valid.    I don't know if similar examples still occur
and with what frequency, but...

In addition, if we handle EAI-extended VRFY and EXPN with
parameters, we end up in the odd situation in which the
no-parameter versions can be issued without mail session
initialization (the EHLO command) but the extended,
parameterized ones require a previously-initialized session.
The MAIL and RCPT commands dn't have an issue because they
require session initialization anyway.  That restriction on
extended VRFY and EXPN certainly works technically, but isn't
going to be the easiest thing to explain and might cause some
"interesting" behavior if implementations screw it up or
clients, having not sent EHLO, assume that they are talking with
legacy 821 servers.   By contrast, a rule that requires session
initialization prior to using a new command is a no-brainer to
specify and implement cleanly, with no excuses about, e.g.,
ancient legacy environments.

I would welcome your thoughts on this tradeoff between adding
new commands (whose syntax would be identical to 5321 VRFY/EXPN
but expanded to UTF-8 input and/or output) and the current
proposal for a parameterized multiple-case situation, presumably
fixed up more or less as you have suggested.

    john

---

From: Ned Freed <[email-masked]>
Date: Sun, 26 Dec 2010 14:53:46 -0800 (PST)

> any parameters at all.

You certainly need to send an EHLO and process the result to find out what
extensions are allowed - guessing what combination of extensions will work most
certainly is not a sound operational policy. And implementations are certainly
free to only offer the extensions that have been returned in an EHLO repsonse.

That said, I don't see an actual requirement in section 2.2 stating that EHLO
must be issued before any parameters can be used. (Maybe it's somewhere else,
but a cursory search didn't find it.) Our MTA certainly doesn't enforce this
and AFAIK neither does sendmail, PostFix, Apache James, or qmail. (Exim, OTOH,
appears to.)

> RFC 5321, Section 4.1.4, explicitly
> this that should almost certainly be made explicit.

There certainly should be a note saying that you need to issue an EHLO
and check the response before attaching any parameters to VRFY/EXPN. I don't
think a requirement that parameters not be accepted before an EHLO
has been issued is needed.

> (I think

It's part of it, I think.

> persuasive case, but I can't speak for the WG.

It doesn't have to be a new code. OTOH, it needs to be a code that EXPN/VRFY
don't return for other purposes, and I think if you look at the list of
available codes a new code is needed.

> decision.

I considered suggesting new commands, but given the need to specify
syntax for the new commands I don't see this as any simplier
specification-wise. In terms of implementation, a new command is actually
a somewhat more complex - the code changes are less localized, and I suspect
other implementations will be comparable in this regard.

But the minute you need some other variant on EXPN and VRFY, the alternate
command appraoch becomes the far more complex way to do it. Now, we haven't had
a specification come along that needed additional EXPN/VRFY parameters before
this one, but these things have a way of showing up once you open the door and
get people thinking about them. So I'm strongly inclined to stick with the
parameter approach.

> With other commands, such as MAIL and RCPT, we've

OTOH, a lot of servers back in the days prior to RFC 2821 didn't support the
821 thing of allowing literal spances with a \ and no enclosing quotes. So this
never could be said to have interoperated. And I think the time has come to
abandon it (although I note that if you really want to continue supporting it,
the added parameter doesn't prevent you from doing so).

				Ned

---

From: "Al Costanzo" <[email-masked]>
Date: Mon, 27 Dec 2010 00:00:09 -0500

> the added parameter doesn't prevent you from doing so).

Ned,

If my memory servers me, (and it may not),

There were good reasons why VRFY supported this.  I dont know if any are 
valid anymore but one idea was that you could send the persons name, not the 
email address, to find out if that person had an email address on that 
system.

Was this not the reason Ned to allow blanks (whitespace) and unquoted 
strings?  If you remove this from the spec, this functionality would be 
lost, I don't really see a reason to lose something that works.

>> this that should almost certainly be made explicit.

I do not remember VRFY command requiring the EHELO, there are a number of 
mailing list verifiers that do not send the EHELO first. Verifiers, since 
they do not actually send email, usually dont send the EHELO first. (our 
application does) and I think requiring the EHELO first as part of a 
handshake it the proper way to do this.

Only my humble opinon.

-Al Costanzo

----- Original Message ----- 
From: "Ned Freed" <[email-masked]>
To: "John C Klensin" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>; "Ned Freed" 
<[email-masked]>; <[email-masked]>; 
<[email-masked]>
Sent: Sunday, December 26, 2010 5:53 PM
Subject: Re: [apps-discuss] [EAI] RFC5336 and VRFY/EXPN (was: Re: apps-team 
review of draft-ietf-eai-rfc5336bis)

> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

---

From: Ned Freed <[email-masked]>
Date: Mon, 27 Dec 2010 08:17:27 -0800 (PST)

> email address, to find out if that person had an email address on that

You've gotten this exactly backwards. Yes, VRFY and EXPN accept names as
arguments, not addresses. The problem is this specification changes them to
accept addresses as well, which is both unnecessary and out of scope.

> strings?

Sure, but it doesn't change the fact that the only changes this WG is supposed
to be making is to add support for UTF-8. Even if it were a good idea to change
the syntax of VRFY or EXPN from what's in RFC 5321  either to accept addresses
or to go back to the syntaxes that were allowed in RFC 821, doing so is not in
scope. If you want to argue about changes in that area, the place for that is
the ietf-smtp list.

> If you remove this from the spec, this functionality would be

Please reread my original message. Again, you have this exactly backwards -
currently the EAI specification makes an unnecessary and out of scope syntax
change change from what's in RFC 5321 that goes far beyond supporting utf-8.
Either form of alternative ABNF I proposed returns us to RFC 5321 syntax plus
utf-8.

> mailing list verifiers that do not send the EHELO first.

That's exactly what John is saying. The problem is you can't send a new
parameter on VRFY/EXPN and expect it to work until you know it's safe to do so.
And the way to know that is to look at the EHLO response.

> Verifiers, since

Which is what is being proposed, and AFAICT is entirely noncontroversial. The
specific issue being discussed is whether or not a server MUST refuse to allow
the extended form if it hasn't gotten an EHLO. I see no point in containing
servers in this fashion.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Mon, 27 Dec 2010 13:49:06 -0500

--On Monday, December 27, 2010 12:00 AM -0500 Al Costanzo
<[email-masked]> wrote:

> There were good reasons why VRFY supported this.  I dont know
> if any are valid anymore but one idea was that you could send
> the persons name, not the email address, to find out if that
> person had an email address on that system.

and what that address was, of course.

This was heavily used for a while with both VRFY and, separately
with finger, to get from name to email address.  The advantage
of the VRFY approach was that it would permit a user to enter an
address that might look like "John [email-masked]" at which
point the MUA would send 
   VRFY John Klensin
to the mail server at example.com, get back an address, and then
open a mail session.

Of course, Ned's observations about lousy support (with which I
agree) notwithstanding, the variation of 
   VRFY "John\ Klensin"
would have been fully 821-compliant.  I vaguely remember one MUA
that would send the first (escape-less) version and then send
the second if the reply from the first was "no 'John; here" or
"syntax error" rather than "no 'John Klensin' here".  The
heuristic is pretty trivial.

That use of VRFY became less popular as MUAs became more
separate from first-hop full service MTA (aka submission
servers) because that sort of use of VRFY pretty much requires
at least some SMTP capability (including domain name lookups,
coping with MXs, etc.) in the MUAs, ability to use port 25
connections from just about anywhere, etc.   I don't know
whether the usage is gone entirely, but it is certainly a lot
less common than it appeared to be many years ago.

> Was this not the reason Ned to allow blanks (whitespace) and
> unquoted strings?  If you remove this from the spec, this
> functionality would be lost, I don't really see a reason to
> lose something that works.

Especially if the clock could be turned back, one could allow
only the single form

  VRFY "John Klensin"

just getting rid of the bare, unquoted, backslash escape that
was permitted by 821 but that 5321 already excludes.

> I do not remember VRFY command requiring the EHELO, there are
> a number of mailing list verifiers that do not send the EHELO
> first.

It never has been a requirement although sending EHLO (or HELO)
first has always been harmless.  Either 5321 or 2821 simply made
it very clear that there was no such requirement.

> Verifiers, since they do not actually send email,
> usually dont send the EHELO first. (our application does) and
> I think requiring the EHELO first as part of a handshake it
> the proper way to do this.

That is one issue on the side of "new command".  If one now
decides to require EHLO first, then all existing uses of VRFY
without it (which conform perfectly well to 5321 and all of its
predecessors) become invalid,  That is pretty uncomfortable to
me.  By contrast, if one creates the new command, then, if the
response to EHLO includes some sort of EXTENDED-VRFY capability
(see below),  one could have:

	-- VRFY without EHLO as requesting the 5321 behavior
	(with or without non-conforming extensions or
	flexibility and with responses limited to ASCII).
	
	-- EHLO, then VRFY as requesting the 5321 behavior,
	identical to that above.  
	
	-- EHLO, then New-VRFY as requiring a strict Atom or
	Quoted-string optionally followed by a parameter list.

Note that, if the non-conforming client behavior we know is out
there is used, then 
    VRFY Costanzo 8BITREPLIES-OK
or equivalent could be interpreted by the server as either of
    VRFY "Constanzo\ 8BITREPLIES-OK"
or as coming with a parameter, or as a syntax error, in either
VRFY case.   Not really a good idea, IMO, and at least one
reason we have not permitted the use of extended parameters with
older SMTP commands with EHLO and capability permission.

Note too that the last paragraph of RFC 5321 Section 3.5.1 is,
in part and while not explicit, a caution against treating the
standard syntax too literally or narrowly. 

We could, in principle, do a completely new command without
involving the extension mechanism at all, thereby allowing
sending it before or without EHLO, but that strikes me as a
really scary idea.

> Only my humble opinon.

Only mine also -- I really don't have my mind made up about this
and am just trying to get the various arguments in front of the
WG.

     john

p.s. I agree with what I take to be at least a corollary to
Ned's remarks: having EAI create special-purpose commands that
are not very narrowly tied to i18n issues would be a bad idea.
If there is any perceived need for a new command, we ought to
define in a separate I-D, as (impossible keyword is deliberate
to clarify that this is not a proposal)
   NewVRFY String *(SP esmtp-param)
with String given only the 5321 interpretation, not the 821 one,
and then process that command through the SMTP list, etc.

If that appeals to anyone, I might generate a strawman just to
facilitate the discussion.

---

From: John C Klensin <[email-masked]>
Date: Mon, 27 Dec 2010 14:39:15 -0500

Hi.

Long, careful, and thoughtful reviews deserve long --and I hope
careful and thoughtful-- responses.  This note attempts to
identify the main substantive points in the reviews and react
to them.  Some may consider it a special case of no good deed
going unpunished; if so, I'm sorry.

I'm sending the note separately to the EAI WG mailing list
([email-masked]) and the Apps Discuss one.  I hope that we can
move discussion to the WG list.  Put differently, please do not
respond to this note on the apps-discuss list -- if anyone has
additional comments, please take them to the WG list.  My
apologies to those who will get it twice -- there seemed no
better way to avoid, or at least reduce, multiple threads of
discussions that were not visible to WG participants.

Writing only as a participant in the WG and co-author of the
"EAI framework" document, I have several comments on the
various Applications Area Review Team comments (my co-chair
helped a bit with part of this note but I'll take the blame
as needed).  To be certain we all start with the same
background assumptions, please remember that the original form
of the "EAI Framework" supported the Experimental version of
these protocols document was published in July 2007 as RFC
4952.  Its replacement, draft-ietf-eai-frmwrk-4952bis-10, is
more to the point today.  That draft is intended to be aligned
with the documents that are now under consideration (or to have
them aligned with it), has completed IETF Last Call, was
approved for publication by the IESG in late September, and has
been in the RFC Editor queue awaiting normative references to
the current set of documents and resolution of a naming issue
that the WG decided to leave until after the approval process
completed (more on that below).

Speaking very personally because I don't know whether the WG
participants will agree, I think these specifications are
important enough that I'd rather have them right than stand on
any sort of procedural argument.  However, I think the
community also needs to consider when, newly-discovered
showstoppers aside, it becomes too late to raise fundamental
design issues that were reviewed extensively by the WG prior to
forwarding the first-round documents to the IESG, went through
IETF Last Call in 2007 and 2008, have been implemented and
tested experimentally, and where the basic framework and
changes from the 2007-2008 specs were reviewed by the WG,
passed through IETF Last Call, and were approved by the IESG in
September of this year.

There are places in the comments below where I've tried to
summarize what I believe is WG opinion or WG consensus, but
even those remarks should be taken as personal opinion rather
than the formal opinion of the WG.  The WG has deliberately not
been given a preview of these notes and is seeing them for the
first time at the same time as reviewers and others.  I do hope
that some of the notes will act as a basis for discussion.

For convenience, I'm going to try to consider the review
comments by topic (there is some overlap among reviews) rather
than responding to each review comment separately.  When it
seems useful to identify particular comments, I've used the
first name of the author of the comments and the date of the
posting.  To avoid a hugely long note, this one is strictly
about 5336bis (the SMTP extensions); a separate note will be
posted about each of the other relevant documents.  Note that,
again speaking personally, I agree with some of the comments
even though WG consensus, as reflected in the current drafts,
does not.  

I have not commented on, or even listed, specific issues whose
consequences appear to be entirely editorial, even if the
editorial issues are quite broad.  I'll leave those comments to
the authors/editors (subject, as usual, to WG approval).  I
have also omitted most issues that seem to have been more than
adequately covered in mailing list discussions.

The balance of this note is about the reviews and on-list
comments about draft-ietf-eai-rfc5336bis.  It does not include
comments that appeared to be redundant unless listing different
versions seemed to add perspective or other information.  While
I've commented briefly on the VRFY and EXPN issues below, those
are being much better covered in the discussions between Dave
and Shawn, Ned's analysis, and the subsequent than I could do
or have attempted here.  Separate notes about the other reviews
will follow.

(1) Overview, scope, and summary:

Claudio Allocchio 2010-12-22 wrote: 

> The document specifies how to apply internationalization to
> some SMTP elements, and to some Header elements which
> depends from the SMTP elements.  However there seem to be a
> discrepancy from what the abstract states, and what is
> elaborated into the document details: the document also
> deals with quite a number of elements which involve
> bodyparts, and affect the whole email legacy system, and
> beyond, such as logs, traces, and wherever email addresses
> are used (certificates, etc.). This seems thus to go beyond
> what the summary states, even this fact is not explicitly
> discussed. It is unclear thus if this "update" of RFC5321
> and RFC5322 is only limited in scope or not. My reading of
> the document say it goes beyond, and implicitly affects also
> what is being transported into the email system.  This is
> thus a serious issue to be solved before the document can go
> to the Standard Track.

With the understanding that the abstract, and several other
editorial issues with the text, could certainly be improved,
please note that specifications for header fields that contain
trace-related information (and that are inserted by MTAs) have
appeared in all of RFCs 821/ 2821/ 5321.  The trend over time,
starting with some explicit DRUMS decisions, has been to move
more of that information into the x21 member of the pair rather
than having it split between x21 and x22.  So the choice with
this document is between updating information that appears in
5321 and creating a split between envelope and header
information that is different than the base email documents.
The WG concluded that tracking the organization and information
content of 5321 would be less confusing than trying to create a
new model.

By contrast, the information about certificates, etc., was
introduced into the predecessor of this document at the
insistence of the community and the IESG during Last Call on
RFC 5336.  We could take it out of here, possibly by reopening
the Framework document and inserting additional "other
consequences for email and the Internet" sections there.  FWIW,
my personal opinion is that doing so would not add
significantly to either document comprehensibility or protocol
quality, but that is a matter for WG and community consensus.

Dave Crocker 2010-12-22 wrote:

> * Beyond <encoded-word> -- The work, here, appears to have
> two goals. One is to add support for Unicode in a
> <local-part>; that is, support for internationalized
> addresses.  However note that <encoded-word> and <A-label>
> already accomplish this in a way that is transparent to the
> existing email infrastructure; only the end-systems need to
> understand it. 

When the discussions that led to the establishment of the EAI
WG started, the goal was seen as expanding the syntax of
<local-part>, as you suggest.  It turns out that doing that is
not really feasible, at least without leading to horrible
kludges, so the WG was chartered (now four or five years ago)
to extend header fields as well.  As mentioned elsewhere, the
IETF has had repeated opportunities to review those decisions
and has formally approved them.  Even mapping back and forth
between A-labels and U-labels is, in the real world, not as
transparent as one would like (or obtain if one knew that
everything that looks like a domain name is a domain name in
the public DNS -- see draft-iab-idn-encoding or review the
slides from the Hiroshima technical plenary for more details on
that issue).  

Curiously, if the WG had not been constrained by very old (and
necessary) rules about how the email environment works, its job
might have been much more simple and more in line with at least
one extrapolation from the comment above: if we had
standardized local-part formats sufficiently to avoid having to
prohibit interpreting or tampering with those fields enroute,
the WG could, in principle, have written extremely short
specifications allowing encoded-words (possibly restricted to
encoded UTF-8) in local-parts and essentially been done.  While
I doubt that solution would have been acceptable to many
implementers and user constituencies (in part because "only the
end-systems need to understand it" has never worked as well as
we often hope it does), it would have been much easier in terms
of specification simplicity.

In case it is useful for understanding the WG's choices in this
area, the WG in principle had a choice between adopting a
minimal change --relying as much as possible on trick encodings
and dependencies on endpoint handling as possible -- and moving
in the direction of relatively comprehensive use of
natively-encoded (see below) Unicode in email with some
transition provisions that would gradually atrophy.  In
practice, there was no choice at all: internal discussions made
it clear that a standard based on the former approach would
simply be ignored and localized or proprietary solutions
adopted and only one based on the latter had an reasonable
chance of implementation and global deployment.  Extrapolation
from the independent-of-EAI work reflected in
draft-iab-idn-encoding -- work that originated in the concerns
of a couple of operating systems that dominate the network
today-- also suggests that more dependency on exotic,
IETF-specific encodings is not a direction we should be going
in unless it is absolutely necessary.

> The second goal is to support Unicode in the
> more "native" form of UTF-8. (The quotation marks are
> because UTF-8 is not native Unicode, either; it is a highly
> encoded form of Unicode...)

As others have commented on the mailing list, the above and
some of the comments that followed in the review demonstrate a
fairly profound misunderstanding of Unicode and Unicode
terminology.  The only encoding form more "native" than UTF-8
(or UTF-16 or UTF-32) is a set of abstract integers that have
no intrinsic computer encoding form.  The IETF has established
a number of "non-native" encoding forms, including UTF-7 and
the Punycode-derived ACE form, but they are avoided where
possible in these specifications.

See the comments under #7 below for an additional aspect of the
ASCII-versus-Unicode distinctions.

(2) Explicitness versus discovery in MIME and related topics,
including a possible parameter on the MAIL command.

Claudio Allocchio 2010-12-22 wrote: 

> Another big issue is about blending the boundaries of
> servers in the transport system (MTAs), clients (UAs) and
> their roles, when the document specify that servers shall
> use guessing techniques from looking into bodyparts to
> understand if what they're transporting is internationalized
> according to this specification or not.  This is IMHO a
> basic violation of the role distiction in the legacy email
> system, and affects the MIME model al well.   

Yes, except that this "blending" already occurs with MIME and
other SMTP extensions.  Over the years, we have made some
decisions to be very explicit about parameters and
pre-identification of content and other decisions to force
systems to inspect or parse content because we did not believe
that such parameters could be trusted (or that any sensible
implementer would trust them).  I hope we have gotten most of
those historical decisions right although I have some doubts,
at least in retrospect, about some of them.

Given that history, I don't think it is appropriate to assume
that the EAI effort is leading us into a role distinction sin
or not because we are already there.  The question is what the
right choice is for this particular situation.  The WG
considered that choice at great length (albeit in the 2006-2008
period rather than more recently, but that part of the
specification has not changed) and came to the conclusion
that forcing inspection was the better plan.  Testing with the
Experimental versions of the protocols did not expose serious
problems with that choice. 

I am sure that the WG would be happy to reconsider this topic
is appropriate, but some solid evidence that there is a problem
and that the current decision is wrong would be far more
helpful in that regard than assertions based on a philosophy of
email that IETF protocols have already violated.  That
assertion, and some evidence, have been made for a MAIL command
parameter:

> I have problems in understanding why an SMTP server should
> peek extensively into the message geing transferred to
> understand that it is handling an interationalzed message.
> "There is no ESMTP parameter to assert that a message is an
> internationalized message." in my opinion should be fixed in
> another way, by ADDING an appropriate ESMTP parameter to
>...

Dave Crocker 2010-12-22 wrote:

> 4.  The SMTP extension needs to have the client use an
> explicit signal when it is sending a message encoded in
> UTF-8. This is easiest as a parameter to the MAIL command.
> The current specification creates a more complex and almost
> heuristic model for distinguishing ASCII from UTF-8 use.

Ned Freed 2010-12-22 wrote:

> (0) I cannot emphasize strongly enough how important it is
> to have a UTF8SMTP sort of parameter on the MAIL FROM to
> indicate that a message employing this extension is being
> submitted/relayed/delivered.  
>...

Ned Freed 2010-12-23 wrote (small excerpt):

> Again, headers don't have the proper semantics. A MAIL FROM
> parameter, OTOH, does.

The working group did consider the tradeoffs between (i)
forcing a server (really relay) to do a complete MIME parse if
it received an extended message and had to decide what to do if
the next hop did not support the extensions and (ii) putting in
a parameter whose presence or absence might not be trustworthy
given a history of, e.g., text/plain body parts being sent with
distinctly non-ASCII content but without charset parameters,
misuse of "name extensions" in lieu of adherence to
Content-Type, etc.  The decision of the WG was reflected in RFC
5336 (after IETF Last Call, etc.) and I note that none of the
known implementers of the RFC 5336 spec are objecting to that
decision now.  The consensus of the reviewers now seems to be
that the WG got it wrong.  The WG needs to review this again;
the change is certainly easy to make should it be appropriate.

(3) Fundamentals of handling syntax changes from other
specifications (especially 5321/5322):

Claudio Allocchio 2010-12-22 wrote: 

> The document quite often repeats rules from other
> specifications, AND re-describes these rules. This is
> dangerous (it makes very difficult to keep aligned
> documents, and will lead to different interpretations) and
> not needed.

The tradeoff, I hope obviously, is with the reader having to
flip back and forth repeatedly between documents to have any
hope of understanding the specification accurately.  I'm
personally sympathetic to your concern, but many WG
participants disagree. 

Specific suggestions that could be used as a model would be
helpful to the WG and the editors.

Claudio Allocchio 2010-12-22 wrote: 

> 3.6.1 "the server normally sends..." --"the server sends"
> (sorry... we do not like ABnormal servers!) :-)

This is another example of a narrative/ informative comment.
We know about exceptions.   Perhaps the sentence would be
better stated as "an RFC 5321-conforming server sends...".

(4) Gateways and related boundary systems

Claudio Allocchio 2010-12-22 wrote: 
 
> This specification updates a number of SMTP and Headers
> fields from RFC5321 and RFC5322. However these fields are
> also used in Gatewaying to other email systems, which are
> still existing. Section 3.7 in RFC5321 describes specific
> cases. However this memo provides no clue on possible effect
> that internationalized fields can have in case gatewaying
> occurs.  Gatewaying functions strongly rely on these values,
> and the introduction of these modification should describe
> this scenario, too. I suggest to include a specific section
> about these issues, and suggest at least a number of

I'm not sure I understand why Gateways are different from
anything else.  If a gateway doesn't advertise the extension,
then it should never see a message that contains either
non-ASCII envelope addresses or non-ASCII content in the
headers (other than as encoded-words, of course).   The WG does
have some placeholder benchmarks for advice of various sorts;
IMO it would be quite useful to supply some of that advice for
gateways once sufficient operational experience had
accumulated.

But the only really hard gateway problem I see is if a given
gateway MTA interfaces to two back-end systems, one of which is
ASCII-only and one of which is broadly UTF-8 capable.  It will
either need to be reconfigured into two virtual gateways, one
of which advertises the extensions and one of which does not,
or it will simply have to reject mail with UTF-8 header fields
that is addressed to a system that can't handle it.  I note,
however, that such a gateway already has a problem that it
presumably has some mechanism to deal with: if it receives mail
with UTF-8 (or other non-ASCII) headers from one of its
i18n-capable systems with an Internet destination, it already
has to reject the message or perform some sort of conversions
that are, necessarily, not described in IETF specifications.

(5) The legacy of the Experimental protocols

Claudio Allocchio 2010-12-22 wrote: 

> The EAI WG charter states that in handling email message we
> shall be "consistent with the long-term view that email with
> invalid addresses or syntax should be rejected, rather than
> fixed up in transit between submission servers and delivery
> servers.". However a number of sections in the specification
> seem to contradict this, where they suggest actions, by
> exammining message elements (in the SMTP and in the Headers
> and in the Bodyparts) in order to try to deliver anyhow.
> This is a subtle line between maximising the service chance
> to deliver, and "fixing" things during the transaction, and
> it can encourage implementors to do many of these acions,
> instead of stiching to the idea "not supported --non
> delivery".

The experimental protocols described in RFC 5336 and elsewhere
included a rather complex mechanism for in-transit downgrading.
I'd encourage you to go look it up if you are interested, but
the general design was based on the 8BITMIME model -- a sender
discovering that the next-hop server wouldn't accept the option
could either reject the message or engage in a downgrading
process that was distinctly not lightweight.  The conclusion
from implementations and experiments was that it was too
heavyweight, too hard to get right, and that the number of
cases in which the downgrade procedure could not be applied
would be very confusing to users.  All vestiges of that
machinery should have been removed from the current versions of
the documents.  But I found some in an earlier review and it
may well be that some of the other review comments have
identified others.  If so, they clearly need to come out; the
editors may need help in identifying specific examples.

(6) Server actions when UTF8SMTP[bis] is not supported (and
Section 3.6.2).

Claudio Allocchio 2010-12-22 wrote: 

> The specification does NOT suggest a preferred order among
> the 3 possible options when the SMTP server state it does
> not support UTF8SMTPbis. Having 3 "MAY" options at the same
> level can create a very uncertain email service behaviour,
> and a very bad user's experience! This is consistent with
> what happens in other cases for other "options" in the email
> system (like DSNs and MDNs), but there should be at least an
> informative document describing the various scenarios, and
> giving some guidance to implementors in order to avoid
> totally incosistent behaviours.

Small bit of calibration about this: remember that the EAI
specs cannot do anything to dictate the behavior of an SMTP
server that does not support the EAI extensions and may have
never heard of them.  There is also a general design
assumptions, shared with other [E]SMTP assumption, that the
specification should not make any changes to general email
behavior that are not necessary consequences of what the
extension is trying to accomplish.

On rereading the document after going through these comments,
it is clear to me that the document would be improved by being
much more clear about what text is a change with EAI and
normative, what is a necessary consequence of provisions in RFC
5321 or other applicable standards, and what is just
informative narrative.  I don't know whether the WG shares that
point of view, nor whether it is desirable to try to rewrite
the document along those lines at this stage (in general, these
are not "known technical defects" so, if we could be assured
that Draft Standard versions of the documents would appear on a
timely basis, it might make sense to let them go for the
present).

These three options are simple extrapolations of rules that
already exist in RFC 5321: SMTP-senders must not send invalid
messages into the public Internet and must reject or bounce
such messages if they cannot be fixed up; only Submission
servers (and gateways) are permitted to make significant
changes to fix things up; and, implicitly, if mail is not
deliverable along the normal path and one knows a legitimate
path that would work, one is permitted to use it.  

Note that the first (MSA) option is useful for non-ASCII
address local-parts iff the MSA has an external source of
information that permits substituting ASCII-only addresses for
non-ASCII ones.  For address domain-parts and miscellaneous
header fields, there are typically encoded-word (or A-label)
options available.  Perhaps the document should be more
explicit about that.

If that flexibility "just in practice break[s] the concept of
universal email common service, creating islands which do not
talk to each other", then we would already have seen that
problem.  For example, the specification of 8BITMIME can lead
to the same situation without EAI: if a relay receives a
message with C-T-E 8bit or binary, and the first-preference
reachable MX doesn't support that option, it can downgrade the
message, reject it, or, IIR, try other servers at the same
preference level to see if one of them will provide support for
the extension.

Ned Freed 20101222 wrote:

>> 1.  If and only if the SMTP client (sender) is a Message
>> Submission Server ("MSA") [RFC4409], it MAY, consistent
>> with the general provisions for changes by such servers,
>> rewrite the envelope, headers, or message material to
>> make them entirely ASCII and consistent with the
>> provisions of RFC 5321 [RFC5321] and RFC 5322 [RFC5322].

> This is a bad idea no matter how you slice it. Such
> transformations depend on the availability of address
> mapping (from a UTF-8 address to an equivalent ASCII
> address) information; an agent being an MSA is neither

However, RFC 5321 already requires that an MSA not inject a
message into the public Internet until/unless it is valid.  And
it (and RFC 4409) effectively give the MSA permission to do
anything an MUA can do in order to accomplish that.  There is
no (intentional at least) assertion in 5336bis that the MSA (or
the MUA, for that matter) must have access to address mapping
information, just no requirement that it not use that
information if it has access to it.  If it doesn't have the
needed information, then there are clearly a lot of things it
cannot do; the quoted statement above is then just a
reiteration of the 5321 prohibition on passing trash into the
network (whether it is worth reiterating is a separate matter).

Of course, if a message is transferred from an MUA to the MSA
that contains non-ASCII header information in UTF-8 form but no
non-ASCII addresses, the above provisions don't imply the MSA
doing anything more than mapping that header information into
encoded words.  Again, there is no requirement that the MSA do
that mapping; the statement simply reiterates, in an i18n
context, the general permission for the MSA to take something
that works in a user->MUA->MSA environment and fix it to
conform to public Internet specifications.

It is worth noting that, during WG discussions, several strong
claims were made about intentions to implement both UTF-8 to
encoded word mappings and tables to support precisely the type
of address mapping suggested above in selected MSAs.  Assuming
people were telling the truth (and I certainly assume they
were), the conditions specified above, including address
mapping, are going to be exercised.

> But the bigger problem by far is defining what
> transformations are permissible. The present framework
> document is basically silent on what's allowed, so I for one
> have no idea what it means to be "consistent with the
> general provisions for changes by such servers" actually
> means. And unless you're willing to spell this out, I see
> this as practically guarnteeing a slew of interop problems.

With the understanding that my comments above go much further
than the existing text, that the WG hasn't agreed to any
expansion, and that Dave seems to be arguing for less text that
is just repetition of existing rules in 5321/5322, how much
further would you suggest going in spelling things out?

(7) Suboptimal use of ABNF and ABNF constructions

Dave Crocker 2010-12-22 wrote:

> * Rule Meta-Naming -- The 'u' preface for revised rules is a
> reasonable idea, but appears to be problematic.  These rules
> replace existing rules in other specifications. There needs
> to be an explicit and decision about the handling of this,
> and it needs to be applied consistently, either directly
> replacing the rules or else re-naming them consistently,
>...
> { On reflection, I think the "u" rule-naming convention is
> problematic.  If the current specification is replacing a
> rule previously defined elsewhere, it needs to use the same
> rulename.  This is simpler and clearer.

(and several comments by others, noting that Dave's second
recommendation appears to contradict Claudio's)

As I commented at more length in a note to Dave, Claudio, and the
EAI list on 2010-12-24 14:20 -0500, we've had a lot of trouble
getting ABNF for these documents right, much less optimal.  A
significant part of the optimality problem is that that there
seems to be no established good way to update one specification
by extending its syntax -- a problem that the IETF community
will almost certainly have to deal with again as we
internationalize more protocols.

In the particular case of ASCII and UTF-8 constructions, even
though ASCII is a compatible proper subset of UTF-8, there
is often a need to distinguish the two because plain-ASCII
works in legacy protocols (and does not require special
normalization or comparison rules that might, themselves, be
controversial) while UTF-8 does not work in the legacy cases
and raises more complex normalization, comparison, and
collation issues.   FWIW, we had the same problem in the
IDNA2008 specs because it was necessary to distinguish between
a traditional DNS "preferred syntax" name (aka "LDH") that did
not start with an ACE prefix and one that did use that prefixed
form, in spite of the fact that the latter, by design, would be
treated as a traditional "LDH" name by any application that was
not aware of the special IDNA constructions.

For the particular case of EAI, the WG decided to go with the
"u" prefix on new rules that superseded 5321 or 5322 rules.
The WG tried to be consistent with that prefix on new rules,
and preserve most of the original rules.  All rules were
extended rather than overwritten.  The WG had discussed,
somewhat earlier, the possibility of simply coping the entire
ABNF collection from RFC5321/5322 and then extending or
modifying rules as needed.  That would have eliminated the need
to reference back to those documents for ABNF but the idea was
rejected, largely to avoid exactly the problem with updating
that several reviewers commented on.

Again, this is a general problem: if the community has a clear
preference as to how it should be dealt with, I'm nearly sure
that the EAI WG will conform.  In the absence of a general
solution, my personal guess is that the "u" convention (applied
accurately and consistently) may turn out to be the best we can
do.

(8) MIME restrictions and nested encodings

Dave Crocker 2010-12-22 wrote:

> 5.  The SMTP extension needs to remove all restrictions it
> imposes on MIME content-type.  A major reason that MIME was
> successful was that it was transparent to the transfer
> infrastructure.  The current SMTP extension specification
> changes this model, which actually increases the barrier to
> adoption of Unicode in email.  It needs to be easy for two
> MUAs that support Unicode in the email header to exchange
> mail even when the infrastructure does not support UTF-8.

Right.  Unfortunately, there is already a restriction in MIME
that links Content-Type to Content-Transfer-Encoding, the "no
nested encodings" rule.  That rule is imposed primarily as a
restriction on message/ subtypes.  And it works because MIME
assumed that all header information was going to be ASCII and
hence able to be transmitted in C-T-E: 7bit.

So now we have a little problem: MIME is designed to permit
forwarding messages in encapsulated form using message/rfc822.
But message/rfc822 uses ASCII headers only and has to obey the
"no nested encodings" rule.  One would like to be able to
encapsulate an EAI-conformant message and even forward it into
a legacy environment.  There are actually a number of options
for doing that, including deciding that a message with
non-ASCII headers cannot be forwarded into a 5321/5322
environment (pretty obviously a bad idea to prohibit that),
creating a new top-level MIME type with different rules (the
transition issues and implications for legacy MUAs are pretty
astonishing), or creating a new "message/" subtype with
different rules and what may be seen as an additional
Content-Type/ Content-Transfer-Encoding entanglement.  After
considerable discussion, the WG selected the latter (see #13
below) and got IETF signoff in RFC 5336.

(9) Strong conformance language

> - bullet point 3. I do not see any need to state that
> clients MUST reject...  and MUST be fully compliant... etc.
> This is implicit in any specification.  If there is a
> specific need to add this detailed behaviour (to prevent
>...

Our general practice with email specifications has been to
specify conforming behavior and simply leave non-conforming
behavior "outside the scope of the specification".  That
doesn't work in this case because, according to the WG's
analysis, a partial implementation --whether outside the spec
or otherwise-- would be bad news for interoperability.  So it
seemed necessary to be very clear about what clients were
expected to do and what servers could reasonably expect.
Whether that was better done with this text, with an
"Implementer's Note" (which has no normative status), or with
an extended explanation of the problems and a threat or two
would seem to me to be largely matters of editorial taste,
especially for a Proposed Standard.

(10) Security-related issues

> there is one more issue: as internationalized items will
> also result into logs and traces, their presence may affects
> trouble ticket systems used by security operations teams
> (CERTs/CSIRTs). And they also can affect quick handling of
> incidents, has it may require more time to correctly read
>....

Agreed.   I assume the editors will consider your proposed text.

(11) Editorial issues with significant substantive
implications.

Dave Crocker 2010-12-22 wrote:

> { Note - the term "mailbox names" is not defined here or in
> RFC 5321.  In RFC 5321 it appears to be used to mean
> local-part; however because its precise meaning is unclear,
> I strongly urge NOT using it here at all. Instead I suggest
> using whatever ABNF rulename is appropriate.  This
> guarantees clarity.

Well, while getting rid of "mailbox name" or defining it very
precisely, would probably be an improvement, there is no ANBF
rulename that would really be appropriate, at least without
additional decoration.  As you and others have pointed out
recently, rules for <mailbox> appear in both 5321 (where it
refers to the mailbox name (sic) itself, e.g.,
local-part@domain-part) and in 5322 (where it refers to
something that might include a display name and assorted
spacing).   I have no doubt that this document should be as
clear and precise as possible, but its authors should not be
blamed for trying to work around the errors of the authors of
5321, 5322, and their predecessors. 

Dave Crocker 2010-12-22 wrote:

>> The value of "uDomain" SHOULD be verified by IDNA
>> definitions [RFC5890].  If

> { I do not understand what "SHOULD be verified by IDNA
> definitions" means.

> If it means that the uDomain "SHOULDbe verified" then it is
> out of scope for this specification. There is no reason that
> SMTP rules concerning verification of Unicode-based domains
> needs to be different from ASCII-based domains.  In
> particular, verification of a Domain name is specified
> elsewhere.
>...

Editorial issues aside, 5321 contains some very
carefully-written language whose intent is to require that
domain-parts be validated lexically but to avoid requiring
that they be looked up in the DNS until/unless that is
necessary for some other reason.  The "lookup" side of
IDNA2008 (in RFC 5392) makes more or less the same circle
around this issue but with the understanding that lexical
validation of U-labels is a somewhat more complex process than
checking against a short list of characters, positions, and
lengths.   I believe that is what the author was trying to
capture.  Certainly it could have been expressed differently.

Dave Crocker 2010-12-22 wrote:

>> Otherwise, surprising rejections
>> can happen during temporary failures, which users might
>> perceive as a serious 

> { Probably not just temporary failures.  Having services
> that are meant to be redundant with each other actually
> provide different semantic behavior is just plain dangerous.
> It is essentially guaranteed that they will cause problems.}

Sure.  But you argued extensively elsewhere for not duplicating
text that appears elsewhere and, IMO, more important, for not
making statements that go beyond a rather narrow reading of the
requirements for internationalized email.  

Ned Freed 20101222 wrote:

> So you can use alternate MXes, but not alternate A records?
> I actually agree with this - an alternate A is presumably
> another address for the same host, so no point in trying it.
> But this needs to be spelled out.
 
> security considerations text. The issue, I hope, is obvious:
> There are sites aplenty with large numbers of MX records and
> which won't support this extension any time soon. And all
> too often those MXes are *slow* to respond. So sending a
> message that causes them all to be tried is ... you get the
> idea.
 
> In fact I'm tempted to go one further and say that if an MX
> at a given priority is found not to support this extension,
> you SHOULD assume that MXes at lower priority don't either.

And this is where we descend into a situation that extends far
beyond EAI.  The problem, as Ned's note almost suggests, is
actually quite general.  For example, if we had

   foo.example.com. MX 0  foomail0.example.com.
   foo.example.com. MX 10 foomail1.example.com.
   foo.example.com. MX 10 foomail2.example.com.

and "foomail1" supported 8BITMIME with downgrading and
"foomail2" did not, a system sending a message containing
C-T-E: 8bit that reached foomail1 would have the message go
through with downgrading if needed but would get a rejection or
option not supported condition if it happened to get to
foomail2.  Or it might get lucky and hit foomail0 and find out
what it actually supported.  Similar situations could apply
--at least wrt rejection versus bounces-- if foomail1 supported
a different set of antispam provisions than foomail2, or, for
that matter, if both deferred antispam processing to foomail0.

As far as I can tell after a very brief scan, there is no
current normative IETF specification that discusses this
situation and gives guidance about it.  In retrospect, we
probably should have at least described the issues with
non-identity of services in RFC 1425, but we didn't do that,
much less make specific recommendations.  

I agree with Ned that it would make a lot of sense to
explicitly say "no alternate A records because, if they don't
represent the same host, you should be using MXs" and "if the
first MX you try doesn't support the extension, assume the
others don't either and stop".  But, because of the generality
of the problem, this document doesn't seem to me to be the
right place to say that.  5321bis probably would be a
reasonable if we think it is a clarification, but especially
with YAM in a coma and no mention of this topic in the
pre-approval document, the odds of my getting to 5321bis before
this EAI work is closed out are in the "low to none" range.
Worse, while someone could quickly cons up an I-D to specify
this behavior and try to push it through as a Proposed Standard
that 5336bis could reference, processing of such a document
would essentially constitute a claim that the spec is new and
not a clarification to 5321, precluding folding it into 5321bis
later, at least without marching that document along the
standards track to Draft Standard first.  Suggestions about how
to get out of that mess are welcome, but I don't think it is an
EAI problem (even though 5336bis might plausibly be modified
--or the Framework document reopened and modified -- to explain
just how ambiguous the whole situation is).

For RFC 5336, it was important to call the issue out because
the "downgrade or not" behavior is even more complicated than
the potential behavior variations with 8BITMIME (remembering
that one can offer the option but, if the next hop doesn't
support 8bit transport, one can either downgrade or reject) and
other extensions..  Whether it is necessary to say anything at
all in 5336bis, given how little we have said in places that
perhaps establish precedents, is debatable.  But please let's
not suggest expanding on this case in this document to cover
other email environments, especially in the context of urging
the the spec keep its scope and assertions narrow.

Dave Crocker 2010-12-22 wrote:

>> 3.6.4.2.  VRFY and EXPN Commands and the UTF-8REPLY
>>Parameter 

>> If the VRFY and EXPN commands are transmitted with the
>> optional parameter "UTF-8REPLY", it indicates the client
>> can accept UTF-8 strings in replies to those commands.
>> This allows the server to use UTF-8 strings in mailbox

> { Is this extension trying to to say that a client might
> support UTF8SMTPbis but not be able to access UTF-8
> replies???  Under what circumstance is this reasonable? }
>...

Not what it is intending to say (could have been expressed
better).  Consider the following scenario: Client sends EHLO,
server offers UTF8SMTPbis.  Client sends VRFY as its next
command, using an ASCII-only argument and no parameter.  The
server has no clue as to whether or not the client is
EAI-compliant or otherwise has any idea what to with a
reply containing a non-ASCII address.  For the server to return
a non-ASCII string under those circumstances would be in
violation of 5321.

(12) The keyword denoted as "UTF8SMTPbis" in the drafts.

While the changes between the Experimental protocols and the
present draft are not large, they are also not trivial.  After
fairly extensive discussion, the WG concluded that these new
specifications should be denoted by a different SMTP Extension
keyword than the "UTF8SMTP" that was used by the Experimental
version (RFC 5336).  In large part because of concerns that
some implementation would be deployed in advance of the
specification becoming final, the WG concluded that the new/
final keyword should be assigned only as part of the IESG
approval process.  "UTF8SMTPbis" is only a placeholder and will
not be that final keyword.  It may be worth noting that, if the
protocol is changed to require a MAIL command parameter, that
WG decision will have turned out to be quite important.

Anyone who strongly believes that some variation on the
"UTF8SMTP" theme misrepresents what is actually happening in
these specifications is welcome to suggest other keywords to
the IESG.

(13) New content types for embedded messages.

Dave Crocker 2010-12-22 wrote:

> 3.  There probably needs to be definition of MIME
> message/uni-rfc822, specifying Unicode support within a
> message contained in MIME.  I'm less clear whether there
> needs to be a MIME Content-Transfer-Encoding form specific
> to UTF-8.

See message/global in draft-ietf-eai-rfc5335bis.  This is an
area where the WG's decisions about what material belonged in
which document followed the advice to keep header and body part
discussions in 5335bis rather than 5336bis.

(14) Procedural and quasi-procedural issues

Dave Crocker 2010-12-22 wrote:

> 1.  The Framework document is normative and needs to be
> completed along with the other two specifications, since
> they quite reasonably state a normative dependence on the
> Framework document.
>...
> * Framework -- The Framework document[1] is (correctly)
> referenced as required reading.  It supplies essential
> terminology and architecture for this specification.  In
> fact it is a specification, complete with formally normative
> vocabulary.  This means that it must be a normative
> reference by rfc5336bis.  It therefore also means that the
> Framework document needs to be completed before the current
> specification can be standardized.

Note that the Framework document went through IETF Last Call
and was approved by the IESG last September for publication as
Informational.  The WG was agnostic on whether it should be
Informational or Standards Track.   Since it is still in the
RFC Editor queue awaiting references to the documents now under
review, I assume that, if there were sufficient consensus that
it is normative and should be on the Standards Track, the IESG
could be persuaded to revisit that decision.  In addition,
while I don't believe that anything in these reviews requires
doing so, the document could, in principle, be recalled by the
IESG and material changed or altered.

Dave Crocker 2010-12-22 wrote:

> * Infrastructure Requirement -- Unless this option is in
> force, carrying internationalized email in a MIME part is
> prohibited.  This is out of scope for the working group and
> it is a counter-productive rule.  Imagine if the same type
>...

There may well be an editorial problem with how this is written
up or even with how "internationalized mail" is defined.  But I
see nothing that supports the apparent hyperbole in the
comments above.  Remember that 5322 and the base MIME specs
prohibit non-ASCII headers, so messages containing such headers
(as the message header or MIME headers) are prohibited by those
specs without some new extension(s).   There is no
(intentional) prohibition in these documents against carrying
non-ASCII text in mail messages.  And, if one wanted to
pseudo-encapsulate a message and its headers by treating it as,
e.g., text/plain with charset=utf8, rather than as
message/rfc822 (or message/global), these specifications don't
prohibit that either (might not be a good idea because of the
loss of recognition and handling of boundary markers and internal
Content-Types and Content-Transfer-Encoddings, but that is not
a problem for this specification).

So, unless the comment above is just the result of confusion
brought on by an editorial problem, I don't see where you are
going with it.

(15) Future of rfc5336bis.

Ned Freed, 20101222 wrote: 

> One final comment. Several of the issues that have been raised
> are so fundamental that once they are addressed (point (0)
> especially), this document is almost certainly going to
> require another full review cycle.

Sadly, I agree.  I even believe that, were none of these
fundamental issues present, the number of editorial corrections
(including ABNF matters) that are required will be sufficiently
extensive that the risk of introducing new problems would
require such a review cycle.

     john

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 28 Dec 2010 20:09:56 +0000

[With my MTA server developer hat on, not speaking as an AD]

Ned Freed wrote:

>and AFAIK neither does sendmail, PostFix, Apache James, or qmail.
>
Isode M-Switch doesn't enforce this either.

>(Exim, OTOH, appears to.)
>  
>
+1. Maybe the client has cached EHLO response from a previous connection.

>*It doesn't have to be a new code. OTOH, it needs to be a code that EXPN/VRFY
>*
+1.

>get people thinking about them. So I'm strongly inclined to stick with the
+1.