Subject: [EAI] apps-team review of draft-ietf-eai-rfc5336bis

From: Claudio Allocchio <[email-masked]>
Date: Wed, 22 Dec 2010 18:24:11 +0100 (CET)

I have been selected as the Applications Area Review Team reviewer for this
draft (for background on apps-review, please see
http://www.apps.ietf.org/content/applications-area-review-team).

Please resolve these comments along with any other Last Call comments you
may receive. Please wait for direction from your document shepherd or AD
before posting a new version of the draft.

Document: draft-ietf-eai-rfc5336bis
Reviewer: Claudio Allocchio (GARR, Italian Research and Academic Network)
Review Date: 2010-12-22
IETF Last Call Date:  N/A
IESG Telechat Date:  N/A

Summary:

The document specifies how to apply internationalization to some SMTP
elements, and to some Header elements which depends from the SMTP elements.
However there seem to be a discrepancy from what the abstract states, and
what is elaborated into the document details: the document also deals with
quite a number of elements which involve bodyparts, and affect the whole
email legacy system, and beyond, such as logs, traces, and wherever email
addresses are used (certificates, etc.). This seems thus to go beyond what
the summary states, even this fact is not explicitly discussed. It is
unclear thus if this "update" of RFC5321 and RFC5322 is only limited in
scope or not. My reading of the document say it goes beyond, and implicitly
affects also what is being transported into the email system. This is thus a
serious issue to be solved before the document can go to the Standard Track.
Another big issue is about blending the boundaries of servers in the
transport system (MTAs), clients (UAs) and their roles, when the document
specify that servers shall use guessing techniques from looking into
bodyparts to understand if what they're transporting is internationalized
according to this specification or not. This is IMHO a basic violation of
the role distiction in the legacy email system, and affects the MIME model
al well.

The document quite often repeats rules from other specifications, AND
re-describes these rules. This is dangerous (it makes very difficult to keep
aligned documents, and will lead to different interpretations) and not
needed.

The full specification probably needs a serious "english language" revision,
because the language issues which I specified in "Minor Issues" seems to be
a non neglegible number. I'll try to identify as many as possible and
suggest changes.

Major Issues:

This specification updates a number of SMTP and Headers fields from RFC5321
and RFC5322. However these fields are also used in Gatewaying to other email
systems, which are still existing. Section 3.7 in RFC5321 describes specific
cases. However this memo provides no clue on possible effect that
internationalized fields can have in case gatewaying occurs. Gatewaying
functions strongly rely on these values, and the introduction of these
modification should describe this scenario, too. I suggest to include a
specific section about these issues, and suggest at least a number of
possible actions which Gateways can take to survive this change.

The EAI WG charter states that in handling email message we shall be
"consistent with the long-term view that email with invalid addresses or
syntax should be rejected, rather than fixed up in transit between
submission servers and delivery servers.". However a number of sections in
the specification seem to contradict this, where they suggest actions, by
exammining message elements (in the SMTP and in the Headers and in the
Bodyparts) in order to try to deliver anyhow. This is a subtle line between
maximising the service chance to deliver, and "fixing" things during the
transaction, and it can encourage implementors to do many of these acions,
instead of stiching to the idea "not supported --> non delivery".

Major Issues by sections.

Abstract:

" This document specifies an SMTP extension for transport and delivery
    of email messages with internationalized email addresses or header
    information."

"internationalized" is not defined here in details. Sections 1 and 1.2 try 
to do this, but the language is not crystal clear: "to permit 
internationalized email addresses in envelopes, and UNICODE characters 
(encoded in UTF-8) [RFC3629] in headers." Are addresses in envelopes (SMTP 
envelopes!) also via UNICODE characters encoded in UTF-8? The reference to 
"the framework document" seem to come too late in the text to ensure the 
reader fully understand the relationship.

  3.2 The UTF8SMTPbis Extension

"An SMTP server that announces this extension MUST be prepared to
accept a UTF-8 string [RFC3629] in any position in which RFC 5321
specifies that a mailbox can appear."

s/mailbox/<mailbox>/  (it's a definition!).

The specification does NOT suggest a preferred order among the 3 possible
options when the SMTP server state it does not support UTF8SMTPbis. Having 3
"MAY" options at the same level can create a very uncertain email service
behaviour, and a very bad user's experience! This is consistent with what
happens in other cases for other "options" in the email system (like DSNs
and MDNs), but there should be at least an informative document describing
the various scenarios, and giving some guidance to implementors in order to
avoid totally incosistent behaviours.

If we do not specify a consistent procedure to try the 3 options, in many
cases we will just in practice break the concept of universal email common
service, creating islands which do not talk to each other. The 3.6.2 section
gives some suggestions about this, but this is limited to MX listed servers
for a single domain. My concern is more general. In 3.6.2 anyhow the
language should be more precise. It states "orgnisations", which something
totally undefined from a protocol point of view. MX records refer to
"domains", not to "organisations".

   3.3.  Extended Mailbox Address Syntax

The definition of

              uAtom = 1*ucharacter
                ; Replace Atom in RFC 5321, Section 4.1.2

              ucharacter = atext / UTF8-non-ascii

              atext = <See Section 3.2.3 of RFC 5322>

permits that uAtom is either an atext OR an UTF8-non-ascii... which in the
end allows a uMailbox to be a mixed contruction where some uAtom is in
atext, while some other uAtom is in UTF8-non-ascii AT THE SAME TIME, e.g. a
construct like

      [email-masked]

is allowed. However the distinction here seems to be unneeded, as ASCII is a
subset of UTF8.

Section 3.6 tries to clarify that this is not allowed; owever there is
normative language there apart on SHOULD for the domain name. I think this
shold be fixed with a clearer normative specfication.

This of course applies also to all other tokens where uAtom is used.

3.5 Body Parts and SMTP Extensions

I have problems in understanding why an SMTP server should peek extensively
into the message geing transfered to understand that it is handling an
interationalzed message. "There is no ESMTP parameter to assert that a
message is an internationalized message." in my opinion should be fixed in
another way, by ADDING an appropriate ESMTP parameter to state this! This
will solve already in the SMTP dialogue many isses which in this
specification are handeled by guessing techinques.

3.6.1 "the server normally sends..." --> "the server sends" (sorry... we do
not like ABnormal servers!) :-)

   Minor Issues:

1. Introduction

The sentence "This document use this mechanism to support an
internationalized email address." defintly needs a better form; if you do
not know already know what we are talking about, it can be misleading.

"This document uses this mechanism to allow internationalized email
addresses support in SMTP elements."

3.1 Framework for the Internationalization Extension

- bullet point 3. I do not see any need to state that clients MUST reject...
and MUST be fully compliant... etc. This is implicit in any specification.
If there is a specific need to add this detailed behaviour (to prevent wrong
implementions? why?) either we shall ensure the text is clear and
unambiguous, or we shall add an Implementers Note, as it was done in some
other RFCs.

3.2 The UTF8SMTPbis Extension

same general comment as per 3.1: in general the section states behaviours
which do not need to be specified again, because they're alredy described
seomewhere else. This might also become a Major issue if the specification
of such behaviours are changed elsewhere. Just poit to extrnal references.
(it seems editors are worried of implmentors which do not fully undesrtand
or know the other email specifications... )

3.3.  Extended Mailbox Address Syntax

What does the sentence "..., using the production for a mailbox and those
productions on which it depends." mean? I assume you meant <mailbox>, is
this correct?

5.  Security Considerations

there is one more issue: as internationalized items will also result into
logs and traces, their presence may affects trouble ticket systems used by
security operations teams (CERTs/CSIRTs). And they also can affect quick
handling of incidents, has it may require more time to correctly read and
understand logs and traces if they are not in current ascii form. This might
even be a Major Issue. I propose adding this text in the section, besides
pointing to the exteral reference:

---------------------- start of added text
Beyond the use inside the email global system (in SMTP envelopes and message
headers), internationalized email addresses will also show up inside other
scenarios, in particular:

  - the logging systems of SMTP transactions and other logs to monitor
    the email systems;

  - the trouble ticket systems used by Security Teams to manage security
    incidents, when an email address is involved

This will likely require extending support for full UTF8 also into these
systems, in order to avoid problems, which could cause also important loss
of data, or require to provide an adequate mechanism to map non ASCII
strings into them.

Another Security aspect to consider is related to the ability by Security
Team members to quickly understand/read and identify email addresses from
the logs, when they are tracking an incident. Mechanims to automatically and
quickly provide the origin or ownership of an internationalized email
address shall implemented for use also by log readers which cannot read
easily non-ASCII information.
--------------------------- end of added text

Also, in Security Consideration, there is no mention of the fact that we are
changing VRFY/EXPN commands to include internationalized addresses.
VRFY/EXPN commands can exist ALSO in an SMTP transaction which is not
supoused to transfer an email messages between MTAs. And their use has also
do to with Security. Here is an additional text for this:

---------------------- start of added text
The SMTP commands VRFY and EXPN are sometimes used in SMTP transactions
where there is no message to transfer (by tools used to take automated
actions in case potential spam messages are identified). RFC5321 section 3.5
and 7.3 give some detailed description of use and possible behaviours.
Implementation of internationalized addrsses can affect also logs and
actions by these tools.
--------------------------- end of added text

   Nits:

see
http://tools.ietf.org/idnits?url=http://tools.ietf.org/id/draft-ietf-eai-rfc5336bis-07.txt

Best Regards!

------------------------------------------------------------------------------
Claudio Allocchio             G   A   R   R          [email-masked]
                         Senior Technical Officer
tel: +39 040 3758523      Italian Academic and       G=Claudio; S=Allocchio;
fax: +39 040 3758565        Research Network         P=garr; A=garr; C=it;

            PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca

---

From: Ned Freed <[email-masked]>
Date: Wed, 22 Dec 2010 18:01:15 -0800 (PST)

I was asked to review draft-ietf-eai-rfc5336bis-07.txt. I was also provided
with a preliminary copy of the review Dave did, so these notes should be viewed
as an addendum to Dave's review, not as a standalone critique. In particular,
any failure to reiterate Dave's various points here should not be taken as
disagreement with him; on the contrary, I am in agreement with almost all of
his comments.

Now for the specifics:

(0) I cannot emphasize strongly enough how important it is to have a
    UTF8SMTP sort of parameter on the MAIL FROM to indicate that a message
    employing this extension is being submitted/relayed/delivered. This is
    especially important in the relay case - we do not want to require
    that relays grub through every message they receive looking for possible
    utf-8 so they can carry the negotiation of this extension forward, but
    that's exactly what the present proposal forces them to do.

(1) 3.1 item 4 and 3.6.4.2 add an optional UTF8REPLY parameter to VRFY and
    EXPN. They do this by redefining the syntax to be:

    vrfy = "VRFY" SP ( uLocal-part / uMailbox )
                     [ SP "UTF8REPLY" ] CRLF

    expn = "EXPN" SP ( uLocal-part / uMailbox )
                     [ SP "UTF8REPLY" ] CRLF

    The original syntax for VRFY and EXPN given in RFC 5321 is:

    vrfy = "VRFY" SP String CRLF
    expn = "EXPN" SP String CRLF
    String = Atom / Quoted-string
    Atom = 1*atext

    It's not immediately obvious, but the new syntax is in fact a superset of
    the old. (uLocal-part subsumes local-part, which in turn allows atoms
    and quoted strings.) However, this change does much more than hang a
    parameter on the command; for the first time an entire address is
    permitted as an EXPN/VRFY argument. But no semantics are given for
    this significant syntax expansion, nor does this expansion appear to be
    within scope for EAI.

    A more reasonable syntax would be:

    vrfy = "VRFY" SP uString [ SP "UTF8REPLY" ] CRLF

    expn = "EXPN" SP uString [ SP "UTF8REPLY" ] CRLF

    uString = uAtom / uQuoted-string

    But this now fails along a different dimension: When adding a parameter
    capability to a command, it should be done in a general way, so that any
    future extensions can build on a cleanly specified foundation. Like so:

    vrfy = "VRFY" SP uString [ SP vrfy-parameters ] CRLF
    vrfy-parameters = esmtp-param *(SP esmtp-param)

    expn = "EXPN" SP uString [ SP expn-parameters ] CRLF
    expn-parameters = esmtp-param *(SP esmtp-param)

    uString = uAtom / uQuoted-string

    Of course this will add 555 as a possible response to VRFY/EXPN, and that
    needs to be noted.

(2) In response to Dave's critique/query about why the parameter on VRFY and
    EXPN is needed, it's because these commands can appear outside of a
    transaction and thus there's not necessarily an indication that the client
    is prepared to accept UTF-8 responses. However, this brings up another
    issue, which is that a server that supports this extension needs a way
    to tell a client that doesn't that it cannot return a proper response
    to VRFY/EXPN without that parameter being specified. A new SMTP error
    status is needed, but the document only defines an extended status
    for this purpose; it doesn't specify a regular status code to use.

(3) Section 3.2 says:

   1.  If and only if the SMTP client (sender) is a Message Submission
       Server ("MSA") [RFC4409], it MAY, consistent with the general
       provisions for changes by such servers, rewrite the envelope,
       headers, or message material to make them entirely ASCII and
       consistent with the provisions of RFC 5321 [RFC5321] and RFC 5322
       [RFC5322].

    This is a bad idea no matter how you slice it. Such transformations
    depend on the availability of address mapping (from a UTF-8 address to
    an equivalent ASCII address) information; an agent being an MSA is neither
    necessary nor sufficient reason to have this information.

    But the bigger problem by far is defining what transformations are
    permissible. The present framework document is basically silent on
    what's allowed, so I for one have no idea what it means to be
    "consistent with the general provisions for changes by such servers"
    actually means. And unless you're willing to spell this out, I see this
    as practically guarnteeing a slew of interop problems.

(4) Also from 3.2:

  3.  It may find an alternate route to the destination that permits
       UTF8SMTPbis.  That route may be discovered by trying alternate
       Mail eXchanger (MX) hosts (using preference rules as specified in
       RFC 5321) or using other means available to the SMTP-sender.

    So you can use alternate MXes, but not alternate A records? I actually
    agree with this - an alternate A is presumably another address for
    the same host, so no point in trying it. But this needs to be spelled
    out.

    At an absolute minimum this needs to be accompanied by some
    security considerations text. The issue, I hope, is obvious: There
    are sites aplenty with large numbers of MX records and which won't
    support this extension any time soon. And all too often those MXes
    are *slow* to respond. So sending a message that causes them all to
    be tried is ... you get the idea.

    In fact I'm tempted to go one further and say that if an MX at a given
    priority is found not to support this extension, you SHOULD assume that
    MXes at lower priority don't either.

(5) Consider this text from 3.5:

   Assuming that the server advertises UTF8SMTPbis and 8BITMIME, and
   receives at least one non-ASCII address, the precise interpretation
   of "BODY=8BITMIME", and "BODY=BINARYMIME" in the MAIL command is:

   Since when are the addresses in the envelope a predictor of the addresses
   that appear in the outermost header, let alone any enclosed message/rfc822
   headers? Really, this could not be more fundamental: There's no requirement
   that envelope addresses appear in the outermost header or vice versa, and
   circumstances abound where the addresses in the header and envelope end up
   being partly or entirely disjoint. As such, while the use of UTF-8 in the
   envelope can be taken as an indication that this is an EAI message, the
   absence tells you nothing.

   Addrressing item (0) above and having a parameter on MAIL FROM addresses the
   technical side of this one, but the entire notion of coupling and
   confusion between envelope and header needs to be stamped out in this
   specification.

(6) A bunch of the references are to bis-versions, but the actual cite is to
    the current RFC.

One final comment. Several of the issues that have been raised are so
fundamental that once they are addressed (point (0) especially), this document
is almost certainly going to require another full review cycle.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Wed, 22 Dec 2010 18:03:33 -0800 (PST)

+1 to all of Claudio's points raised here.

				Ned

> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 23 Dec 2010 17:51:07 -0000

>     that's exactly what the present proposal forces them to do.

Well there was a header for that purpose in the early days of the WG but,  
IIRC, it was omitted because it could not be relied upon as regards  
leakage, and also because we concluded that, although the MTA would have  
to make a complete pass through the body in order to detect the presence  
on non-ascii, most MTAs already had to make that pass anyway as part of  
the 8BITMIME checks, and the extra code now required was neither here not  
there.

But it is some years since we had that discussion. Perhaps someone else  
could recall the exact arguments.

But, having said that, I would not be averse to reinventing that header -   
but be warned that it might turn out not to be as reliable an indication  
as might first appear.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Ned Freed <[email-masked]>
Date: Thu, 23 Dec 2010 16:33:32 -0800 (PST)

> Well there was a header for that purpose in the early days of the WG but,

A header doesn't work for this purpose. It needs to be announced in the
envelope. Just as one example, this allows messages sent to an address
that is known not to be able to support utf-8 addresses to be rejected
immediately.

> the 8BITMIME checks, and the extra code now required was neither here not

That's completely nonsensical on several fronts. First, 8bitMIME is announced
in the envelope so there is no need to scan the message looking for it unless
you want to be especially careful. Second, a check to see if the message
contains 8bit characters is fairly cheap, but a scan to see if the 8bit appears
specifically in addresses is much more expensive - now you're talking about a
full MIME parse. Maybe this doesn't matter when you're running an MTA that
processes a handful of messages a day, but things are a little different when
you're processing several hundred messages a second, which for some MTA is a
*conservative* estimate.

> but be warned that it might turn out not to be as reliable an indication

Again, headers don't have the proper semantics. A MAIL FROM parameter, OTOH,
does.

				Ned

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 23 Dec 2010 19:23:07 -0800

On 12/23/2010 9:51 AM, Charles Lindsey wrote:
> On Thu, 23 Dec 2010 02:01:15 -0000, Ned Freed <[email-masked]> wrote:
>> employing this extension is being submitted/relayed/delivered.
...
> Well there was a header for that purpose in the early days of the WG but, IIRC,

The difference between attaching information to a header field WITHIN an object, 
versus attaching it to meta-data OUTSIDE OF the object is fundamental.

By the time you are processing the header field that is within the object, you 
have already established the processing context for that object.

If you want to switch the basic set of characters that apply to the header of a 
message, you must declare the new set BEFORE you start processing the header.

THAT is why UTF-8 must be declared in the SMTP protocol and not later.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: John C Klensin <[email-masked]>
Date: Sat, 25 Dec 2010 15:20:07 -0500

(writing as a personal opinion, not speaking for the WG.  Shawn
and others have already addressed parts of this, but, for
completeness...)

--On Wednesday, December 22, 2010 18:01 -0800 Ned Freed
<[email-masked]> wrote:

>...
> (1) 3.1 item 4 and 3.6.4.2 add an optional UTF8REPLY parameter
> to VRFY and     EXPN. They do this by redefining the syntax to
> be:
> 
>     It's not immediately obvious, but the new syntax is in
> fact a superset of     the old. (uLocal-part subsumes
> local-part, which in turn allows atoms     and quoted
> strings.) However, this change does much more than hang a
> parameter on the command; for the first time an entire address
> is     permitted as an EXPN/VRFY argument. But no semantics
> are given for     this significant syntax expansion, nor does
> this expansion appear to be     within scope for EAI.
> 

Yes.  I don't think the above mess was intentional.   In
fairness to the authors, the ABNF has been changed too many
times by too many people and I think the situation above is a
side-effect.   As you point out, the syntax changes may not be
adequate (see below).

>     But this now fails along a different dimension: When
> adding a parameter     capability to a command, it should be
> done in a general way, so that any     future extensions can
> build on a cleanly specified foundation. Like so:
> 
>     Of course this will add 555 as a possible response to
> VRFY/EXPN, and that needs to be noted.

Right. See below.

In addition, one must have a session for which mail transactions
are permitted (i.e., EHLO must have been sent) in order to use
any parameters at all.  RFC 5321, Section 4.1.4, explicitly
permits sending VRFY or EXPN without first having EHLO in the
session.  So there is another implicit requirement in all of
this that should almost certainly be made explicit.   (I think
this is the topic about handshaking that Dave and Shawn have
been circling around, but I'm not sure).

> (2) In response to Dave's critique/query about why the
> parameter on VRFY and     EXPN is needed, it's because these
> commands can appear outside of a     transaction and thus
> there's not necessarily an indication that the client     is
> prepared to accept UTF-8 responses. 

Yes, exactly.

> However, this brings up
> another     issue, which is that a server that supports this
> extension needs a way     to tell a client that doesn't that
> it cannot return a proper response     to VRFY/EXPN without
> that parameter being specified. A new SMTP error     status is
> needed, but the document only defines an extended status

If we go down this path, the WG clearly needs to reexamine
whether such a code is necessary.  I think you have made a
persuasive case, but I can't speak for the WG.

Indeed, this has all gotten sufficiently complex that I think
perhaps the WG should reexamine its decision to handle the "VRFY
and EXPN with possible non-ASCII responses" situation by
parameterizing those commands rather than introducing new
commands, say VRF8 and EXP8.  IIR, the WG did consider that
alternative (long ago) and conclude that adding a parameter was
the more straightforward option.  But, perhaps, given the
various complexities that have emerged, that was not the right
decision.  With other commands, such as MAIL and RCPT, we've
already got a history of parameters and the base (821/5321)
syntax is very clearly delimited.  By contrast, I certainly
remember seeing VRFY followed by unquoted strings with blanks in
them in the wild (e.g., arrangements like 
    VRFY Ned Freed
) and some servers tolerating them in the name of robustness or
in memory of 821, which had
    VRFY <SP> <string> <CRLF>
    <string> ::= <char> | <char> <string>
    <char> ::= <c> | "\" <x>
    <c> ::= any one of the 128 ASCII characters, 
        but not any <special> or <SP>
    <x> ::= any one of the 128 ASCII characters
       (no exceptions)
which makes
    VRFY Dave\ Crocker
perfectly valid.    I don't know if similar examples still occur
and with what frequency, but...

In addition, if we handle EAI-extended VRFY and EXPN with
parameters, we end up in the odd situation in which the
no-parameter versions can be issued without mail session
initialization (the EHLO command) but the extended,
parameterized ones require a previously-initialized session.
The MAIL and RCPT commands dn't have an issue because they
require session initialization anyway.  That restriction on
extended VRFY and EXPN certainly works technically, but isn't
going to be the easiest thing to explain and might cause some
"interesting" behavior if implementations screw it up or
clients, having not sent EHLO, assume that they are talking with
legacy 821 servers.   By contrast, a rule that requires session
initialization prior to using a new command is a no-brainer to
specify and implement cleanly, with no excuses about, e.g.,
ancient legacy environments.

I would welcome your thoughts on this tradeoff between adding
new commands (whose syntax would be identical to 5321 VRFY/EXPN
but expanded to UTF-8 input and/or output) and the current
proposal for a parameterized multiple-case situation, presumably
fixed up more or less as you have suggested.

    john

---

From: "Al Costanzo" <[email-masked]>
Date: Mon, 27 Dec 2010 00:00:09 -0500

> OTOH, a lot of servers back in the days prior to RFC 2821 didn't support 
> the
> 821 thing of allowing literal spances with a \ and no enclosing quotes. So 
> this
> never could be said to have interoperated. And I think the time has come 
> to
> abandon it (although I note that if you really want to continue supporting 
> it,
> the added parameter doesn't prevent you from doing so).
>

Ned,

If my memory servers me, (and it may not),

There were good reasons why VRFY supported this.  I dont know if any are 
valid anymore but one idea was that you could send the persons name, not the 
email address, to find out if that person had an email address on that 
system.

Was this not the reason Ned to allow blanks (whitespace) and unquoted 
strings?  If you remove this from the spec, this functionality would be 
lost, I don't really see a reason to lose something that works.

>> RFC 5321, Section 4.1.4, explicitly
>> this that should almost certainly be made explicit.

I do not remember VRFY command requiring the EHELO, there are a number of 
mailing list verifiers that do not send the EHELO first. Verifiers, since 
they do not actually send email, usually dont send the EHELO first. (our 
application does) and I think requiring the EHELO first as part of a 
handshake it the proper way to do this.

Only my humble opinon.

-Al Costanzo

----- Original Message ----- 
From: "Ned Freed" <[email-masked]>
To: "John C Klensin" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>; "Ned Freed" 
<[email-masked]>; <[email-masked]>; 
<[email-masked]>
Sent: Sunday, December 26, 2010 5:53 PM
Subject: Re: [apps-discuss] [EAI] RFC5336 and VRFY/EXPN (was: Re: apps-team 
review of draft-ietf-eai-rfc5336bis)

>> any parameters at all.
> You certainly need to send an EHLO and process the result to find out what
> extensions are allowed - guessing what combination of extensions will work 
> most
> certainly is not a sound operational policy. And implementations are 
> certainly
> free to only offer the extensions that have been returned in an EHLO 
> repsonse.
>
> That said, I don't see an actual requirement in section 2.2 stating that 
> EHLO
> must be issued before any parameters can be used. (Maybe it's somewhere 
> else,
> but a cursory search didn't find it.) Our MTA certainly doesn't enforce 
> this
> and AFAIK neither does sendmail, PostFix, Apache James, or qmail. (Exim, 
> OTOH,
> appears to.)
>
>> RFC 5321, Section 4.1.4, explicitly
>> this that should almost certainly be made explicit.
>
> There certainly should be a note saying that you need to issue an EHLO
> and check the response before attaching any parameters to VRFY/EXPN. I 
> don't
> think a requirement that parameters not be accepted before an EHLO
> has been issued is needed.
>
>> (I think
> It's part of it, I think.
>
>> > However, this brings up
> It doesn't have to be a new code. OTOH, it needs to be a code that 
> EXPN/VRFY
> don't return for other purposes, and I think if you look at the list of
> available codes a new code is needed.
>
>> decision.
> I considered suggesting new commands, but given the need to specify
> syntax for the new commands I don't see this as any simplier
> specification-wise. In terms of implementation, a new command is actually
> a somewhat more complex - the code changes are less localized, and I 
> suspect
> other implementations will be comparable in this regard.
>
> But the minute you need some other variant on EXPN and VRFY, the alternate
> command appraoch becomes the far more complex way to do it. Now, we 
> haven't had
> a specification come along that needed additional EXPN/VRFY parameters 
> before
> this one, but these things have a way of showing up once you open the door 
> and
> get people thinking about them. So I'm strongly inclined to stick with the
> parameter approach.
>
>> With other commands, such as MAIL and RCPT, we've
> OTOH, a lot of servers back in the days prior to RFC 2821 didn't support 
> the
> 821 thing of allowing literal spances with a \ and no enclosing quotes. So 
> this
> never could be said to have interoperated. And I think the time has come 
> to
> abandon it (although I note that if you really want to continue supporting 
> it,
> the added parameter doesn't prevent you from doing so).
>
> Ned
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 27 Dec 2010 12:24:45 -0000

On Fri, 24 Dec 2010 00:33:32 -0000, Ned Freed <[email-masked]>  
wrote:

>> This
>> > is

Ah! the WG considered a header for this purpose, but not a specific  
announcement in the envelope. So we ought to look at that now.

However, I do not quite see the necessity for it. If some MTA announces  
the capability UTF8SMTP, then it is declaring its willingness to accept  
messages destined for utf-8 addresses and to do the "right thing" with any  
problems that might arise (such as generating a bounce if some MTA further  
down the chain finds a problem). Essentially the same as current MTAs do  
with ASCII messages that turn out to be undeliverable.

If some client tries to send such a message without first checking that  
the UTF8SMTP capability has been advertised, then it is in breach of our  
proposal, and is just like people who try to "just send 8-bits" at the  
moment - if they strike lucky it might work, but there is no guarantee.

So I think the only situation where a "this message uses UTF-8 in headers"  
parameter in the MAIL FROM would confer any actual benefit is when some  
particular message within some environment where UTF-8 is the  norm just  
happens to be in pure ASCII (I am on a visit to China, but I am just  
sending an ASCII-only message home). In that case, the _absence_ of that  
parameter in the MAIL FROM would indicate to the MTA that it could happliy  
forward the message to other MTAs that did not advertise UTF8SMTP.

So it would be up to originating MUA (or maybe to submission servers) to  
ensure that the said parameter was provided, or not, according to the  
content of the particular message, and for all subsequent agents to  
propagate that parameter as received when forwarding it on.

But note that introducing this parameter now whould create problems for  
those who have already implemented the experimental proposal without such  
a parameter. That should not be a show stopper, but that transition needs  
some thought.

---

From: John C Klensin <[email-masked]>
Date: Mon, 27 Dec 2010 11:22:48 -0500

--On Monday, December 27, 2010 12:24 PM +0000 Charles Lindsey
<[email-masked]> wrote:

>> A header doesn't work for this purpose. It needs to be
>> announced in the envelope. Just as one example, this allows
>> messages sent to an address that is known not to be able to
>> support utf-8 addresses to be rejected immediately.
> 
> Ah! the WG considered a header for this purpose, but not a
> specific announcement in the envelope. So we ought to look at
> that now.

Actually, the WG did consider an envelope announcement at one
point.  But that is not particularly relevant at this stage.
Several of the reviewers have made a strong argument for an
envelope parameter, so we need to consider whether, in the light
of those arguments, we want to change our collective minds and
add such a parameter.

> However, I do not quite see the necessity for it. If some MTA
> announces the capability UTF8SMTP, then it is declaring its
> willingness to accept messages destined for utf-8 addresses
> and to do the "right thing" with any problems that might arise
> (such as generating a bounce if some MTA further down the
> chain finds a problem). Essentially the same as current MTAs
> do with ASCII messages that turn out to be undeliverable.
> 
> If some client tries to send such a message without first
> checking that the UTF8SMTP capability has been advertised,
> then it is in breach of our proposal, and is just like people
> who try to "just send 8-bits" at the moment - if they strike
> lucky it might work, but there is no guarantee.

If I recall, that was more or less the WG's position wrt both
the header and envelope suggestions.

> So I think the only situation where a "this message uses UTF-8
> in headers" parameter in the MAIL FROM would confer any actual
> benefit is when some particular message within some
> environment where UTF-8 is the  norm just happens to be in
> pure ASCII (I am on a visit to China, but I am just sending an
> ASCII-only message home). In that case, the _absence_ of that
> parameter in the MAIL FROM would indicate to the MTA that it
> could happliy forward the message to other MTAs that did not
> advertise UTF8SMTP.
> 
> So it would be up to originating MUA (or maybe to submission
> servers) to ensure that the said parameter was provided, or
> not, according to the content of the particular message, and
> for all subsequent agents to propagate that parameter as
> received when forwarding it on.
> 
> But note that introducing this parameter now whould create
> problems for those who have already implemented the
> experimental proposal without such a parameter. That should
> not be a show stopper, but that transition needs some thought.

I think this last is a non-issue: precisely the reason we agreed
to change the ESMTP announcement keyword was to protect new,
standards-conforming, implementations (and to delay assigning
that keyword until after we were sure the standards-track specs
were stable) from different behavior in experimental
implementations.  

Advertise new EHLO keyword; get new behavior (or no EAI).
Advertise old/experimental EHLO keyword and one is on one's own.
Oddly, another argument for adding the MAIL command client
announcement is that it would tell a server who had chosen to
advertise both keywords what was expected.  I would hope that,
instead, we would see no such servers (as far as I know, none of
the experimental implementations have been released into the
wild) but, if folks decided to do that to support some sort of
transition requirement, a MAIL from keyword that would imply,
among other things, "don't try to apply RFC 5504 downgrading",
might be beneficial.

     john

> 
> --Charles H. Lindsey ---------At Home, doing my own th
> ing------------------------
> Tel: +44 161 436 6131

---

From: John C Klensin <[email-masked]>
Date: Mon, 27 Dec 2010 13:49:06 -0500

--On Monday, December 27, 2010 12:00 AM -0500 Al Costanzo
<[email-masked]> wrote:

> There were good reasons why VRFY supported this.  I dont know
> if any are valid anymore but one idea was that you could send
> the persons name, not the email address, to find out if that
> person had an email address on that system.

and what that address was, of course.

This was heavily used for a while with both VRFY and, separately
with finger, to get from name to email address.  The advantage
of the VRFY approach was that it would permit a user to enter an
address that might look like "John [email-masked]" at which
point the MUA would send 
   VRFY John Klensin
to the mail server at example.com, get back an address, and then
open a mail session.

Of course, Ned's observations about lousy support (with which I
agree) notwithstanding, the variation of 
   VRFY "John\ Klensin"
would have been fully 821-compliant.  I vaguely remember one MUA
that would send the first (escape-less) version and then send
the second if the reply from the first was "no 'John; here" or
"syntax error" rather than "no 'John Klensin' here".  The
heuristic is pretty trivial.

That use of VRFY became less popular as MUAs became more
separate from first-hop full service MTA (aka submission
servers) because that sort of use of VRFY pretty much requires
at least some SMTP capability (including domain name lookups,
coping with MXs, etc.) in the MUAs, ability to use port 25
connections from just about anywhere, etc.   I don't know
whether the usage is gone entirely, but it is certainly a lot
less common than it appeared to be many years ago.

> Was this not the reason Ned to allow blanks (whitespace) and
> unquoted strings?  If you remove this from the spec, this
> functionality would be lost, I don't really see a reason to
> lose something that works.

Especially if the clock could be turned back, one could allow
only the single form

  VRFY "John Klensin"

just getting rid of the bare, unquoted, backslash escape that
was permitted by 821 but that 5321 already excludes.

> I do not remember VRFY command requiring the EHELO, there are
> a number of mailing list verifiers that do not send the EHELO
> first.

It never has been a requirement although sending EHLO (or HELO)
first has always been harmless.  Either 5321 or 2821 simply made
it very clear that there was no such requirement.

> Verifiers, since they do not actually send email,
> usually dont send the EHELO first. (our application does) and
> I think requiring the EHELO first as part of a handshake it
> the proper way to do this.

That is one issue on the side of "new command".  If one now
decides to require EHLO first, then all existing uses of VRFY
without it (which conform perfectly well to 5321 and all of its
predecessors) become invalid,  That is pretty uncomfortable to
me.  By contrast, if one creates the new command, then, if the
response to EHLO includes some sort of EXTENDED-VRFY capability
(see below),  one could have:

	-- VRFY without EHLO as requesting the 5321 behavior
	(with or without non-conforming extensions or
	flexibility and with responses limited to ASCII).
	
	-- EHLO, then VRFY as requesting the 5321 behavior,
	identical to that above.  
	
	-- EHLO, then New-VRFY as requiring a strict Atom or
	Quoted-string optionally followed by a parameter list.

Note that, if the non-conforming client behavior we know is out
there is used, then 
    VRFY Costanzo 8BITREPLIES-OK
or equivalent could be interpreted by the server as either of
    VRFY "Constanzo\ 8BITREPLIES-OK"
or as coming with a parameter, or as a syntax error, in either
VRFY case.   Not really a good idea, IMO, and at least one
reason we have not permitted the use of extended parameters with
older SMTP commands with EHLO and capability permission.

Note too that the last paragraph of RFC 5321 Section 3.5.1 is,
in part and while not explicit, a caution against treating the
standard syntax too literally or narrowly. 

We could, in principle, do a completely new command without
involving the extension mechanism at all, thereby allowing
sending it before or without EHLO, but that strikes me as a
really scary idea.

> Only my humble opinon.

Only mine also -- I really don't have my mind made up about this
and am just trying to get the various arguments in front of the
WG.

     john

p.s. I agree with what I take to be at least a corollary to
Ned's remarks: having EAI create special-purpose commands that
are not very narrowly tied to i18n issues would be a bad idea.
If there is any perceived need for a new command, we ought to
define in a separate I-D, as (impossible keyword is deliberate
to clarify that this is not a proposal)
   NewVRFY String *(SP esmtp-param)
with String given only the 5321 interpretation, not the 821 one,
and then process that command through the SMTP list, etc.

If that appeals to anyone, I might generate a strawman just to
facilitate the discussion.

---

From: John C Klensin <[email-masked]>
Date: Mon, 27 Dec 2010 14:39:31 -0500

Hi.

Long, careful, and thoughtful reviews deserve long --and I hope
careful and thoughtful-- responses.  This note attempts to
identify the main substantive points in the reviews and react
to them.  Some may consider it a special case of no good deed
going unpunished; if so, I'm sorry.

I'm sending the note separately to the EAI WG mailing list
([email-masked]) and the Apps Discuss one.  I hope that we can
move discussion to the WG list.  Put differently, please do not
respond to this note on the apps-discuss list -- if anyone has
additional comments, please take them to the WG list.  My
apologies to those who will get it twice -- there seemed no
better way to avoid, or at least reduce, multiple threads of
discussions that were not visible to WG participants.

Writing only as a participant in the WG and co-author of the
"EAI framework" document, I have several comments on the
various Applications Area Review Team comments (my co-chair
helped a bit with part of this note but I'll take the blame
as needed).  To be certain we all start with the same
background assumptions, please remember that the original form
of the "EAI Framework" supported the Experimental version of
these protocols document was published in July 2007 as RFC
4952.  Its replacement, draft-ietf-eai-frmwrk-4952bis-10, is
more to the point today.  That draft is intended to be aligned
with the documents that are now under consideration (or to have
them aligned with it), has completed IETF Last Call, was
approved for publication by the IESG in late September, and has
been in the RFC Editor queue awaiting normative references to
the current set of documents and resolution of a naming issue
that the WG decided to leave until after the approval process
completed (more on that below).

Speaking very personally because I don't know whether the WG
participants will agree, I think these specifications are
important enough that I'd rather have them right than stand on
any sort of procedural argument.  However, I think the
community also needs to consider when, newly-discovered
showstoppers aside, it becomes too late to raise fundamental
design issues that were reviewed extensively by the WG prior to
forwarding the first-round documents to the IESG, went through
IETF Last Call in 2007 and 2008, have been implemented and
tested experimentally, and where the basic framework and
changes from the 2007-2008 specs were reviewed by the WG,
passed through IETF Last Call, and were approved by the IESG in
September of this year.

There are places in the comments below where I've tried to
summarize what I believe is WG opinion or WG consensus, but
even those remarks should be taken as personal opinion rather
than the formal opinion of the WG.  The WG has deliberately not
been given a preview of these notes and is seeing them for the
first time at the same time as reviewers and others.  I do hope
that some of the notes will act as a basis for discussion.

For convenience, I'm going to try to consider the review
comments by topic (there is some overlap among reviews) rather
than responding to each review comment separately.  When it
seems useful to identify particular comments, I've used the
first name of the author of the comments and the date of the
posting.  To avoid a hugely long note, this one is strictly
about 5336bis (the SMTP extensions); a separate note will be
posted about each of the other relevant documents.  Note that,
again speaking personally, I agree with some of the comments
even though WG consensus, as reflected in the current drafts,
does not.  

I have not commented on, or even listed, specific issues whose
consequences appear to be entirely editorial, even if the
editorial issues are quite broad.  I'll leave those comments to
the authors/editors (subject, as usual, to WG approval).  I
have also omitted most issues that seem to have been more than
adequately covered in mailing list discussions.

The balance of this note is about the reviews and on-list
comments about draft-ietf-eai-rfc5336bis.  It does not include
comments that appeared to be redundant unless listing different
versions seemed to add perspective or other information.  While
I've commented briefly on the VRFY and EXPN issues below, those
are being much better covered in the discussions between Dave
and Shawn, Ned's analysis, and the subsequent than I could do
or have attempted here.  Separate notes about the other reviews
will follow.

(1) Overview, scope, and summary:

Claudio Allocchio 2010-12-22 wrote: 

> The document specifies how to apply internationalization to
> some SMTP elements, and to some Header elements which
> depends from the SMTP elements.  However there seem to be a
> discrepancy from what the abstract states, and what is
> elaborated into the document details: the document also
> deals with quite a number of elements which involve
> bodyparts, and affect the whole email legacy system, and
> beyond, such as logs, traces, and wherever email addresses
> are used (certificates, etc.). This seems thus to go beyond
> what the summary states, even this fact is not explicitly
> discussed. It is unclear thus if this "update" of RFC5321
> and RFC5322 is only limited in scope or not. My reading of
> the document say it goes beyond, and implicitly affects also
> what is being transported into the email system.  This is
> thus a serious issue to be solved before the document can go
> to the Standard Track.

With the understanding that the abstract, and several other
editorial issues with the text, could certainly be improved,
please note that specifications for header fields that contain
trace-related information (and that are inserted by MTAs) have
appeared in all of RFCs 821/ 2821/ 5321.  The trend over time,
starting with some explicit DRUMS decisions, has been to move
more of that information into the x21 member of the pair rather
than having it split between x21 and x22.  So the choice with
this document is between updating information that appears in
5321 and creating a split between envelope and header
information that is different than the base email documents.
The WG concluded that tracking the organization and information
content of 5321 would be less confusing than trying to create a
new model.

By contrast, the information about certificates, etc., was
introduced into the predecessor of this document at the
insistence of the community and the IESG during Last Call on
RFC 5336.  We could take it out of here, possibly by reopening
the Framework document and inserting additional "other
consequences for email and the Internet" sections there.  FWIW,
my personal opinion is that doing so would not add
significantly to either document comprehensibility or protocol
quality, but that is a matter for WG and community consensus.

Dave Crocker 2010-12-22 wrote:

> * Beyond <encoded-word> -- The work, here, appears to have
> two goals. One is to add support for Unicode in a
> <local-part>; that is, support for internationalized
> addresses.  However note that <encoded-word> and <A-label>
> already accomplish this in a way that is transparent to the
> existing email infrastructure; only the end-systems need to
> understand it. 

When the discussions that led to the establishment of the EAI
WG started, the goal was seen as expanding the syntax of
<local-part>, as you suggest.  It turns out that doing that is
not really feasible, at least without leading to horrible
kludges, so the WG was chartered (now four or five years ago)
to extend header fields as well.  As mentioned elsewhere, the
IETF has had repeated opportunities to review those decisions
and has formally approved them.  Even mapping back and forth
between A-labels and U-labels is, in the real world, not as
transparent as one would like (or obtain if one knew that
everything that looks like a domain name is a domain name in
the public DNS -- see draft-iab-idn-encoding or review the
slides from the Hiroshima technical plenary for more details on
that issue).  

Curiously, if the WG had not been constrained by very old (and
necessary) rules about how the email environment works, its job
might have been much more simple and more in line with at least
one extrapolation from the comment above: if we had
standardized local-part formats sufficiently to avoid having to
prohibit interpreting or tampering with those fields enroute,
the WG could, in principle, have written extremely short
specifications allowing encoded-words (possibly restricted to
encoded UTF-8) in local-parts and essentially been done.  While
I doubt that solution would have been acceptable to many
implementers and user constituencies (in part because "only the
end-systems need to understand it" has never worked as well as
we often hope it does), it would have been much easier in terms
of specification simplicity.

In case it is useful for understanding the WG's choices in this
area, the WG in principle had a choice between adopting a
minimal change --relying as much as possible on trick encodings
and dependencies on endpoint handling as possible -- and moving
in the direction of relatively comprehensive use of
natively-encoded (see below) Unicode in email with some
transition provisions that would gradually atrophy.  In
practice, there was no choice at all: internal discussions made
it clear that a standard based on the former approach would
simply be ignored and localized or proprietary solutions
adopted and only one based on the latter had an reasonable
chance of implementation and global deployment.  Extrapolation
from the independent-of-EAI work reflected in
draft-iab-idn-encoding -- work that originated in the concerns
of a couple of operating systems that dominate the network
today-- also suggests that more dependency on exotic,
IETF-specific encodings is not a direction we should be going
in unless it is absolutely necessary.

> The second goal is to support Unicode in the
> more "native" form of UTF-8. (The quotation marks are
> because UTF-8 is not native Unicode, either; it is a highly
> encoded form of Unicode...)

As others have commented on the mailing list, the above and
some of the comments that followed in the review demonstrate a
fairly profound misunderstanding of Unicode and Unicode
terminology.  The only encoding form more "native" than UTF-8
(or UTF-16 or UTF-32) is a set of abstract integers that have
no intrinsic computer encoding form.  The IETF has established
a number of "non-native" encoding forms, including UTF-7 and
the Punycode-derived ACE form, but they are avoided where
possible in these specifications.

See the comments under #7 below for an additional aspect of the
ASCII-versus-Unicode distinctions.

(2) Explicitness versus discovery in MIME and related topics,
including a possible parameter on the MAIL command.

Claudio Allocchio 2010-12-22 wrote: 

> Another big issue is about blending the boundaries of
> servers in the transport system (MTAs), clients (UAs) and
> their roles, when the document specify that servers shall
> use guessing techniques from looking into bodyparts to
> understand if what they're transporting is internationalized
> according to this specification or not.  This is IMHO a
> basic violation of the role distiction in the legacy email
> system, and affects the MIME model al well.   

Yes, except that this "blending" already occurs with MIME and
other SMTP extensions.  Over the years, we have made some
decisions to be very explicit about parameters and
pre-identification of content and other decisions to force
systems to inspect or parse content because we did not believe
that such parameters could be trusted (or that any sensible
implementer would trust them).  I hope we have gotten most of
those historical decisions right although I have some doubts,
at least in retrospect, about some of them.

Given that history, I don't think it is appropriate to assume
that the EAI effort is leading us into a role distinction sin
or not because we are already there.  The question is what the
right choice is for this particular situation.  The WG
considered that choice at great length (albeit in the 2006-2008
period rather than more recently, but that part of the
specification has not changed) and came to the conclusion
that forcing inspection was the better plan.  Testing with the
Experimental versions of the protocols did not expose serious
problems with that choice. 

I am sure that the WG would be happy to reconsider this topic
is appropriate, but some solid evidence that there is a problem
and that the current decision is wrong would be far more
helpful in that regard than assertions based on a philosophy of
email that IETF protocols have already violated.  That
assertion, and some evidence, have been made for a MAIL command
parameter:

> I have problems in understanding why an SMTP server should
> peek extensively into the message geing transferred to
> understand that it is handling an interationalzed message.
> "There is no ESMTP parameter to assert that a message is an
> internationalized message." in my opinion should be fixed in
> another way, by ADDING an appropriate ESMTP parameter to
>...

Dave Crocker 2010-12-22 wrote:

> 4.  The SMTP extension needs to have the client use an
> explicit signal when it is sending a message encoded in
> UTF-8. This is easiest as a parameter to the MAIL command.
> The current specification creates a more complex and almost
> heuristic model for distinguishing ASCII from UTF-8 use.

Ned Freed 2010-12-22 wrote:

> (0) I cannot emphasize strongly enough how important it is
> to have a UTF8SMTP sort of parameter on the MAIL FROM to
> indicate that a message employing this extension is being
> submitted/relayed/delivered.  
>...

Ned Freed 2010-12-23 wrote (small excerpt):

> Again, headers don't have the proper semantics. A MAIL FROM
> parameter, OTOH, does.

The working group did consider the tradeoffs between (i)
forcing a server (really relay) to do a complete MIME parse if
it received an extended message and had to decide what to do if
the next hop did not support the extensions and (ii) putting in
a parameter whose presence or absence might not be trustworthy
given a history of, e.g., text/plain body parts being sent with
distinctly non-ASCII content but without charset parameters,
misuse of "name extensions" in lieu of adherence to
Content-Type, etc.  The decision of the WG was reflected in RFC
5336 (after IETF Last Call, etc.) and I note that none of the
known implementers of the RFC 5336 spec are objecting to that
decision now.  The consensus of the reviewers now seems to be
that the WG got it wrong.  The WG needs to review this again;
the change is certainly easy to make should it be appropriate.

(3) Fundamentals of handling syntax changes from other
specifications (especially 5321/5322):

Claudio Allocchio 2010-12-22 wrote: 

> The document quite often repeats rules from other
> specifications, AND re-describes these rules. This is
> dangerous (it makes very difficult to keep aligned
> documents, and will lead to different interpretations) and
> not needed.

The tradeoff, I hope obviously, is with the reader having to
flip back and forth repeatedly between documents to have any
hope of understanding the specification accurately.  I'm
personally sympathetic to your concern, but many WG
participants disagree. 

Specific suggestions that could be used as a model would be
helpful to the WG and the editors.

Claudio Allocchio 2010-12-22 wrote: 

> 3.6.1 "the server normally sends..." --"the server sends"
> (sorry... we do not like ABnormal servers!) :-)

This is another example of a narrative/ informative comment.
We know about exceptions.   Perhaps the sentence would be
better stated as "an RFC 5321-conforming server sends...".

(4) Gateways and related boundary systems

Claudio Allocchio 2010-12-22 wrote: 
 
> This specification updates a number of SMTP and Headers
> fields from RFC5321 and RFC5322. However these fields are
> also used in Gatewaying to other email systems, which are
> still existing. Section 3.7 in RFC5321 describes specific
> cases. However this memo provides no clue on possible effect
> that internationalized fields can have in case gatewaying
> occurs.  Gatewaying functions strongly rely on these values,
> and the introduction of these modification should describe
> this scenario, too. I suggest to include a specific section
> about these issues, and suggest at least a number of

I'm not sure I understand why Gateways are different from
anything else.  If a gateway doesn't advertise the extension,
then it should never see a message that contains either
non-ASCII envelope addresses or non-ASCII content in the
headers (other than as encoded-words, of course).   The WG does
have some placeholder benchmarks for advice of various sorts;
IMO it would be quite useful to supply some of that advice for
gateways once sufficient operational experience had
accumulated.

But the only really hard gateway problem I see is if a given
gateway MTA interfaces to two back-end systems, one of which is
ASCII-only and one of which is broadly UTF-8 capable.  It will
either need to be reconfigured into two virtual gateways, one
of which advertises the extensions and one of which does not,
or it will simply have to reject mail with UTF-8 header fields
that is addressed to a system that can't handle it.  I note,
however, that such a gateway already has a problem that it
presumably has some mechanism to deal with: if it receives mail
with UTF-8 (or other non-ASCII) headers from one of its
i18n-capable systems with an Internet destination, it already
has to reject the message or perform some sort of conversions
that are, necessarily, not described in IETF specifications.

(5) The legacy of the Experimental protocols

Claudio Allocchio 2010-12-22 wrote: 

> The EAI WG charter states that in handling email message we
> shall be "consistent with the long-term view that email with
> invalid addresses or syntax should be rejected, rather than
> fixed up in transit between submission servers and delivery
> servers.". However a number of sections in the specification
> seem to contradict this, where they suggest actions, by
> exammining message elements (in the SMTP and in the Headers
> and in the Bodyparts) in order to try to deliver anyhow.
> This is a subtle line between maximising the service chance
> to deliver, and "fixing" things during the transaction, and
> it can encourage implementors to do many of these acions,
> instead of stiching to the idea "not supported --non
> delivery".

The experimental protocols described in RFC 5336 and elsewhere
included a rather complex mechanism for in-transit downgrading.
I'd encourage you to go look it up if you are interested, but
the general design was based on the 8BITMIME model -- a sender
discovering that the next-hop server wouldn't accept the option
could either reject the message or engage in a downgrading
process that was distinctly not lightweight.  The conclusion
from implementations and experiments was that it was too
heavyweight, too hard to get right, and that the number of
cases in which the downgrade procedure could not be applied
would be very confusing to users.  All vestiges of that
machinery should have been removed from the current versions of
the documents.  But I found some in an earlier review and it
may well be that some of the other review comments have
identified others.  If so, they clearly need to come out; the
editors may need help in identifying specific examples.

(6) Server actions when UTF8SMTP[bis] is not supported (and
Section 3.6.2).

Claudio Allocchio 2010-12-22 wrote: 

> The specification does NOT suggest a preferred order among
> the 3 possible options when the SMTP server state it does
> not support UTF8SMTPbis. Having 3 "MAY" options at the same
> level can create a very uncertain email service behaviour,
> and a very bad user's experience! This is consistent with
> what happens in other cases for other "options" in the email
> system (like DSNs and MDNs), but there should be at least an
> informative document describing the various scenarios, and
> giving some guidance to implementors in order to avoid
> totally incosistent behaviours.

Small bit of calibration about this: remember that the EAI
specs cannot do anything to dictate the behavior of an SMTP
server that does not support the EAI extensions and may have
never heard of them.  There is also a general design
assumptions, shared with other [E]SMTP assumption, that the
specification should not make any changes to general email
behavior that are not necessary consequences of what the
extension is trying to accomplish.

On rereading the document after going through these comments,
it is clear to me that the document would be improved by being
much more clear about what text is a change with EAI and
normative, what is a necessary consequence of provisions in RFC
5321 or other applicable standards, and what is just
informative narrative.  I don't know whether the WG shares that
point of view, nor whether it is desirable to try to rewrite
the document along those lines at this stage (in general, these
are not "known technical defects" so, if we could be assured
that Draft Standard versions of the documents would appear on a
timely basis, it might make sense to let them go for the
present).

These three options are simple extrapolations of rules that
already exist in RFC 5321: SMTP-senders must not send invalid
messages into the public Internet and must reject or bounce
such messages if they cannot be fixed up; only Submission
servers (and gateways) are permitted to make significant
changes to fix things up; and, implicitly, if mail is not
deliverable along the normal path and one knows a legitimate
path that would work, one is permitted to use it.  

Note that the first (MSA) option is useful for non-ASCII
address local-parts iff the MSA has an external source of
information that permits substituting ASCII-only addresses for
non-ASCII ones.  For address domain-parts and miscellaneous
header fields, there are typically encoded-word (or A-label)
options available.  Perhaps the document should be more
explicit about that.

If that flexibility "just in practice break[s] the concept of
universal email common service, creating islands which do not
talk to each other", then we would already have seen that
problem.  For example, the specification of 8BITMIME can lead
to the same situation without EAI: if a relay receives a
message with C-T-E 8bit or binary, and the first-preference
reachable MX doesn't support that option, it can downgrade the
message, reject it, or, IIR, try other servers at the same
preference level to see if one of them will provide support for
the extension.

Ned Freed 20101222 wrote:

>> 1.  If and only if the SMTP client (sender) is a Message
>> Submission Server ("MSA") [RFC4409], it MAY, consistent
>> with the general provisions for changes by such servers,
>> rewrite the envelope, headers, or message material to
>> make them entirely ASCII and consistent with the
>> provisions of RFC 5321 [RFC5321] and RFC 5322 [RFC5322].

> This is a bad idea no matter how you slice it. Such
> transformations depend on the availability of address
> mapping (from a UTF-8 address to an equivalent ASCII
> address) information; an agent being an MSA is neither

However, RFC 5321 already requires that an MSA not inject a
message into the public Internet until/unless it is valid.  And
it (and RFC 4409) effectively give the MSA permission to do
anything an MUA can do in order to accomplish that.  There is
no (intentional at least) assertion in 5336bis that the MSA (or
the MUA, for that matter) must have access to address mapping
information, just no requirement that it not use that
information if it has access to it.  If it doesn't have the
needed information, then there are clearly a lot of things it
cannot do; the quoted statement above is then just a
reiteration of the 5321 prohibition on passing trash into the
network (whether it is worth reiterating is a separate matter).

Of course, if a message is transferred from an MUA to the MSA
that contains non-ASCII header information in UTF-8 form but no
non-ASCII addresses, the above provisions don't imply the MSA
doing anything more than mapping that header information into
encoded words.  Again, there is no requirement that the MSA do
that mapping; the statement simply reiterates, in an i18n
context, the general permission for the MSA to take something
that works in a user->MUA->MSA environment and fix it to
conform to public Internet specifications.

It is worth noting that, during WG discussions, several strong
claims were made about intentions to implement both UTF-8 to
encoded word mappings and tables to support precisely the type
of address mapping suggested above in selected MSAs.  Assuming
people were telling the truth (and I certainly assume they
were), the conditions specified above, including address
mapping, are going to be exercised.

> But the bigger problem by far is defining what
> transformations are permissible. The present framework
> document is basically silent on what's allowed, so I for one
> have no idea what it means to be "consistent with the
> general provisions for changes by such servers" actually
> means. And unless you're willing to spell this out, I see
> this as practically guarnteeing a slew of interop problems.

With the understanding that my comments above go much further
than the existing text, that the WG hasn't agreed to any
expansion, and that Dave seems to be arguing for less text that
is just repetition of existing rules in 5321/5322, how much
further would you suggest going in spelling things out?

(7) Suboptimal use of ABNF and ABNF constructions

Dave Crocker 2010-12-22 wrote:

> * Rule Meta-Naming -- The 'u' preface for revised rules is a
> reasonable idea, but appears to be problematic.  These rules
> replace existing rules in other specifications. There needs
> to be an explicit and decision about the handling of this,
> and it needs to be applied consistently, either directly
> replacing the rules or else re-naming them consistently,
>...
> { On reflection, I think the "u" rule-naming convention is
> problematic.  If the current specification is replacing a
> rule previously defined elsewhere, it needs to use the same
> rulename.  This is simpler and clearer.

(and several comments by others, noting that Dave's second
recommendation appears to contradict Claudio's)

As I commented at more length in a note to Dave, Claudio, and the
EAI list on 2010-12-24 14:20 -0500, we've had a lot of trouble
getting ABNF for these documents right, much less optimal.  A
significant part of the optimality problem is that that there
seems to be no established good way to update one specification
by extending its syntax -- a problem that the IETF community
will almost certainly have to deal with again as we
internationalize more protocols.

In the particular case of ASCII and UTF-8 constructions, even
though ASCII is a compatible proper subset of UTF-8, there
is often a need to distinguish the two because plain-ASCII
works in legacy protocols (and does not require special
normalization or comparison rules that might, themselves, be
controversial) while UTF-8 does not work in the legacy cases
and raises more complex normalization, comparison, and
collation issues.   FWIW, we had the same problem in the
IDNA2008 specs because it was necessary to distinguish between
a traditional DNS "preferred syntax" name (aka "LDH") that did
not start with an ACE prefix and one that did use that prefixed
form, in spite of the fact that the latter, by design, would be
treated as a traditional "LDH" name by any application that was
not aware of the special IDNA constructions.

For the particular case of EAI, the WG decided to go with the
"u" prefix on new rules that superseded 5321 or 5322 rules.
The WG tried to be consistent with that prefix on new rules,
and preserve most of the original rules.  All rules were
extended rather than overwritten.  The WG had discussed,
somewhat earlier, the possibility of simply coping the entire
ABNF collection from RFC5321/5322 and then extending or
modifying rules as needed.  That would have eliminated the need
to reference back to those documents for ABNF but the idea was
rejected, largely to avoid exactly the problem with updating
that several reviewers commented on.

Again, this is a general problem: if the community has a clear
preference as to how it should be dealt with, I'm nearly sure
that the EAI WG will conform.  In the absence of a general
solution, my personal guess is that the "u" convention (applied
accurately and consistently) may turn out to be the best we can
do.

(8) MIME restrictions and nested encodings

Dave Crocker 2010-12-22 wrote:

> 5.  The SMTP extension needs to remove all restrictions it
> imposes on MIME content-type.  A major reason that MIME was
> successful was that it was transparent to the transfer
> infrastructure.  The current SMTP extension specification
> changes this model, which actually increases the barrier to
> adoption of Unicode in email.  It needs to be easy for two
> MUAs that support Unicode in the email header to exchange
> mail even when the infrastructure does not support UTF-8.

Right.  Unfortunately, there is already a restriction in MIME
that links Content-Type to Content-Transfer-Encoding, the "no
nested encodings" rule.  That rule is imposed primarily as a
restriction on message/ subtypes.  And it works because MIME
assumed that all header information was going to be ASCII and
hence able to be transmitted in C-T-E: 7bit.

So now we have a little problem: MIME is designed to permit
forwarding messages in encapsulated form using message/rfc822.
But message/rfc822 uses ASCII headers only and has to obey the
"no nested encodings" rule.  One would like to be able to
encapsulate an EAI-conformant message and even forward it into
a legacy environment.  There are actually a number of options
for doing that, including deciding that a message with
non-ASCII headers cannot be forwarded into a 5321/5322
environment (pretty obviously a bad idea to prohibit that),
creating a new top-level MIME type with different rules (the
transition issues and implications for legacy MUAs are pretty
astonishing), or creating a new "message/" subtype with
different rules and what may be seen as an additional
Content-Type/ Content-Transfer-Encoding entanglement.  After
considerable discussion, the WG selected the latter (see #13
below) and got IETF signoff in RFC 5336.

(9) Strong conformance language

> - bullet point 3. I do not see any need to state that
> clients MUST reject...  and MUST be fully compliant... etc.
> This is implicit in any specification.  If there is a
> specific need to add this detailed behaviour (to prevent
>...

Our general practice with email specifications has been to
specify conforming behavior and simply leave non-conforming
behavior "outside the scope of the specification".  That
doesn't work in this case because, according to the WG's
analysis, a partial implementation --whether outside the spec
or otherwise-- would be bad news for interoperability.  So it
seemed necessary to be very clear about what clients were
expected to do and what servers could reasonably expect.
Whether that was better done with this text, with an
"Implementer's Note" (which has no normative status), or with
an extended explanation of the problems and a threat or two
would seem to me to be largely matters of editorial taste,
especially for a Proposed Standard.

(10) Security-related issues

> there is one more issue: as internationalized items will
> also result into logs and traces, their presence may affects
> trouble ticket systems used by security operations teams
> (CERTs/CSIRTs). And they also can affect quick handling of
> incidents, has it may require more time to correctly read
>....

Agreed.   I assume the editors will consider your proposed text.

(11) Editorial issues with significant substantive
implications.

Dave Crocker 2010-12-22 wrote:

> { Note - the term "mailbox names" is not defined here or in
> RFC 5321.  In RFC 5321 it appears to be used to mean
> local-part; however because its precise meaning is unclear,
> I strongly urge NOT using it here at all. Instead I suggest
> using whatever ABNF rulename is appropriate.  This
> guarantees clarity.

Well, while getting rid of "mailbox name" or defining it very
precisely, would probably be an improvement, there is no ANBF
rulename that would really be appropriate, at least without
additional decoration.  As you and others have pointed out
recently, rules for <mailbox> appear in both 5321 (where it
refers to the mailbox name (sic) itself, e.g.,
local-part@domain-part) and in 5322 (where it refers to
something that might include a display name and assorted
spacing).   I have no doubt that this document should be as
clear and precise as possible, but its authors should not be
blamed for trying to work around the errors of the authors of
5321, 5322, and their predecessors. 

Dave Crocker 2010-12-22 wrote:

>> The value of "uDomain" SHOULD be verified by IDNA
>> definitions [RFC5890].  If

> { I do not understand what "SHOULD be verified by IDNA
> definitions" means.

> If it means that the uDomain "SHOULDbe verified" then it is
> out of scope for this specification. There is no reason that
> SMTP rules concerning verification of Unicode-based domains
> needs to be different from ASCII-based domains.  In
> particular, verification of a Domain name is specified
> elsewhere.
>...

Editorial issues aside, 5321 contains some very
carefully-written language whose intent is to require that
domain-parts be validated lexically but to avoid requiring
that they be looked up in the DNS until/unless that is
necessary for some other reason.  The "lookup" side of
IDNA2008 (in RFC 5392) makes more or less the same circle
around this issue but with the understanding that lexical
validation of U-labels is a somewhat more complex process than
checking against a short list of characters, positions, and
lengths.   I believe that is what the author was trying to
capture.  Certainly it could have been expressed differently.

Dave Crocker 2010-12-22 wrote:

>> Otherwise, surprising rejections
>> can happen during temporary failures, which users might
>> perceive as a serious 

> { Probably not just temporary failures.  Having services
> that are meant to be redundant with each other actually
> provide different semantic behavior is just plain dangerous.
> It is essentially guaranteed that they will cause problems.}

Sure.  But you argued extensively elsewhere for not duplicating
text that appears elsewhere and, IMO, more important, for not
making statements that go beyond a rather narrow reading of the
requirements for internationalized email.  

Ned Freed 20101222 wrote:

> So you can use alternate MXes, but not alternate A records?
> I actually agree with this - an alternate A is presumably
> another address for the same host, so no point in trying it.
> But this needs to be spelled out.
 
> security considerations text. The issue, I hope, is obvious:
> There are sites aplenty with large numbers of MX records and
> which won't support this extension any time soon. And all
> too often those MXes are *slow* to respond. So sending a
> message that causes them all to be tried is ... you get the
> idea.
 
> In fact I'm tempted to go one further and say that if an MX
> at a given priority is found not to support this extension,
> you SHOULD assume that MXes at lower priority don't either.

And this is where we descend into a situation that extends far
beyond EAI.  The problem, as Ned's note almost suggests, is
actually quite general.  For example, if we had

   foo.example.com. MX 0  foomail0.example.com.
   foo.example.com. MX 10 foomail1.example.com.
   foo.example.com. MX 10 foomail2.example.com.

and "foomail1" supported 8BITMIME with downgrading and
"foomail2" did not, a system sending a message containing
C-T-E: 8bit that reached foomail1 would have the message go
through with downgrading if needed but would get a rejection or
option not supported condition if it happened to get to
foomail2.  Or it might get lucky and hit foomail0 and find out
what it actually supported.  Similar situations could apply
--at least wrt rejection versus bounces-- if foomail1 supported
a different set of antispam provisions than foomail2, or, for
that matter, if both deferred antispam processing to foomail0.

As far as I can tell after a very brief scan, there is no
current normative IETF specification that discusses this
situation and gives guidance about it.  In retrospect, we
probably should have at least described the issues with
non-identity of services in RFC 1425, but we didn't do that,
much less make specific recommendations.  

I agree with Ned that it would make a lot of sense to
explicitly say "no alternate A records because, if they don't
represent the same host, you should be using MXs" and "if the
first MX you try doesn't support the extension, assume the
others don't either and stop".  But, because of the generality
of the problem, this document doesn't seem to me to be the
right place to say that.  5321bis probably would be a
reasonable if we think it is a clarification, but especially
with YAM in a coma and no mention of this topic in the
pre-approval document, the odds of my getting to 5321bis before
this EAI work is closed out are in the "low to none" range.
Worse, while someone could quickly cons up an I-D to specify
this behavior and try to push it through as a Proposed Standard
that 5336bis could reference, processing of such a document
would essentially constitute a claim that the spec is new and
not a clarification to 5321, precluding folding it into 5321bis
later, at least without marching that document along the
standards track to Draft Standard first.  Suggestions about how
to get out of that mess are welcome, but I don't think it is an
EAI problem (even though 5336bis might plausibly be modified
--or the Framework document reopened and modified -- to explain
just how ambiguous the whole situation is).

For RFC 5336, it was important to call the issue out because
the "downgrade or not" behavior is even more complicated than
the potential behavior variations with 8BITMIME (remembering
that one can offer the option but, if the next hop doesn't
support 8bit transport, one can either downgrade or reject) and
other extensions..  Whether it is necessary to say anything at
all in 5336bis, given how little we have said in places that
perhaps establish precedents, is debatable.  But please let's
not suggest expanding on this case in this document to cover
other email environments, especially in the context of urging
the the spec keep its scope and assertions narrow.

Dave Crocker 2010-12-22 wrote:

>> 3.6.4.2.  VRFY and EXPN Commands and the UTF-8REPLY
>>Parameter 

>> If the VRFY and EXPN commands are transmitted with the
>> optional parameter "UTF-8REPLY", it indicates the client
>> can accept UTF-8 strings in replies to those commands.
>> This allows the server to use UTF-8 strings in mailbox

> { Is this extension trying to to say that a client might
> support UTF8SMTPbis but not be able to access UTF-8
> replies???  Under what circumstance is this reasonable? }
>...

Not what it is intending to say (could have been expressed
better).  Consider the following scenario: Client sends EHLO,
server offers UTF8SMTPbis.  Client sends VRFY as its next
command, using an ASCII-only argument and no parameter.  The
server has no clue as to whether or not the client is
EAI-compliant or otherwise has any idea what to with a
reply containing a non-ASCII address.  For the server to return
a non-ASCII string under those circumstances would be in
violation of 5321.

(12) The keyword denoted as "UTF8SMTPbis" in the drafts.

While the changes between the Experimental protocols and the
present draft are not large, they are also not trivial.  After
fairly extensive discussion, the WG concluded that these new
specifications should be denoted by a different SMTP Extension
keyword than the "UTF8SMTP" that was used by the Experimental
version (RFC 5336).  In large part because of concerns that
some implementation would be deployed in advance of the
specification becoming final, the WG concluded that the new/
final keyword should be assigned only as part of the IESG
approval process.  "UTF8SMTPbis" is only a placeholder and will
not be that final keyword.  It may be worth noting that, if the
protocol is changed to require a MAIL command parameter, that
WG decision will have turned out to be quite important.

Anyone who strongly believes that some variation on the
"UTF8SMTP" theme misrepresents what is actually happening in
these specifications is welcome to suggest other keywords to
the IESG.

(13) New content types for embedded messages.

Dave Crocker 2010-12-22 wrote:

> 3.  There probably needs to be definition of MIME
> message/uni-rfc822, specifying Unicode support within a
> message contained in MIME.  I'm less clear whether there
> needs to be a MIME Content-Transfer-Encoding form specific
> to UTF-8.

See message/global in draft-ietf-eai-rfc5335bis.  This is an
area where the WG's decisions about what material belonged in
which document followed the advice to keep header and body part
discussions in 5335bis rather than 5336bis.

(14) Procedural and quasi-procedural issues

Dave Crocker 2010-12-22 wrote:

> 1.  The Framework document is normative and needs to be
> completed along with the other two specifications, since
> they quite reasonably state a normative dependence on the
> Framework document.
>...
> * Framework -- The Framework document[1] is (correctly)
> referenced as required reading.  It supplies essential
> terminology and architecture for this specification.  In
> fact it is a specification, complete with formally normative
> vocabulary.  This means that it must be a normative
> reference by rfc5336bis.  It therefore also means that the
> Framework document needs to be completed before the current
> specification can be standardized.

Note that the Framework document went through IETF Last Call
and was approved by the IESG last September for publication as
Informational.  The WG was agnostic on whether it should be
Informational or Standards Track.   Since it is still in the
RFC Editor queue awaiting references to the documents now under
review, I assume that, if there were sufficient consensus that
it is normative and should be on the Standards Track, the IESG
could be persuaded to revisit that decision.  In addition,
while I don't believe that anything in these reviews requires
doing so, the document could, in principle, be recalled by the
IESG and material changed or altered.

Dave Crocker 2010-12-22 wrote:

> * Infrastructure Requirement -- Unless this option is in
> force, carrying internationalized email in a MIME part is
> prohibited.  This is out of scope for the working group and
> it is a counter-productive rule.  Imagine if the same type
>...

There may well be an editorial problem with how this is written
up or even with how "internationalized mail" is defined.  But I
see nothing that supports the apparent hyperbole in the
comments above.  Remember that 5322 and the base MIME specs
prohibit non-ASCII headers, so messages containing such headers
(as the message header or MIME headers) are prohibited by those
specs without some new extension(s).   There is no
(intentional) prohibition in these documents against carrying
non-ASCII text in mail messages.  And, if one wanted to
pseudo-encapsulate a message and its headers by treating it as,
e.g., text/plain with charset=utf8, rather than as
message/rfc822 (or message/global), these specifications don't
prohibit that either (might not be a good idea because of the
loss of recognition and handling of boundary markers and internal
Content-Types and Content-Transfer-Encoddings, but that is not
a problem for this specification).

So, unless the comment above is just the result of confusion
brought on by an editorial problem, I don't see where you are
going with it.

(15) Future of rfc5336bis.

Ned Freed, 20101222 wrote: 

> One final comment. Several of the issues that have been raised
> are so fundamental that once they are addressed (point (0)
> especially), this document is almost certainly going to
> require another full review cycle.

Sadly, I agree.  I even believe that, were none of these
fundamental issues present, the number of editorial corrections
(including ABNF matters) that are required will be sufficiently
extensive that the risk of introducing new problems would
require such a review cycle.

     john

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 27 Dec 2010 11:58:05 -0800

On 12/27/2010 4:24 AM, Charles Lindsey wrote:
> On Fri, 24 Dec 2010 00:33:32 -0000, Ned Freed <[email-masked]> wrote:
>> envelope.
...
> However, I do not quite see the necessity for it. If some MTA announces the
> capability UTF8SMTP, then it is declaring its willingness to accept messages
> destined for utf-8 addresses and to do the "right thing" with any problems that
> might arise

Any protocol model that requires a participant to do deep inspection of the data 
or to do some form of guessing is likely to be problematic. It can invite errors 
and/or higher handling overhead.

The current specification actually suffers both of these.  It requires deep 
inspection by the server, AND it requires the server to guess whether an 
ASCII-only message is meant to be part of the legacy processing environment or 
whether it might really be subject to Unicode processing, in spite of having 
only UTF-8 characters below x07f.  (The latter is part of the fun of having 
Unicode be a superset of ASCII.)

While it might be safe to treat an ASCII-only message as being part of the 
legacy system, it might not.  For example, the set of recipients that support 
legacy vs. Unicode might be different and this can change the nature of a 
conversational exchange.  Finding out about the limitation with the first 
posting is better than with a later posting.

That is, the fact that an SMTP option is involved means that the goal is to move 
the entire email infrastructure over to Unicode support.  Both sides of the SMTP 
exchange need to know whether that move has taken place, without having to guess 
or do deep inspection.

It also happens that an option to the MAIL command is cheap, so why not do it?

d/
-- 

   Dave Crocker

---

From: Barry Leiba <[email-masked]>
Date: Mon, 27 Dec 2010 15:58:44 -0500

> destined for utf-8 addresses and to do the "right thing" with any problems
> that might arise (such as generating a bounce if some MTA further down the
> chain finds a problem). Essentially the same as current MTAs do with ASCII
> messages that turn out to be undeliverable.
>
> If some client tries to send such a message without first checking that the
> UTF8SMTP capability has been advertised, then it is in breach of our

That isn't quite enough.  If a message goes through a couple of MTAs
OK, and then there comes an MTA that doesn't do UTF8SMTP, how is this
MTA to know that the message has UTF8 and the situation needs to be
dealt with? The submitting client knew, but we're well past that, and
the only way to tell now, lacking some clue in the protocol, is by
inspection of the message.

As Dave says, the parameter is cheap; let's do it.

Barry

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 27 Dec 2010 16:31:52 -0800

Folks,

I'm sending this under separate Subject line "cover" because I think there is a 
basic issue that needs some careful attention:

and

On 12/27/2010 11:39 AM, John C Klensin wrote:
 > collation issues.

One of the most difficult aspects of integrating work from outside the IETF into 
protocols done inside the IETF is the multicultural "diversity" between the 
outside and inside communities. The two communities have different terminology 
and different technical perspectives, yet the result must be something viable to 
both worlds.

In this case, a key point to realize is that this is not merely work within the 
Unicode community.  It has to make sense for Internet/IETF protocol reviewers, 
implementers and users.

That challenge is exacerbated by the fact that the current work is being grafted 
onto a long-running system with its own peculiar realities involving underlying 
data "representation" versus superficial data "encoding".

And then there are the further opportunities for confusion by the fact that the 
older environment is based on a data form that is a subset of the newer form and 
the fact that some encodings can sit ON TOP OF some other encodings!

All of this should make everyone extremely cautious about terminology that gets 
used:  It needs to be very precise and very accurate, and it needs to work for 
readers who are not experts in one culture or the other.  Consequently the 
observation that a reader might be relatively naive about one of the cultures 
ought to serve as a challenge to the writer, not a means of dismissing comments 
from the reader.

In fact, at least some of us reading the current specifications found things 
confused between ASCII vs. Unicode -- that is between the range of text 
characters being supported -- and between ASCII vs. UTF-8 -- that is between a 
7-bit encoding and an 8-bit encoding.  The former is semantic while the latter 
is syntactic.[1]

The fact that the Unicode world is used to saying UTF-8 to mean both semantics 
and syntax is not enough for the current work, in my view.  It needs to do the 
additional work of making sure that the reader has no legitimate opportunity to 
be confused between the two.

This is why I think the specifications should use Unicode when they mean the 
rich set of text characters, and UTF-8 when talking about the most popular 
encoding of those characters into an 8-bit form.  (As noted, there really are a 
variety of other encodings, including some that are idiosynchratic.)

d/

[1] For what it's worth, I'll note that before I sent my previous note, I did do 
some review of Unicode and UTF-8 terminology and could readily see that 
representation vs. encoding are not maintained as strict distinctions and that 
UTF-8 is used to mean both.  Rather than taking this a reducing the validity of 
my concern, it strengthened my feeling that the specifications need additional 
effort at clarity about the distinction.
-- 

   Dave Crocker

---

From: John C Klensin <[email-masked]>
Date: Mon, 27 Dec 2010 21:38:17 -0500

--On Monday, December 27, 2010 4:31 PM -0800 Dave CROCKER
<[email-masked]> wrote:

>...
> This is why I think the specifications should use Unicode when
> they mean the rich set of text characters, and UTF-8 when
> talking about the most popular encoding of those characters
> into an 8-bit form.  (As noted, there really are a variety of
> other encodings, including some that are idiosynchratic.)

Dave,

In case it wasn't clear, I agree that the above would be a good
idea.  The more difficult problem is how to discuss "ASCII
versus Unicode" since the collection of characters that make up
ASCII is a proper subset of the collection that makes up
Unicode, just as the set of ASCII single-octet code points as
described, e.g., in RFC 20, is a subset of Unicode expressed in
UTF-8 (where the ASCII subset of Unicode is the only set of code
points expressed in exactly one octet.

So, to be more specific, while I agree with you that Unicode
should be used for the abstract set of character and UTF-8 only
when that particular encoding is intended, "ASCII versus
Unicode" and "ASCII versus Unicode are ultimately equally
incorrect.

   john

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 27 Dec 2010 20:06:23 -0800

On 12/27/2010 6:38 PM, John C Klensin wrote:
>     The more difficult problem is how to discuss "ASCII
> Unicode,

That's why I cited that fact in my review and my recent note.

Nonetheless, "ASCII" refers both to a semantic and a syntax for text characters, 
which indeed invites confusion.

Still, ASCII versus Unicode refers to the semantic distinction between legacy 
email versus the enhanced version that is the goal of the working group.

> when that particular encoding is intended,

ok.

>  "ASCII versus

Well, it is essential to make the distinction between legacy and new semantics, 
since it is the essence of the current work.

The choice of terms is a separate issue.  If you have alternative labeling, that 
would be great to consider.

d/
-- 

   Dave Crocker

---

From: John C Klensin <[email-masked]>
Date: Mon, 27 Dec 2010 23:17:32 -0500

--On Monday, December 27, 2010 8:06 PM -0800 Dave CROCKER
<[email-masked]> wrote:

> Well, it is essential to make the distinction between legacy
> and new semantics, since it is the essence of the current work.

yes.  That was what people were trying to do with "ASCII versus
UTF-8".  As you point out, that doesn't work well.  "ASCII
versus Unicode" really doesn't work any better.

> The choice of terms is a separate issue.

Indeed.

>  If you have
> alternative labeling, that would be great to consider.

I hope someone in the WG will have a good suggestion.   If not,
we may be back to "ASCII" versus "non-ASCII" as the disjoint
pair, "Unicode"  to describe the international CCS (inclusive of
ASCII), and "UTF-8" to describe the particular encoding that is
used for this protocol and recommended by RFC 2277 (a
recommendation that is reinforced by draft-iab-idn-encoding and
other more recent work).

    john

---

From: Yangwoo Ko <[email-masked]>
Date: Tue, 28 Dec 2010 15:23:41 +0900

> recommendation that is reinforced by draft-iab-idn-encoding and

I hope so too. But I don't expect that we are going to have a nice
answer to this soon enough. If it still significantly matters, will it
be useful to list up the cases of possible confusions when current
terminogies are used? With the list, we may be able to add some notes
to one of our documents (possibly framework doc)  to address the known
possible confusions.

-- 
a human known as [email-masked] @ gtalk

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 28 Dec 2010 12:21:40 -0000

On Mon, 27 Dec 2010 19:58:05 -0000, Dave CROCKER <[email-masked]> wrote:

> That is, the fact that an SMTP option is involved means that the goal is  
> to move the entire email infrastructure over to Unicode support.  Both  
> sides of the SMTP exchange need to know whether that move has taken  
> place, without having to guess or do deep inspection.

Clearly the client needs to know what the server will accept, but I am  
still not clear why the server needs to know what the client supports  
(i.e. why a full two-way handshake is needed. Perhaps you could provide an  
example.
> It also happens that an option to the MAIL command is cheap, so why not  

Yes, I have no great aversion to a MAIL option as suggested, and the way  
this discussion is going it seems likely that the WG is going to change  
its mind on that. My aim in this thread is to establish exactly what will  
be gained by so doing, since in most situations it is not needed.

I mentioned one case where it _would_ help in my reply to Ned, and I see  
Barry has now identified another.

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 28 Dec 2010 07:23:25 -0500

> recommendation that is reinforced by draft-iab-idn-encoding and

It seems to me that one could take a cue from the IDNA work, which
differentiated A-label, U-label, LDH-label, and so on.

I haven't thought about this too carefully, have barely had time to
follow the discussion, and won't have time to think about it much more
for the next 3 weeks, but perhaps something like "A-set" for the
ASCII-only characters, "U-set" for all Unicode characters, and (ugly)
"X-set" for the set of characters that are in Unicode eXcluding the
ASCII subset?  I know that introducing and rigorously defining new
terms is hard (to return to the IDNA analogy, it took a _lot_ of time
to work out those precise definitions), but it seems that confusion
may continue otherwise.

Feel free to point and laugh.  This might just be a bad idea.

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 28 Dec 2010 12:37:39 -0000

> 	Quoted-string optionally followed by a parameter list.

I am still not clear what the precise difficulty is that we are  
addressing. Writing
     VRFY John Klensin
and finding it doesn't work is a problem with the present legacy setup and  
it is not our job to fix it (though fixing it as a byproduct of some other  
fix would be OK).

What we _are_ concerned with is

     VRFY [email-masked]

for which we _do_ need some extension. But why cannot we just say that if  
you want to send that VRFY then you MUST do an EHLO first to ensure that  
UTF8SMTPbis is available, in which case you know that the extension  
permitting "rené" is available. This leaves the legacy behaviour (not  
sending EHLO first and hence not being allowed to say "rené") untouched.  
And no need for any New-VRFY.

Or is what I am suggesting what you intended to convey by your second  
option above?

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 28 Dec 2010 09:06:03 -0800

Folks,

It occurs to me that we could make do with existing terms.  The challenge is to 
use them carefully:

               | Representation |  Encoding  |
               +----------------+------------+
               |                |            |
        Legacy |     ASCII      |   7-bit    |
               |                |            |
               +----------------+------------+
               |                |            |
International |    Unicode     |   UTF-8    |
               |                |            |
               +----------------+------------+

The change in our reference is to stop using ASCII to refer to an encoding.  For 
Internet ASCII, the formal definition usually is "7-bit ASCII in an 8-bit field, 
with the high bit off", but "7-bit" should suffice...

Thoughts?

d/
-- 

   Dave Crocker

---

From: John C Klensin <[email-masked]>
Date: Tue, 28 Dec 2010 13:25:35 -0500

--On Tuesday, December 28, 2010 3:23 PM +0900 Yangwoo Ko
<[email-masked]> wrote:

>> I hope someone in the WG will have a good suggestion.   If
>> not, we may be back to "ASCII" versus "non-ASCII" as the
>> disjoint pair, "Unicode"  to describe the international CCS
>> (inclusive of ASCII), and "UTF-8" to describe the particular
>> encoding that is used for this protocol and recommended by
>> RFC 2277 (a recommendation that is reinforced by
>> draft-iab-idn-encoding and other more recent work).
> 
> I hope so too. But I don't expect that we are going to have a
> nice answer to this soon enough. If it still significantly
> matters, will it be useful to list up the cases of possible
> confusions when current terminogies are used? With the list,
> we may be able to add some notes to one of our documents
> (possibly framework doc)  to address the known possible
> confusions.

If we try to push this definitional issue very far, it is far
more complicated than has been suggested so far.  For the work
of the EAI WG, I suggest that there are two practical
possibilities:

	(1) The current terminology is not precisely correct
	given other, Standard, usage but, with small
	adjustments, can be used in the documents with some
	explicit local definitions.
	
	(2) In the tradition of John Tukey and others, we invent
	entirely new language that is precisely-defined locally.
	As Andrew points out, that is precisely what IDNA2008
	ended up having to do after struggling with a variation
	on this problem in the DNS space.

Despite Andrew's misgivings, the definitional problem here is
not nearly as hard as that for IDNA.  I think either of the
above would be possible without significantly delaying the WG.
However, the first would be a lot safer in terms of the risk of
introducing new errors as terminology is redefined and then
applied (a problem we certainly got into with IDNA).

Dave suggested in a later note...

>...
> The change in our reference is to stop using ASCII to refer to
> an encoding.  For Internet ASCII, the formal definition
> usually is "7-bit ASCII in an 8-bit field, with the high bit
> off", but "7-bit" should suffice...

Unfortunately, this doesn't work either if one aspires to the
kind of precision for which you seem to be asking.  First of
all, the ANSI definition of ASCII contains both a character
repertoire and an encoding and is, if I recall, fairly explicit
about that.   ANSI also didn't define anything about ASCII in
longer data fields than 7 bits, so the encoding itself is
two-levels: (i) the ASCII repertoire specified as a seven-bit
encoding and (ii) the embedding of [intrinsically 7-bit] ASCII
into an 8-bit field.

While there has never been such a thing as "8-bit ASCII" from a
standards point of view, that term, or "ASCII-8", have often
been used both to describe embedding of ASCII into octets _and_
as the name of an extended, ASCII-based, character repertoire
and encoding that is essentially Latin-1 or ISO 8859-1.   While
the ASCII repertoire is a proper subset of the Unicode
repertoire and (i) the ASCII code points have the same integer
values as the corresponding Unicode code points if the left
octet is stripped from the latter and (ii) when Unicode is
encoded in UTF-8, the octets corresponding to the ASCII
repertoire have the same values as the ASCII code points
embedded in octets with a leading zero bit, there are other CCSs
for which ASCII (both the repertoire and the leading-zero
imbedding) are proper subsets --including the entire ISO 8859
family.   All of the character repertoires of the 8859 standards
are subsets of the Unicode repertoires, but only 8859-1 bears an
obvious relationship to the Unicode code point assignments (and,
in UTF-8, the top half of the 8859-1 repertoire turns into two
octets and the relationship to the original encodings disappears
entirely.

We have usually used "ASCII [embedded] in an 8-bit field, with
the high bit off" on the Internet (indeed, that is part of the
problem that RFC 20 tried to solve), but we have had ASCII
floating around in 7-bit form (not stored in octets) and there
is have been two other octet embeddings that puts the spare bit
somewhere in the middle (one of them as a zero and one as a
parity bit).  There are also 7-bit encodings of other things
floating around, including the encoding described in RFC 2152
(and its predecessor 1642) and the 1642 modification described
in IMAP4rev1 [RFC 2060], a modification not incorporated in the
1642->2152 update.

So, no, "7-bit" doesn't do it either.

Dave's note suggest that a general terminology solution is
needed, a view with which I agree.  But, just as the EAI WG has
been urged to avoid doing work that extends beyond the narrow
scope of what is specifically needed to email address and header
internationalization, an effort to sort out character set
repertoire, definition, encoding, and embedding issues in a
general and appropriate way should not be imposed on the EAI WG.
It might be appropriate to do so by reopening and revising RFC
3536 (which I note uses some terminology that is now considered
obsolete enough to have caused loud protests when used in other
documents) or by initiating a separate effort, but it would be,
IMO, unreasonable and unfair to delay EAI progress for that work
to be initiated and completed.  As Andrew said (and the above
explanation should reinforce), "introducing and rigorously
defining new terms is hard".

    john

---

From: John C Klensin <[email-masked]>
Date: Tue, 28 Dec 2010 14:44:41 -0500

--On Tuesday, December 28, 2010 12:37 PM +0000 Charles Lindsey
<[email-masked]> wrote:

>> without it (which conform perfectly well to 5321 and all of
>> its predecessors) become invalid,  That is pretty
>> uncomfortable to me.  By contrast, if one creates the new
>> command, then, if the response to EHLO includes some sort of
>> EXTENDED-VRFY capability (see below),  one could have:
>> 
> I am still not clear what the precise difficulty is that we
> are addressing. Writing
>      VRFY John Klensin
> and finding it doesn't work is a problem with the present
> legacy setup and it is not our job to fix it (though fixing it
> as a byproduct of some other fix would be OK).

Right.

> What we _are_ concerned with is

We are actually concerned with two cases (if one considers the
above to be the same case if "@example.com" is left off).  The
other one is 
       VRFY Rene
(note no accent), with the understanding that, e.g., either
       [email-masked]
or
       [email-masked]
could come back.

> for which we _do_ need some extension. But why cannot we just
> say that if you want to send that VRFY then you MUST do an
> EHLO first to ensure that UTF8SMTPbis is available, in which
> case you know that the extension permitting "rené" is
> available. 

The problem is (mostly) not what the client sends but what the
server can return.   Suppose the server advertises UTF8SMTPbis
and the client sends "VRFY rene" (all-ASCII).  Now the server
doesn't know whether it is plausible to return an address
containing non-ASCII characters or not: the client has to
somehow give it permission to do that.

Modulo however one feels about heuristics, one would be ok with
UTF8SMTP in the EHLO response to _send_ "VRFY rené" (the client
knows that the server supports the EAI extensions) and the
server could deduce that, if the client is sending non-ASCII, it
can accept a non-ASCII response.

But, if the client sense all-ASCII, the server can't figure out
what it can accept in a response unless the client also sends
along some sort of "I can accept a non-ASCII response if you
would like to give me one".  Hence the need for a parameter.

Even in the "rené" case, the argument about whether or not the
parameter is desirable comes down to a situation much like the
discussion of parameters on the MAIL command.  The difference is
that they heuristics are easy for "rené" and don't work at all
for "rene"; for the MAIL case, they are very easy if either the
MAIL command or the first RCPT command contains a non-ASCII
address and are possible, but very hard, if the message has
non-ASCII in the headers but no clue in the envelope.

The problem with 
   VRFY John Klensin
is that it often does "work" in the legacy setup.  As a leftover
from 821 and massive confusion about the "\" escape, some will
treat it as equivalent to
   VRFY "John\ Klensin"
while others will treat it as a syntax error.  The first is a
little more liberal application of the robustness principle than
I would recommend, but it "works".  The second works too -- it
is a syntax error.

But, if the server advertises UTF8SMTPbis, it could certainly be
telling the client that it expects an 5321 interpretation of the
VRFY command, for which the syntax is unambigious.   For a
client who actually understands EAI, that gives
   VRFY John
an unambiguous interpretation (don't return non-ASCII in the
address)
   VRFY John UTF8SMTPbis
is unambiguous too -- the server is being asked to verify "John"
but can return non-ASCII in the address if needed. 
   VRFY John Klensin
also has an unambiguous interpretation as a syntax error and
   VRFY rené UTF8SMTPbis
is just fine too.

But suppose the client doesn't know anything about i18n and just
ignores the UTF8SMTPbis announcement (as it is obligated to
ignore any extension announcement it doesn't recognize).   Now,
when it sends 
    VRFY John UTF8SMTPbis
the server really doesn't know --because it doesn't know whether
the client is EAI-aware or not-- whether this was intended to be
the EAI form, with a parameter enabling a non-ASCII reply, as
above or whether it is a broken (but common) way to express a
request for information about "John" with a rather odd-looking
second name.

If you are still confused about that much, the next part will
not make sense, but let me try anyway.

There is a problem here that we just didn't have when we added
parameters to MAIL and RCPT (and, for that matter, to DATA).  In
the first two cases, 821 and its successors defined the base
syntax is containing only an address in brackets.  All of the
weird backslash-escape stuff could appear there too, but it was
rare and one still had the brackets, so anything past the right
bracket was guaranteed to be a parameter (or so severely broken
that it could be discounted).  

Some folks were stupid and/or careless enough to omit the
brackets in their implementations, but I have never seen an
instance in the wild of "no brackets but embedded space in the
address"; when SMTP server implementations decided to accept
unbracketed strings, they would generally give up if they then
encountered spaces.   (I do remember writing a test case once
for the equivalent of 
   MAIL FROM: foo [email-masked]
my recollection is that I didn't learn anything interesting,
i.e., that most server implementations rejected it as a syntax
error although there might have been one that resolved it into 
   MAIL FROM:<foo\ [email-masked]> 
)

But VRFY isn't like that.  We don't have a clear,
widely-respected, "if there is anything past this point in the
string it is clearly a syntax error or EMSMTP-enabled
parameter", so there is always a possibility of the parameter
being misinterpreted as part of the string to be verified
because the client sends something odd (and non-conforming but
common) no matter what the server says.  If you want to
guarantee that whatever the client sends can be interpreted
unambiguously, then one needs an unambiguous client announcement
prior to the VRFY command and separate from whatever shows up in
it, e.g., a MAIL parameter (requiring that VRFY be sent, not
only after EHLO but after MAIL) or a separate
"I'm-an-EAI-conforming-client" command.   Or one eliminates the
ambiguity completely with a separate command that cannot be sent
unless some relevant extension is specified that comes with a
rigid syntax and provision for parameters.

On the other hand, something bizarre-looking like "UTF8SMTP" is
going to be safe enough in practice because it will always (or
almost always) yield an error message if interpreted as part of
the name to be verified.   If the WG concludes on that basis
that 
   VRFY Potentially-Quoted-String UTFSMTP
is good enough that it can safely be used after the server has
announced EAI support, I'm not going to lose a lot of sleep over
the risk and edge cases.

> This leaves the legacy behaviour (not sending EHLO
> first and hence not being allowed to say "rené") untouched.

Again, the problem is not having the client send "rené".  It is
having the server send it when the client sent pure ASCII.

    john

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 28 Dec 2010 20:09:56 +0000

[With my MTA server developer hat on, not speaking as an AD]

Ned Freed wrote:

>extensions are allowed - guessing what combination of extensions will work most
>certainly is not a sound operational policy. And implementations are certainly
>free to only offer the extensions that have been returned in an EHLO repsonse.
>
>That said, I don't see an actual requirement in section 2.2 stating that EHLO
>must be issued before any parameters can be used. (Maybe it's somewhere else,
>but a cursory search didn't find it.) Our MTA certainly doesn't enforce this
>and AFAIK neither does sendmail, PostFix, Apache James, or qmail.
>
Isode M-Switch doesn't enforce this either.

>(Exim, OTOH, appears to.)
>  
>
>and check the response before attaching any parameters to VRFY/EXPN. I don't
+1. Maybe the client has cached EHLO response from a previous connection.

>*It doesn't have to be a new code. OTOH, it needs to be a code that EXPN/VRFY
>*
>a somewhat more complex - the code changes are less localized, and I suspect
+1.

>command appraoch becomes the far more complex way to do it. Now, we haven't had
>a specification come along that needed additional EXPN/VRFY parameters before
>this one, but these things have a way of showing up once you open the door and
+1.

---

From: [email-masked]
Date: Tue, 28 Dec 2010 14:21:11 -0800 (PST)

> inspection of the message.

This is indeed the main value of having such a parameter. But it provides
other benefits as well - for example, an MTA may know that the forward
path for certain recipients doesn't support this extension. Such recipients
can be rejected at RCPT TO time. And the earlier the rejection happens, the
better, because the further into the infrastructure you travel, the more likely
problems are to arise on getting the notification back.

More generally, this group seems to be having difficulty on at least two
fronts. First, the current specifications still show some attachment to the
previous model, where messages were supposed to contain the information
necessary to downgrade them. The new model, where bouncing is going to be the
only option most of the time, has very different operational characteristics.
For one thing, stuff that's merely nice to have in a scenario where downgrading
is possible becomes a lot more important when downgrading is no longer an
option.

Second, there's a big difference between designing an experimental protocol and
specifying protocols for a production service. In the former case, it may be OK
to do stuff that has implementors scratching their heads and wondering how to
put it all together. But in the latter case, the shoe is really on the other
foot: The service has to sell itself on a benefits versus costs basis. So,
while there's certainly no rule in the IETF about avoiding deep inspection,
requiring it may result in serious deployment problems. And deployment of
new services is tough enough even for very simple, straightforward things,
which this set of specifications most certainly is NOT.

				Ned

---

From: Barry Leiba <[email-masked]>
Date: Tue, 28 Dec 2010 19:54:36 -0500

I agree with both things that Ned says, below.  I think we do have to
think of this not as revising the experiment, but as using the results
of the experiment to start over and re-design.  I'm sorry that those
of us who were following this from the start have not been following
it closely enough to get this right.

Barry

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 28 Dec 2010 22:12:24 -0800

On 12/28/2010 10:25 AM, John C Klensin wrote:
> kind of precision for which you seem to be asking.

1.  A 2x2 matrix isn't all that precise.

2.  Retaining ambiguous terms doesn't work very well, either.

3.  The terms I suggested all have established use.

4.  RFC 20 (October, 1969) which has a version of the same language I used for 
citing ASCII as an encoding form, as you also cite:

        "7-bit ASCII embedded in an 8 bit byte whose high order bit is always 0"

In other words, "7-bit" is a long-established term for an encoding of ASCII.

So, in spite of the length of your note, I could not discern what you consider 
to be deficient in the terminoloy matrix I suggest, nor what alternative(s) you 
suggest.

>  an effort to sort out character set

That does indeed sound quite ambitious.  So it's probably good that that is not 
the task being suggested.  There is only the need to have the working group 
invoke existing terms or define new ones, for its own usage to ensure clarity 
and accuracy.

Once again:  A 2x2 matrix is not much of all that ambitious, especially when 3 
of the 4 terms appears to be already acceptable...

d/
-- 

   Dave Crocker

---

From: Yangwoo Ko <[email-masked]>
Date: Wed, 29 Dec 2010 17:28:04 +0900

Let me think about the other extreme.

What about explicitly saying that "in this document, we use the terms,
ASCII and UTF-8, to mean either encoding or representation and they
should be interpreted according to the context they are used" ?

Then what specific parts of the document will confuse readers?

On Wed, Dec 29, 2010 at 3:12 PM, Dave CROCKER <[email-masked]> wrote:
> 4.  RFC 20 (October, 1969) which has a version of the same language I used
> for citing ASCII as an encoding form, as you also cite:
>
> So, in spite of the length of your note, I could not discern what you
> consider to be deficient in the terminoloy matrix I suggest, nor what
> alternative(s) you suggest.
>
>
> That does indeed sound quite ambitious.  So it's probably good that that is
> not the task being suggested.  There is only the need to have the working
> group invoke existing terms or define new ones, for its own usage to ensure
> clarity and accuracy.
>
> Once again:  A 2x2 matrix is not much of all that ambitious, especially when
> 3 of the 4 terms appears to be already acceptable...
>
> d/
> --
>
>  Dave Crocker

---

From: "Jiankang YAO" <[email-masked]>
Date: Wed, 29 Dec 2010 17:01:40 +0800

----- Original Message ----- 
From: "Andrew Sullivan" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, December 28, 2010 8:23 PM
Subject: Re: [EAI] Unicode vs. UTF-8 / Encoding vs. Representation

> differentiated A-label, U-label, LDH-label, and so on.

yes, I think that we may borrow the ideas from RFC5890.

 A "U-label" is an IDNA-valid string of Unicode characters, in
      Normalization Form C (NFC) and including at least one non-ASCII
      character, expressed in a standard Unicode Encoding Form (such as
      UTF-8). 

I suggest to define the following terms:

U-character:
 A "U-character" is a non-ASCII Unicode character, in
      Normalization Form C (NFC) and expressed in a standard Unicode Encoding Form of UTF-8. 

U-string:
 A "U-string" is a string of Unicode characters, in
      Normalization Form C (NFC) and including at least one non-ASCII
      character, expressed in a standard Unicode Encoding Form of UTF-8. 

A-character:
  An  "A-character" is an ASCII Unicode character, expressed in a standard Unicode Encoding Form of UTF-8. 

A-string:
 An  "A-string" is  a string of Unicode characters,  including ASCII
      characters only, expressed in a standard Unicode Encoding Form of UTF-8. 

are  the definitions above clear?

another question is that we define these new terms in rfc5336bis or in rfc4952bis?

thanks.

Jiankang Yao

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 29 Dec 2010 11:21:40 -0000

> somehow give it permission to do that.

I think I would take the view that if the client behaves as a pure legacy  
client, then the server should respond like a pure legacy server. So if we  
want a server to offer [email-masked] in response to a request for (pure  
ASCII) 'rene', then we need some extended syntax (either a parameter or a  
new command). But I am not sure we need that facility anyway (and I  
suspect René himself would be most offended to be addressed as Rene).
> can accept a non-ASCII response.

Indeed, but I think you missed the real point of my message, which was  
suggesting that a client MUST do an EHLO (and check for UTF8SMTPbis)  
_before_
issuing "VRFY rené", and that any VRFY issued _without_ a prior EHLO would  
be constrained to pure ASCII only. That would remove any need for any  
'new-VRFY' command AFAICS.
> VRFY command, for which the syntax is unambigious. ...

No, it would not tell a legacy client that. It has to assume a legacy  
client unless it receives explicit information (parameter or new command)  
to the contrary.

> also has an unambiguous interpretation as a syntax error and

Use of such a parameter would be my preferred solution. It is consistent  
with the existing syntax of 5321, and hence confirms (to my view) the  
position that
    VRFY John Klensin
is a syntactic error, and should remain so.
> the server really doesn't know -....

Then to a legacy server it is a syntactic error. If the server chooses to  
guess some intended meaning, then that is not our problem (it would most  
likely return some 553, which is fine by me). An I18N server would  
interpret the UTF8SMTPbis as a parameter and
     VRFY "René Déscartes" UTF8SMTPbis
would work just fine (assuming the client had checked that EHLO allowed  
UTF8SMTPbis).

If we go ahead with some UTF8SMTPbis parameter, as Ned is asking, then it  
may as well be allowed for all commands (VRFY EXPN MAIl RCPT DATA).

---

From: John C Klensin <[email-masked]>
Date: Wed, 29 Dec 2010 14:30:34 -0500

> 1.  A 2x2 matrix isn't all that precise.

I'm not objecting to the matrix, only to the terms, even though
I think we need to be sure that we don't confuse encoding with
embedding (ASCII in an eight bit field isn't the only problem;
technically UTF-32 is an embedding of the Unicode code point
integer space (0 ... 0x10FFFF) in a 32-bit field.  Worse, both
UTF-16 and UTF-32 have big-endian and little-endian variations.
 
> 3.  The terms I suggested all have established use.

And, by the parameters of this conversation, established
somewhat ambiguous use.

> 4.  RFC 20 (October, 1969) which has a version of the same
> language I used for citing ASCII as an encoding form, as you
> also cite:
> 

I can't get from that longer statement to your conclusion.  The
reality is that we have been sloppy about these differences for
years and have gotten away with it because everyone knew how to
extrapolate from ASCII-the-repertoire to ASCII-the-encoding to
ASCII-the-embedding, especially since TENEX/TOPS-20 (ASCII
stored in 7 bit fields) and Multics (ASCII stored in 9 bit
fields with two leading zeros) stopped being important forces in
the network.  I didn't have enough contact with the relevant
systems to know whether IBM OS/360 ASCII (parity bit in the
middle, if I recall) was ever actually used in ARPANET/Internet
contexts, but, if it was, it should be added to that "no longer
an important force" list.

> So, in spite of the length of your note, I could not discern
> what you consider to be deficient in the terminoloy matrix I
> suggest, nor what alternative(s) you suggest.

See above.

Suggested alternative #1:

	Define new, precise, terms on an IETF-wide basis rather
	than trying to make patches that eliminate some of the
	imprecision and reusing terms that are in use elsewhere
	with other meanings.  If we are going to do that, I
	think Andrew's terms would be a reasonable starting
	point.  But it is clear to me that definitional work is
	not an appropriate task for the EAI WG to take on.

Suggested alternative #2: 

	We localize the definitions, using YangWoo's suggestion
	as a starting point and ignoring any cells of your
	matrix (or my slightly larger one) that we don't need.

     john

---

From: John C Klensin <[email-masked]>
Date: Wed, 29 Dec 2010 14:40:33 -0500

--On Wednesday, December 29, 2010 11:21 AM +0000 Charles Lindsey
<[email-masked]> wrote:

> I think I would take the view that if the client behaves as a
> pure legacy client, then the server should respond like a pure
> legacy server. So if we want a server to offer
> [email-masked] in response to a request for (pure ASCII)
> 'rene', then we need some extended syntax (either a parameter
> or a new command).

yes

> But I am not sure we need that facility
> anyway (and I suspect René himself would be most offended to
> be addressed as Rene).

For a better example, suppose one had
   VRFY mueller
and the response should have been
   [email-masked]
Then, while Muller might not like it, or even be offended, she
(and almost everyone else) would certainly anticipate the
linkage between that query and that result.   But we agree that
a parameter is needed in that case, so perhaps it is a non-issue.
 
> Indeed, but I think you missed the real point of my message,
> which was suggesting that a client MUST do an EHLO (and check
> for UTF8SMTPbis) _before_
> issuing "VRFY rené", and that any VRFY issued _without_ a
> prior EHLO would be constrained to pure ASCII only. That would
> remove any need for any 'new-VRFY' command AFAICS.

Again, the reason for either the new command (presumably with a
parameter) and a parameter on VRFY has to do with possible
responses, not what appears as a command argument.    That is
where John Levine's argument comes in: only a command that
explicitly starts an EAI session from the client side and is
then followed by VRFY can eliminate the need for the parameter.

>> But, if the server advertises UTF8SMTPbis, it could certainly
>> be telling the client that it expects an 5321 interpretation
>> of the VRFY command, for which the syntax is unambigious. ...
> 
> No, it would not tell a legacy client that. It has to assume a
> legacy client unless it receives explicit information
> (parameter or new command) to the contrary.

Ok. We agree.

I've covered your other comments in other notes, I think.

   john

---

From: Joseph Yee <[email-masked]>
Date: Wed, 29 Dec 2010 17:33:15 -0500

> an important force" list.

Echo to John's comment, using "7-bit" as the word for encoding is in the category of "you know what I mean" in writing.

> 	matrix (or my slightly larger one) that we don't need.

I agreed that Unicode and UTF-8 are great terms to start with.  Maybe something like "ASCII" on representation (I just wish we could use 'English' instead) and "ASCII Encoding" or "ASCII-Encoding" to make the distinction from representation.

And we still don't know how to represent the set of "Unicode excluding ASCII".

And expanding from Andrew's idea, maybe using "mailbox" instead of "set"?
So it's "A-mailbox", "U-mailbox" and "X-mailbox"?

(hoping few more iterations would generate good idea/result)

best
Joseph

---

From: Yangwoo Ko <[email-masked]>
Date: Thu, 30 Dec 2010 11:53:29 +0900

> And we still don't know how to represent the set of "Unicode excluding ASCII".

What is this set for?

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 30 Dec 2010 10:40:44 -0000

> UTF-16 and UTF-32 have big-endian and little-endian variations.

If you are looking for a term to describe some UTF-8 text that is not also  
an ASCII text, then normal mathematical usage is to use the word "proper".

E.g., a is A is a "proper subset" of B, then A is not identical to B.

So something like "proper-utf8" would follow that concention. You might  
even use "PROPER_UTF8" as that parameter of the MAIL command.

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 30 Dec 2010 06:27:44 -0800

Folks,

This issue is worth exploring specifically, since it is a touchstone to very 
different approaches in specification style.  It is an issue that occurs 
frequently in discussions about specification writing.  In my experience, 
repeating normative text is a mistake, and sometimes a very bad one.

The simplistic response, to the concern about having to switch back and forth 
between specifications, is that a serious implementer is going to have to do 
that anyhow, unless the new extension specification incorporates ALL of the base 
specification.

It is also likely that having to switch back and forth frequently, between 
specification, could mean that there is a  need for cleaner specification of the 
extension.  Good modularization has clean interfaces.  Clean interfaces do not 
requiring constantly checking back and forth from one side of the interface to 
the other.

That is, the general practice, in writing Internet specifications, is 
modularity.  Think of it as a form of layering violation, when one specification 
repeats normative text from another.

There are a number of dangers in repeating text.  One is that it invites 
divergence.  If the controlling specification is changed, the document repeating 
the text could diverge from the primary standard.  In addition, the 
specification repeating the normative text might misrepresent the original 
specification.

At the least repeated text is taken out of context.  Repeating part of a 
specification invites the reader to be unaware of the context in which that part 
takes place.

The third issue is sloppiness.  Good specification writing is an exercise in 
organization and discipline.  Having to repeat text from another specification 
could indicate a general issue with organization of the new specification.

d/

-- 

   Dave Crocker

---

From: Joseph Yee <[email-masked]>
Date: Thu, 30 Dec 2010 10:39:56 -0500

I try to mean non-ascii, and I write an incorrect term.  Sorry for the noise.

Joseph

---

From: [email-masked]
Date: Thu, 30 Dec 2010 10:18:42 -0800 (PST)

+1 on all points.

				Ned

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 30 Dec 2010 16:33:44 -0800

> I try to mean non-ascii, and I write an incorrect term.  Sorry for the noise.

That's ok.  The question remains:  Why is "non-ascii" important?

There is some benefit in separating ASCII from the rest of Unicode, for some of 
the BNF, in order to facilitate adding Unicode on top of the existing ASCII 
definitions.  But beyond that, Unicode is Unicode.  ASCII is part of Unicode and 
the fact that a collection of Unicode happens to also be all ASCII does not make 
it "not" Unicode.

If a message is in legacy form, then it is ASCII (or, rather, ASCII-only).  If 
it is in Unicode form, then it is in Unicode, even if all the characters are 
x07F or below...

d/

-- 

   Dave Crocker

---

From: Yangwoo Ko <[email-masked]>
Date: Fri, 31 Dec 2010 10:08:47 +0900

Thanks Dave. That is exactly what I tried to convey. And that is the
reason why I do not understand the necessity of defining a term for
non-ASCII Unicode characters. There need to be only two kinds, legacy
and more than legacy.

On Fri, Dec 31, 2010 at 9:33 AM, Dave CROCKER <[email-masked]> wrote:
> There is some benefit in separating ASCII from the rest of Unicode, for some
> of the BNF, in order to facilitate adding Unicode on top of the existing
> ASCII definitions.  But beyond that, Unicode is Unicode.  ASCII is part of
> Unicode and the fact that a collection of Unicode happens to also be all
> ASCII does not make it "not" Unicode.
>
> If a message is in legacy form, then it is ASCII (or, rather, ASCII-only).
>  If it is in Unicode form, then it is in Unicode, even if all the characters
> are x07F or below...
>
> d/
>
> --
>
>  Dave Crocker

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 31 Dec 2010 11:45:17 -0000

On Fri, 31 Dec 2010 00:33:44 -0000, Dave CROCKER <[email-masked]> wrote:

> There is some benefit in separating ASCII from the rest of Unicode, for  
> some of the BNF, in order to facilitate adding Unicode on top of the  
> existing ASCII definitions.  But beyond that, Unicode is Unicode.  ASCII  
> is part of Unicode and the fact that a collection of Unicode happens to  
> also be all ASCII does not make it "not" Unicode.
>
> If a message is in legacy form, then it is ASCII (or, rather,  
> ASCII-only).  If it is in Unicode form, then it is in Unicode, even if  
> all the characters are x07F or below...

But you often need to recognise, or to talk about, that case, because a  
message in that form can safely be sent to a legacy server (which might be  
the only way to reach its destination).

So we need a nice, simple, uncomplicated and memorable term for that  
situation and for its opposite. I have suggested "ASCII" and  
"proper_utf8", which are disjoint sets.