Subject: Re: [EAI] Comments on draft-ietf-eai-frmwrk-4952bis-01

From: Shawn Steele <[email-masked]>
Date: Mon, 12 Jul 2010 06:08:30 +0000

> The second sentence is not very clear. Is the following what is meant?

>    In this case, the client will not be able to transmit any messages
>    which contain non-ASCII characters in the local-part of any email
>    addresses. If the non-ASCII characters are confined only to the
>    domain portion of the address, such a client will be able to
>    transmit messages if it uses [RFC3490] or [RFC5890] ToAscii() to
>    encode the domain portions of these addresses.

They may have ASCII aliases (eg: transliterations) available, so I think that's a tad too restrictive.

   In this case, the client will not be able to transmit any messages
   which contain non-ASCII characters in the local-part of any email
   addresses. If the client has an ASCII only address available, such
   as an alias, or by applying [RFC3490bis] ToAscii to the domain
   portion of the address, then it may be able to send the mail in
   ASCII as per [RFC2821].

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Mon, 12 Jul 2010 04:38:44 -0400

Shawn,

--On Monday, July 12, 2010 06:08 +0000 Shawn Steele
<[email-masked]> wrote:

>>    In this case, the client will not be able to transmit any
>>    messages which contain non-ASCII characters in the
>>    local-part of any email addresses. If the non-ASCII
>>    characters are confined only to the domain portion of the
>>    address, such a client will be able to transmit messages
>>    if it uses [RFC3490] or [RFC5890] ToAscii() to encode the
>>    domain portions of these addresses.

I can't even find the above text in 4952bis-01.  To what are you
referring?

> so I think that's a tad too restrictive.

And I don't understand this comment.  If an ASCII alias is
substituted for the non-ASCII local part, then the message
proposed to be transmitted no longer contains a non-ASCII local
part.  Either the local-part in the _message the client is going
to transmit_ is all-ASCII or it isn't.  If it isn't, the above
is true.  If it is all-ASCII, the above is irrelevant.  

If you are suggesting that two messages are somehow the same
because one uses ASCII aliases for addresses found in the other,
I think that takes us directly into the mess of having to assert
an externally-verifiable identity binding between the two
addresses.  Only the delivery MTA or the destination user can
know that, and they aren't obligated to tell (even on VRFY).

>    In this case, the client will not be able to transmit any
> messages    which contain non-ASCII characters in the
> local-part of any email    addresses. If the client has an
> ASCII only address available, such    as an alias, or by
> applying [RFC3490bis] ToAscii to the domain    portion of the
> address, then it may be able to send the mail in    ASCII as
> per [RFC2821].

That statement would be true, but a lot less helpful than I
think you want, if the last part of it read something like
"...then it may be able to send a similar, transformed, message
in ASCII as..."

   best,
      john

---

From: "Jiankang YAO" <[email-masked]>
Date: Mon, 12 Jul 2010 16:50:46 +0800

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: "Shawn Steele" <[email-masked]>; <[email-masked]>
Sent: Monday, July 12, 2010 4:38 PM
Subject: Re: [EAI] Comments on draft-ietf-eai-frmwrk-4952bis-01

> I can't even find the above text in 4952bis-01.  To what are you

He may reference to rfc5336bis

---

From: Shawn Steele <[email-masked]>
Date: Mon, 12 Jul 2010 16:57:20 +0000

Sorry, I cut too much context.  I was replying to Ernie's reply where he said:

	3.2. The UTF8SMTPbis Extension
	(last paragraph)  

	   If a server advertises UTF8SMTP and the client does not recognize the
	   extension, the client may send a regular message [RFC2821 <http://tools.ietf.org/html/rfc2821>] and
	   [RFC2822 <http://tools.ietf.org/html/rfc2822>].  In this case, the client may continue to use the
	   [RFC3490 <http://tools.ietf.org/html/rfc3490>] or [RFC5890 <http://tools.ietf.org/html/rfc5890>] ToAscii() to encode the domain portion of an
	   address.

	The second sentence is not very clear. Is the following what is meant

> address, then it may be able to send the mail in    ASCII as

Where you replied:

	That statement would be true, but a lot less helpful than I think you want, if the last part of it read something like "...then it may be able to send a similar, transformed, message in ASCII as..."

The original idea seems to be that "If UTF8SMTP isn't supported, you might be able to try the old way by doing XXXXX".  Since we expect at least some users to have ASCII addresses, then XXXXX has more options than what Ernie's suggestion allowed.  That's all I was trying to say.  Even using Punycode as in the original idea would be a "transformed" message.

I'm not sure we need this text, since, of course, if the server doesn't support UTF8SMTP, the existing standards still work :)  It seems like there's no harm in pointing that out, but I don't want to be overly restrictive in our wording.

Re: transliterations, I think I was confusing.  We're expecting admins/users to create a Unicode address for their mailbox, and, likely, an ASCII address, which, in some cases, is probably a transliteration chosen by the admin or user.  (Eg: Tarek, my manager, instead of however that's spelled in Arabic).

- Shawn

-----Original Message-----
From: John C Klensin [mailto:[email-masked]] 
Sent: Monday, July 12, 2010 1:39 AM
To: Shawn Steele; [email-masked]
Subject: Re: [EAI] Comments on draft-ietf-eai-frmwrk-4952bis-01

Shawn,

>>    domain portions of these addresses.

I can't even find the above text in 4952bis-01.  To what are you referring?

> think that's a tad too restrictive.

And I don't understand this comment.  If an ASCII alias is substituted for the non-ASCII local part, then the message proposed to be transmitted no longer contains a non-ASCII local part.  Either the local-part in the _message the client is going to transmit_ is all-ASCII or it isn't.  If it isn't, the above is true.  If it is all-ASCII, the above is irrelevant.  

If you are suggesting that two messages are somehow the same because one uses ASCII aliases for addresses found in the other, I think that takes us directly into the mess of having to assert an externally-verifiable identity binding between the two addresses.  Only the delivery MTA or the destination user can know that, and they aren't obligated to tell (even on VRFY).

> address, then it may be able to send the mail in    ASCII as

That statement would be true, but a lot less helpful than I think you want, if the last part of it read something like "...then it may be able to send a similar, transformed, message in ASCII as..."

   best,
      john

---

From: Joseph Yee <[email-masked]>
Date: Mon, 12 Jul 2010 14:00:59 -0400

Shawn,

The discussion on encoding domain part with RFC3490 or RFC5890 is focus on what may/can do immediately with ascii-only@utf8-domain address (following the discussion that narrow down the scenario).  Your concern of too restrictive choices was addressed on section 7.1 in Framework-02 of what MUA and user can do.  The text there mentions wide range of approaches.  

I believed that Section 7.1 (in Framework-02) address your concern in Section 3.2 (UTF8SMTPbis).

Best
Joseph

On 2010-07-12, at 12:57 PM, Shawn Steele wrote:

> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima

---

From: Shawn Steele <[email-masked]>
Date: Mon, 12 Jul 2010 18:03:11 +0000

I'll look, I'm rereading, but not quite there yet.  Thanks :)

-----Original Message-----
From: Joseph Yee [mailto:[email-masked]] 
Sent: Monday, July 12, 2010 11:01 AM
To: Shawn Steele
Cc: John C Klensin; [email-masked]
Subject: Re: [EAI] Comments on draft-ietf-eai-frmwrk-4952bis-01 (and -02)

Shawn,

Best
Joseph

---

From: Shawn Steele <[email-masked]>
Date: Mon, 12 Jul 2010 18:52:54 +0000

Confirmed, yes, I'm happy with it.  (mentioning again as it might get lost in my longer reply I just replied).

-Shawn

-----Original Message-----
From: Joseph Yee [mailto:[email-masked]] 
Sent: Monday, July 12, 2010 11:01 AM
To: Shawn Steele
Cc: John C Klensin; [email-masked]
Subject: Re: [EAI] Comments on draft-ietf-eai-frmwrk-4952bis-01 (and -02)

Shawn,

Best
Joseph

---

From: Jason Nelson <[email-masked]>
Date: Mon, 12 Jul 2010 18:58:29 +0000

In the current design, with a message destined to two recipients, A and B, A supports UTF8 and B does not - what should B's P2 content look like?  I get that the RFC specifically deals with addresses in transit, and that we should not be mucking about with messages when not the submission server, but resolving how the mail goes to both the guys in the US and the your business partner in Taiwan with both being able to reply across the board, that's giving me a few headaches.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Shawn Steele
Sent: Monday, July 12, 2010 11:53 AM
To: Joseph Yee
Cc: [email-masked]
Subject: Re: [EAI] Comments on draft-ietf-eai-frmwrk-4952bis-01 (and -02)

-Shawn

-----Original Message-----
From: Joseph Yee [mailto:[email-masked]] 
Sent: Monday, July 12, 2010 11:01 AM
To: Shawn Steele
Cc: John C Klensin; [email-masked]
Subject: Re: [EAI] Comments on draft-ietf-eai-frmwrk-4952bis-01 (and -02)

Shawn,

Best
Joseph

---

From: Ernie Dainow <[email-masked]>
Date: Mon, 12 Jul 2010 15:04:02 +0000

The confusion here is that it actually applies to a different thread. 
The wording I suggested and is being amended by Shawn (with which I 
agree) is from the thread

More comments on draft-ietf-eai-rfc5336bis-00

not from Re: [EAI] Comments on draft-ietf-eai-frmwrk-4952bis-01

    -Ernie

---

From: Shawn Steele <[email-masked]>
Date: Mon, 12 Jul 2010 19:09:02 +0000

The whole part is different (or I missed it) in the new 02 draft, so I think it's fine now.

Sorry, I'm getting "digests", so I have to cut & paste the subject, and obviously didn't get it right :)

-Shawn

-----Original Message-----
From: Ernie Dainow [mailto:[email-masked]] 
Sent: Monday, July 12, 2010 8:04 AM
To: Shawn Steele
Cc: John C Klensin; [email-masked]
Subject: Re: [EAI] Comments on draft-ietf-eai-rfc5336bis-00 (not frmwrk-4952bis-01)

    -Ernie

---

From: John C Klensin <[email-masked]>
Date: Mon, 12 Jul 2010 15:12:00 -0400

--On Monday, July 12, 2010 16:57 +0000 Shawn Steele
<[email-masked]> wrote:

> 	   If a server advertises UTF8SMTP and the client does not
> recognize the 	   extension, the client may send a regular
> message [RFC2821 <http://tools.ietf.org/html/rfc2821>] and
> [RFC2822 <http://tools.ietf.org/html/rfc2822>].  In this case,
> the client may continue to use the 	   [RFC3490
> <http://tools.ietf.org/html/rfc3490>] or [RFC5890
> <http://tools.ietf.org/html/rfc5890>] ToAscii() to encode the
> domain portion of an 	   address.

Aha.  I rewrote that very severely before putting it into -02,
so couldn't find what you were quoting.

> 	>>    In this case, the client will not be able to transmit
> any 	>>    messages which contain non-ASCII characters in the
> 	>>    local-part of any email addresses. If the non-ASCII 	>>
> characters are confined only to the domain portion of the 	>>
> address, such a client will be able to transmit messages 	>>
> if it uses [RFC3490] or [RFC5890] ToAscii() to encode the 	>>
> "...then it may be able to send a similar, transformed,
> message in ASCII as..."
> 
> The original idea seems to be that "If UTF8SMTP isn't
> supported, you might be able to try the old way by doing
> XXXXX".  Since we expect at least some users to have ASCII
> addresses, then XXXXX has more options than what Ernie's
> suggestion allowed.  That's all I was trying to say.  Even
> using Punycode as in the original idea would be a
> "transformed" message.

I don't know what Ernie intended, but I took it as closer to
"you either support the extensions or you are on your own with
the traditional/legacy system".

> I'm not sure we need this text, since, of course, if the
> server doesn't support UTF8SMTP, the existing standards still
> work :)  It seems like there's no harm in pointing that out,
> but I don't want to be overly restrictive in our wording.

Right.   Joseph's interpretation is, I think, close to mine.
Have a look at the -02 text and see if you like it any better.
I do have one observation about his comments -- see below.

> Re: transcriptions, I think I was confusing.  We're
> expecting admins/users to create a Unicode address for their
> mailbox, and, likely, an ASCII address, which, in some cases,
> is probably a transcription chosen by the admin or user.
> (Eg: Tarek, my manager, instead of however that's spelled in
> Arabic).

I think that, as you move around the world and look at names
written in Roman characters for people whose native language
isn't normally written that way, you will find transcriptions,
transcriptions, translations, and more or less fanciful
alternatives in some proportions.  Note that, strictly speaking,
Chinese cannot be transliterated, only transcribed (Japanese
Kana     can be transliterated into Romanji).  If you take the
three active participants in the WG who are of Chinese descent
as examples, we see (I hope they will correct me if I have this
wrong):

 o Joseph (probably not a transcription*; I don't know
	whether it is a translation of something else or not)
 o Jiankang (I presume a transcription*; but he has
	also sometimes used "Health", which I understand to be a
	translation)
 o XiaoDong (a transcription*; but he sometimes also
	has used "Sheldon", which sounds somewhat alike but it
	neither a translation or transcription)

	* Note that while Hanyu Pinyin is now the standard
	transcription system in the PRC and some other places,
	there are a huge number of mutually-incompatible
	transcription systems for Chinese.  And, of course,
	transcription systems, by definition, reflect spoken
	languages, not written ones.

Similar issues apply with other languages and names.  Knowing
the personal name doesn't tell you what convention is being used
for a name in a different language/cultural environment.   As an
example that you and I should find mutually amusing, Иван
usually makes it into Roman characters as "Ivan", but Johann,
John, and even Sean or Shawn, (but not Jon or Jonathan) are not
unreasonable: all derive down different paths from יוחנן.

--On Monday, July 12, 2010 14:00 -0400 Joseph Yee
<[email-masked]> wrote:

> The discussion on encoding domain part with RFC3490 or RFC5890
> is focus on what may/can do immediately with
> ascii-only@utf8-domain address (following the discussion that
> narrow down the scenario).  Your concern of too restrictive
> choices was addressed on section 7.1 in Framework-02 of what
> MUA and user can do.  The text there mentions wide range of
> approaches.  

In going through what became -02 over the weekend, I discovered
several clusters of text that are somewhat redundant with each
other or that might be organized to be more easily found or
understood.  While I fixed those that I found and for which the
fixes seemed straightforward, there are almost certainly others
left.  Suggestions about what material can safely be eliminated
as covered elsewhere in document, what might better be
reorganized and how, and where cross-references are needed would
be most welcome.

    john

---

From: Joseph Yee <[email-masked]>
Date: Mon, 12 Jul 2010 15:14:41 -0400

The follow text is what Ernie suggested for 5336bis

> 	>>    domain portions of these addresses.

Which Shawn expressed concern, and I believed (and Shawn agreed) that text in Section 7.1 in Framework-02 addressed Shawn's concern.  So the text above is good (from consensus) for SMTPbis section 3.2 last paragraph.

Hope this is clear, because it involves two docs to discuss two "inter-related" issues at the same time.

Joseph

---

From: Shawn Steele <[email-masked]>
Date: Mon, 12 Jul 2010 19:32:01 +0000

Like I (sort of didn't) say:  My use of "transcription" muddied the water and lost my point :)

Anyway, I'm (still) happy with -02.

You said:
"In going through what became -02 over the weekend, I discovered several clusters of text that are somewhat redundant with each other or that might be organized to be more easily found or understood.  While I fixed those that I found and for which the fixes seemed straightforward, there are almost certainly others left.  Suggestions about what material can safely be eliminated as covered elsewhere in document, what might better be reorganized and how, and where cross-references are needed would be most welcome."

I think it's mostly fine.  My (longer mail) comments are mostly nitpicking.  I mentioned that I think 9.1 goes into more detail than is strictly necessary, but that need not hold up the doc.  It also skips other interesting cases (full/half width, hiragana/katakana/kanji, etc.)  If I was touching that part, I'd yank most of it and just say "you need to be careful with addresses" and maybe list a few examples of why (like the bullets).

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Mon, 12 Jul 2010 18:39:43 -0400

--On Monday, July 12, 2010 18:58 +0000 Jason Nelson
<[email-masked]> wrote:

>...
> In the current design, with a message destined to two
> recipients, A and B, A supports UTF8 and B does not - what
> should B's P2 content look like?  I get that the RFC
> specifically deals with addresses in transit, and that we
> should not be mucking about with messages when not the
> submission server, but resolving how the mail goes to both the
> guys in the US and the your business partner in Taiwan with
> both being able to reply across the board, that's giving me a
> few headaches.

I wonder how many others on this mailing list are familiar
enough with that other Standard messaging architecture to know
what "P2 Content" is?   :-)

In any event, it is these mixed cases that are the reason why
many of us hoped that we could devise an in-transit downgrading
mechanism that would be easy to implement and work universally
in a predictable and interoperable way.   There are, I believe,
three such cases, each with a slightly different scenario:

	(i) Mixed ASCII and non-ASCII destination
	(forward-pointing) addresses with the ASCII address
	sites presumably being non-EAI-capable, as in your
	example.
	
	(ii) ASCII-only destination addresses, some of which are
	EAI-capable, and a backward-pointing ("sender")
	non-ASCII address.
	
	(iii) Mailing lists with mixed ASCII and non-ASCII
	addresses.

Even if it is fully deliverable, with all information preserved
in headers, replying to a message in one of these categories can
easily create one of the others --and a less-tractable situation.

For purposes of that classification,
ASCII-local-part@Punycoded-IDN is an ASCII-only address.  If one
had code that immediately converted IDN A-labels into U-label
form, there are an additional set of cases having to do with
systems that are IDNA-aware but perhaps not EAI-aware.

Anyway, downgrading failed.  We could review/ debate degrees of
failure (and waste a lot of time), but it was clear that we
weren't going to get the hoped-for predictability as soon as we
discovered that messages for which alternate ASCII addresses
existed for some of the non-ASCII ones but not all of them.
Relative to your example, please note the relationship between
"one ASCII destination, one non-ASCII destination" and "two
non-ASCII destinations, one with an alternate address available
and the other without one".

But the problem really isn't new.  Ignore the addresses for a
moment and assume that A is not only UTF8-capable but prefers
message content in Lower Slobbovian.  B can handle only ASCII
and English.  The sender, miraculously, can speak and write both
Lower Slobbovian and English.  Now, what is to be sent?

	(i) If the message content is sent in English, A will be
	unhappy and may reply in Lower Slobbovian with parts of
	the original message copied.   B will receive the
	message from the sender with everything intact and
	comprehensible.  When A replies, the reply message will
	be unreadable to B will not be able to read (even though
	it can be received) and that may be displayed as a low
	of question marks (since B presumably doesn't have Lower
	Slobbovian fonts installed and might not even be able to
	handle "text/plain; charset=utf8").
	
	(ii) If the message content is sent in Lower Slobbovian,
	B is out of luck.  The original message will be received
	but will be incomprehensible -- only the "From" and "To"
	header fields and their envelope counterparts are likely
	to be useful and readable.  And any reply from A won't
	even contain traces of English/ASCII.
	
	(iii) If the message is sent with both charsets (ASCII
	and UTF8) and languages (English and Lower Slobbovian)
	using multipart/alternative...  Well, we had high hopes
	for that one, but it was never as useful as expected
	(and is not supported well, unlike the use of
	multipart/alternative for text and HTML body parts).
	But assume it worked as intended and think about what
	happens to A and B.  Each sees the message in her
	preferred language and characters, is only vaguely aware
	(if at all) that the other language and characters are
	there.  So far, so good.  Now A replies to the message
	she is seeing (a Lower Slobbovian one), in the language
	she sees.  And B responds in the language she is seeing
	(English) in that language.    In a way, this is the
	worst scenario of all because a slight change in the
	example results in A and B seeing content in languages
	that the other one can't read at all.

The address case affects whether messages that might be
unreadable (or that would lead to unreadable replies) can be
delivered or reflected properly in headers, but it doesn't
change things as compared to those nasty content example.  And,
while there are some advantages of having messages the user
can't read delivered (as long as they can be rendered, one might
be able to find a translator), they are pretty marginal in most
cases... especially when they cannot be rendered.

I think the bottom line is:

(1) Internationalization is a messy business that really is not
going to be helpful other than for communication within language
communities.  EAI is no exception in that regard.

(2) A decision to send messages in a possible-mixed environment
wrt addressing, like a decision to send messages in a language
that some recipients might not be able to read (or even render)
is high-risk -- especially in a world in which a popular source
of high spam likelihood scores is "message in a language or
script I don't normally read or can't read at all".

(3) If someone is communicating internationally -- with people
who use multiple languages and writing systems -- the
conservative approach is going to be to use ASCII addresses and
least-common-denominator languages and scripts.  EAI is going to
work best within a language community where all (or almost all)
of the systems are EAI-aware, capable of rendering the relevant
script, and all of the addresses are in that script too,
regardless of how they are encoded.  

While we may need to do some education, those principles are
user-level common sense even if they are lots better understood
about content than about addresses today.  We might or might not
be able to reinforce the education with automated tools (and
might or might not want to).  But those users who violate the
principles will sooner or later get burned with messages that
either don't reach all recipients or replies to those messages
that are incomprehensible.  Like small children and hot stoves,
few of them will need to repeat the experience very many times.

I wish I had a more optimistic answer.  But we need to remember
that every step we take toward a non-ASCII/ non-English (or at
least non-Western-European-language) Internet is a step that
enables some users and language communities at the price of
reducing the degree to which everyone connected to the Internet
can communicate with all the others.  IMO, that is a good
tradeoff, but not one that is without costs.

FWIW, we've had variations on the above conversation/ comments/
analysis enough times that maybe it should be written into a
more public and permanent place.  If there is WG consensus that
it should go into 4952bis, I'm ok with that although I'd
appreciate it if someone else would suggest text.   If not, we
should think about other places and opportunities.

    john

---

From: Shawn Steele <[email-masked]>
Date: Tue, 13 Jul 2010 01:22:22 +0000

> I wonder how many others on this mailing list are familiar enough with that other Standard messaging architecture to know

I had to ask Jason :)

- Shawn

---

From: "Jiankang YAO" <[email-masked]>
Date: Tue, 13 Jul 2010 10:42:31 +0800

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: "Shawn Steele" <[email-masked]>; "Joseph Yee" <[email-masked]>; <[email-masked]>
Sent: Tuesday, July 13, 2010 3:12 AM
Subject: Re: [EAI] Comments on draft-ietf-eai-frmwrk-4952bis-01

> languages, not written ones.

you know more than me even I am a Chinese.
thanks.