Subject: Re: [EAI] draft-ietf-eai-frmwrk-4952bis-02

From: Shawn Steele <[email-masked]>
Date: Mon, 12 Jul 2010 18:47:15 +0000

> Joseph, please do a review of your own as needed and then
> initiate a WG Last Call.  That Last Call should be terminated if
>substantive protocol changes that would require text in this
> document are discussed seriously.

:)

>   o  Discussion of RFC 5825 ("downgraded display") has been removed per
>      the earlier note and on-list discussion.  Any needed discussion
Good

> (as far as I know, an Informational spec can
>      obsolete or update Experimental ones, so no downref problem here
 >     --JcK).

Sounds reasonable.

Anchor 5
I'm confused about the timing/process for renaming UTF8SMTPbis - I guess it needs generated before this doc is published, but this doc is scheduled to happen before the SMTP doc.  Does it need a name before last call?  How late can we defer creating a name? 

4.3
"In particular, the use of non-ASCII
   message content is an integral part of the MIME specifications
   [RFC2045] and does not require these extensions (although it is
   compatible with them)."

My understanding is that MIME isn't required for UTF-8 UTF8SMTP content.  I'm not sure if that's clear here.  (Though I imagine lots of mail is MIME anyway).

6.1 bullet 1:
"use of ASCII-compatible (ACE)
       encoding [RFC3492] [RFC5890] on the right hand side is
       discouraged"

This is fine for this doc, however in an EAI mail, punycode should be forbidden.  So I'd like to see "MUST NOT", or at least "SHOULD NOT" have punycode on the RHS if the LHS has non-ASCII in the UTF8SMTP spec.  (Which qualifies as "discouraged", which is stated here).

6.1 bullet 4.  "no other code pages"
Addresses my earlier question, and I'm fine with this text, although, in practice, it's sort of like closing the barn door after the animals have left. :)  

7.1 does address my "sender can use legacy mail, if they can figure it out" concern.  

[anchor15] - Yao's been working on documents regarding the choice of addresses, which helps, I think.  Though an interesting problem, I don't think we need this note.  Some of the most interesting parts.  (How does my client learn that Shawn-Unicode and ShawnSte are the same address?, and how do I enable the user to choose which address to send with) are probably more usability type things.  I'm not sure we need to provide much more than general address selection/ways we expect them to be used guidelines.

Anyway, I'd remove [anchor15] in the interest of not holding up the doc.  I think that 9 adequately captures the idea that app developers need to consider User Interface and Configuration challenges.

9.1 - The charter has the "addresses" spec, I'm not sure you need to go into this much depth here.  (don't see that it hurts either)

* Normalization - I think the key point is that servers should be expected to see addresses in different normalization (or unnormalized) forms, and they should be prepared to normalize them.  I think it is less important for us to specify that they must use Form C.  (They could achieve the same behavior if they used NFD internally).  

Editorial:  "Stated differently" restates the 1st bullet, but then lists two new concepts that were alluded to in the text earlier, but not the bullets.  (so it's not really "stating differently".)  I'd remove "stated differently, and let the C0/C1 and "other kinds of punctionation" be top level bullets:

   o  In general, it is wise to support addresses in Normalized form.
      Servers should normalize with a consistent Unicode normalization
      form.  Compatibility forms may only be appropriate in unusual circumstances.

   o  It may be wise to support other forms of the same local-part
      string, either as aliases or by normalization of strings reaching
      the delivery server, in the event that the sender does not send
      the strings in normalized form.

   o  C0 (and presumably C1) controls (see The Unicode Standard
         [Unicode52]) are prohibited, the first in RFC 5321 and the
         second by an obvious extension from it [RFC5198].

    o  Other kinds of punctuation, spaces, etc., are risky practice.
         Perhaps they will work, and SMTP receiver code is required to
         handle them, but creating dependencies on them in mailbox names
         that are chosen is usually a bad practice and may lead to
         interoperability problems.

12 - Security considerations.
I think this does a nice job of balancing people's concerns about homographs without becoming too unreasonable, and I wouldn't change it :)  *IF* something did have to change, it's worth recognizing that email local parts and domain names have different problems.  It's very unclear whether I'm supposed to be [email-masked] or, maybe, shawnst (who gets most of my typoe'd mail :))  It's way worse for places like live mail and gmail where people end up as shawn1234 and shawn6666.  Those are quite confusable regardless of script.  In short, I think that enabling Unicode does not increase the risk for an enterprise that controls their own assignments (as long as they're careful).  And, I think there's no further risk for the large places that already have quite confusing names.  As said, I think this doc does a decent job of balancing the paranoia without itself becoming a fear monger.  

In summary, I can live with the document as it is :)  I have asked the Exchange folks to review some of the parts I'm less familiar with, but I don't expect major concerns.

Thanks John,

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Mon, 12 Jul 2010 16:06:28 -0400

--On Monday, July 12, 2010 18:47 +0000 Shawn Steele
<[email-masked]> wrote:

>...
> Anchor 5
> I'm confused about the timing/process for renaming UTF8SMTPbis
> - I guess it needs generated before this doc is published, but
> this doc is scheduled to happen before the SMTP doc.  Does it
> need a name before last call?  How late can we defer creating
> a name? 

Based on experience with IDNA2003, we can delay up until the
time the first document that actually has to include the string
is published.  That is some time after IETF Last Call and even
after the usual definition of AUTH48 (the final review after the
RFC Editor gets through editing).  Whether that is desirable or
not is a question about which the WG needs to reach some
conclusion.   My own opinion is that deferring the assignment of
the keyword/ extension parameter will help with interoperability
by just about guaranteeing that anyone who uses the final
keyword will have at least had an opportunity to see the final
specs rather than making a guess and jumping the gun.

That interacts with the publication plan for 4952bis:  In
principle, we can do any of the following:

	(1) Complete WG Last Call with the placeholders for the
	keyword and the other documents retained, but then let
	the document sit, not moving toward IETF Last Call until
	the other documents (or at least some of them) are ready.
	
	(2) Complete WG Last Call as above and IETF Last Call.
	Assuming the IESG approves, let the document sit in the
	RFC Editor queue in "MISSREF" (normative references
	missing) and/or IANA-hold (waiting for the keyword to be
	assigned) states until the other documents are ready.
	
	(3) Edit the document to hand wave around the references
	to the other documents and the precise name of the
	keyword, process it through the IESG, and publish it.

Note that early assignment of the keyword doesn't make much
difference to (1) or (2): we still need to wait for the other
documents.  The main difference between (1) and (2) is where it
waits and whether we have IESG confirmation about the document
and strategy.

While we could decide on (1) in principle, the decision among
those three is ultimately up to Alexey.  I have great confidence
that he will pay careful attention to whatever advice the WG
gives, but it is his decision.

>    message content is an integral part of the MIME
> specifications    [RFC2045] and does not require these
> extensions (although it is    compatible with them)."
> 
> My understanding is that MIME isn't required for UTF-8
> UTF8SMTP content.  I'm not sure if that's clear here.  (Though
> I imagine lots of mail is MIME anyway).

Yes, it is actually a requirement.  You should look through the
document and see if it needs to be spelled out more clearly, but
for an EAI message to be transmitted, the server must support
(and announce) and the client must announce 8BITMIME.  And
8BITMIME requires MIME structuring and body parts.   Remember
that, absent 8BITMIME, you can't transmit UTF-8 without some
flavor of ASCII encoding and that without MIME, encoded words
are a non-starter.  

We more or less made the same decision that was made about more
basic kinds of messages and SMTP extensions that involve
advanced features:  you want the advanced stuff, supporting MIME
is a prerequisite.  Non-MIME messages are pretty much bound to
821/822.

> This is fine for this doc, however in an EAI mail, punycode
> should be forbidden.  So I'd like to see "MUST NOT", or at
> least "SHOULD NOT" have punycode on the RHS if the LHS has
> non-ASCII in the UTF8SMTP spec.  (Which qualifies as
> "discouraged", which is stated here).

This is another discussion we had the first time around.  It may
not be sensible to try to prohibit it because an A-label is just
another ASCII name.  In other words, at least under IDNA2003 (to
which 4952 was written), if the domain-part is an ASCII string,
an EAI application really doesn't need to know whether that
string has the special properties of an A-label or whether it is
just an ASCII string.  IDNA2008 changes things a bit because it
fairly strongly encourages (but does not require) applications
that are about to look up an A-label to validate it, which just
about requires conversion to a U-label.  I can see some basis
there for a "SHOULD NOT", but I'd like to hear from others and
see a slightly stronger justification than an assertion.  Put
differently, it is not clear to me that having A-labels in the
domain-part is harmful.  Stupid and ugly, yes, but not
necessarily harmful.

> Addresses my earlier question, and I'm fine with this text,
> although, in practice, it's sort of like closing the barn door
> after the animals have left. :)  

Not sure how much non-ASCII text in non-UTF-8 charsets have
actually been used in email headers.  People thinking they want
them is another matter, but a non-trivial number of MTAs and
anti-spam filters/evaluators will apparently reject such
messages as bogus or worse.   So, if you think the animals are
out of the barn, I'd be very interested in a census of the
relevant animals.  That said, I don't think it will change this
provision.  To paraphrase a colleague of ours, the late thing we
need is to introduce more places where people can validly use
funky encodings.

>...
> [anchor15] - Yao's been working on documents regarding the
> choice of addresses, which helps, I think.  Though an
> interesting problem, I don't think we need this note.  Some of
> the most interesting parts.  (How does my client learn that
> Shawn-Unicode and ShawnSte are the same address?, and how do I
> enable the user to choose which address to send with) are
> probably more usability type things.  I'm not sure we need to
> provide much more than general address selection/ways we
> expect them to be used guidelines.
> 
> Anyway, I'd remove [anchor15] in the interest of not holding
> up the doc.  I think that 9 adequately captures the idea that
> app developers need to consider User Interface and
> Configuration challenges.

Good.  I'd like to hear from others, but this is consistent with
my personal instincts.

> 9.1 - The charter has the "addresses" spec, I'm not sure you
> need to go into this much depth here.  (don't see that it
> hurts either)

The answer may depend on when we want to push this document out
the door and in what state.  See above.
 
> * Normalization - I think the key point is that servers should
> be expected to see addresses in different normalization (or
> unnormalized) forms, and they should be prepared to normalize
> them.  I think it is less important for us to specify that
> they must use Form C.  (They could achieve the same behavior
> if they used NFD internally).  

Not quite, I think.  We need to do the case analysis carefully
here given that NFC sometimes decomposes (NFD never composes)
and that this interacts with the 5321 limits on string lengths.

> Editorial:  "Stated differently" restates the 1st bullet, but
> then lists two new concepts that were alluded to in the text
> earlier, but not the bullets.  (so it's not really "stating
> differently".)  I'd remove "stated differently, and let the
> C0/C1 and "other kinds of punctionation" be top level bullets:
> 
>    o  In general, it is wise to support addresses in
> Normalized form.       Servers should normalize with a
> consistent Unicode normalization       form.  Compatibility
> forms may only be appropriate in unusual circumstances.
> 
>    o  It may be wise to support other forms of the same
> local-part       string, either as aliases or by normalization
> of strings reaching       the delivery server, in the event
> that the sender does not send       the strings in normalized
> form.
> 
>          [Unicode52]) are prohibited, the first in RFC 5321
> and the          second by an obvious extension from it
> [RFC5198].
> 
>     o  Other kinds of punctuation, spaces, etc., are risky
> practice.          Perhaps they will work, and SMTP receiver
> code is required to          handle them, but creating
> dependencies on them in mailbox names          that are chosen
> is usually a bad practice and may lead to

I like this organization.  I'd like to hear from others before I
start making changes (which will not get into the queue today in
any event -- see below).

> I think this does a nice job of balancing people's concerns
> about homographs without becoming too unreasonable, and I
> wouldn't change it :)  *IF* something did have to change, it's
> worth recognizing that email local parts and domain names have
> different problems.  It's very unclear whether I'm supposed to
> be [email-masked] or, maybe, shawnst (who gets most of
> my typoe'd mail :))  It's way worse for places like live mail
> and gmail where people end up as shawn1234 and shawn6666.
> Those are quite confusable regardless of script.  In short, I
> think that enabling Unicode does not increase the risk for an
> enterprise that controls their own assignments (as long as
> they're careful).  And, I think there's no further risk for
> the large places that already have quite confusing names.  As
> said, I think this doc does a decent job of balancing the
> paranoia without itself becoming a fear monger.

Actually, in some parts of the world and at the second and third
level, foo123, foo124, and foo321 are all seen as domain name
labels, so perhaps the situations are not quite as different as
you suggest.   But I think --and you seem to have confirmed--
that we could spend a lot of effort tuning that text to make it
only slightly better.  So, unless others have objections or
concerns, we are probably at a good stopping point.
  
> In summary, I can live with the document as it is :)  I have
> asked the Exchange folks to review some of the parts I'm less
> familiar with, but I don't expect major concerns.

Thanks.

For planning purposes, I'm traveling tomorrow and Wednesday.
I'm going to try to read and respond to Jason's note, but I'm
pretty much out of bandwidth for this document between now and
the posting deadline, so the odds of my being able to get a -03
up before IETF are small to none.    My going silent in a little
while isn't indifference, it is just IAB and "day job"
priorities catching up with me.

    john

---

From: Joseph Yee <[email-masked]>
Date: Mon, 12 Jul 2010 16:18:37 -0400

On 2010-07-12, at 4:06 PM, John C Klensin wrote:

[snip]

> Good.  I'd like to hear from others, but this is consistent with
 
Just want to make quick reply to anchor15.  I am in favour to remove it too.  They are more into product design than protocol design.

Joseph

[snip]

---

From: Shawn Steele <[email-masked]>
Date: Tue, 13 Jul 2010 01:52:47 +0000

>  The main difference between (1) and (2) is where it waits

I would prefer that the document wait in the step as close to final as possible (eg: 2 if I understand correctly).  Or be a bit handwavy and don't hold up this doc.  Handwavey has another advantage in that it forces someone to look at the actual standard instead of just thinking "oh, support UTF-8, check".

> and the client must announce 8BITMIME.

I guess I missed where the client must use 8BITMIME.  I'd thought that the server announcing 8BITMIME would be sufficient to guarantee 8bitness, and that MIME structuring wouldn't necessarily be required.  It's not quite clear to me why MIME structure would be necessary for a simple "hi there" message if the server announced UTF8SMTP?  Clearly UTF8SMTP is permitting non-ASCII?

>        encoding [RFC3492] [RFC5890] on the right hand side is

I'm happy with it for this doc.  I think servers have to recognize it anyway (since the sender might not be EAI aware), however an EAI aware system should recognize ACE, so presumably it should be able to send clean UTF-8 w/o ACE.  It's not a big deal, but mostly I would like to try to keep the stuff on the wire "clean", which entails trying to forbid the clients from being lazy :)  So however much we can discourage laziness is what I'd like. MUST NOT may be too strong.

> Addresses my earlier question, and I'm fine with this text, although, 
> in practice, it's sort of like closing the barn door after the animals 
> have left. :)

> So, if you think the animals are out of the barn, I'd be very
> interested in a census of the relevant animals.

I believe really, really, really old copies of sendmail :)  (& maybe it was a funky sco version as well, I can't speak to it's pedigree).

>  That said, I don't think it will change this provision.

That's fine.  Just saying that adding "no other code pages" isn't going to fix any two decade old system out there (shudder) that depends on this.  

> > 9.1 - The charter has the "addresses" spec, I'm not sure you need to 
> > go into this much depth here.  (don't see that it hurts either)

> The answer may depend on when we want to push this document out the door and in what state.  See above.

Sorry, no clue which "above".  Hear from others?  That's fine.
 
> > * Normalization - I think the key point is that servers should be 

> Not quite, I think.

If NFC(x) == NFC(y), then NFD(x) === NFD(y), so equality is fine.

> We need to do the case analysis carefully here given that NFC
> sometimes decomposes (NFD never composes) and that this
> interacts with the 5321 limits on string lengths.

I'd missed the length thing, however my feeling is that people will send NFD (or something neither NFC nor NFD) on the wire, whether or not we think that's smart.  So, either way, a server really should be applying either NFC or NFD so they can appropriately match.  At that point I really don't care what their internal architecture is like, I think.  Obviously if I want to send NFD on the wire and it's too long, then I'm stuck, but we already have been SHOULDing NFC on the wire.

Anyway, I don't really care, I'd be using NFC anyway if I was building something.  I just wonder if it might be easier for some platforms to use a native NFD internally, and whether we were overly specifying what they might do internally.

> Actually, in some parts of the world and at the second and third level, foo123,
> foo124, and foo321 are all seen as domain name labels, 

Same reasoning though, when you see those, it's because there're a billion people getting free blogs or whatever, and then the ASCII gets confusable as well.  I really think there's too much fear about homographs (see http://blogs.msdn.com/b/shawnste/archive/2009/11/23/the-fud-of-idn-and-homographs.aspx ), however since that's the question managers like to ask, it's good to address it here.

>  But I think --and you seem to have confirmed--
> that we could spend a lot of effort tuning that text to make it only slightly better.
>  So, unless others have objections or concerns, we are probably at a good stopping point.

Agreed :)  

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Tue, 13 Jul 2010 13:14:12 -0400

--On Tuesday, July 13, 2010 01:52 +0000 Shawn Steele
<[email-masked]> wrote:

> I would prefer that the document wait in the step as close to
> final as possible (eg: 2 if I understand correctly).  Or be a
> bit handwavy and don't hold up this doc.  Handwavey has
> another advantage in that it forces someone to look at the
> actual standard instead of just thinking "oh, support UTF-8,
> check".

But it means that they may not be able to _find_ that "actual
standard": no forward reference is possible, the I-D is
published under an RFC number that is widely separated from the
rest of the documents, the other I-Ds get listed only as "work
in progress" without even a file name, etc.  As editor, I will
do whatever the WG says to do.  As co-chair, I will merely
advise the WG that this is not an easy decision.

In some sense, getting it published faster makes it less useful
in the long term, simply because of what needs to be taken out
or made non-specific.

> I guess I missed where the client must use 8BITMIME.  I'd
> thought that the server announcing 8BITMIME would be
> sufficient to guarantee 8bitness, and that MIME structuring
> wouldn't necessarily be required.  It's not quite clear to me
> why MIME structure would be necessary for a simple "hi there"
> message if the server announced UTF8SMTP?  Clearly UTF8SMTP is
> permitting non-ASCII?

It depends on what you mean by "permitting".  One cannot apply a
content-transfer-encoding to headers.  Encoded words are
forbidden in a lot of contexts, including actual mailbox names.
I can construct a few examples that would work but, if there is
actually going to be _any_ UTF-8 in the headers, it has to be
transmitted in 8bit form, and that means 8BITMIME MUST be
enabled.    We had a long discussion many months ago about
whether to incorporate the effect of 8BITMIME in UTF8SMTP or
whether to require that, if UTF8SMTP was used, 8BITMIME was
required.  We selected the latter, largely because it reduces
complexity and saved us working through the downgrade cases for
8BITMIME itself again.

The simple "hi there" message doesn't require 8BITMIME
transmission (although we still require that the server announce
8BITMIME if it is going to announce UTF8SMTP[bis]), but only if
the client doesn't use any EAI facilities: no non-ASCII
addresses, no header field elements in UTF-8, etc.

> I'm happy with it for this doc.  I think servers have to
> recognize it anyway (since the sender might not be EAI aware),
> however an EAI aware system should recognize ACE, so
> presumably it should be able to send clean UTF-8 w/o ACE.
> It's not a big deal, but mostly I would like to try to keep
> the stuff on the wire "clean", which entails trying to forbid
> the clients from being lazy :)  So however much we can
> discourage laziness is what I'd like. MUST NOT may be too
> strong.

See earlier note to Ernie about the tradeoffs in this.   And I
would really, really, like to be sure that whatever is said in
this doc is consistent with whatever is (or is not) said in
5336bis.

>> although,  in practice, it's sort of like closing the barn
>> door after the animals  have left. :)
> 
> I believe really, really, really old copies of sendmail :)  (&
> maybe it was a funky sco version as well, I can't speak to
> it's pedigree).

The really, really, really old versions of sendmail had a rather
elegant and efficient way to deal with octets with the
high-order bit turned on.  They zeroed the bit without comment
and proceeded.  So alternate code pages may have worked (and
national variations on ASCII (really ISO 646) definitely did,
but nothing involving 8bit codings including 8859 variations.

> That's fine.  Just saying that adding "no other code pages"
> isn't going to fix any two decade old system out there
> (shudder) that depends on this. 

If there are two decade old versions of sendmail that exhibit
that behavior, I'd assume they are locally-patched forked
copies.   Making provisions for them gets us rather far into the
territory of permanent support for bugs and bad local ideas.

> > 9.1 - The charter has the "addresses" spec, I'm not sure
>> > you need to  go into this much depth here.  (don't see that
>> > it hurts either)
> 
>> The answer may depend on when we want to push this document
>> out the door and in what state.  See above.
> 

Hear from others and those three options about handling the
document, how long it waits, etc.

> I'd missed the length thing, however my feeling is that people
> will send NFD (or something neither NFC nor NFD) on the wire,
> whether or not we think that's smart.  So, either way, a
> server really should be applying either NFC or NFD so they can
> appropriately match.  At that point I really don't care what
> their internal architecture is like, I think.  Obviously if I
> want to send NFD on the wire and it's too long, then I'm
> stuck, but we already have been SHOULDing NFC on the wire.
> 
> Anyway, I don't really care, I'd be using NFC anyway if I was
> building something.  I just wonder if it might be easier for
> some platforms to use a native NFD internally, and whether we
> were overly specifying what they might do internally.

We haven't said a thing about what is done internally, only that
testing has to be equivalent under NFC.  "The length thing" may
cause some strings to be valid under NFC but not under NFD (and
vice versa) so being clear about the canonical test is
important.  

>> Actually, in some parts of the world and at the second and
>> third level, foo123, foo124, and foo321 are all seen as
>> domain name labels, 
> 
> Same reasoning though, when you see those, it's because
> there're a billion people getting free blogs or whatever, and
> then the ASCII gets confusable as well.  I really think
> there's too much fear about homographs (see
> http://blogs.msdn.com/b/shawnste/archive/2009/11/23/the-fud-of
> -idn-and-homographs.aspx ), however since that's the question
> managers like to ask, it's good to address it here.

Exactly.

>> that we could spend a lot of effort tuning that text to make
>> it only slightly better. So, unless others have objections or

thx
    john