Subject: [EAI] Rechartering

From: Shawn Steele <[email-masked]>
Date: Mon, 13 Jul 2009 18:37:42 +0000

I was wondering if there were any thoughts about rechartering and any standards-track schedule.  The info on http://www.ietf.org/html.charters/eai-charter.html is clearly out of date :)

In particular, I've heard that China is interested in publishing Chinese standards based on the experimental RFCs already released by this working group (I've heard they intend to publish in November.)  It seems like it'd be good if there were standards-track RFCs that addressed the Chinese standards' topics at least in-progress by then.

-Shawn

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 13 Jul 2009 22:49:44 +0100

Hi Shawn,

>In particular, I've heard that China is interested in publishing Chinese standards based on the experimental RFCs already released by this working group (I've heard they intend to publish in Novembe…
Thank you for the information.

I think the WG is going to be discussing rechartering-related topics at 
length in Stockholm.

---

From: Shawn Steele <[email-masked]>
Date: Mon, 13 Jul 2009 22:07:39 +0000

That's good.  Unfortunately I can't attend :( If the WG can provide a roadmap/schedule at Stockholm, that'd help my planning efforts :)

Thanks,

-Shawn

-----Original Message-----
From: Alexey Melnikov [mailto:[email-masked]]
Sent: Monday, July 13,  2009 14:50
To: Shawn Steele
Cc: [email-masked]
Subject: Re: [EAI] Rechartering

Hi Shawn,

---

From: Shawn Steele <[email-masked]>
Date: Mon, 13 Jul 2009 22:27:02 +0000

I didn't see any discussion of my mailto: comments.

I think that it's good for the mailto: doc to provide escaping of the UTF-8 code points, however I also think that many clients will be quite liberal in what they allow.  (Safari & IE on Windows already seem to allow mailto:café(at)example.org and I don't see any reason to break that)

In other words in 2) I'd like to see the language:

"Applications MAY process UTF-8 encoded values in mailbox and hvalue directly even if they aren't percent-encoded."

Also it looks like hvalue SHOULD be percent encoded, but in the mailbox section it says "Percent-encoding can be used to denote non-ASCII characters" without any SHOULDs that I can see.

I'm happy for hvalue (and mailbox) to have SHOULD be percent-encoded for IRI [RFC3987] compatibility, though I note that humans typing mailto values directly aren't likely to do this.

4.1 also says "angle brackets...are mandatory", but doesn't use MUST language.  I'd like it to be SHOULD:

   Please note that if the left-hand side of the mail address contains
   non-ASCII characters, the less-than and greater-than sign (angle
   brackets, escaped as %3C and %3E) SHOULD be included.

(Why are <> necessary?)

I'd also like similar language if the angle brackets are missing

"Applications MAY process UTF-8 encoded mailbox values directly even if the RECOMMENDED angle brackets are not present"

I don't think it's realistic to expect current systems (Browsers, OS, etc.) to be more restrictive than they already are.  Specifically UTF-8 domain names tend to work fine with mailto in IDN aware systems, regardless of percent encoding or angle brackets.

Any comments this time? :)

-Shawn

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 17 Jul 2009 18:42:51 +0200

> That's good.  Unfortunately I can't attend :( If the WG can provide a roadmap/schedule at Stockholm, that'd help my planning efforts :)
The biggest obstacle to getting a roadmap/schedule has been that we have 
very little feedback from implementors on when they expect to feel that 
they have tested the implementations in practice enough to give feedback 
on the documents.

The most critical part is getting the experience of how non-EAI users 
(and their software) react to downgraded messages.

I'm very unwilling to put up a schedule just to have a schedule - 
experience shows that this easily gets into the mode where the chair 
pushes the writers, who then write whatever is needed to get the work 
done, no matter what the "right" result should be.

We need more mail from our implementation teams.

                         Harald

---

From: Shawn Steele <[email-masked]>
Date: Fri, 17 Jul 2009 20:32:21 +0000

We definitely want the "right" result :)  My request isn't "just to have a schedule", but rather so that there's something we can plan to.  Also, if "right" takes much longer, it won't matter.  There's a huge user segment that currently doesn't have effective email support because they aren't literate in the Latin script.  It'd be nice if those people could be enabled to experience what we take for granted, and IMHO it's worth risk of not being perfect in order to support them in the next year instead of longer.

I have two concerns regarding scheduling:

1) I hear from CNNIC that China is creating standards based on the Experimental RFCs, due to be published in November.  We may not care or approve of that behavior, but it may be a standard my products have to support, so it'd be nice if it was more in-sync with IETF.  I realize this isn't this working group's problem, but it does impact the behavior of mail in China.

2) Several of our product units are starting to look at "vNext"  For planning purposes I have to guess about when EAI'll finally become a standard.  Mostly it comes down to "is this standard going to happen soon enough that we need to address it in this product cycle?  Or is it going to be uncertain enough that we'll punt it to next release cycle."

As you're probably aware, some products' release cycles take several years.  So if there aren't standards, or at least nearly finished drafts of standards, in the next 4-6 months, then it is likely that EAI will miss our next release cycle.  Even if we started now, it could take some time for EAI support in our products.  If this release cycle is missed, it could be 5+ YEARS before our customers see complete support.  EAI touches almost every product we make, even XBox.  It's a huge amount of work to "do it right".

I think everyone's customers need EAI support sooner rather than later, and I'm trying to build a case for earlier support, but there's no point to us working hard to build something that isn't standardized.  And it's hard for me to tell some executive "We've gotta jump on this" if the only schedule the WG has posted is obviously years behind schedule and there's no end-game in sight.

My thinking:

* Some drafts are in last-call, so that "phase" is pretty much complete.
* 1-2 months for experimenting & creating the report, maybe Aug-Sept?
* Could rechartering happen in parallel?, so end of Sept?
* Then, assuming the Experimental RFC's didn't reveal any gaping holes, could they be fast tracked in 3 months or so?  Maybe by Jan 2010?

Picking months:

* August 2009 - Finish Last Calls
* August - September 2009, continue experimenting and create the report
* September 2009, Recharter for standards track.
* January 2010.  Core RFCs stable, preferably done or last-call.

If the WG doesn't think we're going to have anything by middle 2010, please let me know & I'll stop bugging our product units.  I know IETF efforts are typically very slow, but I think that this schedule would be achievable, with the "right" results, IF we are willing to commit to the work and solving non-ASCII users needs sooner rather than later.

-Shawn

-----Original Message-----
From: Harald Alvestrand [mailto:[email-masked]]
Sent: Friday, July 17,  2009 9:43
To: Shawn Steele
Cc: Alexey Melnikov; [email-masked]
Subject: Re: [EAI] Rechartering

                         Harald

---

From: Xiaodong Lee <[email-masked]>
Date: Sun, 19 Jul 2009 17:55:38 +0800

Hi, Shawn,

On behalf of CNNIC, not as co-chair of this WG, any questions are
welcome.

Shawn Steele wrote:

You have the right information that there are official proposed
standard which are on going in China now, those Chinese standard
adopted the IETF standard on EAI totally, except for adding some
special process for Chinese characters, just for convenient of Chinese
email vendors to follow up the standard. If Microsoft decide to follow
the IETF standard, it will be okay.

If Microsoft can adopt the EAI protocol, it will be fine for end users,
especially for Chinese users, that is why we make those RFCs into China
national industry standard.

My thinking:

Picking months:

-Shawn

-----Original Message-----
From: Harald Alvestrand [mailto:[email-masked]]
Sent: Friday, July 17,  2009 9:43
To: Shawn Steele
Cc: Alexey Melnikov; [email-masked]
Subject: Re: [EAI] Rechartering

Shawn Steele wrote:

Harald

_______________________________________________
IMA mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Sun, 19 Jul 2009 16:55:22 -0400

--On Friday, July 17, 2009 8:32 PM +0000 Shawn Steele 
<[email-masked]> wrote:

> We definitely want the "right" result :)  My request isn't
> "just to have a schedule", but rather so that there's
> something we can plan to.  Also, if "right" takes much longer,
> it won't matter.  There's a huge user segment that currently
> doesn't have effective email support because they aren't
> literate in the Latin script.  It'd be nice if those people
> could be enabled to experience what we take for granted, and
> IMHO it's worth risk of not being perfect in order to support
> them in the next year instead of longer.
>...

Shawn,

Personal opinion only...

First, we know from the IDNA experience what the costs are of 
getting into a hurry because of some real or imagined deadline, 
taking shortcuts about decision-making, and getting things even 
a little bit wrong.  We discover that those decisions cause 
problems somewhere, problems that are serious to some 
communities who very much need to have them fixed.  And then we 
find that discussions of fixing them cause someone to say, 
essentially, "no matter how bad the earlier decision was, we 
cannot make an incompatible change, so that community just 
loses".  In the long term, that sort of situation costs us 
global interoperability and everyone loses.

That said, there are conceptually two design elements in the EAI 
work.  On has to do with the basic idea of non-ASCII addresses, 
header fields, and related information.   It is relatively 
straightforward and, although I think more testing is needed, 
all of the evidence so far is that it just works between EAI 
implementations (and communication with non-EAI implementations 
gets handled the same way any "the server doesn't support the 
capability you need" situation --with the exception of 
8BITMIME-- is handled, and that model is very well tested).

The other is the idea of downgrading.  It involves exploration 
of a new area -- new syntax in addresses, new header fields, 
comments that aren't quite comments, header fields with closer 
relationships to other header fields than we are usually 
comfortable about, non-obvious rules about when downgrading is 
permitted, and so on.  We've got some empirical evidence from 
Ernie's tests that different implementations interpret the 
specification a bit differently and that some combinations of 
systems will lose information.

Before the WG can meaningfully move toward standardization, we 
have to sort the downgrading issue out.  If there is strong 
consensus one way or the other as to what to do, then I think we 
can move forward quickly.   If there isn't, we will will 
probably need a lot of empirical evidence about what works and 
what doesn't, what needs respecification, and so on.  That is 
going to take time, especially if we continue at our current 
pace.

In particular, with the understanding that I am not proposing or 
recommending this and don't favor it, suppose the WG reached 
agreement in the next few weeks that speed was more important 
than a downgrade capability, independent of what various of us 
believe about whether downgrading could be sorted out in the 
long run and about how useful it would be.  We'd need to look 
carefully at the IMAP and POP documents, but excising 
downgrading from the rest would be fairly straightforward and 
the dominant concern for "use the experimental docs and 
implementations to see if this can really work in the hostile 
real email world" would vanish.  I'd guess that, if the various 
authors got motivated and moving, we could have a charter 
modified for standards track and updated documents that were 90% 
final by Hiroshima.

But, we keep downgrading in the program and if testing continues 
to be as leisurely as it has been in the past, with results that 
are as ambiguous, it may be a long time.

I note that, while you've made your wishes clear, you haven't 
offered to get a test implementation together on a platform 
entirely different from the ones that have been used so far and 
then both make that test platform available for interoperability 
testing and do significant testing yourself/yourselves.  That is 
the sort of action that would help things move forward more 
quickly.

again, just my opinion.
    john

---

From: Shawn Steele <[email-masked]>
Date: Tue, 21 Jul 2009 06:37:18 +0000

I appreciate the concerns about moving too fast, and I don't disagree with your assessment of the risk.

My concern is the standardization of UTF-8 email in China.  I'd rather that the IETF standards, including downgrade, were compatible with China's standards.  Otherwise we could end up with a de-facto standard of whatever China is (or isn't) doing with downgrade.

I'm also concerned that the quirks of downgrade aren't going to be very discoverable in an laboratory setting :(.  I don't foresee any problems that can't be corrected with the current approach to downgrade, however it's the unforeseen that's the problem.  At the leisurely pace it's been proceeding, I'm afraid that the industry won't wait for the working group, particularly if the Chinese standards proceed without the IETF WG standards.

I also fear a Chinese EAI standard without downgrade more than a downgrade with quirks.  Also, hopefully, "downgrade" will eventually stop being used, so even if it's really bad, at least it should be limited :)

So IMO moving cautiously/slowly is also a risk, and it's rapidly becoming a larger risk than the proposed documents in their current state.  I'm aware of at least 2 other attempts/approaches to "international email" that were shot down/deferred to wait for a real EAI standard from the IETF.  I doubt the users in those communities will wait forever.

-Shawn

________________________________________
From: John C Klensin [[email-masked]]
Sent: Sunday, July 19, 2009 1:55 PM
To: Shawn Steele; Harald Alvestrand
Cc: [email-masked]
Subject: Re: [EAI] Rechartering

Shawn,

Personal opinion only...

again, just my opinion.
    john

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 21 Jul 2009 15:12:09 +0100

On Tue, 21 Jul 2009 07:37:18 +0100, Shawn Steele  
<[email-masked]> wrote:

> I'm also concerned that the quirks of downgrade aren't going to be very  
> discoverable in an laboratory setting :(.  I don't foresee any problems  
> that can't be corrected with the current approach to downgrade, however  
> it's the unforeseen that's the problem.  At the leisurely pace it's been  
> proceeding, I'm afraid that the industry won't wait for the working  
> group, particularly if the Chinese standards proceed without the IETF WG  
> standards.

I see no problem if the Chinese were to issue a standard without  
downgrade, provided it was otherwise compatible with the format we have  
decided.

Ideally, the IETF should standardise RFC 5335 now, so that all the  
necessary headers etc get defined and we have a fixed email format (a  
plain upgrade of RFC 5332), with the downgrade standard to follow later.  
Of course that is not possible with the documents worded as they are now,  
but there must be some way to achieve that effect. That would send a clear  
signal that internationalised email is fixed THAT WAY, and a few  
subsequent trimmings of the downgrade machanism within that framework  
should not cause problem (especially as downgrade may well disappear in  
the Long Term).

So if that is, in essence, what the Chinese standard is going to do, then  
why not?

> I also fear a Chinese EAI standard without downgrade more than a  
> downgrade with quirks.  Also, hopefully, "downgrade" will eventually  
> stop being used, so even if it's really bad, at least it should be  
> limited :)

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: John C Klensin <[email-masked]>
Date: Tue, 21 Jul 2009 14:25:08 -0400

--On Tuesday, 21 July, 2009 06:37 +0000 Shawn Steele
<[email-masked]> wrote:

> I appreciate the concerns about moving too fast, and I don't
> disagree with your assessment of the risk.
> 
> My concern is the standardization of UTF-8 email in China.
> I'd rather that the IETF standards, including downgrade, were
> compatible with China's standards.  Otherwise we could end up
> with a de-facto standard of whatever China is (or isn't) doing
> with downgrade.

Well, I don't know how to go much further in speculation in that
direction.  There are people on the list, including the
co-Chair, who presumably know what is going on with Chinese
standardization efforts and they can speak up if they consider
that appropriate.

There are a number of areas in which we've discovered, in more
ways than I (at least) would have initially predicted, that
Chinese is different from other scripts.   Some of them, like
the effects of Han unification and SC/TC relationships, are
perhaps artifacts of Unicode coding decisions, others are more
basic.  It is more difficult in some ways, and easier in others.
Almost every time someone has extrapolated from CJK to other
scripts or in the other direction, they have gotten themselves
into trouble.  

That doesn't change your basic concern, but is an extra argument
why we should proceed with due caution and why an international
standard might be very different from one for use within China
for communication among those who are Chinese-literate.   I
think our Chinese colleagues have recognized that distinction
from the beginning and, again, can speak for themselves.

> I'm also concerned that the quirks of downgrade aren't going
> to be very discoverable in an laboratory setting :(.  

We share that concern.  It is why the "testing" I'm most anxious
to see involves turning EAI loose in the general Internet email
environment and seeing what happens.  That is exactly the kind
of testing Ernie has been trying to do.   While the specific
problems that have been identified can be fixed (some may
require revisiting documents to be sure we all really intend the
same thing), the trend is not encouraging even after only a
handful of tests.

> I don't
> foresee any problems that can't be corrected with the current
> approach to downgrade, however it's the unforeseen that's the
> problem.  At the leisurely pace it's been proceeding, I'm
> afraid that the industry won't wait for the working group,
> particularly if the Chinese standards proceed without the IETF
> WG standards.

While I'm actually anxious to move this along as quickly as we
reasonably can, at this point, I can only remind you and others
about the success rate of proprietary (and even national and
regional) email models and standards that do no interoperate
smoothly with the existing Internet base.  I'm sure you, or
those around you, remember both MSMail and the original,
incompatible with even X.400, Exchange Server model.  Harald
doesn't need to be reminded about X.400 itself, and that list
goes on and on.

> I also fear a Chinese EAI standard without downgrade more than
> a downgrade with quirks.  Also, hopefully, "downgrade" will
> eventually stop being used, so even if it's really bad, at
> least it should be limited :)

See above above "quirks".  And, whether "downgrade" is used or
not,  it makes  sufficient changes to the fundamental syntax of
Internet email addresses, and sufficient requirements on systems
processing headers, that it will be with us forever even if no
one is actually using it.  Conversely, it makes implementation
of i18n mail sufficiently more difficult that we just can't
estimate whether having it will speed deployment (by making it
slightly easier for EAI-compliant systems to communicate with
legacy ones) or slow it (by making implementation and deployment
of EAI-compliant systems much more difficult.   The issue with
implementation and deployment are pervasive and may extend into
things that the documents don't talk about.  For example, the
typical address book today maps a name onto one email address
(or maybe "home" and "work" email addresses).  EAI downgrading
essentially requires that address book support more than one
address with the same role.  Is that a big deal?  I don't know--
it will certainly depend on the address book design.  But it is
almost certainly going to require more work than simply making
provision for the existing fields to be in UTF-8 and more
instruction to the user about what is going on.   Part of that
discussion is closely related to one we are having in the
IDNABIS context, which is how much we really expect users to
know and adapt to in order to keep computer systems and networks
happy.  I don't know the answer, but, unlike a few years ago,  I
now suspect that alternate addresses in the protocol will put
more of a knowledge burden on users than just having messages
returned if non-ASCII address doesn't go through.  

As another example of a side effect of the downgrading
architecture, consider the huge number of web sites out there
that ask for email addresses.  We know that many of them,
probably a majority, can't even get a local part that contains a
"+"-delimited subaddress right.  Can they be converted to handle
UTF-8 local parts?  I think so, but, for some systems, it will
require more work and education than we probably anticipate.
But I personally think that the odds of getting a site that
can't manage "[email-masked]" to be able to handle 
  <[email-masked] <[email-masked]>>
within my lifetime are pretty small -- the pointed brackets will
be rejected, the alternate address will confuse it beyond any
hope, etc.   On the other hand, I think a simple
   [email-masked]
will work before I convince them to handle that "+".

> So IMO moving cautiously/slowly is also a risk, and it's
> rapidly becoming a larger risk than the proposed documents in
> their current state.  I'm aware of at least 2 other
> attempts/approaches to "international email" that were shot
> down/deferred to wait for a real EAI standard from the IETF.
> I doubt the users in those communities will wait forever.

Sure.  But see above.

I'm actually getting very sympathetic to the idea in Charles's
note of doing the rest and then letting downgrading sort itself
out later.  But, in practice, I think that dooms downgrading, at
least along the lines of the current specification.   Without
it, we get to simplify the syntax to make i18n address syntax
just traditional syntax with non-ASCII characters -- not only
does the alternate address syntax go away but so does the
requirement for brackets, etc.  Syntactically, SMTP is actually
pretty easy either way.  But, once you get the provisions for
downgrading out of the syntax of addresses in the mail headers
--and, in practical terms, the addresses that users see-- those
extensions are, in practice, probably gone for good: it won't be
easy to put them back in if systems don't even recognize the
syntax as valid.

But, to partially agree with Charles and say it differently, I
think one can have "quick" without Downgrade or somewhat more
slowly (a lot more slowly if the WG continues at its recent
pace) with it.   I don't have any idea how to choose but I am
determined that, if we have Downgrading, it will be "right" and
its interoperability and implementation characters well-tested
and understood.

    john

     john

---

From: Shawn Steele <[email-masked]>
Date: Thu, 23 Jul 2009 01:16:14 +0000

>environment and seeing what happens.

On a Windows PC that pretty much involves "everything".  OpenType apparently even has a field for an email address.  I can see some partial tests being possible, but I can't imagine that a full test of "everything" would be feasible, unfortunately.  (I can see where I could talk myself into a corner here, I don't mean to not be careful, I just mean that we aren't going to catch everything.)

> within my lifetime are pretty small.

I'd agree, but [email-masked] should still work :)0

So, I pretty much agree with your points, but you aren't giving me much hope of EAI being adaquate to those criteria in the next decade.  I still think some sort of schedule would be interesting to help us figure out how to proceed in addressing your concerns.

-Shawn

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 23 Jul 2009 07:49:30 +0200

> So, I pretty much agree with your points, but you aren't giving me much hope of EAI being adaquate to those criteria in the next decade.  I still think some sort of schedule would be interesting to…
Would it be interetsing to (for at least 5 minutes) explore the idea of 
a test-driven schedule, rather than a calendar-driven one?

That is, specify a number of criteria that should be met before 
standards-track, and see if we can work backwards:

tests that need to be done -> people volunteering for the work -> 
reasonable schedule?

                    Harald

---

From: Shawn Steele <[email-masked]>
Date: Thu, 23 Jul 2009 06:25:51 +0000

Yes, that would be good :)  Especially understanding which criteria people feel are necessary, some could be assumed and people could have different views.

I wasn't trying to suggest that a schedule must fit a certain date.  There are possibly dates I have to make decisions by, but that shouldn't drive a schedule.  Any schedule would help me understand if EAI is going to take another 6 months, year, or 10 years :)  And if a schedule misses milestones, then it still helps understand how long the process may take.

Also your suggestion is good because if people feel that when such a schedule is derived it is "too slow", then maybe it'll be clear what could be done to speed it up.  (like maybe whomever feels it isn't happening enough could step up as volunteers.)

-Shawn

________________________________________
From: Harald Alvestrand [[email-masked]]
Sent: Wednesday, July 22, 2009 10:49 PM
To: Shawn Steele
Cc: John C Klensin; [email-masked]
Subject: Test - driven schedule (Re: [EAI] Rechartering)

                    Harald

---

From: "YAO Jiankang" <[email-masked]>
Date: Thu, 23 Jul 2009 16:23:29 +0800

----- Original Message ----- 
From: "Shawn Steele" <[email-masked]>
To: "John C Klensin" <[email-masked]>; "Harald Alvestrand" <[email-masked]>
Cc: <[email-masked]>
Sent: Tuesday, July 21, 2009 2:37 PM
Subject: Re: [EAI] Rechartering

> My concern is the standardization of UTF-8 email in China.  I'd rather that the IETF standards, including downgrade, were compatible with China's standards.  >Otherwise we could end up with a de-facto standard of whatever China is (or isn't) doing with downgrade.

As Xiaodong has already pointed out it, China's standards will be full compatible with IETF EAI WG RFCs. we will also follow the downgrade rfc. Actually, many chinese care for the downgrade issue very much. 

> 
> I'm also concerned that the quirks of downgrade aren't going to be very discoverable in an laboratory setting :(.  I don't foresee any problems that can't be >corrected with the current approach to downgrade, however it's the unforeseen that's the problem. 

it is better that industry powers such Microsoft, yahoo, google can join this beta test.

> At the leisurely pace it's been proceeding, I'm afraid that the industry won't wait for the working group, particularly if the Chinese standards proceed without the >IETF WG standards.

The first chinse email standard is mainly based on RFC4952. ohter standards will follow RFC5335 5336 5337 5504. if we can move the standard track soon,
china will follow the new IETF standard. Chinese are eager to IETF EAI standard track too. 

> 
> I also fear a Chinese EAI standard without downgrade more than a downgrade with quirks.  Also, hopefully, "downgrade" will eventually stop being used, so >even if it's really bad, at least it should be limited :)

I am also involved in the chinese EAI standards' work. the chinese standard will support downgrade.

Yao Jiankang

---

From: "YAO Jiankang" <[email-masked]>
Date: Thu, 23 Jul 2009 16:33:47 +0800

----- Original Message ----- 
From: "Harald Alvestrand" <[email-masked]>
To: "Shawn Steele" <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, July 23, 2009 1:49 PM
Subject: [EAI] Test - driven schedule (Re: Rechartering)

> a test-driven schedule, rather than a calendar-driven one?

It is a good news. 
I think that both the test work and standard track progress work can be drivend in the parallel way.

CNNIC
Yao Jiankang

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 23 Jul 2009 10:44:09 +0100

Seems I sent my mail to John directly, rather than to the list. Here it  
is, together with his reply and my further views.

On Wed, 22 Jul 2009 16:58:11 +0100, John C Klensin <[email-masked]> wrote:

> --On Wednesday, 22 July, 2009 10:42 +0100 Charles Lindsey
> <[email-masked]> wrote:
>
>> What I had in mind was that the headers document (and probably
>> the smtp one as well) would fix the syntax of addresses with
>> alternatives (extra <...> and all). So any future work on
>> downgrade would have to live with that syntax. If that
>> syntactic change is to happen at all, it needs to happen NOW
>> because, as John points out, an awful lot of systems all over
>> the place are going to have to learn to allow it (at least to
>> the extent that they accept it and store it and/or pass it on
>> to further systems as an opaque bundle, even if they cannot
>> make use of it themselves).
>
> That is plausible.  However, the syntax change is a huge price
> to pay in terms of difficulty of changing systems and
> reeducating users unless it is going to support a downgrade
> mechanism that works and is significantly useful.   So I can't
> agree that changing the syntax to support downgrade without
> being sure that we have a downgrade model that we are also
> willing to standardize is a good idea.   One or the other --
> syntax change with a fully-fleshed-out and tested downgrade or
> no syntax change, no downgrade.

Sure. If we standardize the format, together with alt-addresses, then  
there must be a firm commitment to follow it up with a downgrade standard.  
But since the downgrade protocol is likely to need further tweaks, it  
could be better to leave it as Experimental until it has stablized.

I envisage that major pieces of software, such as servers, will be written  
with the downgrade part as an optional plug-in. The default plugin would  
be to "bounce everything" (which is a legitimate tactic under the present  
proposal). One up from that is a plugin that will recognize and forward it  
to the alt-address, but not much more.

Nobody wants to rewrite a full server each month, which is why the basic  
functionality must be fixed from the start. But upgrading your downgrade  
plugin (well, probably not monthly) is a task that could be more easily  
envisaged, and would be driven by the needs of the particular community  
you were trying to serve.

---

From: John C Klensin <[email-masked]>
Date: Thu, 23 Jul 2009 11:41:29 -0400

--On Thursday, July 23, 2009 16:33 +0800 YAO Jiankang
<[email-masked]> wrote:

>>>> can't manage "[email-masked]" to be able to
>>>> handle <[email-masked] <[email-masked]>>
>>> So, I pretty much agree with your points, but you aren't
>>> giving me much hope of EAI being adaquate to those criteria
>>> in the next decade.  I still think some sort of schedule
>>> would be interesting to help us figure out how to proceed in
>>> addressing your concerns.
>>>   
>> Would it be interetsing to (for at least 5 minutes) explore
>> the idea of  a test-driven schedule, rather than a
>> calendar-driven one?
> 
> It is a good news. 

I didn't see "parallel" in Harald's suggestion.  And I do think
the suggestion is a good one.  

I think it is also worth pointing out that some of the recent
discussions on this list may suggest changes in the SMTP
extension and header specs.  Those changes may or may not be
necessary, which is another testing and thinking topic, but they
represent decisions that need to be made and it would be better
to make them with data than without.

For example, experience elsewhere (including in the IDNABIS
discussions) makes it fairly clear that either NFC or NFD
normalization is a necessity if one is going to compare two
Unicode strings.  Without one or the other, two different ways
of entering exactly the same character will not compare equal.
A given user may have only one way to enter a particular
character while another user may have only the other way
available.

That suggests that we should _at least_ recommend normalization
in the SMTP server (much as 5321 recommends case-insensitive
matching) and that we should think about requiring it.  But,
again, data would help.

     john

---

From: John C Klensin <[email-masked]>
Date: Thu, 23 Jul 2009 11:59:10 -0400

--On Thursday, July 23, 2009 01:16 +0000 Shawn Steele
<[email-masked]> wrote:

>... 
> On a Windows PC that pretty much involves "everything".
> OpenType apparently even has a field for an email address.  I
> can see some partial tests being possible, but I can't imagine
> that a full test of "everything" would be feasible,
> unfortunately.  (I can see where I could talk myself into a
> corner here, I don't mean to not be careful, I just mean that
> we aren't going to catch everything.)

Of course not.  That would require nearly proving a universal
negative. On the other hand, one only needs to find a few
problematic cases to know that there is a problem.

Your Windows "nearly everything" example is an interesting one
simply because one of the ways of making users crazy (another
one of those intra-system examples) is that if there is not a
standard way to parse addresses that either (i)looks for a
balanced/ matching ">" rather than either getting into error
states if a "<" is encountered or getting into problems with
">>" or (ii) is sufficiently centralized in an API that
everything uses so as to be easily changed...

Then it seems to me that you are in big trouble because
deploying EAI with Downgrade --even in the syntax-only form
Charles suggests-- means that users will encounter it working
one way in some places (e.g., accepting non-ASCII addresses but
not alternate address syntax) and in other ways (e.g., accepting
non-ASCII and alternate addresses) in others, even within the
same system.   That is, IMO, much worse even that having
extended (non-ASCII) addresses work in some places but not in
others.

> I'd agree, but [email-masked] should still work :)0

Sure.  But unless it can be parsed out of the above, that isn't
interesting.   Note that, if one had a Unicode-native system and
simply removed the "ASCII address" restriction,
    [email-masked]
would parse as the address the user would expect while
    <[email-masked] <[email-masked]>>
would almost certainly get some sort of "invalid end of address"
or "invalid unquoted space in the middle of the address" error.
And, if it tried to "fix" that error, as many MUAs and some
other systems would, life would get interesting.

> giving me much hope of EAI being adaquate to those criteria in
> the next decade.  I still think some sort of schedule would be
> interesting to help us figure out how to proceed in addressing
> your concerns.

Again, if you are in a hurry, the key question is whether we
need downgrading.  If we don't, a large fraction of the issues
I'm raising just disappear and I have no trouble working through
a schedule (in your model) or a requirements plan (in Harald's).

    john

---

From: Shawn Steele <[email-masked]>
Date: Fri, 24 Jul 2009 04:48:56 +0000

> As Xiaodong has already pointed out it, China's standards will be full
> compatible with IETF EAI WG RFCs. we will also follow the downgrade
> rfc. Actually, many chinese care for the downgrade issue very much.

I think that'll "work", however the current RFCs are EXPERIMENTAL, not STANDARD.  So any future RFCs need not be compatible with the experimental RFCs.  Should that (IMO very bad thing) happen, then the Chinese standard and the IETF standards would diverge :(

I think it would be "safest" if the IETF RFCs were updated to standards first, then the Chinese standards follow those, however I don't believe that'll happen in practice.  Then the IETF will have to choose to either just turn the experimental RFCs into real standards with minimal changes to conform to the defacto Chinese standard, or to deviate from the Chinese standard and break what might already be implimented.

- Shawn

---

From: Shawn Steele <[email-masked]>
Date: Fri, 24 Jul 2009 04:54:02 +0000

> matching) and that we should think about requiring it.  But,

I think that's fairly irrelevent.  It won't break the RFCs for the rest of the system.  It'd only break servers that registered both NFC and NFD (or similar variations) of names.  Then they'd pretty much deserve to be broken :)

-Shawn

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 24 Jul 2009 09:22:40 +0200

> I think that's fairly irrelevent.  It won't break the RFCs for the rest of the system.  It'd only break servers that registered both NFC and NFD (or similar variations) of names.  Then they'd prett…
The more dangerous brokenness is the business card data entry situation, 
where people enter an address from a business card and get a "no such 
mailbox" response, without there being any visual difference between the 
two strings.

Requiring normalization at the server (without placing a requirement on 
what one normalizes to) would at least remove this class of issue.

---

From: "YAO Jiankang" <[email-masked]>
Date: Fri, 24 Jul 2009 15:40:26 +0800

----- Original Message ----- 
From: "Shawn Steele" <[email-masked]>
To: "YAO Jiankang" <[email-masked]>; "John C Klensin" <[email-masked]>; "Harald Alvestrand" <[email-masked]>
Cc: <[email-masked]>
Sent: Friday, July 24, 2009 12:48 PM
Subject: RE: [EAI] Rechartering

>I think that'll "work", however the current RFCs are EXPERIMENTAL, not STANDARD.  So any future RFCs need not be compatible with the experimental >RFCs  Should that (IMO very bad thing) happen, then the Chinese standard and the IETF standards would diverge :(

China eai standards are a series of standards. every eai rfc is supposed to be published as a standard.
the first chinese standard based on RFC4952 is supposed to be published at the end of this year. others will be published next year. So if during these 2 years, we move to the standard track progress, Chinese standard will be updated to follow these new updates. 

>I think it would be "safest" if the IETF RFCs were updated to standards first, then the Chinese standards follow those, however I don't believe that'll happen in >practice.  

as you know, China is eager to have the EAI standard to guide the implmentation.

as said above,  if during these 2 years,  the first version of Chinese standard will be updated to follow these new updates. 
if more than 2 years, china may publish other eai standard and promote the industry to use.

according to the current  discussion, it seems that there is a dilemma: no industry to join the test or deployment  may mean not enough tests of eai; more industry to join the tests of eai means that there is a possibility of a defacto standard of eai.

>Then the IETF will have to choose to either just turn the experimental RFCs into real standards with minimal changes to conform to the defacto Chinese >standard, or to deviate from the Chinese standard and break what might already be implimented.

in some sense,  I share the concerns with you. we will try to make the other  china standard publication based on RFC 5335 5336 or other new standard track rfc  to catch up the new progress of EAI WG.  if there is no standard track progress or plan in IETF, the industry will have to follow rfc5335 5336 5504 to implement eai because there is market needs even that there are no chinese standard to guide that implementation.

so I appreciate your points: moving too fast is risk; moving cautiously/slowly is also a risk.

Yao Jiankang
CNNIC

---

From: John C Klensin <[email-masked]>
Date: Fri, 24 Jul 2009 04:34:09 -0400

--On Friday, July 24, 2009 09:22 +0200 Harald Alvestrand
<[email-masked]> wrote:

>>...
>> I think that's fairly irrelevent.  It won't break the RFCs
>> for the rest of the system.  It'd only break servers that
>> registered both NFC and NFD (or similar variations) of names.
>> Then they'd pretty much deserve to be broken :)
>>   
> The more dangerous brokenness is the business card data entry
> situation, where people enter an address from a business card
> and get a "no such mailbox" response, without there being any
> visual difference between the two strings.
> 
> Requiring normalization at the server (without placing a
> requirement on what one normalizes to) would at least remove
> this class of issue.

Exactly.
     john

---

From: John C Klensin <[email-masked]>
Date: Fri, 24 Jul 2009 04:57:45 -0400

--On Friday, July 24, 2009 04:48 +0000 Shawn Steele
<[email-masked]> wrote:

>...
> I think it would be "safest" if the IETF RFCs were updated to
> standards first, then the Chinese standards follow those,
> however I don't believe that'll happen in practice.  Then the
> IETF will have to choose to either just turn the experimental
> RFCs into real standards with minimal changes to conform to
> the defacto Chinese standard, or to deviate from the Chinese
> standard and break what might already be implimented.

I would state your second sentence very differently, but "yes".
Note that the normalization question is an example here ... NFC
is almost irrelevant for Chinese characters because there are no
combining forms, but very important for scripts in which
combining characters are used and precomposed alternatives are
common.

     john

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 23 Jul 2009 11:16:51 +0200

Harald Alvestrand wrote:

>> So, I pretty much agree with your points, but you aren't giving me 
>> much hope of EAI being adaquate to those criteria in the next 
>> decade.  I still think some sort of schedule would be interesting to 
>> help us figure out how to proceed in addressing your concerns.  
>
> Would it be interetsing to (for at least 5 minutes) explore the idea 
> of a test-driven schedule, rather than a calendar-driven one?
>

I think this is a great idea. So yes, let's talk about it in Stockholm.

---

From: Shawn Steele <[email-masked]>
Date: Fri, 24 Jul 2009 15:48:51 +0000

> combining characters are used and precomposed alternatives are

Yes, but it's irrelevent on the client side.  Servers "should" be smart, but I wouldn't want to force NFC on a server that thinks its architecture is best served by NFD (for example).  I think it'd be great to strongly recommend that they consider normalization and the best mapping practices for their needs, but I don't think we need to force servers.

Turkish i is a good place where IDN style mappings may not be appropriate for local parts.  As it "only" impacts a particular server's users, and not interoperability, I'd rather not over-specify that part of the standard.

-Shawn

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 24 Jul 2009 10:59:37 +0200

John C Klensin writes:
>> situation, where people enter an address from a business card and 
>> get a "no such mailbox" response, without there being any visual 
>> difference between the two strings.
>>
>>  Requiring normalization at the server (without placing a requirement 
>>  on what one normalizes to) would at least remove this class of 
>>  issue.
>
> Exactly.

I wonder if that's a bit of a fallacy... anyone who expects humans to 
read and reenter 42-69 differently from 42‒69 has a problem. IMO that's 
a perfectly natural mistake, it involves a character often used in 
today's email addresses, and I've never heard anyone use this 
particular example before so I doubt it would be implemented quickly.

Alternative proposal: Extend VRFY to optionally return a fuzzily 
correctified version of an email address. If VRFY is disabled or 
doesn't return an address, correction isn't available for that address. 
I think this is better because it permits the fuzzy fuzziness to be 
invoked when the address is entered into e.g. an address database, 
instead of when a message is transmitted. As a bonus it's 99% 
independent of EAI.

Arnt

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 27 Jul 2009 16:00:04 +0200

Arnt Gulbrandsen wrote:
>>> requirement  on what one normalizes to) would at least remove this 
>>> class of  issue.
>>
>> Exactly.
>
> read and reenter 42-69 differently from 42‒69 has a problem. IMO 
> that's a perfectly natural mistake, it involves a character often used 
> in today's email addresses, and I've never heard anyone use this 
and in this case, NFC wouldn't help either....
> doesn't return an address, correction isn't available for that 
> address. I think this is better because it permits the fuzzy fuzziness 
> to be invoked when the address is entered into e.g. an address 
> database, instead of when a message is transmitted. As a bonus it's 
> 99% independent of EAI.
Fuzzy matching is a very interesting subject, and I'd argue that it's 
100% orthogonal to EAI - I still remember the mail I got as a new 
postmaster after a new fuzzy-match algorithm had rerouted a misspelling 
of one man's wife's mail to his mailbox - "how did the system know we 
are married?".

I'd like to punt this issue to the Single Global Directory, but that's 
just reducing it to a previously unsolved problem.....

                      Harald

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 27 Jul 2009 16:16:42 +0200

Sorry about my address. Problems I can't solve right now.

> and in this case, NFC wouldn't help either....

Exactly. Nothing is equivalent to the typos humans make.

> postmaster after a new fuzzy-match algorithm had rerouted a 
> misspelling of one man's wife's mail to his mailbox - "how did the 
> system know we are married?".
>

(One solved long ago by spammers, *sigh* ;)

I'm serious, though: EAI should suggest that before storing an address, 
the software might attempt to check its accuracy using RCPT or VRFY, 
and do _nothing_ more.

The heuristics fit in VRFY, and at that data entry time it's possible to 
ask "uh, was there a typo, did you mean <x>?" if the VRFY responds 
suitably.

Arnt

---

From: Joseph Yee <[email-masked]>
Date: Mon, 27 Jul 2009 11:28:08 -0400

IMO this set MUST normalize first, punctuation and number: dot,  
underscore, plus, minus, number, and any other punctuation characters  
allowed in email now. Some normalization may be solved by internal  
mapping, alias, etc.  I'm concerned that punctuation marks are not  
(especially characters with special meaning in email format).

It sounds like I'm spelling NFKC/D, I'm not.  I want to focus on  
punctuation mark normalization in regard to email address only.  It  
can be special rule to apply first before other normalization, no  
punctuation mark allowed beyond ASCII set, NF-something for the whole  
email address (if it addressed the punctuation mark concern).

Joseph

On 27-Jul-09, at 10:00 AM, Harald Alvestrand wrote:

>> I wonder if that's a bit of a fallacy... anyone who expects humans  
>> to read and reenter 42-69 differently from 42‒69 has a problem.  
>> IMO that's a perfectly natural mistake, it involves a character  
>> often used in today's email addresses, and I've never heard anyone  
>> use this particular example before so I doubt it would be  
>> implemented quickly.
>> address. I think this is better because it permits the fuzzy  
>> fuzziness to be invoked when the address is entered into e.g. an  
>> address database, instead of when a message is transmitted. As a  
>> bonus it's 99% independent of EAI.
> Fuzzy matching is a very interesting subject, and I'd argue that  
> it's 100% orthogonal to EAI - I still remember the mail I got as a  
> new postmaster after a new fuzzy-match algorithm had rerouted a  
> I'd like to punt this issue to the Single Global Directory, but  
> that's just reducing it to a previously unsolved problem.....
>
>                     Harald
>
>
>

---

From: Tony Finch <[email-masked]>
Date: Mon, 27 Jul 2009 17:04:23 +0100

On Fri, 24 Jul 2009, Arnt Gulbrandsen wrote:
> correctified version of an email address.

That's already part of the VRFY spec :-)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

---

From: John C Klensin <[email-masked]>
Date: Mon, 27 Jul 2009 13:11:31 -0400

--On Friday, July 24, 2009 10:59 +0200 Arnt Gulbrandsen
<[email-masked]> wrote:

> Alternative proposal: Extend VRFY to optionally return a
> fuzzily correctified version of an email address. If VRFY is
> disabled or doesn't return an address, correction isn't
> available for that address. I think this is better because it
> permits the fuzzy fuzziness to be invoked when the address is
> entered into e.g. an address database, instead of when a
> message is transmitted. As a bonus it's 99% independent of EAI.

First, as Tony points out, that has been part of VRFY forever
and, historically, systems used to fairly commonly generate real
addresses.  In particular, it was common in some systems for one
to be able to send
   VRFY John Klensin
and get an address back, and VRFY was designed to accommodate
that use when systems wanted to support it.

Second, what you are suggesting is a member of a family of
solutions we've discussed before.  Some of them would fall under
Harald's Global Universal Directory project, others don't, but
all are members of the set of "if downgrading of
forward-pointing addresses is needed, it needs to be done at the
originating MUA or Submission server and downgrading does not
need to be done in transit (at SMTP time).

    john