Subject: [xmpp] i18n summary

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 21 Dec 2010 08:55:58 -0700

Yesterday I had separate chats about internationalization with both
Alexey Melnikov and Joe Hildebrand. In this message I attempt to
summarize the issues we face.

1. Multiple Stringprep Profiles

In XMPP, we have multiple stringprep profiles: Nameprep (inherited from
IDNA2003), Nodeprep, and Resourceprep. In this regard I think that XMPP
is unique, because as far as I can see all other stringprep customers
use only one profile (e.g., Nameprep or SASLprep or LDAPprep or a custom
profile, but never more than one of those) -- see for instance the
issues logged in the PRECIS WG:

http://trac.tools.ietf.org/wg/precis/trac/report/6

2. Migration

The fact that we have multiple stringprep profiles means that our
migration from stringprep-based technologies to non-stringprep-based
technologies will involve at least two aspects: migration of XMPP
domainparts from IDNA2003 to IDNA2008 (i.e., for server FQDNs) and
migration of XMPP localparts from Nodeprep to something new (i.e., for
registered account names, MUC room names, etc.). I don't think we'll
need to migrate resourceparts because they are fairly ephemeral, but
there might be migration issues there, too.

This implies that *if* we were to legislate an upgrade from IDNA2003 to
IDNA2008 for domainparts before we have a way to upgrade from Nodeprep
to something new for localparts, then we would need to complete two
migrations. IMHO that would be highly sub-optimal. Migration will be a
lot of work (the scope is yet to be determined, but I think it will be
fairly large, requiring updates to client and server software, creation
of migration tools, scrubbing of data at XMPP service providers, etc.).
Migrating twice would place a significant burden on the XMPP community.

3. Preparation and Validation

One aspect of the move from IDNA2003 to IDNA2008 was a change in the
"dividing line" between protocol and user interface (see RFC 5895). This
places more responsibility in the client. In fact, we seem to be coming
to the conclusion that only the client knows enough about the user's
intentions, Unicode version, input methods, locale, and other context to
interpret and prepare strings for sending over the network in the
various "JID slots" that I mentioned in a previous message ('to' and
'from' addresses, roster items, MUC room names, etc.). The server might
be able to validate some of that input (e.g., because certain Unicode
codepoints are disallowed), but it doesn't know enough about the user's
context to prepare the JIDs in every JID slot (indeed, it doesn't even
have access to many JID slots unless it performs some kind of "deep
stanza inspection"). This is a major change in the i18n model for XMPP.
We haven't yet had a chance to think through all the implications of
that change, nor to define the "dividing line" as it applies to XMPP
clients and servers (IDNA2008 talks about entities like DNS resolvers
and domain name registrars, not application clients and servers).

4. Desire for Change

Alexey seemed to be questioning whether the XMPP community wants to
upgrade from IDNA2003 to IDNA2008. I don't think there is a lack of
will. To the extent that folks in the XMPP community understand the
issues (and I don't think they understand i18n all that well, although
to be fair that's true of most application technologies), they want to
do the right thing and are not bitterly clinging to their old stringprep
profiles. Unicode agility is a good thing, and everyone who thinks about
the topic wants to stay current instead of being tied to Unicode as it
was defined back in 2002 (i.e., version 3.2). As far as I can see, there
is no desire in the XMPP community to ignore IDNA2008. We want to move
forward to a post-stringprep world. The big question is how to make that
happen, not "if".

5. Setting a Precedent

The IETF community put a lot of work into IDNA2008. If one of the big
customers of IDNA technology were to ignore IDNA2008, that would set a
bad precedent and encourage other customers to ignore it, too. But it's
clear to me that the XMPP community doesn't want to ignore IDNA2008.
Instead, it simply wants to migrate all of its i18n technologies
simultaneously, with a full understanding of the implications, and in a
way that makes sure we get it right this time. There's no precedent
being set here, because the XMPP community is unique among stringprep
customers.

6. The Address Spec

If we need to add more text to draft-ietf-xmpp-address explaining the
situation I've summarized in this email, I would be happy to do that.
However, given the vast improvement in quality between RFCs 3920/3921
and 3920bis/3921bis, I would not be happy to have the address spec stuck
in limbo, thus preventing publication of revised specs for the core
until 2012 or 2013.

7. Making the Transition

As mentioned, the big question is how to make a successful transition to
a post-stringprep world. Some members of the XMPP community are actively
contributing to the PRECIS WG. I'm working on a tutorial for presenting
at the XMPP Summit in February so that more XMPP developers can
strengthen their understanding of i18n issues. We need to continue
discussions in the XMPP WG about the topics that have emerged in recent
mailing list threads. I encourage the group, and the chairs, to make it
a priority for us to solve these problems in 2011 -- and I will commit
to putting significant energy into these efforts.

In short, we have a lot of work to do, so let's get busy. :)

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Kurt Zeilenga <[email-masked]>
Date: Tue, 21 Dec 2010 08:22:49 -0800

> issues logged in the PRECIS WG:

Aside from LDAPprep actually being a family of profiles referred to by a single name, LDAP also uses SASLprep (in handling of passwords) and likely Nameprep (in handling of certificates) (though this will go away in favor of the latest IDNA).  One could even imagine a JID holding directory attribute being introduced that required use of Nodeprep and Resourceprep.

I think multiple string preparation algorithms per protocol is the rule (especially when one considered normative references), not the exception.

-- Kurt

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 21 Dec 2010 09:30:40 -0700

On 12/21/10 9:22 AM, Kurt Zeilenga wrote:
>> In XMPP, we have multiple stringprep profiles: Nameprep (inherited
>> from IDNA2003), Nodeprep, and Resourceprep. In this regard I think
>> that XMPP is unique, because as far as I can see all other
>> stringprep customers use only one profile (e.g., Nameprep or
>> SASLprep or LDAPprep or a custom profile, but never more than one
>> of those) -- see for instance the issues logged in the PRECIS WG:
> 
> Aside from LDAPprep actually being a family of profiles referred to
> by a single name, LDAP also uses SASLprep (in handling of passwords)
> and likely Nameprep (in handling of certificates) (though this will
> go away in favor of the latest IDNA).  One could even imagine a JID
> holding directory attribute being introduced that required use of
> Nodeprep and Resourceprep.

Great, I'm looking forward to contributions from the LDAP community in
the PRECIS WG so we can discuss these issues at greater length there.

> I think multiple string preparation algorithms per protocol is the
> rule (especially when one considered normative references), not the
> exception.

The issue tracker page I pointed to in my previous message does not
support your assertion. However, the PRECIS WG has yet to complete its
reviews of the referenced documents, so I might be missing something.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Dave Cridland <[email-masked]>
Date: Tue, 21 Dec 2010 16:39:09 +0000

> strengthen their understanding of i18n issues. We need to continue

Can we make the XMPP Summit (or at least a day of it) formally a  
joint IETF XMPP WG Interim as well?

 From the XSF side, I think the only question mark would be the Note  
Well, but I doubt that anyone would object to that.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 21 Dec 2010 10:10:55 -0700

On 12/21/10 9:39 AM, Dave Cridland wrote:
> Can we make the XMPP Summit (or at least a day of it) formally a joint
> IETF XMPP WG Interim as well?

Interim meetings need to be announced 4 weeks in advance:

http://www.ietf.org/iesg/statement/interim-meetings.html

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 21 Dec 2010 10:36:31 -0700

> http://www.ietf.org/iesg/statement/interim-meetings.html

In addition, there would be joint scheduling issues, questions about
whether things the XSF cares about deserve to be on the agenda for a
joint meeting, etc. I'm not saying it can't be done, but work would be
required to organize it correctly, and I'm not sure how much work will
be completed before the end of the year (e.g., I have only ~36 hours of
work time left before taking a break).

FWIW, some proposed topics for the Summit were listed here:

http://mail.jabber.org/pipermail/summit/2010-December/000602.html

Just my gram of silver... :)

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Dave Cridland <[email-masked]>
Date: Tue, 21 Dec 2010 20:54:07 +0000

> work time left before taking a break).
Sure, but an agenda only needs publication after the announcement, so  
I'd imagine the IESG would cut us a bit of slack given that we're  
making this a joint thing.

Besides, the only thing we really need face-to-face time for is i18n,  
and the community input will be invaluable, plus we'll have pretty  
much every major server implementor present in the room and most of  
the clients, too.

As for the relative jointness of it, you could simply request an  
afternoon from the XSF for a formally distinct IETF Interim if that  
proves to be a sticking point.

So while it's not entirely non-tight, schedule-wise, I think it's  
very much achievable and advantageous to do so, even if our agenda is  
limited to just "i18n".

Is anyone objecting to this? I'll drop a note to the XSF folk as  
well, if there's nobody screaming soon. (Or you can, I hear you may  
have a contact or two with the XSF as well).

Dave.

---

From: Ben Campbell <[email-masked]>
Date: Wed, 22 Dec 2010 17:16:36 -0600

(as half of the chairs)

An interim meeting would be useful to hash out an issue that needs high-bandwidth communication to resolve _inside_ the working group. Now, maybe I'm misreading this, but I don't see a lot of controversy on this topic _inside_ the working group. Of course that could change if the IESG insists on a move to IDNA2008 prior to publication of 3920bis, but we're not quite to that bridge yet.

Do people think there is such a controversy?

Given the holiday season, we may find it difficult to get all the ducks in a row in time to make the 4 week announcement deadline, and i actually don't expect the IESG to offer much leeway on that. That doesn't mean we can't give it a try if we really think we need to. OTOH, we can still have conference calls or xmpp conferences later if we think we need to.

On Dec 21, 2010, at 2:54 PM, Dave Cridland wrote:

> Sure, but an agenda only needs publication after the announcement, so I'd imagine the IESG would cut us a bit of slack given that we're making this a joint thing.
> 
> Besides, the only thing we really need face-to-face time for is i18n, and the community input will be invaluable, plus we'll have pretty much every major server implementor present in the room and most of the clients, too.
> 
> As for the relative jointness of it, you could simply request an afternoon from the XSF for a formally distinct IETF Interim if that proves to be a sticking point.
> 
> So while it's not entirely non-tight, schedule-wise, I think it's very much achievable and advantageous to do so, even if our agenda is limited to just "i18n".
> 
> Is anyone objecting to this? I'll drop a note to the XSF folk as well, if there's nobody screaming soon. (Or you can, I hear you may have a contact or two with the XSF as well).
> 
> Dave.
> -- 
> _______________________________________________
> xmpp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xmpp

---

From: Ben Campbell <[email-masked]>
Date: Wed, 22 Dec 2010 17:19:53 -0600

(As individual)

My understanding of the original reason for splitting out xmpp-address was exactly to allow us to close on the rest of 3920bis and then take the time to handle the IDNA2008 migration right. Hopefully no one takes that to indicate that the group does not care about this. But would it help if we updated the charter milestone list to include a "move xmpp-address to IDNA2008" work item, at a fairly high priority?

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 22 Dec 2010 16:45:51 -0700

On 12/22/10 4:19 PM, Ben Campbell wrote:
> (As individual)
> 
> My understanding of the original reason for splitting out
> xmpp-address was exactly to allow us to close on the rest of 3920bis
> and then take the time to handle the IDNA2008 migration right.
> Hopefully no one takes that to indicate that the group does not care
> about this. But would it help if we updated the charter milestone
> list to include a "move xmpp-address to IDNA2008" work item, at a
> fairly high priority?

I think it would help. Even better, I think it would accurately describe
reality. :)

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 22 Dec 2010 16:50:31 -0700

On 12/22/10 4:16 PM, Ben Campbell wrote:
> (as half of the chairs)
> 
> An interim meeting would be useful to hash out an issue that needs
> high-bandwidth communication to resolve _inside_ the working group.
> Now, maybe I'm misreading this, but I don't see a lot of controversy
> on this topic _inside_ the working group. Of course that could change
> if the IESG insists on a move to IDNA2008 prior to publication of
> 3920bis, but we're not quite to that bridge yet.
> 

I think there's plenty of controversy. The devil is very much in the
details here, and folks in the WG don't really understand the details.
We need to both educate and deliberate. This internationalization stuff
can make your head hurt in a hurry and it requires a high-bandwidth
connection to come to reasonable conclusions. A perfect use for an
interim meeting, I think.

> Given the holiday season, we may find it difficult to get all the
> ducks in a row in time to make the 4 week announcement deadline, and
> i actually don't expect the IESG to offer much leeway on that. That
> doesn't mean we can't give it a try if we really think we need to.
> OTOH, we can still have conference calls or xmpp conferences later if
> we think we need to.

I leave that up to the chairs. The interim meeting, if held, would
likely occur on Monday, February 7. That would give folks a few days in
the first week of January to work out the details. With intensive focus,
it just might happen. And remember that only the announcement needs to
be sent, not the agenda.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Kevin Smith <[email-masked]>
Date: Thu, 23 Dec 2010 08:45:52 +0000

> it just might happen. And remember that only the announcement needs to

This seems to be largely an irrelevancy to me (although I could be
missing something) - XMPP people will be meeting and discussing this
regardless of if it has the traditional XSF stamp on it, or gets a
shiny new IETF stamp, won't they?

/K

---

From: Dave Cridland <[email-masked]>
Date: Thu, 23 Dec 2010 09:00:15 +0000

On Wed Dec 22 23:16:36 2010, Ben Campbell wrote:
> Now, maybe I'm misreading this, but I don't see a lot of  
> controversy on this topic _inside_ the working group. Of course  
> that could change if the IESG insists on a move to IDNA2008 prior  
> to publication of 3920bis, but we're not quite to that bridge yet.
> 
You know, I can honestly say I don't understand the problem enough to  
answer that. I have no clear understanding of the problem, although  
thanks to chats with Joe and Peter, I'm beginning to get terrified  
over the whole thing.

> ducks in a row in time to make the 4 week announcement deadline,  
> and i actually don't expect the IESG to offer much leeway on that.  
> That doesn't mean we can't give it a try if we really think we need  
> to. OTOH, we can still have conference calls or xmpp conferences  
> later if we think we need to.
> 
> 
There is a gethering of the XMPP community happening already. Really,  
I'm saying we should take advantage of this existing gathering.

In practical terms, we have a venue, a date, and a single agenda  
item. What more do we need to make an announcement?

Dave.

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 23 Dec 2010 11:48:39 +0000

Peter Saint-Andre wrote:

>I think there's plenty of controversy.
>
+1.

>The devil is very much in the
I agree. Even showing people examples of problems would be very useful.

>it just might happen. And remember that only the announcement needs to
I can ask IESG about the interim. That might be easier to arrange that 
people think.

---

From: Bernard Aboba <[email-masked]>
Date: Thu, 23 Dec 2010 08:02:42 -0800

On Thu, Dec 23, 2010 at 6:52 AM, Bernard Aboba <[email-masked]>wrote:

>
>
> On Tue, Dec 21, 2010 at 7:55 AM, Peter Saint-Andre <[email-masked]>wrote:
>
>>
>> 2. Migration
>>
>>
>> migrations. IMHO that would be highly sub-optimal.
>>
>
>
> [BA] From what I can tell, on a number of platforms "upgrading" from IDNA
> 2003
> to 2008 will occur by upgrading libraries.  Therefore, at least with
> respect to the
> domainpart, a client calling upgraded libraries will not necessarily know
> the
> difference.
>
>
>> "dividing line" between protocol and user interface (see RFC 5895).
>
>
> [BA] While some of the wording has changed, I don't believe that this
> requires
> a change in client application architecture (at least with respect to
> handling of the
> domainpart).  As noted above, on some platforms it probably will not
> require changes
> in client code.
>
> On the server side, substantial changes are required within "domain name
> registrars"
> (e.g. DNS servers and in some architectures closely allied components such
> as LDAP
> servers).  However, it's not clear to me that XMPP servers need necessarily
> worry about
> those changes, in terms of their handling of the domainpart.
>
>
>> 4. Desire for Change
>>
>> upgrade from IDNA2003 to IDNA2008... there is no desire to ignore IDNA
>> 2008.
>
>
> [BA] To the extent that the upgrade from IDNA 2003 to IDNA 2008 occurs via
> library upgrades that don't affect applications, the domainpart upgrade
> could
> turn out to be relatively painless.  So ignoring the IDNA 2003/2008
> transition
> (for domainparts) could turn out to be quite a viable transition strategy.
>
>
>
>> To the extent that folks in the XMPP community understand the
>> to be fair that's true of most application technologies)
>
>
>
> [BA] Overall, there is a high degree of variance in the quality of i18n
> implementations
> within applications.  This is because correct handling of i18n is quite
> difficult at present.
>
>> bad precedent and encourage other customers to ignore it, too.
>
>
> [BA] As noted earlier, ignorance on the part of application developers
> isn't necessarily
> a bad strategy for the domainpart, assuming that platform and library
> developers do
> their job to make the migration as painless as possible.  Time will tell if
> such a strategy
> is feasible for localparts. However, at least at the developer level, I
> think we've learned
> that any migration strategy which requires large scale changes within
> applications will
> face resistance.
>
>
>> 6. The Address Spec
>>
> [BA] Yes, that would be unfortunate.
>
>
>>
>> I'm working on a tutorial for presenting
>> strengthen their understanding of i18n issues.
>>
>
> [BA] Judging by the variance in i18n implementations within XMPP clients,
> that
> could be quite helpful.
>

---

From: Joe Hildebrand <[email-masked]>
Date: Thu, 23 Dec 2010 10:55:25 -0700

On 12/23/10 9:02 AM, "Bernard Aboba" <[email-masked]> wrote:

> 2003 to 2008 will occur by upgrading libraries.  Therefore, at least with
> respect to the domainpart, a client calling upgraded libraries will not
> necessarily know the difference.

Could you say more about how the upgrade might happen with a library change?
Nameprep doesn't exist in IDNA2008, so I'm not sure what good a library
update would do.

> registrars" (e.g. DNS servers and in some architectures closely allied
> components such as LDAP servers).  However, it's not clear to me that XMPP
> servers need necessarily worry about those changes, in terms of their
> handling of the domainpart.

I'm most worried about roster entries from domains whose representation
changes under the new rules.  They *might* not be a problem if the
nameprep'd domainpart is always valid IDNA2008, and if you've stored the
nameprep'd version instead of what the original user entered, but further
analysis is likely required.

-- 
Joe Hildebrand

---

From: Bernard Aboba <[email-masked]>
Date: Thu, 23 Dec 2010 12:53:53 -0800

> Could you say more about how the upgrade might happen with a library
> change? Nameprep doesn't exist in IDNA2008, so I'm not sure what good a
> library update would do.
>

[BA] It is quite possible to design an XMPP client that doesn't call
Nameprep directly for processing of the domainpart, only functions such as
ToASCII() or ToUnicode() (or IdnToAscii() and IdnToUnicode() on Windows).
Were these functions to be upgraded to support IDNA 2008, the application
would automatically become IDNA 2008 compliant with respect to handling of
the domainpart.

For API details, see:
http://msdn.microsoft.com/en-us/library/dd318142%28VS.85%29.aspx
http://www.gnu.org/software/libidn/

> analysis is likely required.

[BA] It really shouldn't matter if the nameprep'd domainpart is valid
IDNA2008 or not, as long as the client or server don't attempt to "enforce"
this.  That's one reason why "enforcement" is best handled at the domain
registrar.