Subject: [Emu] Revised sections for Issue #18 (Internationalization)

From: "Joseph Salowey (jsalowey)" <[email-masked]>
Date: Tue, 22 Sep 2009 16:18:11 -0700

"4.3.6. Internationalization of Display Strings

The payload MAY provide a standard attribute format that supports
international strings.  This attribute format MUST support encoding
strings in UTF-8 [RFC3629] format.  Any strings sent by the server
intended for display to the user MUST be sent in UTF-8 format and
SHOULD be able to be marked with language information and adapted to
the user's language preference as indicated by RFC 4646 [RFC4646].  
Note that in some cases, such as when transmitting error codes, it 
is acceptable to exchange numeric codes that can be translated by the 
client to support the particular local language.  These numeric codes
are not subject internationalization during transmission. "

"4.5.2. Internationalization

   The password authentication exchange MUST support user names and
   passwords in international languages.  It MUST support encoding of
   user name and password strings in UTF-8 [RFC3629] format. The method 
   MUST specify how username and password normalization is performed in 
   reference to SASLPrep [RFC4013] or Net-UTF-8 [RFC5198].

   Any strings sent by the server during the password exchange and
intended
   for display to the user MUST be sent in UTF-8 format and SHOULD be
   able to be marked with language information and adapted to the user's
   language preference as indicated by RFC 4646 [RFC4646].  Note that in
some 
   cases, such as when transmitting error codes, it is acceptable to
exchange 
   numeric codes that can be translated by the client to support the 
   particular local language.  These numeric codes are not subject 
   internationalization during transmission. "

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 23 Sep 2009 10:48:54 +0200

>    reference to SASLPrep [RFC4013] or Net-UTF-8 [RFC5198].

There is some discussion about SASLprep on the [email-masked] mailing
list right now that you may want to review.

One of my experiences with SASLprep is that it makes sense to normalize
passwords (because they are input to hashes so some ambiguities needs to
be resolved) but it does not always make sense to normalize usernames.
Some characters that may occur in a username are converted to other
characters by SASLprep, a simple example is SASLprep('ª') = 'a.'  So
using SASLprep for usernames may prevent otherwise reasonable usernames
from being usable.

Another concern with SASLprep is that it is based on Unicode 3.2 and
does not support newer Unicode versions.

Perhaps you want to consider an approach where passwords are normalized
using Net-UTF-8 and usernames are sent as-is but mandate that
username-to-username comparisons use Net-UTF-8 on both strings before
comparing them.

Username and passwords strings are somewhat different beasts, and it
does not necessarily make sense to use the same normalization or
comparison algorithms for both strings.

/Simon

---

From: Alan DeKok <[email-masked]>
Date: Wed, 23 Sep 2009 15:28:34 +0200

> username-to-username comparisons use Net-UTF-8 on both strings before

  I would like to see some more details around this suggestion.  Issues
of data entry (user -> keyboard -> ?? -> EAP) and user store (admin ->
keyboard -> ??? -> database) are important.

  Something like:

The canonical form of the username and password as carried in EAP or
placed in an authentication store SHOULD be Net-UTF-8.

  In practice, this means that the supplicant converts both to Net-UTF-8
prior to placing them in an EAP packet.  It means that an authentication
system converts them to Net-UTF-8 prior to placing them in a data store.
 It means that the data store treats them as an opaque sequence of
bytes, and does *not* mangle them in any way.  It means that any
cryptographic calculations based on the passwords must be done on the
Net-UTF-8 form.

  i.e. whatever a supplicant places in the packet is considered by the
authenticator to be the "canonical" form of the username and password.

  Having the authenticator canonicalize the username or password that is
takes from an EAP packet is a recipe for disaster.

> comparison algorithms for both strings.

  Why not?  memcmp(s1, s2, length) would be fine if we assume that the
strings are both canonicalized *before* the comparison is done.  Stating
*who* does the canonicalization ensures that only a limited number of
entities do the canonicalization, which minimizes errors and
inter-operability.

  The nice thing about that requirement is that it is very compatible
with sites choosing to not use Net-UTF-8.  If they choose to put
something else in a DB as the authentication credentials, it will work
so long as the supplicant puts exactly the same data into the EAP packet.

  Alan DeKok.

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 23 Sep 2009 16:17:46 +0200

> keyboard -> ??? -> database) are important.

Yes, more details are definitely needed.

> placed in an authentication store SHOULD be Net-UTF-8.

Should we suggest that passwords are sent over the wire at all?  In good
systems that should be the exception.  I think there are three different
kind of strings that needs to be discussed.

1) Usernames.  Should be sent over the wire as-is, or possibly
Net-UTF-8.  It is not clear to me that the even the weak NFC part of
Net-UTF-8 is always a good idea for usernames, you could just send the
username as-is and require that COMPARISONS are done in a way that
reduces ambitious.  There are some indication that the algorithm of
comparing two strings for equality is simpler to make robust and
ambigious-free than an algorithm that normalize strings.

2) Passwords sent to servers.  This has similar properties as 1) but
there could be differences in how they are treated.

3) Passwords sent to servers only in hashed form.  This has stricter
normalization requirements than 1) and 2) -- NFC is most certainly
needed, and probably something more, similar to SASLprep but preferably
without some of the problems that SASLprep has.

> entities do the canonicalization, which minimizes errors and

The problem with normalization early is that you end up with a interop
problem where both the client and server needs to implement the exact
same normalization algorithms for things to be robust.  If normalization
only happens in one implementation, things will be predictable.

Of course, it is not possible to avoid this problem for hashed password
strings.  They need to be normalized before hashing happens, on all
systems.

> so long as the supplicant puts exactly the same data into the EAP packet.

Right, it is a tradeoff between different neat properties.  I'm not sure
we have a good understanding of all the factors that are important yet.

/Simon

---

From: Alan DeKok <[email-masked]>
Date: Thu, 24 Sep 2009 15:56:26 +0200

Simon Josefsson wrote:
> systems that should be the exception.

  It is widely deployed today with TTLS.  I think that allowing this
practice to continue is a requirement.

> reduces ambitious.

  Ouch.  I fundamentally disagree.  See RFC 4282, and my attempt to
update it, based on real-world deployment experience:

http://tools.ietf.org/html/draft-dekok-radext-nai-00

  My document explains why it's difficult (i.e. impossible) for third
parties to "normalize" the data.

>  There are some indication that the algorithm of

  Yes.  The systems which do end-user input likely have enough
information to normalize the data.  No one else does.  As a result,
everyone else should compare the two strings for equality.  They should
not normalize the strings.

> without some of the problems that SASLprep has.

  I beleive that (2) and (3) are the same case.  For (3), we will end up
checking for:

	hash(p1) == hash(p2)

  For strong hashes, the simplest way to ensure this is true is if:

	p1 == p2

> only happens in one implementation, things will be predictable.

  RFC 5198 specifies a normalized form.  If two implementations
"normalize" to the RFC 5198 form differently, one of them is wrong.

  Alan DeKok.

---

From: Simon Josefsson <[email-masked]>
Date: Thu, 24 Sep 2009 16:14:36 +0200

> practice to continue is a requirement.

I agree, but that does not necessarily mean that
passwords-sent-over-the-wire and passwords-sent-hashed must have the
same internationalization treatment or considerations.

> parties to "normalize" the data.

Interesting!  It is good to see work in this area, and your reasons
appears good to me.  I believe it is desirable to reduce the amount of
string normalization that happens, and instead use comparison-rules when
possible, but sometimes normalization is required for one reason or the
other.  As soon as normalization rules are suggested, I believe it is
worthwhile to evaluate why comparison-rules are not sufficient, and your
document address that.

> "normalize" to the RFC 5198 form differently, one of them is wrong.

Sure, but see section 4 of RFC 5198.  If it happens that NFC backwards
compatibility is broken, you end up with the interop problem.  This has
happened before for NFKC which has caused (theoretical) interop problems
for IDNA and SASLprep -- if two implementations of RFC 3454 use
different Unicode versions, they don't necessarily interop.

/Simon

---

From: Alan DeKok <[email-masked]>
Date: Thu, 24 Sep 2009 17:22:13 +0200

> same internationalization treatment or considerations.

  That isn't a requirement, but it does make things easier.

> compatibility is broken, you end up with the interop problem.

  ? Section 4 says explicitly:

   if a string does not contain any unassigned
   characters, and it is normalized according to NFC, it will always be
   normalized according to all future versions of the Unicode Standard.

  So there is no backwards compatibility problem.

  My interpretation is that every system taking user input needs to
perform this normalization.  Once that's done, string comparison is
essentially memcmp().

  If both user && authenticator are using the same version of Unicode,
then they inter-operate.

  If one is using a newer version, then it either

	(a) creates the same output string, because the *old* characters
            have not been re-assigned in the *new* standard

 	(b) creates a different output string than the old system,
	    because either the standard is not backwards compatible,
	    or the implementation is wrong.

  Is there anything I'm missing?

  Alan DeKok.

---

From: Simon Josefsson <[email-masked]>
Date: Thu, 24 Sep 2009 17:31:17 +0200

>   So there is no backwards compatibility problem.

See this part:

   Were Unicode to be changed in a way that violated these assumptions,
   i.e., that either invalidated the byte string order specified in RFC
   3629 or that changed the stability of NFC as stated above, this
   specification would not apply.

If that happens, there is a backwards compatibility problem with old
implementations.

> perform this normalization.  Once that's done, string comparison is

Right.  My point is that the one needs to weight this approach to a
system which does not use normalization but instead use
internationalized comparison rules.

>   Is there anything I'm missing?

Yes: there is typically no version negotiation of SASLprep or RFC 5198
version used in protocols.  So a client would typically not know what
Unicode version the server uses.  This would be one reason to put the
Unicode version depended code into one place (where the comparison
happens) rather than in several places (normalization in the client and
in the server).

Which approach to use needs to be decided understanding these
trade-offs.  I'm not saying that the comparison-approach is always
better, but my experience with normalization in client/server-systems is
that it does not work well enough that we should consider if other
approaches may work better.

/Simon

---

From: Alan DeKok <[email-masked]>
Date: Fri, 25 Sep 2009 11:59:03 +0200

> internationalized comparison rules.

  How do you do internationalized comparisons on hashed passwords?

  All you have is the hash.  And if the passwords input to the hash
aren't the same (i.e. non-normalized), then you're *guaranteed* that the
hashes won't match.

  Alan DeKok.

---

From: Simon Josefsson <[email-masked]>
Date: Fri, 25 Sep 2009 12:55:32 +0200

> aren't the same (i.e. non-normalized), then you're *guaranteed* that the

Right.  Hashed passwords is one example of when internationalized
comparisons wouldn't work.  I'm sorry if this wasn't clear in my earlier
note.

However there is a risk that normalization _introduce_ differences: if
two systems use different normalization algorithms that leads to
different outputs for the same input, the hashes won't match either.

/Simon

---

From: "Joseph Salowey (jsalowey)" <[email-masked]>
Date: Mon, 28 Sep 2009 11:00:23 -0700

I'd like to see if we can close on this issue soon.  The main use case
we are targeting is one where the password is sent to the server.  We do
not know how the server will do the comparison.   Given that this is a
requirement document I don't think we need to have the full solution
described.  Can you two work out some proposed text to go in the
requirements document on this issue with usernames and passwords?  

Thanks,

Joe 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Simon Josefsson
> Sent: Friday, September 25, 2009 3:56 AM
> To: Alan DeKok
> Cc: [email-masked]
> Subject: Re: [Emu] Revised sections for Issue #18 
> (Internationalization)
> 
> > aren't the same (i.e. non-normalized), then you're 
> *guaranteed* that 
> > the hashes won't match.
> 
> Right.  Hashed passwords is one example of when 
> internationalized comparisons wouldn't work.  I'm sorry if 
> this wasn't clear in my earlier note.
> 
> However there is a risk that normalization _introduce_ 
> differences: if two systems use different normalization 
> algorithms that leads to different outputs for the same 
> input, the hashes won't match either.
> 
> /Simon
> _______________________________________________
> Emu mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/emu
>

---

From: Simon Josefsson <[email-masked]>
Date: Tue, 29 Sep 2009 09:54:03 +0200

> requirements document on this issue with usernames and passwords?  

I think the text proposed is mostly fine.  I would say "normalization
and/or comparison" instead of just "normalization" to allow for
mechanisms that just specify comparison-based rules rather than
normalization-based rules.  So:

OLD:

NEW:
   The password authentication exchange MUST support user names and
   passwords in international languages.  It MUST support encoding of
   user name and password strings in UTF-8 [RFC3629] format. The method
   MUST specify how username and password normalizations and/or
   comparisons is performed in reference to SASLPrep [RFC4013] or
   Net-UTF-8 [RFC5198].

/Simon

---

From: Alan DeKok <[email-masked]>
Date: Tue, 29 Sep 2009 15:21:35 +0200

> normalization-based rules.  So:

  The text looks OK to me.

  My only concern is that standardizing i18n in *each* possible EAP
document is duplicating work.  If we end up with only one tunnel method
that is standardized, then there's less of a problem.

  Alan DeKok.

---

From: Simon Josefsson <[email-masked]>
Date: Tue, 29 Sep 2009 16:09:02 +0200

> that is standardized, then there's less of a problem.

Standardizing multiple tunnel-based methods seems like a bad idea for
many reasons.

I agree that standardizing i18n in every document will lead to
duplication of work (and duplication of discussions and the potential
for mistakes), but placing too specific requirements on all future EAP
documents today is similar to placing too specific crypto requirements
on all future EAP documents today.  The algorithms are evolving and
there needs to be some element of agility in the requirements.  If there
is, any mistakes in this area can be fixed later on.

/Simon