Subject: Re: Use of NFKC in SASLprep

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Fri, 04 Jun 2004 08:54:54 -0700

At 04:19 PM 4/22/2004, Simon Josefsson wrote:
>FYI, as SASLprep is using NFKC per Unicode 3.2, you might be
>interested in the NFKC stability problem discussed on
><http://www.unicode.org/review/pr-29.html>. The page mention a
>strategy to maintain stability across NFKC revisions, which might be
>relevant for SASLPrep.  I'd be interested to hear people's opinions on
>the problem.

I recently consulted with authors of [Stringprep] (namely Paul
Hoffman) as well as our liaison to the UTC (Patrik Fältström)
regarding this [SASLprep] concern.  They recommend that the
SASLprep I-D not attempt to fix the issue discussed in PR#29.
They argue (as I understand their arguments) that it would be
inappropriate for SASLprep, a profile of Stringprep, to apply
PR#29 to a version of Unicode instead of utilize Unicode
normalization as required by Stringprep.  Or, to say it in
another way, PR#29 issues in a consistent manner for all profiles
of Stringprep and hence ought to be handled as part of the
ongoing revision of Stringprep (and its normative references).
They suggest we not wait for this revision.

I fully support with their recommendation (which is consistent
with my prior recommendations to the WG).

We also had a discussion as to wether the SASLprep I-D should
note the existence of this issue (without any fix) and its
impact upon implementations (and use) of SASLprep.  They argued
that such would be inappropriate for SASLprep I-D to include
such as there no issue specific to the SASLprep profile of
Stringprep here.  They noted it **might** be appropriate to
publish an Informational RFC updating [Stringprep].

While I have some mild reservations, I can support this
recommendation.

Please consider these recommendations.  Sam (as chair) and I (as
editor) intend to response to the IESG comment regarding
concern within the next week. 

Kurt

---

From: Simon Josefsson <[email-masked]>
Date: Sat, 05 Jun 2004 14:19:48 +0200

"Kurt D. Zeilenga" <[email-masked]> writes:

> Stringprep here.

So, what do we have here?  There is a known specification ambiguity in
NFKC that, for example, appear to allow implementations to return
three distinct output from the input string U+0B47 U+0300 U+0B3E
U+0323 (see earlier discussion).  This is an interoperability problem.
It appears to make SASLPrep2 more complicated to specify (since
SASLprep is not idempotent), with consequences for SASL user database
migration (if prepared strings are stored in the database, which I
have suggested shouldn't be the case in another thread).  It is
difficult to see that this property is impossible to abuse when
attacking systems that use SASLprep on authentication/authorization
tokens.  And the conclusion?  Let's ignore the issue, and make sure it
is not even mentioned.

Speaking as an implementor, I hesitate to expose my users to this
issue.  The conservative approach appear to be to enforce, in the
implementation, a local policy of rejecting the ambiguous input
strings directly.  This remove the need to worry about this issue,
without any obvious negative consequences (correct me if I'm wrong).

Applying the same conservative approach to the specification seem
possible.  The drawback of doing so?  We have to describe how
implementions should detect and reject the ambiguous combination of
code points.  The advantage?  Implementors aren't exposed to
interoperability and security issues.  Further, migrating to SASLprep2
may be simpler since it is more likely to be able to assume SASLprep
is idempotent.

Below is a proposal for this section, and I hope it can be considered.
If the text, or something similar to it, is not adopted, I feel that
at least acknowledging the issue would be appropriate.  The ambiguity
in NFKC doesn't disappear even if you refuse to acknowledge it.

What do other implementors think?  Have you compared the NFKC
implementation you use with the test vectors in PR-29?

Thanks,
Simon

   2.3.1 Prohibited Output Of Problem Sequences

   The algorithm used by [StringPrep] to normalize sequences have been
   discovered [PR-29] to be non-idempotent.  Fortunately, this appear
   to only affect a restricted set of combination of code points, and
   none of those combinations are believed to occur naturally (due to
   different linguistic uses of the characters).  Idempotency of the
   SASLprep operation is important for interoperability, security, and
   migration to revised versions of SASLprep.  Thus, this section
   prohibit the problem sequences from occurring in SASLprep output.
   The logic to detect and prohibit the problem sequences uses the
   tables from [PR-29].

   The problem sequences begin with a code point from one row in
   column 1 of table 1, followed by one or more of the 163 code points
   with a non-zero Canonical Combination Class (see Table 2), followed
   by one of the code points in the same row, in column 2 of Table,
   corresponding to the row of the first code point.

   [Table 1 is table 2 from <http://www.unicode.org/review/pr-29.html>.]

   [Table 2 is the non-zero entries of
   <http://www.unicode.org/Public/3.2-Update/extracted/DerivedCombiningClass-3.2.0.txt>]

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Sat, 05 Jun 2004 20:39:46 -0400

> tokens.  And the conclusion?  Let's ignore the issue, and make sure it

It strikes me that people are blowing this way out of proportion, and I 
really dislike the repeated attempts in this working group to "fix" 
Unicode, which is _way way way_ out of scope for us.

There is not a problem with NFKC.  There is a TYPO in the document which 
specifies it.  If NFKC is implemented correctly, as is done by multiple 
implementations _including the sample implementation in the same document_, 
there is no problem.

This is not the end of the world.  We've certainly had similar problems in 
IETF documents; the RFC Editor has a long list of errata for published 
RFC's.  I cannot imagine that the Unicode Consortium will not publish a 
corrigendum to correct the erroneous document.

In the meantime, because SASLprep will be used in security-sensitive 
applications including SASL and Kerberos, I think it is appropriate to 
include a reference to PR-29 and a recommendation that implementors insure 
that NFKC is implemented correctly.

I do _not_ think it is appropriate to suggest behaviour _other_ than simply 
implementing NFKC correctly.  For example, I don't think it's appropriate 
to specify that a large class of sequences be prohibited from output.  Any 
implementation which would be in a position to implement such a suggestion 
is _also_ in a position to get NFKC right in the first place, thereby 
avoiding the entire problem.

-- Jeffrey T. Hutzelman (N3NHS) <[email-masked]>
   Sr. Research Systems Programmer
   School of Computer Science - Research Computing Facility
   Carnegie Mellon University - Pittsburgh, PA

---

From: Simon Josefsson <[email-masked]>
Date: Sun, 06 Jun 2004 12:20:03 +0200

Jeffrey Hutzelman <[email-masked]> writes:

> There is not a problem with NFKC.  There is a TYPO in the document
> which specifies it.  If NFKC is implemented correctly, as is done by
> multiple implementations _including the sample implementation in the
> same document_, there is no problem.

I spoke to other implementors of StringPrep, which do implement NFKC
the same way as the reference implementation (i.e., not according to
the letter of the specification), and one of the Stringprep authors
had told them that then they were simply not following the current
specifications.  I think a public statement committing to one of the
two interpretation (i.e., either consider it a typo in the old version
of NFKC, or that the typo corrected NFKC will be a new version, which
StringPrep doesn't use yet) of this problem would be useful.

> include a reference to PR-29 and a recommendation that implementors
> insure that NFKC is implemented correctly.

We agree about this.

Thanks,
Simon

---

From: Sam Hartman <[email-masked]>
Date: Mon, 07 Jun 2004 17:45:23 -0400

>>>>> "Simon" == Simon Josefsson <[email-masked]> writes:

    Simon> appears to make SASLPrep2 more complicated to specify
    Simon> (since SASLprep is not idempotent), with consequences for
    Simon> SASL user database migration (if prepared strings are
    Simon> stored in the database, which I have suggested shouldn't be
    Simon> the case in another thread).  

As someone who tends to believe that prepared strings should be stored
in the database, I think it is worth working through the issues of
what we will do with saslprep2 now.

My presumption is that we go forward assuming that saslprep is
idempotent even though we know it is not.  That is, we ignore this
issue now and for all future versions of saslprep.

--Sam

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Thu, 24 Jun 2004 21:02:47 -0700

>issue now and for all future versions of saslprep. 

I have written the following text, replacing the first paragraph
of the Security Consideration discussion, to make note of the
issue.

  This profile is intended to used to prepare simple user names and
  passwords strings for comparison or use in cryptographic functions
  (e.g., message digests).  The preparation algorithm was specifically
  designed such that its output is canonical, and it is for well-formed.          
  However, due to an anomaly [PR29] in specification of Unicode     
  normalization, canonical equivalence is not guaranteed for a select
  few character sequences.  These sequences, however, do not appear  
  in well-formed text.  This specification was published despite     
  this known technical problem.  It is expected that this specification
  will be revised before further progression on the Standards Track               
  (after [Unicode] and/or [StringPrep] specifications have been updated    
  to address this problem).

Comments?  Alternative wording?

Kurt

---

From: Simon Josefsson <[email-masked]>
Date: Fri, 25 Jun 2004 12:37:49 +0200

>   designed such that its output is canonical, and it is for well-formed.
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^

I have trouble parsing the last part?

> Comments?  Alternative wording?

Otherwise it looks fine to me!

Thanks,
Simon

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Fri, 25 Jun 2004 08:30:15 -0700

At 03:37 AM 6/25/2004, Simon Josefsson wrote:
>>and it is for well-formed.

s/well-formed/well-formed text/

Kurt

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Fri, 04 Jun 2004 08:54:54 -0700

Kurt

---

From: Simon Josefsson <[email-masked]>
Date: Sat, 05 Jun 2004 14:19:48 +0200

Thanks,
Simon

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Sat, 05 Jun 2004 20:39:46 -0400

---

From: Simon Josefsson <[email-masked]>
Date: Sun, 06 Jun 2004 12:20:03 +0200

We agree about this.

Thanks,
Simon

---

From: Sam Hartman <[email-masked]>
Date: Mon, 07 Jun 2004 17:45:23 -0400

--Sam

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Thu, 24 Jun 2004 21:02:47 -0700

Kurt

---

From: Simon Josefsson <[email-masked]>
Date: Fri, 25 Jun 2004 12:37:49 +0200

Thanks,
Simon

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Fri, 25 Jun 2004 08:30:15 -0700

Kurt