Subject: [idn] Re: :Re: Last Call: Preparation of Internationalized Strings

From: [email-masked]
Date: Sun, 02 Jun 2002 14:44:40 +0200

[Resending with different From: address.]

Patrik Fältström <[email-masked]> writes:

> --On 2002-05-30 12.16 +0200 Simon Josefsson <[email-masked]> wrote:
>
>> This is interesting -- has the Unicode consortium promised to always
>> update the CK normalization tables in a forward compatible way?
>
> Yes.

The reference for that statement seem to be (correct me if I'm wrong)
http://www.unicode.org/unicode/standard/policies.html:

,----
| Normalization. Once a character is encoded, its canonical combining
| class and decomposition mapping will not be changed in a way that will
| destabilize normalization.
`----

Which looks good.  However, reading on:

,----
| The decomposition mapping may only be changed at all in the following
| _exceptional_ set of circumstances:
| 
| + when there is a clear and evident mistake identified in the Unicode
|   Character Database (such as a typographic mistake), and
| 
| + when that error constitutes a clear violation of the identity
|   stability policy (#4), and
| 
| + when the correction of such an error does not violate constraints (a)-(d)
`----

So it appears as if the statement isn't strictly true?

A further security consideration of IDNA could be that whenever such
modifications is done in the Unicode standards, they may be exploited
and it should be an operational consideration to never register
domains, issues PKIX certifices for domains, create Kerberos realms,
create PGP keys, etc, for IDNs that contains characters that have
their decomposition mapping changed by the Unicode consortium.

It seems as if a modification of this kind occured between Unicode 3.0
and 3.1: http://www.unicode.org/versions/corrigendum2.html.

The conclusion here is that this isn't a practical problem -- only one
character changed normalization between 3.0 and 3.1 and none between
3.1 and 3.2 AFAIK.  I am more worried about the transcoding problem.

---

From: [email-masked]
Date: Sun, 02 Jun 2002 14:45:53 +0200

Patrik Fältström <[email-masked]> writes:

> --On 2002-05-31 00.26 +0900 Soobok Lee <[email-masked]> wrote:
>
>> This issues were raised at the time of IDN WG last call,  3 months ago.
>
> And my answer was exactly the same then as now. We can not do better. If
> Unicode is updated, you will either get inconsistency with the parties
> using the new version of Unicode (if IDN is not upgraded) or inconsistency
> with old registered names (if IDN is upgraded).
>
> The point is that the current scheme (a) trust Unicode when they say that
> changes will be backward compatble and (b) IF they do an incompatible
> change, we can at that point in time make the desicion -- we don't have to
> decide now what path we take.

Yes.  I think it could be useful to have this discussion in the
specifications, so it is stored in the collective memory.  I am not
sure everyone will remember or understand all subtle problems that
will follow from changes in normalization mapping tables in a few
years (I know I won't).  Is the following reasonable?

,----
| If or when this specification is updated to use a more recent Unicode
| normalization table, the new normalization table must be compared with
| the old to spot backwards incompatible changes.  If there are such
| changes, they must be handled somehow, or there will be security as
| well as operational implications.  Methods to handle the conflicts
| could include keeping the old normalization, or taking care of the
| conflicting characters by operational means, or some other method.
|
| Implementations must never use more recent normalization tables than
| the one referenced from this document, even though more recent
| tables may be provided by operating systems.  If an application is
| unsure of which version of the normalization tables are in the
| operating system, the application should include the normalization
| tables itself.  Using normalization tables other than the one
| referenced from this specification may have security and operational
| implications.
`----

The transcoding problem is still open, it seems.

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 02 Jun 2002 16:04:23 +0200

--On 2002-06-02 14.45 +0200 [email-masked] wrote:

> Is the following reasonable?

I think so personally, but, the way a last call works (and the way I want
to have it working as one of the editors of the documents) is that we
during the last call period collect issues like this one, and _then_ we
editors summarize and together with AD and wg chair rise a list with issues.

I.e. the whole last call period is a "collection of issues" period. If we
start updating the documents now, then comments will be based on multiple
versions of the document -- which doesn't make things easier.

So, I have added this issue to my list of things.

   paf

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 02 Jun 2002 16:01:59 +0200

> So it appears as if the statement isn't strictly true?

This is also true. I am discussing the issue about 3.1 versus 3.2 with
officers of the Unicode Consortium as we speak. I have requested some
conclusion no later than friday this coming week, which I will report back
to this wg (if not they send it to the mailing list directly).

  paf

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 2 Jun 2002 08:27:50 -0700

Simon,

There have been two corrections to normalization since Unicode 3.0.  One
involved a Chinese (Han) compatibility character that was mapped to the
wrong "normal" character by error.  The other involved a Yiddish
(Hebrew) compatibility character that should have had a compatibility
mapping, but did not, also by error.

Both corrections were made to characters that are supposedly "very rare"
in actual use, so that the real-world impact would be minimal.  Neither
one has anything to do with transcoding tables.

I know you are very concerned that Unicode has "broken its promise" by
making changes to the normalization tables after claiming they would not
do so.  I think if the corrections had not been made, there would have
been an equal but opposite reaction that Unicode was too stubborn to
correct its own mistakes, and that NFKC was rendered "useless" because
of these two incorrect mappings.

The pages explaining the corrigenda include lengthy, detailed
explanations of why the Technical Committee felt they were necessary and
justified.  As someone already mentioned, one of the justifications
given for the Yiddish change was that no normative references existed
*yet* for the Unicode normalization tables (i.e. from IDN).  This
implies that once such normative references *do* exist, a similar
decision to correct an error might not be made.

I imagine these were very difficult decisions for the UTC, who knew that
someone would jump on the changes immediately as evidence that
normalization is inherently unstable and Unicode is therefore "not
secure."

It's true that we are relying on "enlightened statesmen" to make the
right decisions and not, say, decide one day to add a compatibility
mapping for U+00C6 LATIN CAPITAL LETTER AE that would break everything.
The UTC has tried to assure us that such a thing will not happen, but in
the end, all we can do is trust.

-Doug Ewell
 Fullerton, California

----- Original Message -----
From: <[email-masked]>
To: <[email-masked]>
Sent: Sunday, June 02, 2002 5:44 am
Subject: [idn] Re: :Re: Last Call: Preparation of Internationalized
Strings

,----

,----
| + when the correction of such an error does not violate constraints
(a)-(d)
`----