Subject: RE: [[email-masked]: Comments on NFSv4 rfc3010bis- 05 draft]

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 24 Jan 2003 09:39:39 -0800

> Oh boy.  I've been thinking about the Unicode issues as well.
>
> My thoughts:
>
> - By specifying _a_ Unicode normalization form for filename components
>   on the wire we can avoid the need to normalize strings in the kernel.
>
>   Checking that inputs are normalized is a lot simpler and less
>   resource consuming than actually performing normalization.
>
>   Obviously, changing utf8str_cs to specify a normalization form may
>   require a new error code and may be an incompatible change.

As far as the error code goes, I would think you could use NFS4ERR_BADNAME.

>   If the WG cannot reach consensus about such a change then a
>   recommendation that a given normalization form be used should help
>   prevent unnecessary normalization operations by NFSv4 clients/servers
>   that heed the recommendation (a norm. form could even be
>   "negotiated", but let's not go there).

My worry about this is the whole issue of existing file systems and
their contents.  I'm sure that there are file systems out there that 
do not normalize and thus there may be directories which contain multiple
files that would map to the same string under any given normalization 
form.  If you impose a normalization form, then you have the issue of
dealing with that legacy data.

> - I prefer normalization forms D or KD.  Normalization forms C and KC
>   involve normalization to form D first (decomposition), followed by
>   recomposition (but only using composed codepoints from Unicode
>   version 3.0).

At least I feel the need for a "Normalization Forms for Dummies" document.
Maybe other working group members do as well.  Any pointers to something
that will explain this stuff to those who have not already immersed
themsleves in this area.

>   All we need for NFSv4 is a canonical normalization form; form D being
>   lighter weight than form C makes form D more desirable.
>
>   On the other hand, the native norm. form produced by input mechanisms
>   at the clients may be preferred by this WG, whatever it may be, as
>   long as all/most clients are consistent.  

Dream on.

By the way does anybody know what CIFS does about the normalization issue?

>   By using the normalization
>   form imposed at the earliest stage possible we may avoid the need to
>   re-normalize most of the time and get away with checking for
>   normalized strings in the libraries and in the kernel with occasional
>   normalization in the libraries.

I think my point is the earliest possible point was when the files were
created, i.e. it may be long gone.

>  - I'm not sure that I care about avoiding the K forms (as suggested by
>    Dan Oscarsson) as I see the K forms as merely slightly reducing the
>    available namespace in exchange for reducing the scope for confusion.
>
>    The typical example of a K normalization is conversion of ligatures
>    such as "fi" or "ae" (one codepoint each) to the visually related
>    codepoints that make them up ('f' and 'i' for "fi" and 'a' and 'e'
>    for "ae"); there may be other form K substitutions that may actually
>    be offensive to speakers of live languages, so further study of the
>    matter may be warranted.  But then, this is a network filesystem
>    protocol, some compromises have to be made.

Although I have not studied this area and so I may be missing the 
justification for some of this, this strikes me as strange.  If
someone names a file with the ligature fi, then maybe he did it for a
reason, such as the fact that it is about the ligature fi.  He could 
have named it ordinary fi, but chose not to for his own reasons.  It 
seems wierd to change it to ordinary fi and that file may also exist.  
Visual similarity seems a strange basis to map characters.  Are you 
going to map Cyrillic Veh to Roman B, because they look the same?

> Nico
> -- 

On Fri, Jan 24, 2003 at 09:14:06AM -0600, Spencer Shepler wrote:
> 
> Forwarding on behalf of Dan.
> 
> -- 
> Spencer
> 

> Date: Sat, 18 Jan 2003 10:45:13 +0100 (CET)
> From: Dan Oscarsson <[email-masked]>
> Reply-To: Dan Oscarsson <[email-masked]>
> Subject: Comments on NFSv4 rfc3010bis-05 draft
> To: [email-masked]
> X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.5 SunOS 5.9 sun4u sparc 
> 
> This is a comment on draft-ietf-nfsv4-rfc3010bis-05
> 
> I have for many years worked with internationalisation and when I
> look at the internationalisation section 11 in the draft I do not
> understand the choice made.
> 
> First you have to separate the format used in the protocol from
> the way to match strings in server or client. It is somewhat
> unclear in the document.
> 
> Looking at the most important string: utf8str_cs which is used in
> naming components and pathnames, I see that you do not specify a
> normalisation form. Why not?
> 
> To create simple and effcient implementations it is very good
> if all strings are normalised. As names are important to people,
> normalisation form KC cannot be used. Instead normalisation
> form C MUST be used. This will allow all special forms of a character
> that a person what to have in a name be preserved. At the same time
> if will give a compact format to transmitt over the net and ONE
> single format to decode/encode in implementations.
> The "stringprep" document was created to handle matching of
> names in DNS, it is not generally suitable.
> 
> If utf8str_cs strings need to be compared case insensitively,
> you have to use form KC and single character lower casing.
> The case mapping in stringprep will result in names that
> should be different will be matched as equivalent (this is due
> to going to far in case mapping).
> The table B.2 which the draft says to use when doing
> case insensitiv matching is only usefull if normalisation form KC
> has been used before. It cannot be used on unnormalised or
> form C text.
> 
> As an implementor of programs handling UCS I want to have
> a simple efficient export/import of text as well as a good
> case insensitive matching. By always requiring normalisation
> form C I get simpler code and that form will never destroy
> any data in the text. In addition to from C I would
> prefer to have most of form KC applied, but only the part
> that equivalences equivalent characters. Form KC unfortunately
> does equivalencing of characters that are not equivalent and
> forcing some to be decomposed.
> 
> I am sure that not mandating a normalisation form in NFSv4
> will result in more complex code and more failurs due to
> mismatch between client/server handling.
> So why not normalise?
> 
> Regards,
> 
>    Dan
>

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 24 Jan 2003 11:59:24 -0600

> As far as the error code goes, I would think you could use NFS4ERR_BADNAME.

*nod*

> dealing with that legacy data.

Er, no, utf8str_cs requires that a normalization form be used, just not
on the wire.  So the problem of legacy filesystems remains even if we do
not act to recommend or require a specific normalization form on the
wire.

As for the legacy filesystems, the problem is far worse than you
suggest, since some filesystems allow arbitrary 8-bit data in filenames,
including arbitrary non-UTF-8 encodings, without any codeset tagging
whatsoever.  But this is not relevant to the matter of whether
utf8str_cs should mandate a specific normalization form for filenames on
the wire.

> that will explain this stuff to those who have not already immersed

There are several books with "Unicode" in the title (none in my office
right now).  And the Unicode home page is a good place to start:

http://www.unicode.org/

> >   long as all/most clients are consistent.  

Heh.  I didn't want to dismiss that out of hand, see?  I don't profess
to know what y'all do :)

[...]

> created, i.e. it may be long gone.

"Normalization form" is a concept specific to Unicode (well, to any
codeset which has multiple equivalent encodings for the same character,
which mostly means Unicode).

Legacy filesystems don't even use a single codeset consistently, much
less a single Unicode normalization form.

Log in to some *nix system with one locale using Latin-1 as the codeset
and create some file with a name including an accented character.  Then
log out and log in again using a locale with a UTF-8 codeset and try it
again.  You should end up with the two files with the same name, each of
which does not display correctly or at all unless you're using the
correct locale.  Fun, fun, fun!

But I insist: the legacy issue has nothing to do with selecting a single
required normalization form for filenames on the wire.  The legacy
filesystem issues should be addressed in a separate thread. See above.

> going to map Cyrillic Veh to Roman B, because they look the same?

If that's what Unicode specifies for the K normalization forms, yes.

Think of a group of users using a shared directory: one user may know
how to enter a ligature, many may not even be able to tell that some
filename uses a ligature - they may not know what a ligature is, how to
recognize one, much less how to type one in.  Doesn't it then make sense
to use the K compatibility mappings for ligatures?  Obviously, for the
single-user case the answer is "no", but for the multi-user case the
answer is not clear.

Cheers,

Nico
--

---

From: Mike Eisler <[email-masked]>
Date: Fri, 24 Jan 2003 10:33:10 -0800

> not act to recommend or require a specific normalization form on the

It requires a normalization form only if the server is case insensitive.

> right now).  And the Unicode home page is a good place to start:

I find each visit to unicode.org ever more confusing. My most
recent visit revealed that "16 bit" Unicode now has over 2^64 characters.
It is absolutely impenetrable.

The nfsv4 i18n follows the lead of IETF's stringprep RFC (which we
were asked to do by IESG). Presumably the folks who wrote it were
experts, and they strongly recommend KC for case insensitive matching.
Now we have two other experts in the last 24 hours disagree, but with
two more opinions.

Clearly, no matter what we do in this area, it is highly
probable we'll get it wrong.

Just as clearly, normalization is not well thought out by the
people specifying this; otherwise it would be much easier to grasp.

So what do we want to do with the i-d that IESG has approved for
publication. My inclination is to leave it alone, since I suspect we
could delay it for 12 more months and still not reach consensus.
Only via real experience will a practical truth emerge ... it may that
IESG is right, it may be that Dan is right, Nico is right, Dave is right. My
guess is it will be none of the above. Fixed in NFSv4.x (x > 0).

	-mre

---

From: Mike Eisler <[email-masked]>
Date: Fri, 24 Jan 2003 10:45:14 -0800

> recent visit revealed that "16 bit" Unicode now has over 2^64 characters.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

try: "over 2^16 characters". Point holds though.

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 24 Jan 2003 14:06:03 -0600

> It requires a normalization form only if the server is case insensitive.

This is not my reading of the draft.

Section 11.1.1 clearly specifies what to do with respect to filenames
with equal names but different encoding [due to different normalization
forms used by the clients that create them].  Section 11.1.1 necessarily
requires that the server perform normalization of client inputs to some
form of the server's choosing.

The other utf8str_* types are not claimed to be useable for filenames:

 - "The utf8str_cis type is a case insensitive string of UTF-8
    characters.  Its primary use in NFS Version 4 is for naming NFS
    servers."

 - "The utf8str_mixed type is a string of UTF-8 characters, with a
    prefix that is case sensitive, a separator equal to '@', and a
    suffix that is fully qualified domain name.  Its primary use in NFS
    Version 4 is for naming principals identified in an Access Control
    Entry."

Only utf8str_cs is said to be useable for filename components.

So, my reading is that utf8str_cs is for filename components, that the
server must normalize client filename component inputs to some
normalization form preferred by the server, and that duplicate filenames
are not allowed.

> It is absolutely impenetrable.

?? There's no such thing as 16-bit Unicode.  There's UTF-16, an encoding
for Unicode; UTF-16 is not limited to 2^16 codepoints: it can, at the
very least, represent 2^21 codepoints, including all of the BMP.

UTF-8 can represent all of the codepoint space (2^31), though it's been
recently restricted to just 2^21, since that now seems to be all that
will ever be necessary (ha! where have we heard that before? still,
seems reasonable).

> Now we have two other experts in the last 24 hours disagree, but with

Er, form KC is the form recommended by the IETF, yes.  That does not
change the fact that form D is lighter weight, nor does the NFSv4 draft
specify any form for utf8str_cs.

> probable we'll get it wrong.

It's not clear that what NFSv4 specifies is wrong; utf8str_cs is
certainly sub-optimal, but it's not incorrect.

> people specifying this; otherwise it would be much easier to grasp.

I disagree.

And note that my preference for form D over C has to do with code bloat
and performance; I think that case-insensitive comparisons can be made
of Unicode strings normalized with form D just as well as with form C
(just to be sure I'll hit the books again).

> guess is it will be none of the above. Fixed in NFSv4.x (x > 0).

I'd like to see the WG RECOMMEND to clients _a_ normalization form for
utf8str_cs.

I think we may not be able to change the spec to require that that norm
form be used on the wire for utf8str_cs types, not at this late a date,
though that is what I would have preferred.  Problem is, I don't think
it will ever be possible or worthwhile anymore to make this change later
if we don't do it now.  Is that a big deal?  Well, only if you think
that having full-blown Unicode normalization facilities in the server
implementations of NFSv4 (which may be required for multi-protocol
servers anyways) is a big deal.  Hindsight is 20/20...

So, if nothing else, we're all aware now that NFSv4 i18n is tricky to
implement correctly; being aware is better than nothing, and if that's
all that comes out of this thread it will still be worth having had it.

Also, at some point we need to test i18n at Connectathon.

Cheers,

Nico
--

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 24 Jan 2003 14:10:36 -0600

> servers anyways) is a big deal.  Hindsight is 20/20...

I would prefer to leave the current I-D as is and let it become an
RFC.  We can continue to build consensus around this issue and either
publish a seperate RFC covering this specific issue and/or incorporate
the work into the Draft version of the NFSv4 RFC.

Spencer

---

From: Mike Eisler <[email-masked]>
Date: Fri, 24 Jan 2003 13:00:43 -0800

> form of the server's choosing.

I don't see the text. Section  11.1.4 says no normalization
form is specified. Stringprep has tables B1 and B2. B1 is required
to be processed, but I didn't see anything in stringprep that said that
normalization went with table B1. Table B2 explicitly says KC, and B2
is for dealing with case insensitive stuff. Section 3 of stringprpe (now
RFC 3454) refers to Appendix B and the various tables. It says nothing
about normization for B1, whereas it does say something about normalization
with respect to B2 and B3.

Now I am admittably quite ignorant on this topic, but it seems rather
odd that B1 requires normalization (and according to RFC 3454, B1 is
mandatory), and yet B3 is specified to allow applications to not use
normalization when case matching. Section 2 of RFC 3454 distinguishes between
the mapping phase and the normalization phase, and mapping is the topic
of section 3 and appendix B. So if your reading is correct,
then it seems to me RFC 3454 has a paradox.

We apparently need an "RFC 3454 for Dummies" document.

	-mre

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 24 Jan 2003 15:12:18 -0600

> with respect to B2 and B3.

Yes, but the requirements from section 11.1.1 about duplicate filenames
imply normalization, if not for stored file names, then during
comparison of file names.

> We apparently need an "RFC 3454 for Dummies" document.

I was only referring to section 11.1.1, specifically: "Two valid
distinct UTF-8 strings might be the same after processing via the
utf8str_cs profile. If the strings are two names inside a directory, the
NFS version 4 server will need to either: ..."

If the server allows two (or more) files to have the same names but
different byte sequences (say, one UTF-8, NFD, the other UTF-8, NFC),
then we'll have the "how do I delete a file named '-i'?" problem
magnified to "I see a file named 'bár' but when I type 'rm bár' it
doesn't work".  Or am I missing some text about utf8str_cs comparisons
having to be done over normalized forms of the comparison inputs??

(yes, above I did not use UTF-8 for "bár" - my apologies)

Cheers,

Nico
--