Subject: [nfsv4] Chapter 12 for next rfc3530bis

From: <[email-masked]>
Date: Mon, 11 Oct 2010 19:42:46 -0400

Attached find my preliminary xml for chapter 12 of the next draft.  I
have proofread it but have not yet run xml2rfc as part of integrating
with the rest of new draft.

 <<ModI18N.xml>> 

I want to thank particularly James Lentini and David Black for reading
through Chapter 12 from the last draft and giving me significant numbers
of comments.  I can't remember those making other suggestions, but I
thank them as well.  I am trying to close down the update process for
this draft but I think it is appropriate to give those who suggested
improvements a chance to look at what I have and see if it matches their
intentions.  I said "opportunity".  I'm not nagging anybody to do it.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 12 Oct 2010 13:08:00 -0400

On Mon, Oct 11, 2010 at 07:42:46PM -0400, [email-masked] wrote:
>           <t hangText="ascii_must/ASCII:">
>             indicating that strings of this type MUST be
>             pure ASCII, and thus automatically UTF-8.
>             The processing of these string must ensure
>             that they are only have ASCII characters but 
			^^^^^^^^^^^^^

I know, you put that in there just to check if anyone was reading....

>         <c>prinpfx4</c>
>           <c>UVMUST</c>
>           <c>NFS</c>
>           <c>
>             Must match one of a list of valid users
>             or groups for that particular domain.  

What have the traditional restrictions on usernames been?  (A subset of
ascii?)

I just want to make sure I'm getting into the business of mapping
non-ascii non-utf8 usernames....

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 12 Oct 2010 13:09:36 -0400

> >             that they are only have ASCII characters but 

(I meant to have a "not" there!)

---

From: <[email-masked]>
Date: Tue, 12 Oct 2010 14:12:55 -0400

Inline. 

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Tuesday, October 12, 2010 1:10 PM
To: Noveck, David
Cc: Black, David; [email-masked]; [email-masked]
Subject: Re: [nfsv4] Chapter 12 for next rfc3530bis

> >             that they are only have ASCII characters but 

Of course.  You pass the test :-)

> What have the traditional restrictions on usernames been?  (A subset
of

I'm guessing there has been some stuff in the 80-ff range for people who
have common European accents, umlauts, tildes, and whatever you call the
angstrom thing.

As far as a subset, probably NULs and such are out.  I haven't seen a
list.

The point is that what is valid depends primarily on presence on the
list.  If there is no name "aaaaa" then it is invalid in exactly the
same way a name consisting of a NUL would be.  They both are not on the
list.  There is no reason to divide this into two errors, at least that
I can see. 

> non-ascii non-utf8 usernames....

UVMUST says it MUST be UTF-8.  So if you get into the business of
mapping non-ascii non-utf8, you are non-compliant and you have only
yourself to blame for having to map stuff that you aren't supposed to
have accepted in the first place.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 12 Oct 2010 14:28:53 -0400

> have accepted in the first place. 

I have some control over nfsd, but none over useradd.  If there are
people out there with /etc/passwd's containing non-utf8 non-ascii
usernames then the only way I'd see to enforce a MUST of utf-8 would be
by taking a stab at what encoding they're using and then mapping to and
from utf-8.  No thanks!

Hm, but non-ascii usernames can't be used in email addresses, can they?
So maybe we'll never see them in practice.

--b.

---

From: <[email-masked]>
Date: Tue, 12 Oct 2010 14:47:40 -0400

You as the server can enforce it by simply checking whether it is UTF-8
and returning INVAL if it isn't.

The client is going to have some understanding of what it is being
presented with and certainly can map an o-umlaut for example, to the
UTF-8 version of that which is two bytes.  It is supposed to send UTF-8.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Tuesday, October 12, 2010 2:29 PM
To: Noveck, David
Cc: Black, David; [email-masked]; [email-masked]
Subject: Re: [nfsv4] Chapter 12 for next rfc3530bis

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 12 Oct 2010 15:05:52 -0400

> and returning INVAL if it isn't.

Alas, no, if I get a getattr for the owner of a file, find the file is
owned by uid 8569, look up 8569 and it's somebody non-ascii and
non-utf8, my choices are:

	- Return an error, or claim not to support the owner attribute
	  on this file.
	- Attempt to guess the encoding and map to utf-8 before
	  returning.
	- Just return the username as it is.

And if I pick 3 (and it's what the code would do now, and seems the
least of evils), and the client tries to send the same name back to me
in a setattr, I can't see returning INVAL.

I could try to avoid getting into the whole situation in the first place
by forbidding those names.  That's not really up to me.

In the end I guess this just means I'll be violating the spec in some
(probably very unlikely?) cases.  Solaris too I bet, if it has
historically allowed such usernames.  It won't keep me up at night.

--b.

---

From: <[email-masked]>
Date: Wed, 13 Oct 2010 10:58:40 -0400

> So maybe we'll never see them in practice.

That's in the works, see draft-ietf-eai-frmwrk-4952bis, which also provides some indirect support for the pragmatic approach we're taking to i18n in rfc3530bis.

Thanks,
--David

---

From: <[email-masked]>
Date: Thu, 14 Oct 2010 09:58:19 -0400

First of all, if it non-UTF8, it is non-ASCII.  That way we only have
one predicate to deal with.  Assuming that in the past Cyrillic, Hebrew,
Nagari, etc. names have not been allowed (Why can't I have a Linear-B
username :-) this only includes one-byte ascii and extended ascii
characters.  If there are extended-ascii characters, then you have
something that is not utf8, but the procedure to covert it to utf8 is
very simple.  You move around some of bits and come up with the two-byte
utf8 equivalent of the extended ascii characters.  The ascii characters
you copy over straight.

Why can't I just say "Problem solved"?  Am I missing something?  I
understand that "Mission accomplished" is out of bounds. 

I think there is a philosophical problem underlying this issue.  If you
treat what is returned by the lookup as an array of bytes, you have all
these problems with non-utf8-ness.  But if you treat it as an array of
characters, then the mapping to UTF-8 is clear.  But, the idea that it
is a simplification to treat everything as an array of bytes is very
prevalent.  It is true that it seems to avoid all that I18N nastiness,
with that being left for (unlucky) others  to deal with.  But that isn't
the philosophy of NFSv4 (for good reason), although for file name
components we have an accommodation to allow non-utf8 to work.  But I
can't see a similar change for user and group names.  They must be UTF-8
and the only way to provide UTF-8 depends on understanding the string as
an array of characters in some known code and converting it to UTF-8.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Tuesday, October 12, 2010 3:06 PM
To: Noveck, David
Cc: Black, David; [email-masked]; [email-masked]
Subject: Re: [nfsv4] Chapter 12 for next rfc3530bis

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 14 Oct 2010 10:42:38 -0400

> 	- Attempt to guess the encoding and map to utf-8 before

option.

My understanding is that converting to utf8 is not so simple: you have
to know which of the multiple possible extended ascii character sets
you're converting from.  /etc/passwd doesn't necessarily come with that
information.

> characters,

I can't assume that whatever password database is using knows about
encodings--so I really am stuck with just an array of bytes.

--b.

> then the mapping to UTF-8 is clear.  But, the idea that it

---

From: "Everhart, Craig" <[email-masked]>
Date: Thu, 14 Oct 2010 11:57:21 -0400

> you're converting from.  /etc/passwd doesn't necessarily come with
that

Is it optional somehow?  Is there a place where it might be found?

How would I go about creating a non-USASCII username?  Am I being
irresponsible if, having done so and not having provided the information
about how to interpret it, I connect it up to a distributed file system
that uses utf-8 file attributes?  Particularly if all the clients of
that file system don't know a-priori what special interpretation your
client is using for its attributes?

Is it possible to document /etc/passwd as being in UTF-8 format, and
whenever entries are found not to be valid UTF-8, NFS generates errors?
This is basically Bruce's choice 1.  Why wouldn't that be a reasonable
choice, other than inertia?

Is there a documentable way to turn this into Bruce's choice 2 without
heuristics?
 
> > and the only way to provide UTF-8 depends on understanding the
string
> as
> > > > yourself to blame for having to map stuff that you aren't
> supposed
> > to
> > > usernames then the only way I'd see to enforce a MUST of utf-8
> would
> > be
> > > by taking a stab at what encoding they're using and then mapping
to
> > and
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: <[email-masked]>
Date: Thu, 14 Oct 2010 12:01:52 -0400

Hi Bruce,

> you're converting from.  /etc/passwd doesn't necessarily come with that

First of all, more than a "guess" is required, and some system help may well be needed.  To take a specific example, if /etc/passwd contains usernames in multiple incompatible ISO 8859 (extended ASCII) character sets (http://www.terena.org/activities/multiling/ml-docs/iso-8859.html), somebody has seriously screwed up the system administration - IMHO, the "system administrator" component is probably defective and should be replaced ;-).  Assuming a competent system administrator, there should be at most one ISO 8859 charset in use and there should be a system locale from which it's possible to determine which one.  In contrast, trying to "guess" in isolation which 8859 charset is in use is dangerous.

> > 	- Just return the username as it is.

Changing this situation will require passing charset info on the wire, and that will be awful, IMHO.

Converting other charsets to/from UTF-8 can in general require locale info, not just knowing which charset, because the 8859 extended code point space is very limited by comparison to Unicode.  For an eye-opening related example see RFC 5895's discussion of converting capital "I" to lower case for Turkish (http://datatracker.ietf.org/doc/rfc5895/).

Thanks,
--David

> Sent: Thursday, October 14, 2010 10:43 AM
> To: Noveck, David

---

From: <[email-masked]>
Date: Thu, 14 Oct 2010 12:53:18 -0400

> Changing this situation will require passing charset 
> info on the wire, and that will be awful, IMHO. 

A) It is awful in my opinion, as well.
B) That isn't the NFSv4 protocol.  We have a design for this stuff and
it is to encode it in UTF-8 and we're not changing it, except if it
absolutely cannot work.

In the past when the issue of converting stings to UTF-8 according to
locale was raised, it was for name components and Linux implementations
would not agree to do anything like that.  There was an objection to
doing this kind of conversion in the kernel, as not appropriate to the
kernel environment.  I believe Linus was quite firm on the matter.

In the case of this lookup function I'm guessing that this happens
outside the kernel.  If that's the case, conversion based on the locale
to UTF-8 could happen before the data was returned for the lookup and
the kernel string conversion issue would not arise.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of [email-masked]
Sent: Thursday, October 14, 2010 12:02 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] Chapter 12 for next rfc3530bis

Hi Bruce,

> you're converting from.  /etc/passwd doesn't necessarily come with
that

First of all, more than a "guess" is required, and some system help may
well be needed.  To take a specific example, if /etc/passwd contains
usernames in multiple incompatible ISO 8859 (extended ASCII) character
sets (http://www.terena.org/activities/multiling/ml-docs/iso-8859.html),
somebody has seriously screwed up the system administration - IMHO, the
"system administrator" component is probably defective and should be
replaced ;-).  Assuming a competent system administrator, there should
be at most one ISO 8859 charset in use and there should be a system
locale from which it's possible to determine which one.  In contrast,
trying to "guess" in isolation which 8859 charset is in use is
dangerous.

As should be clear from prior discussion, NFSv4 is UTF-8 on the wire,

> > 	- Just return the username as it is.

Changing this situation will require passing charset info on the wire,
and that will be awful, IMHO.

Converting other charsets to/from UTF-8 can in general require locale
info, not just knowing which charset, because the 8859 extended code
point space is very limited by comparison to Unicode.  For an
eye-opening related example see RFC 5895's discussion of converting
capital "I" to lower case for Turkish
(http://datatracker.ietf.org/doc/rfc5895/).

Thanks,
--David

> > and the only way to provide UTF-8 depends on understanding the
string as
supposed
would
to

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 14 Oct 2010 14:01:19 -0400

> A) It is awful in my opinion, as well.

And it doesn't help, since the problematic case is that of a preexisting
non-UTF8 username whose encoding we *don't know*.

> it is to encode it in UTF-8 and we're not changing it, except if it

My conclusion for now is: a) I'll probably never get a bug report about
this since presumably email and lots of other things break if you try
it, b) in the unlikely case I do get a complaint, I'm happy to tell the
adminstrator they're out of spec, and c) this is something that drives
everyone who touches it (including, obviously, me) nuts, too much time
has already been spent on it, and I regret bring it up, ok, ok, ok!

> would not agree to do anything like that.

Well, and that nobody else actually attempted that sort of conversion
(e.g., making open() locale-aware) either--it was never just Linux.

--b.

> There was an objection to