Subject: [nfsv4] Proposed internationalization changes for next rfc3530bis drafts

From: "Noveck, David" <[email-masked]>
Date: Thu, 12 Sep 2013 17:49:41 -0400

Here are the changes I've come up with, based mainly on working group discussion.  Thanks to everybody who participated.  I've tried to explain why the changes were made with explanations in italics.

I'd like feedback on all of these proposed changes, particularly as to whether the behavior of client and server implementations that you know about actually are properly reflected in the resulting document.  If people can, I'd hope for the same sort of information about the chapter as a whole.

There are a few cases where I've found something unclear and basically guess at the resolution.  I've underlined these.  Please pay particular attention to these changes.
For draft-ietf-nfsv4-rfc3530bis
Section 2.1:

*         Change the typedef of linktext4 to read
typedef opaque linktext4<>;
This goes with the other changes below, which indicate link text is opaque an does not have the UTF-9-preferred status that component names do.
Section 5.7
*         Change table entry for mimetype:
Change the data type field to ascii_REQUIRED4.
Forgot to do this earlier.
Section 12.1:

*         At the end of the section, add the following new paragraph:
In the case of a MAY, SHOULD, or SHOULD NOT that applies to servers, clients need to be aware that there are servers which either do or don't do the specified action, and they need to be prepared for either eventuality.

Where the servers can behave in different ways, clients need to be aware of this and be prepared to deal with it.
Section 12.2:

*         In first bullet, change "MUST" to "SHOULD".
Discussion on the working group list, revealed that NFSv4 clients on UNIX systems are basically depending on locale-based logic for this, so the "MUST" has to go.
On the other hand, the large and growing use of UTF-8 locales means that "SHOULD" is OK here.

*         After second bullet, add a new bullet which says the following:
The contents of symbolic links (of type linktext4 in the XDR) MUST be treated as opaque data by client and server. Although UTF-8 encoding is often used, it need not be. In this respect, the contents of symbolic links are like the contents of regular files in that their encoding is not within the scope of this specification.

This was prompted by Mike Kupfer's draft.  Although he didn't convince me about component names, it is clear that clients and server treat symbolic links as opaque so the spec might as well say that.
Section 12.3:

*         Add the following text after the second sentence of last paragraph:
A server MUST make the same choice (as to whether to normalize or not, the target form of normalization and whether to do normalization-insensitive string comparisons) in the same way for all accesses to a particular filesystem.
I originally proposed this almost tongue-in-cheek, but, since we are describing the combination of fileserver/filesystem behavior, decided that this is a valid constraint, even if clients don't depend on it.
Section 12.5

*         In the first paragraph, change "SHOULD " to "MAY"
The problem, as revealed by working group discussion, is that this isn't currently common enough for us to say "SHOULD", given that the intent of the current internationalization chapter is to describe current practice.

*         Add a new paragraph after the current first paragraph with the following text:
Requirements for server handling of component names which are not valid UTF-8, when a server does not return NFS4ERR_INVAL in response to receiving these are described in section 12.6.

*         Replace the parenthesized section in the second paragraph by the following:

e.g. names containing slashes or characters that have more than two octets on a filesystem that supports Unicode characters only
The case of slashes was not explicitly mentioned so it should be.  I'm worried that people would have read the old example given, quickly said "that doesn't apply to me") and may be returning NFS$ERR_BADNAME in this name.  Since this chapter is to reflect existing practice, I'd like to know what people actually do.

*         Replace the last sentence of the last paragraph with the following:
This includes such situations as file system prohibitions of "." and ".." as file names for certain operations, and similar constraints.
The big issue with what was there before is that we say (in section 12.3) is that the server "MUST NOT" impose normalization constraints.
New Section 12.6
Based on working group discussion, I've become convinced, that these cases, even though not encouraged, need to be better described.

*         The new section should be the following:
12.6 Handling of component names that are not valid UTF-8 strings
In cases in which the server receives a component name that is not a valid UTF-8 string, the required handling depends on whether a file object is being created or looked up.  Object creation happens for the component name in LINK and CREATE, and the second component name in RENAME.  Object lookup happens for the component name in LOOKUP and REMOVE, and the first component name in RENAME.  The component name in OPEN will result in object lookup and also object creation if the lookup fails and the other OPEN parameters allow file creation.
When creation of a component name which is not valid UTF-8 occurs, and is allowed by the server:

o   A subsequent lookup of the same component name in the same directory MUST result in finding the created file object.

o   A READDIR of the directory in which the name is created MUST result in an entry containing the component name used for the creation.

o   A subsequent lookup using any other string as the component name SHOULD NOT find the originally created object.

o   A subsequent lookup using any valid UTF-8 string MUST NOT  find the originally created object.
When a lookup of a component name which is not valid UTF-8 occurs, and is allowed by the server:

o   The result of the lookup MUST NOT be any directory entry created with a valid UTF-8 component name.

o   The result of the lookup MUST be a directory entry created with the identical invalid ITF-8 string, if one exists in the directory.

For draft-ietf-nfsv4-rfc3530-dot-x

*         Change the typedef of linktext4 to read as follows:

/// typedef opaque        linktext4<>;

Matches the changes in the main spec.

That's all folks

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 13 Sep 2013 10:15:20 -0400

> The contents of symbolic links (of type linktext4 in the XDR) MUST be treated as opaque data by client and server. Although UTF-8 encoding is often used, it need not be. In this respect, the conten…

Yes, agreed in spirit, but then that "MUST" doesn't quite make sense for
clients.  (*Somebody* on the client side certainly interprets the
contents of files and links, and I guess we can argue about whether the
piece of code that, say, follow symlinks, is really part of the "client"
or not, but....  How about instead we just drop "client and" from the
above?)

One other dumb comment about the internationalization section: could we
please drop the attempt to redefine MUST, SHOULD, etc?

If the point is to make it clear this is describing deployed behavior,
it's redundant.  Those words are *always* meant to describe the behavior
required to interoperate with existing implementations.

If the point is to explain why things changed from 3530, let's instead
address that specifically.

--b.

---

From: Mike Kupfer <[email-masked]>
Date: Fri, 13 Sep 2013 11:21:11 -0700

Noveck, David wrote:

> Here are the changes I've come up with, based mainly on working group
> discussion.  Thanks to everybody who participated.  I've tried to
> explain why the changes were made with explanations in italics.

This set of changes mostly addresses my concerns; thanks.

I'm still uneasy about the possibility of normalizing a component string
that looks like UTF-8 but actually used some other encoding.  How about
this text at the end of Section 12.3:

    It is possible that a client may send a string that mimics UTF-8
    (i.e., it satisfies the rules for UTF-8, but a different encoding
    was actually used).  Applying Unicode character normalization to
    such a string can potentially result in a collision, where the
    client expects two names to be distinct and the server maps them to
    a single name.  A similar collision is possible if the server
    returns strings that mimic UTF-8.  The best way to avoid this
    problem is to use UTF-8, as recommended in Section 12.2.

> The contents of symbolic links (of type linktext4 in the XDR) MUST be
> treated as opaque data by client and server. Although UTF-8 encoding
> is often used, it need not be. In this respect, the contents of
> symbolic links are like the contents of regular files in that their
> encoding is not within the scope of this specification.

I agree with Bruce's suggestion to drop "client and".

> 12.6 Handling of component names that are not valid UTF-8 strings
[...]
> o   The result of the lookup MUST be a directory entry created with
>     the identical invalid ITF-8 string, if one exists in the
>     directory.

Typo: ITF -> UTF

cheers,
mike

---

From: Nico Williams <[email-masked]>
Date: Fri, 13 Sep 2013 15:23:48 -0500

> this text at the end of Section 12.3:

We can reason about it: n-i behavior applied to non-UTF-8 that looks
like UTF-8 == aliasing.  Heck, n-i behavior applied to true UTF-8 in
many forms == aliasing.  Aliasing definitely needs to be warned about!

>     problem is to use UTF-8, as recommended in Section 12.2.

Sure.  We should also note the aliasing case where the clients do use
UTF-8, but in different forms.  It is possible to create contrived
examples where such aliasing creates a security vulnerability.

For example, a web service/servlet/... that has a black or white list of
files that some user may not / may create/access/delete, with the app
not doing n-i comparisons...  If the app does not perform n-i lookups in
its list but the filesystem does, then users will be able to bypass the
application's controls for any file whose name has more than one
equivalent form.  In practice this is an extremely unlikely and quite
contrived example.

Interop problems with HFS+ that I had thought were contrived but
realistic enough to warrant n-i behavior... turned out to be real
enough.  My contrived security vulnerability might also turn out to be
real enough; we should definitely warn about it in the security
considerations section.

Nico
--

---

From: "Noveck, David" <[email-masked]>
Date: Sun, 15 Sep 2013 09:07:23 -0400

> Yes, agreed in spirit, but then that "MUST" doesn't quite make sense for clients.  (*Somebody* on 
> the client side certainly interprets the contents of files and links, and I guess we can argue about 
> whether the piece of code that, say, follow symlinks, is really part of the "client"
> or not,

We can argue about that, but anyway, the code that interprets symlinks cannot treat them as
opaque, as that would prevent interpretation.
  
> but....  How about instead we just drop "client and" from the above?)

OK.

> One other dumb comment about the internationalization section: 

If that were a dumb comment, it would be stupid of me to respond.  Since it isn't, I will.

> could we please drop the attempt to redefine MUST, SHOULD, etc?

I don't think I actually try to redefine these terms.  Take a look at the attached to see something that makes 
the point clearer, I hope.

> If the point is to make it clear this is describing deployed behavior, it's redundant.  

I don't think that's the case.

> Those words are *always* meant to describe the behavior required to interoperate with existing implementations.

If you have running code then you should certainly try to  do that, if possible.  The problem here is that, given where 
the protocol is right now, there is no foolproof way to interact if you don't use UTF-8 all the time.  The client behavior
required to interoperate correctly  with existing servers all the time is to always use UTF-8.   But we don't want to say 
"MUST use UTF-8", simply because the real clients which exist are interoperating adequately, if not perfectly, and they 
do not always UTF-8.   

If you don't have running code, then you can use them to describe how you  think clients and servers should work together, 
and that can be done with varying levels of percipience.

> If the point is to explain why things changed from 3530, let's instead address that specifically.

We don't need to explain that right now.  Further, I don't know how to discuss the issue without mentioning that we 
had something fairly close to reality in rfc3010bis-03 and then were asked/forced to change it to something quite dubious.   
That gets us into an argument that I don't think we want to have right now.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Friday, September 13, 2013 10:15 AM
To: Noveck, David
Cc: nfsv4 list ([email-masked])
Subject: Re: [nfsv4] Proposed internationalization changes for next rfc3530bis drafts

> The contents of symbolic links (of type linktext4 in the XDR) MUST be treated as opaque data by client and server. Although UTF-8 encoding is often used, it need not be. In this respect, the conten…

Yes, agreed in spirit, but then that "MUST" doesn't quite make sense for clients.  (*Somebody* on the client side certainly interprets the contents of files and links, and I guess we can argue about whether the piece of code that, say, follow symlinks, is really part of the "client"
or not, but....  How about instead we just drop "client and" from the
above?)

One other dumb comment about the internationalization section: could we please drop the attempt to redefine MUST, SHOULD, etc?

If the point is to make it clear this is describing deployed behavior, it's redundant.  Those words are *always* meant to describe the behavior required to interoperate with existing implementations.

If the point is to explain why things changed from 3530, let's instead address that specifically.

--b.

---

From: "Noveck, David" <[email-masked]>
Date: Sun, 15 Sep 2013 12:23:10 -0400

As to the issue about normalization of UTF-8-seeming strings, I agree something needs to be
said about this, but I'm not clear where.    It could be in 12.2 as a support point for the SHOULD.

I note that I never say explicitly that normalization is not to be done for strings which are not valid
UTF-8.  In 12.6, I focused on behavior rather than mentioning normalization explicitly.

One issue that we'll have to address if we do put something about the issue into 12.3 is what I call the 
"précis-fox-fox problem":  If you have a component name which is the NFD form of "précis" followed 
by the byte 0xff, are you allowed to normalize it to the NFC form of "précis" followed by the 
byte 0xff  (nor match that when doing n-I lookups".  I'd like the answer to be "NO", but since this chapter 
is to be derived from implementations, can I hear from people familiar with the appropriate server/filesystem 
code?  Thinking about possible implementations of n-I lookup, it might be all too easy to match byte-by-byte,
 see that the NFC and NFD versions of "é" match and that everything else matches exactly, never noticing 
that you have fallen (by seeing the end-of-string after 0xff) into the dreaded legacy world of non-UTF-8 
strings.

I'll come up a new version of proposed changes that addresses this and the other issue in this email, 
but not the chapter 12 changes in other emails so far (other than the new 12.1, if we have agreement on
that).

-----Original Message-----
From: Mike Kupfer [mailto:[email-masked]] 
Sent: Friday, September 13, 2013 2:21 PM
To: Noveck, David
Cc: nfsv4 list ([email-masked])
Subject: Re: [nfsv4] Proposed internationalization changes for next rfc3530bis drafts 

I'm still uneasy about the possibility of normalizing a component string that looks like UTF-8 but actually used some other encoding.  How about this text at the end of Section 12.3:

[...]

Typo: ITF -> UTF

cheers,
mike

---

From: Nico Williams <[email-masked]>
Date: Sun, 15 Sep 2013 13:30:53 -0500

But this isn't about NFSv4, really.  Every filesystem protocol that didn't
get this right on day one suffers from the same exact problems now, if they
are still around.

---

From: "Noveck, David" <[email-masked]>
Date: Tue, 17 Sep 2013 08:59:44 -0400

I'm trying to get the next set of i18n changes for rfc35300bis together.

If people are generally OK with the new section 12.1 attached, I'll put that into it.

I'm interested in any comments people have.

-----Original Message-----
From: Noveck, David 
Sent: Sunday, September 15, 2013 9:07 AM
To: 'J. Bruce Fields'
Cc: nfsv4 list ([email-masked])
Subject: RE: [nfsv4] Proposed internationalization changes for next rfc3530bis drafts

> Yes, agreed in spirit, but then that "MUST" doesn't quite make sense 
> for clients.  (*Somebody* on the client side certainly interprets the 
> contents of files and links, and I guess we can argue about whether the piece of code that, say, follow symlinks, is really part of the "client"
> or not,

We can argue about that, but anyway, the code that interprets symlinks cannot treat them as opaque, as that would prevent interpretation.

OK.

> could we please drop the attempt to redefine MUST, SHOULD, etc?

I don't think I actually try to redefine these terms.  Take a look at the attached to see something that makes the point clearer, I hope.

> Those words are *always* meant to describe the behavior required to interoperate with existing implementations.

If you have running code then you should certainly try to  do that, if possible.  The problem here is that, given where the protocol is right now, there is no foolproof way to interact if you don't use UTF-8 all the time.  The client behavior
required to interoperate correctly  with existing servers all the time is to always use UTF-8.   But we don't want to say 
"MUST use UTF-8", simply because the real clients which exist are interoperating adequately, if not perfectly, and they 
do not always UTF-8.   

If you don't have running code, then you can use them to describe how you  think clients and servers should work together, and that can be done with varying levels of percipience.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]]
Sent: Friday, September 13, 2013 10:15 AM
To: Noveck, David
Cc: nfsv4 list ([email-masked])
Subject: Re: [nfsv4] Proposed internationalization changes for next rfc3530bis drafts

--b.

---

From: Mike Kupfer <[email-masked]>
Date: Tue, 17 Sep 2013 15:30:13 -0700

Noveck, David wrote:

> As to the issue about normalization of UTF-8-seeming strings, I agree
> something needs to be said about this, but I'm not clear where.

I suggested Section 12.3 so that this issue would be mentioned along
with other issues related to normalization.  But if there's a better
place, I'm fine with that.

> One issue that we'll have to address if we do put something about the
> issue into 12.3 is what I call the "précis-fox-fox problem": If you
> have a component name which is the NFD form of "précis" followed by
> the byte 0xff, are you allowed to normalize it to the NFC form of
> "précis" followed by the byte 0xff (nor match that when doing n-I
> lookups".  I'd like the answer to be "NO", but since this chapter is
> to be derived from implementations, can I hear from people familiar
> with the appropriate server/filesystem code?  

I wrote a test program for this and ran it on a default configuration of
Solaris (i.e., the local fs is not requiring UTF-8).  The composed and
decomposed versions are treated as distinct.

>From the documentation, it looks like the only way to get n-I lookups is
to create the fs with the utf8only property set to "on".  And if the
utf8only property is "on", then the name with the 0xff byte is rejected
("illegal byte sequence").

mike