Subject: [nfsv4] symlinks: opaque or UTF-8

From: "Noveck, Dave" <[email-masked]>
Date: Sat, 24 Nov 2007 11:07:32 -0500

Both RFC3530 and the v4.1 spec say about symbolic links:

   The data is a UTF-8 string that is opaque to the server.

The two clauses of this sentence are, if not quite contradictory, at the
very least they are in considerable tension.

If the data is opaque to the server, in what sense can it be described
as "a UTF-8 string"?

linktext4 is defined as a UTF-8 string which might lead to the
conclusion that if the client presents something which is not a UTF-8
string, then NFS4ERR_INVAL should result.  That's what we do with other
stuff defined in the XDR as a UTF-8 string, but I know that I currently
don't do that (for v4.0) and suspect that others don't as well.  I get
the sense from some of the surrounding text that, without being too
explicit about it, implementers are being invited to ignore the
definition of a symbolic link as a UTF-8 string.

So the first question is what do we want clients and servers to do with
regard to symbolic links.

If we want them to be treated as UTF-8 strings:

      Servers would check that they are UTF-8 in CREATE.

      Servers might check that they are UTF-8 in READLINK.

      Clients would endeavour to send UTF-8 strings for CREATE.

      If passed strings for links in a context in which the strings
      would be interpreted according to some other code, then they 
      would, as part of making them UTF-8 strings, translated.

      Similarly on READLINK.

      If clients are doing that, then servers would have to translate
      them to UTF-8 where the filesystem stores names in other than
      UTF-8 format and where the symbolic links, stored for example
      by v2 and v3, are not in UTF-8.

On the other hand, if they are opaque, none of the above applies any 
more than it does to data handled by read and write.  But the data for
read and write is not described as UTF-8 strings.  It is described as 
opaque, end of story.

This is a potential interoperability issue, and I think we want to be
absolutely clear about which of these two we mean.  

If we want them to be treated as UTF-8 strings, then we can be explicit
about what clients and servers have to do and mention the errors that
should be returned if they don't do that (e.g. INVAL if CREATE gets
symlink test that is not UTF-8).  We could say that while the data is
opaque to the server as to content, it is not, as a UTF-8 string, opaque
to the server as to form.

In the other case in which we want this data to be treated as opaque, I
would prefer we simply dropped all talk of this being a UTF-8 string, as
confusing.  The XDR speciifcation of linktext4 raises interesting
issues.  You could argue that we are not allowed, in a minor version, to
change an existing XDR definition. However note that in RFC3530:

    linktext4 is typedef'd as utf8str_cs. (in both the text and .x file
portion)

    utf8str_cs is typedef'd as utf8string. (in the .x file portion, the
text typedefs it incorrectly as opaque).
    
    utf8string is typedef'd as opaque<>. (in the .x file portion, the
text typedefs it incorrectly as opaque).

So consider if we typedef'd linktext4 as opaque<>, which is what it is
effectively typedef'd as in v4.0.

Is it really the case that we are obliged to maintain the exact chain of
definition for existing ops, or only the result, which is that linktext4
winds up being opaque<>?  If the former, I'll bet there are a bunch of
cases in which we have changed the existing chain of definitions.  Could
we not legitimately typedef linktext4 as opaque<> in v4.1?  I think we
could.

If we are stuck with the XDR/stringprep definition of this as UTF-8, the
other way to deal with this if we want the opaque interpretation is to
say:

     The data is intended to be a UTF-8 string.  However, this data
     is opaque to the server and it is not the server's responsibility
     to validate this data as UTF-8, either when creating a symbolic
     link or when reading it.

The trouble with this formulation is that if it isn't the server's
responsibility to make this UTF-8 (and the specfication of this as a
UTF-8 string is something other than pure bullshit) then it is the
client's and if it is the client's, he is obliged to do the
translations, but if he did that, he could not inter-operate with v2 and
v3.  So the client can't accept that responsibility and the
responsibility would have to be the application's.  But we have no
business telling the applications what to do and they wouldn't listen to
us anyway.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Jim Rees <[email-masked]>
Date: Sat, 24 Nov 2007 12:54:58 -0500

Noveck, Dave wrote:

I think the problem is that some clients won't know what character set the
linktext is in, as that is considered an application issue in some older
clients.  It may be useful for the client to send arbitrary text to the
server, knowing that it will only be meaningful for the client that created
the symlink.  I doubt we want to allow this but it's worth some thought.

  We could say that while the data is
  opaque to the server as to content, it is not, as a UTF-8 string, opaque
  to the server as to form.

That was my interpretation.  We should make this explicit.

---

From: "Yoder, Alan" <[email-masked]>
Date: Sat, 24 Nov 2007 10:47:45 -0800

A UTF-8 string is a string that 

  (1) contains no non-UTF-8 character sequences
  (2) can be manipulated with the C string library. 
  (3) has a semantics that may or may not be 
      known to the server
  (4) has other properties as well

I suspect that (2) is about the only thing
that implementations have cared about to date.

However, (1) is pertinent if the string is being
converted to Unicode; are there clients that need
to store soft links to Unicode file names in an
interoperable way?

Alan

> -----Original Message-----
> From: Noveck, Dave 
> Sent: Saturday, November 24, 2007 8:08 AM
> To: [email-masked]
> Subject: [nfsv4] symlinks: opaque or UTF-8
>

---

From: "J. Bruce Fields" <[email-masked]>
Date: Sat, 24 Nov 2007 14:37:41 -0500

> If the data is opaque to the server, in what sense can it be described

One of the fundamental requirements for a symlink is that it should be
able to store any path on the client's filesystem.

The only way I can see to allow that is to allow arbitrary strings to be
stored in them.

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Sat, 24 Nov 2007 15:59:37 -0500

> very least they are in considerable tension.

Given that we make no effort whatsoever to define a portable symlink
type (for instance, nowhere is there a definition of a NFSv4 'path
separator') and hence that we assume the symlink is supposed to be
entirely interpreted by the client, then why bother with the UTF-8
constraint?

Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Sat, 24 Nov 2007 18:41:34 -0500

> That was my interpretation. 

But have you acted on this interpretation?

For example, have you as a server validated the UTF8-ness of data to be
stored in a symlink?

Or as a client, taken data presented to you in a different code and
translated it to UTF8 (below the VOP interface presumably).

I think we have two issues here:

      How do we handle this in v4.1?

      Do we have an interoperability issue in v4.0?

-----Original Message-----
From: Jim Rees [mailto:[email-masked]] 
Sent: Saturday, November 24, 2007 12:55 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] symlinks: opaque or UTF-8

Noveck, Dave wrote:

I think the problem is that some clients won't know what character set
the linktext is in, as that is considered an application issue in some
older clients.  It may be useful for the client to send arbitrary text
to the server, knowing that it will only be meaningful for the client
that created the symlink.  I doubt we want to allow this but it's worth
some thought.

  We could say that while the data is
  opaque to the server as to content, it is not, as a UTF-8 string,
opaque
  to the server as to form.

---

From: "Noveck, Dave" <[email-masked]>
Date: Sat, 24 Nov 2007 18:54:35 -0500

I don't know about (2).  A string of zero bytes seems to obey the rules
of UTF-8, which prescibes the identity mapping for characters below 127
which zero is.  If I present a name that has zero byte in it, is the
correct error NFS4ERR_INVAL or NFS4ERR_BADCHAR? 

> I suspect that (2) is about the only thing that implementations have
cared about to date.

I doubt they have cared about (2) for symlinks.  You have a byte count
and if you have treated it as opaque (like it was treated in v3), I
don't think you would have looked for nulls.

> However, (1) is pertinent if the string is being converted to 
> Unicode; are there clients that need to store soft links to 
> Unicode file names in an interoperable way?

The other is interoperability with v2/v3 implementations that are
writing symlinks in a non-UTF8 way.  To deal with that, you eaither
translate the data to/from UTF-8 or treat it as opaque which makes
interoperability with the v2/v3 implmeentation easy and implementation
with those using Unicode just about impossible.

-----Original Message-----
From: Yoder, Alan 
Sent: Saturday, November 24, 2007 1:48 PM
To: Noveck, Dave; [email-masked]
Subject: RE: [nfsv4] symlinks: opaque or UTF-8

I suspect that (2) is about the only thing that implementations have
cared about to date.

However, (1) is pertinent if the string is being converted to Unicode;
are there clients that need to store soft links to Unicode file names in
an interoperable way?

Alan

> The two clauses of this sentence are, if not quite contradictory, at 
> the very least they are in considerable tension.
> 

> string, then NFS4ERR_INVAL should result.  That's what we do with 
> other stuff defined in the XDR as a UTF-8 string, but I know that I 
> currently don't do that (for v4.0) and suspect that others don't as 
> well.  I get the sense from some of the surrounding text that, without

> being too explicit about it, implementers are being invited to ignore 
> the definition of a symbolic link as a UTF-8 string.
> 
> So the first question is what do we want clients and servers to do 
> with regard to symbolic links.
> 

> If we want them to be treated as UTF-8 strings, then we can be 
> explicit about what clients and servers have to do and mention the 
> errors that should be returned if they don't do that (e.g. INVAL if 
> CREATE gets symlink test that is not UTF-8).  We could say that while 
> the data is opaque to the server as to content, it is not, as a UTF-8 
> string, opaque to the server as to form.
> 
> In the other case in which we want this data to be treated as opaque, 
> I would prefer we simply dropped all talk of this being a UTF-8 
> string, as confusing.  The XDR speciifcation of linktext4 raises 
> interesting issues.  You could argue that we are not allowed, in a 
> minor version, to change an existing XDR definition. However note that

> in RFC3530:
>     utf8str_cs is typedef'd as utf8string. (in the .x file portion, 
> the text typedefs it incorrectly as opaque).
>     

> Is it really the case that we are obliged to maintain the exact chain 
> of definition for existing ops, or only the result, which is that 
> linktext4 winds up being opaque<>?  If the former, I'll bet there are 
> a bunch of cases in which we have changed the existing chain of 
> definitions.  Could we not legitimately typedef linktext4 as opaque<> 
> in v4.1?  I think we could.
> 
> If we are stuck with the XDR/stringprep definition of this as UTF-8, 
> the other way to deal with this if we want the opaque interpretation 
> is to
> say:
> 
> translations, but if he did that, he could not inter-operate with v2 
> and v3.  So the client can't accept that responsibility and the 
> business telling the applications what to do and they wouldn't listen 
> to us anyway.
> 
>  
>

---

From: Mike Eisler <[email-masked]>
Date: Wed, 28 Nov 2007 13:22:06 -0800 (PST)

--- "Noveck, Dave" <[email-masked]> wrote:

> the

This is rooted in how symbolic links were first specified when
4.2 BSD introduced them:

- they were parseable by system calls like open(), stat(), rename(),
  link(), i.e. anything that took a pathname as an argument.

- the symlink() system call that created the symbolic did not impose
  a form on symlink, presumably because symbolic links could
  be relative or absolute, and many relative symbolic links do not
  have component separators ("/" in 4.2 BSD), and remote file access
  protocols were not a consideration.

When NFS came along, we were faced with a situation where clients,
and only clients, interpreted the symbolic link (although the
MOUNT protocol could). Note that the symbolic link type was
of type "path" in NFSv2, and "path" was of type "string", i.e.
US-ASCII. This meant that aclient that used say backslashes as
a component separator could create a symbolic link that if read by
another client that used forward slashes as the component separator
would likely result in ENOENT.

In NFSv3, the type of the symbolic link as still ultimately
an string (nfspath3).

In NFSv4, the type was changed to opaque from string. Given that
component names were changed to a type (utf8str_cs) that derived from
opaque, this was the right thing. And symlinks were defined as
linktest4, which is also a typedef of utf8str_cs.

So any suggestion that we change the meaning of symlinks to anything
but utf8 is dead on arrival, unless it is accompanied by a new
operation to read these new symlinks (that NFS has never
specified before in over 20 years), and another operation to
create them. Minor versioning rules.

OK, so let's just consider what is in RFC3530 and the current
NFSv4.1 as errata.

What do we want to say?

We currently have:

  "The data is
   a UTF-8 string that is opaque to the server.  That is, whether
   created by an NFS client or created locally on the server, the data
   in a symbolic link is not interpreted when created, but is simply
   stored."

What was meant by "opaque to the server". I believe what was
meant is that structure of presumed path name in the symolic
link contents is opaque to the server. E.g. if the contents
are a\b or d/c, the server MUST NOT rejected the former if its
native component name separator is /.

  "The data is
   a UTF-8 string that may or may not be a path name constructed
   in a form that may or may be known to the server.  That is, whether
   created by an NFS client or created locally on the server, the data
   in a symbolic link is not interpreted as a path name when created
   but is simply stored, as long as the string is legal UTF-8,
   and UTF-8 characters used are supported by the server. Nor is the
   symbolic link interpreted as a path name when retried via READLINK.
   A process on a client might interpret the results of READLINK as 
   a path name but whether this interpretation succeeds or not 
   depends on whether the symbolic link is a legal absolute or 
   relative symbolic link on the client's operating environment.

In the IMPLEMENTATION section we could have:

   A server might store a symbolic link in an encoding other than
   UTF-8. If so, the server MUST convert the contents of the
   symbolic link to UTF-8 before returning READLINK's result.
   If a server supports a file access protocol other than
   NFSv4.0 and NFSv4.1, such a file access protocol might
   create symbolic links using non-UTF-8 characters. If so,
   it is RECOMMENDED the server record the identity of the character
   set used when the symbolic link is created, so that conversion to
   UTF-8 is possible, and thus interoperability betwenn non
   NFSv4 and NFSv4 clients is possible. Some server implementations
   might assign the character set identity on a per-fsid basis.

   In particular, NFSv3 is a file access protocol that while
   strictly speaking uses only characters that are a proper subset
   of UTF-8 (US-ASCII: 7 bit characters, with the highest order
   8th bit always equal to zero), in practice can non-US-ASCII
   characters. If the NFSv3 client creates symbolic links with
   UTF-8 characters, and the fsid's character set is also UTF-8,
   interoperability is trivial. If the fsid's character set is
   not UTF-8, then the NFSv3 client needs to create symbolic links
   using the character set of the fsid, or the server needs to have
   a method for converting from the client's character set to that
   of the fsid.

> other
> explicit
> I
> of
> the
> and
> to

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 28 Nov 2007 17:10:17 -0500

On Wed, Nov 28, 2007 at 01:22:06PM -0800, Mike Eisler wrote:
> > In the other case in which we want this data to be treated as
> > opaque, I would prefer we simply dropped all talk of this being a
> > UTF-8 string, as confusing.  The XDR speciifcation of linktext4
> > raises interesting issues.  You could argue that we are not allowed,
> > in a minor version, to change an existing XDR definition. However
> > note that in RFC3530:
> > 
> > So consider if we typedef'd linktext4 as opaque<>, which is what it
> > is effectively typedef'd as in v4.0.
> > 
> > Is it really the case that we are obliged to maintain the exact
> > chain of definition for existing ops, or only the result, which is
> > that linktext4 winds up being opaque<>?  If the former, I'll bet
> > there are a bunch of cases in which we have changed the existing
> > chain of definitions.  Could we not legitimately typedef linktext4
> > as opaque<> in v4.1?  I think we could.

That would be great....

Do any servers currently guarantee that they return utf-8 data in
symlinks?  Linux's certainly doesn't, and won't for the forseeable
future, because:

	- Popular Linux on-disk filesystems treat paths as arbitrary
	  binary data (with only NULL and "/" being special).

	- We want the Linux server to be useful to Linux clients.

	- That means letting them store paths to non-nfsv4 filesystems
	  in symlinks on nfsv4 filesystems.  They won't be able to do
	  that reliably if we insist on utf-8 symlinks.

I just don't see a way out of the above.  Help!

--b.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 28 Nov 2007 16:29:54 -0600

> symlinks?  Linux's certainly doesn't, and won't for the forseeable

MacOS X might (it normalizes to NFD on create in NFSv3, and Leopard,
IIRC, has NFSv4 support).  Solaris doesn't [yet], but nowadays
OpenSolaris/Solaris Nevada does have the necessary framework for dealing
with UTF-8 in NFSv4, and so we would have to know how to deal with
symlinks...  soon.

That said, symlinks can and do contain non-pathname contents in many
cases.  Restricting symlinks to UTF-8 may not be the best thing to do.

But when a client follows a symlink it will have to know what to do with
any non-UTF-8 sequences therein -- the only reasonable thing to do, at
first blush, in such cases is to return an error to the application.

But there's the possibility that a symlink might have a pathname that
contains codepoints that are, from a client's p.o.v., unassigned.
Should a client try to follow symlinks with such pathnames?  I think the
answer is "yes."

My proposal:

 - let symlinks be opaque<>

 - recommend that clients refuse to follow symlinks with illegal UTF-8
   octet sequences

 - recommand that clients do follow symlinks with unassigned codepoints,
   provided there are no illegal UTF-8 octet sequences in them

> 	  binary data (with only NULL and "/" being special).

Let's call this behaviour "just-use-8" for simplicity (we use it
elsewhere).

> 	  that reliably if we insist on utf-8 symlinks.

Symlinks are the _least_ of your problem if you have just-use-8 clients
and servers.

> I just don't see a way out of the above.  Help!

See:

http://blogs.sun.com/nico/entry/filesystem_i18n

Short version:

 - Your best bet on the server is to treat all UTF-8-looking octet
   sequences as per NFSv4/NFSv4.1, have a knob to reject/allow
   non-UTF-8, and implement normalization-insenstive LOOKUPs.

 - On the client side you should refrain from using non-UTF-8 locales.
   Alternatively you should implement codeset conversions in your
   libc/glibc system call stubs when in non-UTF-8/ASCII/C locales.

 - Provide tools for finding and converting legacy filesystem contents.

Nico
--

---

From: Mike Eisler <[email-masked]>
Date: Wed, 28 Nov 2007 15:10:41 -0800 (PST)

> 	  binary data (with only NULL and "/" being special).

So what happens when a process opens the symlink? What character
set is used to process the pathname? If the path to the symlink is
 "/mnt/a/b/symlink"
and a/b/symlink are on the serveer, doesn't the client expect the
character set of what's in "symlink" to match that of "a/b/symlink"?

> 	  that reliably if we insist on utf-8 symlinks.

Don't you have the same problem with path names to file objects?
What if a client wants to create a file with non-UTF-8 characters?
What does the Linux server do?

It seems to me that if the creating files with UTF-8 names
is not a problem, creating symlinks with UTF-8 content is not 
problem.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 28 Nov 2007 18:42:31 -0500

On Wed, Nov 28, 2007 at 03:10:41PM -0800, Mike Eisler wrote:
> set is used to process the pathname?

I don't understand what you mean by "use a character set to process a
pathname".  The system calls and posix api's that deal with paths just
take char *'s.  They aren't given anything that would identify the paths
as using a particular character set.

> If the path to the symlink is
>  "/mnt/a/b/symlink"

No, The client just takes a the contents of symlink, splits it at the
"/" characters, and does lookups for each component.  Those components
may fall into different filesystems with different ideas about what a
legal pathname component looks like.

> What does the Linux server do?

We give it to the filesystem and let it decide.  If you want a fully
NFSv4-compliant(TM) server then you should be exporting a filesystem
that enforces the utf-8 requirement.

Absent enforcement from the filesystem, I don't see any hope of making
utf-8 restrictions work in the presence of v2/v3 clients and local
processes.

> is not a problem, creating symlinks with UTF-8 content is not 

It's a different problem.  Even if we could demand that any filesystem
that we exported require utf-8, we couldn't demand that none of our
clients mount a filesystem with non-utf8 pathnames.

--b.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 28 Nov 2007 15:58:50 -0800 (PST)

--- "J. Bruce Fields" <[email-masked]> wrote:

> pathname".  The system calls and posix api's that deal with paths

I was curious if 16 bit or 32 bit characters were being
used.

> just
> "/" characters, and does lookups for each component.  Those

OK, I get it.

> components

So your server's READDIR will return directory entries that
aren't legal UTF-8?

> clients mount a filesystem with non-utf8 pathnames.

I don't follow how it is different. You seem to be saying that
regardless what the specification says, your server treats
component names as 8 bit opaque characters.

How do you expect clients to display the right glyphs for a character
if the server renders illegal utf-8?

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 28 Nov 2007 18:15:31 -0600

> > > set is used to process the pathname?

Usually processes _follow_ symlinks without even knowing they are there.

But when the read the symlink they should get the contents of it as it
exists on the server.

If two users are exchanging non-pathname data via symlinks and they
don't agree on codesets, well, too bad.

If two users are exchanging pathname data via symlinks and they don't
agree on codesets, then as long as the pathname in the symlink exists
then it should be possible to traverse it _provided_ that the client
following the symlink doesn't transform the symlink's contents in any
way.

> I was curious if 16 bit or 32 bit characters were being

Huh?  Oh boy, no, you really need to define a character width and
codeset for use between the system call layer and the filesystem.
Just-use-8 will work provided all users are using UTF-8 locales.  And
just-use-8 will work for US-ASCII names provided that all users are
using single-byte or variable-width codeset locales that are US-ASCII
compatible (ISO-8859-*, UTF-8, ...).  But throw in UTF-16 or UTF-32
locales and I think you've got big problems in a just-use-8 system.

> So your server's READDIR will return directory entries that

Clients need to be ready to deal with such servers, because legacy
filesystem contents exists.  Servers might well reject CREATEs of
non-UTF-8 names while still allowing LOOKUPs of and READDIRs that
include existing non-UTF-8 names.

> > It's a different problem.  Even if we could demand that any
> > filesystem that we exported require utf-8, we couldn't demand that
> > none of our clients mount a filesystem with non-utf8 pathnames.
> 

It is different because symlinks have been defined from the get-go as
being allowed to contain non-pathname data.  Therefore constraining them
to contain UTF-8 seems a bit harsh (though probably livable).

> if the server renders illegal utf-8?

"renders"?

The answer is: the way clients generally tend to do in many other cases,
namely, by replacing them with question marks or some other appropriate
glyph.

Nico
--

---

From: Mike Eisler <[email-masked]>
Date: Wed, 28 Nov 2007 16:17:42 -0800 (PST)

> cases.  Restricting symlinks to UTF-8 may not be the best thing to

So when an app like the ls command reads a symlink and displays
its contents, what does it display when it finds non-utf-8 in it?
 
> first blush, in such cases is to return an error to the application.

What does one do at second blush?

>  - let symlinks be opaque<>

why stop there?

>  - Provide tools for finding and converting legacy filesystem

The problem is that NFSv3 implemented to be 8 bit clean. 
Many, if not most, non-ASCII deployments of
NFSv3 don't use UTF-8. Lots of data today is stored on NAS
devices that allow access via file access protocols like CIFS
that do mandate things like Unicode, no weaseling out permitted.

What to do?

- declare the character set for the fsid

- convert from the character set to Unicode if using
  CIFS

- convert from the character set to UTF-8 if using
  NFSv4

- for NFSv3, if the declared character set uses 8 bit characters
  do not convert. If the declared character set uses 16 bit or
  higher characters, convert to UTF-8 and hope for best.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 28 Nov 2007 19:19:17 -0500

On Wed, Nov 28, 2007 at 03:58:50PM -0800, Mike Eisler wrote:
> > filesystem
> I don't follow how it is different.

The difference is:

	- In one case I just need to insist that all the paths on
	  filesystems I export be utf-8.

	- In the symlink case, I need to insist that all the paths on
	  all the *other* filesystems that my client mounts *also* be
	  utf-8, whether those other filesystems are nfsv4 or ext3.  At
	  least if they want symlinks to work.

Symlinks on an nfsv4-mounted /home should be able to refer to paths on
an ext3-mounted /.

--b.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 28 Nov 2007 18:25:12 -0600

> 	  filesystems I export be utf-8.

All new paths.  In practice you'll have to allow legacy objects to
retain their legacy names.

> 	  least if they want symlinks to work.

Users already need to create symlinks with valid pathnames when that is
what they want.  If a user is running in a non-UTF-8, non-ASCII locale
and they cut-and-paste what should be a UTF-8 path but which has been
codeset converted to another locale, well, they'll create a bogus
symlink.  This is true.

The only way out of that is to have the client convert back to UTF-8
when creating the symlink.  In such an environment the client would have
to be converting to/grom UTF-8 in lots of other places, so might as
well.

That doesn't mean that the spec should REQUIRE that symlink content be
UTF-8 -- it need only REQUIRE or RECOMMEND that symlink content that is
intended to be traversable as a pathname be in the correct codeset,
namely (for an all NFSv4/UTF-8 environment) UTF-8.

> Symlinks on an nfsv4-mounted /home should be able to refer to paths on

Off-topic: can't you use UTF-8 in ext3 filesystems?

Nico
--

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 28 Nov 2007 19:28:56 -0500

> So your server's READDIR will return directory entries that

Yep, it may.  I'm not particularly proud of it.  It's just the least
worst option I see given the requirements and the available time.

(Responding separately because I really think it is a separate
issue--one is about imposing requirements on the server's own path
components, the other is about imposing requirements on client-side
paths which may not touch nfsv4 filesystems at all.)

--b.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 28 Nov 2007 16:29:43 -0800 (PST)

> include existing non-UTF-8 names.

So if servers can return non-UTF-8 in readdir and readlink,
then what's the point?

Either a server can cope with UTF-8 or not.

If it cannot, and is going to ignore the spec, then what difference
does it make if we mandate UTF-8? Bruce will deal with his
unsolvable problem by returning non-UTF-8 every where, and
allowing non0UTF-8 every where. If that's what his server does
and if clients are OK with that, why should we care?

It's not like anyone cared until Dave kicked the hornets' nest.

UTF-8 is a tool, use it or not, but I don't see why we should
weaken the spec because some cannot or will not use it.

> to contain UTF-8 seems a bit harsh (though probably livable).

Sorry, I don't see the difference. User names in NFSv4 don't
have path names data and they are mandated to have UTF-8 
(not that I have any faith that all NFSv4 user names are 
encoded in UTF-8).

> namely, by replacing them with question marks or some other

Well that's useful. :-(

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 28 Nov 2007 19:37:52 -0500

> codeset converted to another locale, well, they'll create a bogus

That's not the example that concerns me.

If some file on another filesystem only exists under a path that
contains non-utf8 characters, how will a user create a symlink to it on
an nfsv4 filesystem?

> Off-topic: can't you use UTF-8 in ext3 filesystems?

Sure.  The filesystem doesn't know or care that you're using utf-8,
though.

--b.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 28 Nov 2007 18:41:56 -0600

> contains non-utf8 characters, how will a user create a symlink to it on

If we define symlink contents as opaque then it should just work, modulo
local implementation details.

> Sure.  The filesystem doesn't know or care that you're using utf-8,

But it might.  What stops you from having an option to reject non-UTF-8
creates?

Nico
--

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 28 Nov 2007 19:49:27 -0500

> and if clients are OK with that, why should we care?

OK with me.

I guess I took for granted the original claim that there was an
interoperability problem if some implementations treated symlinks as
utf-8 and some didn't.  If that's true, then the spec should state that
many (most?  all?) servers don't actually do this, as that's the
information that will help future implementations interoperate with the
ones that are actually deployed.

But probably the interoperability problems aren't a big deal, fair
enough.

Other than the obvious "I tried to cp this symlink from A to B, and B
won't accept it", which I suppose has already existed for filenames for
a while.  Seems a silly to me, but whatever.

--b.

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 28 Nov 2007 21:38:17 -0500

> interoperability problem if some implementations treated symlinks 
> as utf-8 and some didn't.  If that's true, then the spec should 
> state that many (most?  all?) servers don't actually do this, as 
> that's the information that will help future implementations 
> interoperate with the ones that are actually deployed.

Huh?  How can the spec doing a meaningful survey of v4.1 servers
at this point, which is when we are ready to close on it.  We 
could do a survey of v4.0 servers but the v4.0 spec is already
done.

Anyway the v4.1 spec is not intended to be an Informational RFC.
Its job is to define what the clients and servers should do 
in order to interoperate.  If some 75% of the servers are 
treating this as opaque then stating that would help client be
compatible with 75% of the servers but that is not good.
The idea of interoperablity is that you can be compatible
with 100%, if we all obey the protocol. 

> But probably the interoperability problems aren't a big deal, 
> fair enough.

They are a big deal.  If clients don't treat this data as
UTF-8, servers can't either, and vice versa.  The thing we have
to decide is whether that is a desirable (or unavoidable) state
of affairs.

I think Mike should have been explicit about his "<sarcasm>"
and "</sarcasm>" tags.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Wednesday, November 28, 2007 7:49 PM
To: Mike Eisler
Cc: [email-masked]
Subject: Re: [nfsv4] symlinks: opaque or UTF-8

On Wed, Nov 28, 2007 at 04:29:43PM -0800, Mike Eisler wrote:
> does it make if we mandate UTF-8? Bruce will deal with his unsolvable 
> problem by returning non-UTF-8 every where, and allowing non0UTF-8 
> every where. If that's what his server does and if clients are OK with

> that, why should we care?

OK with me.

--b.

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 28 Nov 2007 20:40:41 -0600

>> and if clients are OK with that, why should we care?

Okay with me as well.

> ones that are actually deployed.

This should not be in the specification.  THe NFSv4.1 specification
is not even complete so there are no implementation in existence
and the decision about will be implemented will change over time
so any statement about current capabilities will be stale.

Spencer

---

From: Mike Eisler <[email-masked]>
Date: Wed, 28 Nov 2007 19:18:37 -0800 (PST)

> I think Mike should have been explicit about his "<sarcasm>"

I'm not being sarcastic.

The protocol has an i18n requirement. No one has shown how we can 
meet the i18n requirement and cope with servers that have no clue what
character set their metadata is in. Given that there are servers that
can meet the requirement, why should we water down the protocol, and
make interoperability even harder?

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 28 Nov 2007 21:31:51 -0600

> make interoperability even harder?

Yes, we need to move forward not backwards.

Spencer

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 28 Nov 2007 22:38:09 -0500

> I'm not being sarcastic.

Then I'm totally confused.

> why should we water down the protocol, and make interoperability even
harder? 

Can you stop arguing for your position for a second and focus on making
it clear what it is?

So far, the only thing I had been sure of in this debate is that you and
Bruce disagree on what should be done so when you argue for your
position (with or without sarcasm) and Bruce's response is "OK with me",
I'm at a total loss.

Bruce is has been basically saying that we should treat this as opaque
and that doesn't sound to me like what you are saying, unless I am
totally confused, which is possible.

So help me:

     Bruce, what is it that you are OK with?

     Spencer, what is it that you are OK with?

     Mike, what is it that Spencer and Bruce are OK with? :-)

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Wednesday, November 28, 2007 10:19 PM
To: [email-masked]
Subject: RE: [nfsv4] symlinks: opaque or UTF-8

I'm not being sarcastic.

The protocol has an i18n requirement. No one has shown how we can meet
the i18n requirement and cope with servers that have no clue what
character set their metadata is in. Given that there are servers that
can meet the requirement, why should we water down the protocol, and
make interoperability even harder?

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 28 Nov 2007 21:54:46 -0600

>      Spencer, what is it that you are OK with?

Symlink text (linktext4) remains utf8str_cs just as with
other filesystem object names, component4, are required to be.

If the server or client have issues with UTF8 then they translate
to or from as appropriate for their environments but what flows
over NFSv4.1 is UTF8.

Or in other words, I am agreeing to the text that Mike proposed
earlier in this thread.

Spencer

---

From: Mike Eisler <[email-masked]>
Date: Wed, 28 Nov 2007 20:30:24 -0800 (PST)

>      Mike, what is it that Spencer and Bruce are OK with? :-)

I believe Spencer and Bruce are OK with the NFSv4.1 spec mandating
that the server return components and pathnames always in UTF-8
encoding, whether from READLINK, READDIR, or anything other operation 
that returns UTF-8 data types. 

Bruce is saying he cannot actually implement a Linux server to do this
all the time. I believe Bruce and Spencer are OK with the fact Bruce's
server won't comply with the specification. 

Any possible interest I had in changing the spec to
accommodate Bruce evaporated when he
said that his READDIR implementation returned non-UTF-8 component
names. So for that reason, I am OK with Bruce's server not returning
UTF-8. 

As an aside, UCS-4 has codes for characters numbering from 0x80
through 0xFF does it not? Why can't Bruce's "just-use-8" 
server convert bytes with values 0x80 through 0xFF into their
corresponding UTF-8 two character sequences in the results, 
and similarly accept the UTF-8 two character
sequences that correspond to 0x80 through 0xFF in requests?

I.e., UCS characters 00000080 through 000007FF
are converted to: 110xxxxx 10xxxxxx. 0x80 would then be
  11000010 10000000 or C2 80.
0xFF would be 
  11000011 10111111 or C3 BF

Bruce's server would then reject any UTF-8 encoding in a request
not "between" C2 80 and  C3 BF with NFS4ERR_BADCHAR.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 29 Nov 2007 00:03:37 -0500

> not "between" C2 80 and  C3 BF with NFS4ERR_BADCHAR. 

The standard unix definition of a symlink does not allow for any form of
server proofreading of the string contents. It is entirely up to the
user/application on the client to define something that it thinks will
work. See

http://www.opengroup.org/onlinepubs/009695399/functions/readlink.html

and 

http://www.opengroup.org/onlinepubs/009695399/functions/symlink.html

In fact, the symlink() operation explicitly states that "The string
pointed to by path1 shall be treated only as a character string and
shall not be validated as a pathname.". Note that it does not state that
you are allowed to validate it in terms of a particular character set,
or that you are required to stringprep it either.

Any attempt to require that NFSv4 must stringprep is therefore likely to
confuse applications which will see that the string is either being
refused by the server with an error code that is not sanctioned by the
POSIX/Single Unix Spec for symlink(), or is not being returned correctly
verbatim by readlink().

Trond

---

From: Mike Eisler <[email-masked]>
Date: Wed, 28 Nov 2007 22:12:29 -0800 (PST)

--- Trond Myklebust <[email-masked]> wrote:

> shall not be validated as a pathname.". Note that it does not state

Who wrote on this thread that NFSv4.1's CREATE operation
should validate the symlink as a path.

It doesn't say anything about whether the
character set can be validated or not, and even if it did, that
would be irrelevant, because we are specifying a networking protocol,
and are free to specify the encoding anyway we want.

For example, ever since NFSv2, all pathnames and components can
legally include nul ASCII characters, even though this violates
POSIX. NFS != POSIX.

So everything you claim about readlink and symlink would apply
to open, link, stat, etc., and therefore UTF-8 would not be appropriate
for OPEN, LINK, LOOKUP, etc.

> that

I don't care about stringprep one way or the other.
IESG forced in on us, but if it gets in the way, don't use it.
Normalization strikes me as complex or stupid or both.
If you want start a separate thread on stringprep, go ahead,
we'll see how much enthusiasm there is this time to address it.

But this thread is about UTF-8.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 29 Nov 2007 01:16:08 -0600

> its contents, what does it display when it finds non-utf-8 in it?

The actual contents.  It's up to the terminal to do something
interesting with unexpected octets.

> > But when a client follows a symlink it will have to know what to do
> > with any non-UTF-8 sequences therein -- the only reasonable thing to
> > do, at first blush, in such cases is to return an error to the
> > application.
> 

By "at first blush" I meant that I hadn't taken unassigned codepoints
into consideration (but I wrote about those right below).

> >  - let symlinks be opaque<>

Because symlink content need not be filesystem object names or paths.

Filesystem object names should always be in UTF-8.  We need to allow for
legacy filesystem contents -- it's a reality, so really we need to
restrict at least CREATEs to UTF-8 only.

> The problem is that NFSv3 implemented to be 8 bit clean. 

And many NFSv4 implementations are also just-use-8.

> Many, if not most, non-ASCII deployments of NFSv3 don't use UTF-8.

Indeed.

> Lots of data today is stored on NAS devices that allow access via file
> access protocols like CIFS that do mandate things like Unicode, no
> weaseling out permitted.

Yup.

> What to do?

There's a number of things that can be done.

> - declare the character set for the fsid

I don't think this works well.  You might not know what that codeset
should be, or there may be many in use.

> - convert from the character set to Unicode if using CIFS

This doesn't work well for the same reasons as above.  But if you do
know that a single codeset is in use for a given fs then doing the
codeset conversions in the filesystem itself is the best bet.

> - convert from the character set to UTF-8 if using NFSv4

Same as above.

>   higher characters, convert to UTF-8 and hope for best.

Sure.

Some additional options:

 - just-send-8 for existing non-UTF-8 fs object names and hope for the
   best

 - force the user to convert legacy fs object names

 - perform potentially lossy conversions (e.g., convert all non-ASCII,
   non-UTF-8 octet sequences to question marks, or encode them as
   backslashed octal) and hope for the best (i.e., when the user notices
   they might then go convert the names' codesets)

But this is about legacy fs object names, and that's off-topic for this
thread, which was about symlink contents.  As far as symlink contents is
concerned, using opaque is fine, and probably even the best thing to do,
with the caveats I mentioned earlier.  OTOH, I won't cry if symlink
content is required to be UTF-8.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 29 Nov 2007 01:27:15 -0600

On Wed, Nov 28, 2007 at 04:29:43PM -0800, Mike Eisler wrote:
> > > > We give it to the filesystem and let it decide.  If you want a
> > > > fully NFSv4-compliant(TM) server then you should be exporting a
> > > > filesystem that enforces the utf-8 requirement.
> > > 

Readlink is distinct from READDIR because READIR deals in fs object
names, but readlink deals in symlink contents which need not have
anything to do with fs object naming.

> Either a server can cope with UTF-8 or not.

Servers should definitely reject non-UTF-8 in CREATEs.

> that, why should we care?

Dealing with UTF-8 has been as much a simple matter of specs as of
programming.

I don't support "allowing non-UTF-8" everywhere; I didn't say or imply
that I did.

> weaken the spec because some cannot or will not use it.

Requiring UTF-8 on create, supporting normalization-insensitive LOOKUPs
(or normalization on CREATE), those are all simple matters of
programming (hard, but not all that hard).  Converting legacy
filesystem, OTOH, content is not a simple matter of programming -- it
requires user input, possibly *lots* of user input.

I think the spec should deal with legacy content.  One way would to
allow the server to indicate that some fsid contains non-UTF-8 names.
I described other alternatives just before this post.

> > It is different because symlinks have been defined from the get-go
> > as being allowed to contain non-pathname data.  Therefore
> > constraining them to contain UTF-8 seems a bit harsh (though
> > probably livable).
> 
> Sorry, I don't see the difference. User names in NFSv4 don't have path
> names data and they are mandated to have UTF-8 (not that I have any
> faith that all NFSv4 user names are encoded in UTF-8).

Symlink content need be neither usernames nor pathnames.  They can be
anything, except that they can't contain '\0'.  Historically anyways.
Yes, we could limit them to UTF-8.  Not limiting them to UTF-8 !=
throwing I18N out the window (which you seem to be arguing).

> > > How do you expect clients to display the right glyphs for a
> > > character if the server renders illegal utf-8?
> > 
> > "renders"?
> > 
> > The answer is: the way clients generally tend to do in many other
> > cases, namely, by replacing them with question marks or some other
> > appropriate glyph.
> 
> Well that's useful. :-(

Legacy content is painful to deal with, but it is useful to the user
(otherwise we'd not have legacy content to deal with).

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 29 Nov 2007 01:35:49 -0600

On Wed, Nov 28, 2007 at 10:12:29PM -0800, Mike Eisler wrote:
> Who wrote on this thread that NFSv4.1's CREATE operation should
> validate the symlink as a path.

Noone.

> It doesn't say anything about whether the character set can be
> validated or not, and even if it did, that would be irrelevant,
> because we are specifying a networking protocol, and are free to
> specify the encoding anyway we want.

NFSv4/4.1 impose a stringprep requirement on symlink contents that is
not compatible with the definition of symlinks.  IN PRACTICE I suspect
noone will care -- non-pathname content uses of symlinks that I know
about are all about application locks and aren't likely to run into
codeset issues.

> legally include nul ASCII characters, even though this violates

True.  But there's no need to gratouitously diverge either.

> So everything you claim about readlink and symlink would apply to
> open, link, stat, etc., and therefore UTF-8 would not be appropriate

The definition of symlink content sets symlinks apart from the others.

> IESG forced in on us, but if it gets in the way, don't use it.

I would force it on you also.  That doesn't mean that it's appropriate
for every string slot in a protocol.

> Normalization strikes me as complex or stupid or both.

It's complex, but not stupid.  It may arise from what you might think of
as stupid decisions in the past, but the past is what it is.

> we'll see how much enthusiasm there is this time to address it.

Stringprep should stay in, with unspecified normalization as in NFSv4.0.

> But this thread is about UTF-8.

I thought it was about symlink contents.

> > Any attempt to require that NFSv4 must stringprep is therefore
> > likely to confuse applications which will see that the string is
    ^^^^^^
Well, I don't think it's likely.  Possible, yes.  That's why I wouldn't
mind if symlink content were required to be UTF-8, but I think the
natural thing to do is exempt it.

> > either being refused by the server with an error code that is not
> > sanctioned by the POSIX/Single Unix Spec for symlink(), or is not
> > being returned correctly verbatim by readlink().

Nico
--

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 29 Nov 2007 06:36:24 -0500

Since I don't have to interoperate with Bruce's server, he can violate
the spec all he wants and I won't mind.

The problem is if clients need to adapt to Bruce's server, and as a
result have problems interoperating with a server that does obey the
protocol.

We can't stop people from not obeying the ptotocol, but I want to make
sure we are not somehow agreeing to write down that the link has to be
UTF-8, with the tacit understanding that Bruce (and others?) will treat
it as opaque.

If Bruce violates the protocol as far as READDIR then that's an OK
analogy fro READLINK, but does he violate it on LOOKUP?  If so, and
clients are adapting to that we have a problem.  Simlarly, are we
supposing he will violate it on CREATE?  It this is just a legacy thing,
then fine. 

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Wednesday, November 28, 2007 11:30 PM
To: Noveck, Dave; [email-masked]; [email-masked]
Subject: RE: [nfsv4] symlinks: opaque or UTF-8

>      Mike, what is it that Spencer and Bruce are OK with? :-)

I believe Spencer and Bruce are OK with the NFSv4.1 spec mandating that
the server return components and pathnames always in UTF-8 encoding,
whether from READLINK, READDIR, or anything other operation that returns
UTF-8 data types. 

Any possible interest I had in changing the spec to accommodate Bruce
evaporated when he said that his READDIR implementation returned
non-UTF-8 component names. So for that reason, I am OK with Bruce's
server not returning UTF-8. 

As an aside, UCS-4 has codes for characters numbering from 0x80 through
0xFF does it not? Why can't Bruce's "just-use-8" 
server convert bytes with values 0x80 through 0xFF into their
corresponding UTF-8 two character sequences in the results, and
similarly accept the UTF-8 two character sequences that correspond to
0x80 through 0xFF in requests?

I.e., UCS characters 00000080 through 000007FF are converted to:
110xxxxx 10xxxxxx. 0x80 would then be
  11000010 10000000 or C2 80.
0xFF would be
  11000011 10111111 or C3 BF

Bruce's server would then reject any UTF-8 encoding in a request not
"between" C2 80 and  C3 BF with NFS4ERR_BADCHAR.

---

From: Mike Eisler <[email-masked]>
Date: Thu, 29 Nov 2007 11:58:41 -0800 (PST)

> Because symlink content need not be filesystem object names or paths.

The majority of the time they are. We should design for the 
situation where an app is using symlink for something better served
by a named attribute?

Suppose the locale on the client is set to something other
than US-ASCII or UTF-8. A process on the client creates a symlink
containing path component separators, and 8 bit characters with the
higher bit set. Another process using the same locale issues an
open() system call on a pathname to symlink, and per
the NFSv4.1 spec, the path *to* the symlink is UTF-8 (so
the char set of locale is converted to UTF-8). How does the NFS
filesystem on the client cope with a logical pathname composed of a
combination of UTF-8 and "just-use-8"? How does the NFS filesystem on
the client even know char set the second half of the logical pathname
is?

> restrict at least CREATEs to UTF-8 only.

Legacy content in the range 0x80 to 0xFF can be converted to
to UTF-8.

> should be, or there may be many in use.

I'm not sure if most of my customers would agree with you.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 29 Nov 2007 15:09:38 -0500

> the client even know char set the second half of the logical pathname

So how, in this scenario, does the NFS filesystem get to figure out what
locale the application is using?

> Legacy content in the range 0x80 to 0xFF can be converted to

That also assumes that the kernel knows what charset the app was using.
Otherwise you are converting blindly 'just-use-8' to UTF-8.

Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 29 Nov 2007 15:10:21 -0500

> situation where an app is using symlink for something better served

This may be a rare exception, but normally yes.  We should design to
support existing applications and API's, and not consider ourselves
absolved of that responsibility because we think they should have used
some other facility (and note that servers need not support named
attributes and client need not provide an API to access them on any
partiuclar OS).

---

From: Mike Eisler <[email-masked]>
Date: Thu, 29 Nov 2007 12:12:52 -0800 (PST)

> result have problems interoperating with a server that does obey the

I see. So my idea of Bruce converting bytes 0x80 through 0xFF won't 
work in general.

So ...

If Bruce accepts UTF-8 on LOOKUP, OPEN, CREATE (including
symlink contents), and returns the UTF-8
the client "wrote" to a directory or symlink via
OPEN and CREATE, then the client can operate the same way to his 
server and a compliant server. 

> UTF-8, with the tacit understanding that Bruce (and others?) will

If Bruce treats it as opaque, then Bruce will accept UTF-8.

> through

---

From: Mike Eisler <[email-masked]>
Date: Thu, 29 Nov 2007 12:15:58 -0800 (PST)

> locale the application is using?

The locale is kept in an LC_* environment variable.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 29 Nov 2007 14:23:18 -0600

> locale the application is using?

First define some suitable boundaries, define what codeset is used
between each, and the answers become clear.  You can push all codeset
conversions into the NFSv4 client module (in a modular system) if you
like, but then you need to pass codeset information up and down the
operating system stack between the application and the filesystem.  Or
you can avoid having to track user-land codeset information in kernel-
land by pushing codeset conversions to the edges of the operating system
(e.g., the system call layer and the VFS<->filesystem interface).  Or
you can just get rid of non-UTF-8 locales.  All of these choices present
various difficulties and/or performance issues, but all are workable.

> That also assumes that the kernel knows what charset the app was using.

Right, and that's not necessarily the case.

A big homedir server in a cosmopolitan company might have content in
many codesets.

> Otherwise you are converting blindly 'just-use-8' to UTF-8.

Either you can't do that at all or you risk a lossy conversion.

Nico
--

---

From: Mike Eisler <[email-masked]>
Date: Thu, 29 Nov 2007 12:38:47 -0800 (PST)

> attributes and client need not provide an API to access them on any

OK, I set my LC_ environment variable to a non-UTF-8, non-ASCII
locale. I issue the symlink() system call using the character set
corresponding to LC_. I then issue an open() on that symlink.
My NFSv4.1 client follows the link, has to issue OPEN in UTF-8 and 
it finds that the symlink content is not legal UTF-8. What do I send
to the server.

If symlinks are opaque, then this means:

- We are mandating processes on clients use the UTF-8 locale.
  They can't even use a supposedly UTF-8 compatible locale like
  UCS-2 or UCS-4.
  
   OR

- they can use non-UTF-8 locales, but they cannot use
  symlinks to for path redirection, unless the symlinks
  contain just 7bit US-ASCII.

All for the sake of rare exceptions.

> > Filesystem object names should always be in UTF-8.  We need to
> allow
> > for

---

From: Jim Rees <[email-masked]>
Date: Thu, 29 Nov 2007 15:43:29 -0500

Mike Eisler wrote:

Is it?

% printenv |grep LC
%

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 29 Nov 2007 15:46:31 -0500

> various difficulties and/or performance issues, but all are workable.

Right, but the point is, that that requirement has been in force since
RFC3530, for path components names.  The only question we are discussing
is whether it should be extended to symlinks, but even it isn't, if you
implement a v4.x client you have to have one of the mechanisms you
describe
to perfore the conversions to and from UTF-8 (or have to ban any use of 
non-UTF8).  Or permaps I should switch tenses and say if you have
implemented
a v4.x client, you must have already done one of those things. 

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]] 
Sent: Thursday, November 29, 2007 3:23 PM
To: Trond Myklebust
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] symlinks: opaque or UTF-8

allow

Nico
--

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 29 Nov 2007 15:49:43 -0500

> The locale is kept in an LC_* environment variable.

Sorry, but those environment variables are not, and have never been,
part of the official Linux syscall interface. Any information they may
contain cannot be used in the filesystem code.
Even if it could, then that would break locale-savvy applications, which
use the glibc setlocale() interface: the latter does not change the
environment variables, nor does it call down into the kernel.

Basically, our syscall interfaces take character string arguments, not
wide character strings, and not multi-byte character strings.

Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 29 Nov 2007 15:57:22 -0500

I'm baffled.  Forget symlinks (How I wish I could).

Someone running a non-UTF8 locale issues a remove or open or rename.
You get a string that is not UTF-8.  You have an NFSv4 client that sends
requests to a server that expects UTF-8 strings (he sends you
NFS4ERR_INVAL if you send them, just as RFC3530 says he should.  I
assume you need to translate them to UTF8 so you can do the required
operation rather than return EINVAL and listen to him scream.  How do
you do it?

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Thursday, November 29, 2007 3:50 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] symlinks: opaque or UTF-8

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 29 Nov 2007 16:05:18 -0500

> operation rather than return EINVAL and listen to him scream.  How do

We don't. We never have, and we never will.

The NFS client code faithfully sends whatever the user sends down in the
syscall, and returns faithfully whatever the server sends us in its
READDIR/READLINK/... reply. There is no stringprep, there is no locale
conversion other than whatever was done in userland by the application.
If the server chooses to enforce a particular encoding, then that is
entirely up to it.

The reason why I'm objecting in the particular case of symlinks, is that
symlinks refer to the _client_ namespace. They are inherently
non-portable, because on my machine, I can mount whatever I like,
whereever I like, and that changes the meaning of the symlink. That is
what makes symlinks special, and is why I don't agree with the server
performing any checking whatsoever on the contents.

Trond

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 29 Nov 2007 15:06:08 -0600

On Thu, Nov 29, 2007 at 03:46:31PM -0500, Noveck, Dave wrote:
> describe to perfore the conversions to and from UTF-8 (or have to ban
> any use of non-UTF8).  Or permaps I should switch tenses and say if
> you have implemented a v4.x client, you must have already done one of
> those things. 

I agree entirely.  For me this thread is primarily about symlink
contents.  There are interesting sub-threads, but as far as NFSv4.1 the
only substantive issue here is what to say about symlink contents.

On that score I recomment opaque, but UTF-8 is OK -- I don't foresee a
lot of problems arising from using UTF-8 for symlink contents.

(All that said, I suspect most NFSv4 implementations haven't gotten the
I18N stuff right, but will eventually.)

Nico
--

---

From: "Yoder, Alan" <[email-masked]>
Date: Thu, 29 Nov 2007 13:09:42 -0800

I thought it was the client's job to translate
non-UTF-8 strings into UTF-8 and back when
communicating with the server?  A Windows NFS
client would certainly have to do this, for
example, as their internal format is Unicode.

Alan

> Sent: Thursday, November 29, 2007 12:57 PM
> To: Trond Myklebust; [email-masked]
> Cc: [email-masked]
> > > > filesystem on the client cope with a logical pathname 
> composed of
> a
> > > > combination of UTF-8 and "just-use-8"? How does the NFS 
> filesystem
> > > on
> > > So how, in this scenario, does the NFS filesystem get to 
> figure out
> > > what

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 29 Nov 2007 16:14:08 -0500

> example, as their internal format is Unicode.

Why? A Windows symlink would only make sense on that windows client.
They don't even use the same pathname separators as UNIX machines and
nor do we impose this upon them in the spec. Symlinks are non-portable.

Trond

---

From: Mike Eisler <[email-masked]>
Date: Thu, 29 Nov 2007 13:48:38 -0800 (PST)

> various difficulties and/or performance issues, but all are workable.

I know from customer experience with NFSv4 that getting rid of
non-UTF-8 locales is a non-starter. People who live in locales that
express their
characters in 8 bits often don't want to move to 16 bit UTF-8
sequences.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 29 Nov 2007 15:49:45 -0600

> it finds that the symlink content is not legal UTF-8. What do I send

Yes, I see your point.

If symlink contents is opaque then you have to be careful to create them
with UTF-8 contents only when their contents is intended to be a path.

> [...]

Sold.  Symlink contents must be UTF-8.

---

From: "Yoder, Alan" <[email-masked]>
Date: Thu, 29 Nov 2007 13:52:08 -0800

I was speaking about file and directory names.

I disagree with the assertion that symlinks should
be noninteroperable.  But if versioning rules won't
let us fix that, I guess there you have it.

Alan 

> Sent: Thursday, November 29, 2007 1:14 PM
> To: Yoder, Alan
> Cc: Noveck, Dave; [email-masked]; [email-masked]

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 29 Nov 2007 15:56:57 -0600

On Thu, Nov 29, 2007 at 01:48:38PM -0800, Mike Eisler wrote:
> express their characters in 8 bits often don't want to move to 16 bit
> UTF-8 sequences.

Indeed, there are good reasons why getting rid of non-UTF-8,
non-US-ASCII locales cannot be part of most vendors' I18N strategy.

That leaves at least two other designs :)  NFSv4 I18N is definitely
feasible.

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 29 Nov 2007 17:00:06 -0500

I think you should consider Alan's remarks as relevant to the more
general question of non-symlink pathname components.

Since you have stated a general (and ostensibly final) position
regarding charracter translation in general, not just for symlinks, I
understood his remarks in that light.

The spec simply says that the strings are UTF8 at the server interface
(i.e. in the protocol).  It doesn't state, whether the responsibilility
to make them so belongs to the client or to the application, although I
always assumed that the client would be doing this translation.  

A widnows client could take a position parallel to Trond's, that it will
not do translation from UCS-2 to UTF-8 and the application should do it.
The spec has nothing to say about that.  I imagine Windows users and
application writers might have a great deal to say about it.

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Thursday, November 29, 2007 4:14 PM
To: Yoder, Alan
Cc: Noveck, Dave; [email-masked]; [email-masked]
Subject: RE: [nfsv4] symlinks: opaque or UTF-8

Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 29 Nov 2007 17:04:31 -0500

Or even 24-bit.  Many quite widely used non-CJK languages are outside
the 
range that fits in 16 bits in UTF-8.  For example, Hindi.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 29 Nov 2007 16:07:28 -0600

> operation rather than return EINVAL and listen to him scream.  How do

Local detail.

The client, somewhere, has all the necessary information.  For most Unix
and Unix-like clients *today* that somewhere is the libc system call
stubs, *provided* that one is willing to declare that below the system
call layer in the kernel all filesystem object names must be in one
particular codeset and encoding (e.g., UTF-8).

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 29 Nov 2007 16:08:50 -0600

> performing any checking whatsoever on the contents.

I18N is not specific to NFSv4.  You should think of it as pervasive,
affecting *all* your filesystems.  Then this problem goes away.

Nico
--

---

From: Mike Eisler <[email-masked]>
Date: Thu, 29 Nov 2007 14:09:10 -0800 (PST)

Yes I understand, but as should be clear from Trond's comments
about how Linux works, someone using an 8 bit character set would
be well advised to stick with just-use-8.

Hindi and other speaks that used Linux are simply out of luck.

--- "Noveck, Dave" <[email-masked]> wrote:

> > > So how, in this scenario, does the NFS filesystem get to figure
> out
> > what

---

From: Mike Eisler <[email-masked]>
Date: Thu, 29 Nov 2007 14:13:43 -0800 (PST)

Given the market position of the Linux client in the NFS
space, and given that it is going to send non-UTF-8 characters,
NFSv4.x is untenable in non-English-speaking locales.

So I think we should do one of:

- change the spec to allow the client and server to indicate
  to each other that both are using a "just-use-8" model

- get rid of UTF-8 all together.

If we decide latter requires us to add OPEN2, CREATE2, etc., 
I'd rather we just moved to NFSv5.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 29 Nov 2007 17:14:15 -0500

> operation rather than return EINVAL and listen to him scream.  How do

OK, I didn't know that there were OS's that would return different
answers to

	open(path,...)

for the same string of bytes "path", depending on the current locale.

My guess would be that the chances of that happening in some future
version of the Linux kernel are, for all practical purposes, zero.

--b.

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 29 Nov 2007 17:16:38 -0500

I'm really thinking of it as pervasive, honest, and I see it affects
all my filesystems.

But the problem hasn't gone away :-)

Maybe I just need to think harder about it being pervasive.

However, that could result in me deciding to get into a different
line of work :-)

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]] 
Sent: Thursday, November 29, 2007 5:09 PM
To: Trond Myklebust
Cc: [email-masked]; Noveck, Dave; [email-masked]
Subject: Re: [nfsv4] symlinks: opaque or UTF-8

Nico
--

---

From: Mike Eisler <[email-masked]>
Date: Thu, 29 Nov 2007 14:20:12 -0800 (PST)

> let us fix that, I guess there you have it.

The outline of a portable symlink would be a symlink create
operation that took an explicit array of components, and
a readlink operation that returned an explicit array of components,
both with an indicate whether the pathname is relative or absolute.

E.g., from a Windows client, aa\bb would be conveyed in CREATE as:

   "aa", "bb"

and stored on a UNIX server as:

   "aa/bb".

A client, UNIX or Windows, would get from READLINK

  "aa", "bb"

The client insert the appropriate component separators in the
result from the readlink system API.

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 29 Nov 2007 17:32:49 -0500

You could have two different object types so that would solve the
versioning problem for CREATE.

For reading I think READLINK would become mandatory to not implement and
a new READPATH would return a flag indicating whether you had the new
interoperable type or the old and union would have the appropriate
return data for each case.

I want it to be clear that I am only speaking of such a thing for v4.x,
x >= 2.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Thursday, November 29, 2007 5:20 PM
To: [email-masked]
Subject: RE: [nfsv4] symlinks: opaque or UTF-8

   "aa", "bb"

   "aa/bb".

  "aa", "bb"

---

From: Mike Eisler <[email-masked]>
Date: Thu, 29 Nov 2007 14:36:35 -0800 (PST)

> I want it to be clear that I am only speaking of such a thing for

Ditto, but given that we are going back to first principles on
NFSv4 (UTF-8), might as well think about v4.2 now, because v4.1
has just been thrown a huge monkey wrench.

I mean, we've been at this 10 years, and Linux developers
waited till *now* to tell us their NFSv4 implementations will
never stop sending non-UTF-8?

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 29 Nov 2007 17:04:30 -0600

> always assumed that the client would be doing this translation.  

That's a local problem.

> not do translation from UCS-2 to UTF-8 and the application should do it.

That's insane, but if some OS vendor wants to do that, sure.

> application writers might have a great deal to say about it.

And many others too.

Nico
--

---

From: Mike Eisler <[email-masked]>
Date: Thu, 29 Nov 2007 15:21:55 -0800 (PST)

> version of the Linux kernel are, for all practical purposes, zero.

What exactly did non-English speaking users of Linux do to deserve
this kind of treatment?

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 29 Nov 2007 18:46:07 -0500

> application writers might have a great deal to say about it.

The problem with having the kernel do the charset conversion is that you
have to convert both incoming and outgoing filenames. Consider if I do a
readdir() on a directory, and end up with a file containing one or more
characters that cannot be translated into the application's charset.
Should I return EINVAL knowing that this will make the file invisible to
that user? Or should I pass the raw string up to the application, and
thus allow it the chance to deal with the problem, for example by
displaying a mangled name, that can still be used to open the file when
the user clicks on it?
In my experience most users would prefer to see some form of
representation of the file, even if they cannot display it perfectly
rather than have the kernel (or the libc front-end for that matter) just
tell them EINVAL.

IOW: yes, applications like browsers and "open" dialog boxes etc need to
be able to deal with charset conversion issues. It might help if
everyone were to get together and state that 'the underlying charset
that you use for your syscalls should really be UTF-8'. However the
reality is that currently there is no such requirement on the Linux
open() interface, and that it is rather late in the day to be talking
about making that a hard requirement almost 20 years after the API was
defined.

Trond

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 29 Nov 2007 18:01:39 -0600

> NFSv4.x is untenable in non-English-speaking locales.

Things change...  Give them time.

>   to each other that both are using a "just-use-8" model

A way to negotiate just-use-8 would be useful, and I think the IETF
should allow no-I18N _options_ when it's useful for dealing with vast
amounts of legacy content.

> - get rid of UTF-8 all together.

Absolutely not.

Nico
--

---

From: Mike Eisler <[email-masked]>
Date: Thu, 29 Nov 2007 16:33:02 -0800 (PST)

> Things change...  Give them time.

Since know Linux isn't going change, you are
apparently predicting that an NS client on an
operating system more friendly to non-English
speakers will prevail. Perhaps after I'm dead.

> amounts of legacy content.

Yet in thinking about it, I don't know how useful it would be.
A Linux client is going to have no choice but to say: just-use-8
when issuing EXCHANGE_ID, because it has no way to predict if
the processes using the client ID will use the UTF-8 locale.
A server that assigns locales per fsid will then store the
component names if the locale is not UTF-8

> > - get rid of UTF-8 all together.

Why?

---

From: Mike Eisler <[email-masked]>
Date: Thu, 29 Nov 2007 16:51:04 -0800 (PST)

> version of the Linux kernel are, for all practical purposes, zero.

Well it turns out the chances are, for all, practical purposes, 100%,
if one is to believe:
   http://linux.die.net/man/8/smbmount
   http://linux.die.net/man/8/mount

Apparently what Bruce and Trond really meant was, the chances are zero
the Linux NFS client and server won't do this. This continues to
beg the question: why?

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 29 Nov 2007 20:15:59 -0500

> the Linux NFS client and server won't do this. This continues to

You did read the details of that 'solution', right? How does a mount
option help you to figure out which charset the application is using?

Trond

---

From: Mike Eisler <[email-masked]>
Date: Thu, 29 Nov 2007 17:22:05 -0800 (PST)

> option help you to figure out which charset the application is using?

In lieu of a system call in Linux to tell the kernel what charset
the process wants to use, a mount option is next best thing.

There were at least two file systems on that list that set
the charset. If Linux maintainers of core file name processing
don't see the trend, then shame on them, but until they "get it",
there is a precedent for how the NFS client and server can proceed.

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 29 Nov 2007 20:47:34 -0500

> representation of the file, even if they cannot display it 
> perfectly rather than have the kernel (or the libc front-end 
> for that matter) just tell them EINVAL. 

OK.

In my experience, users do not want to present a valid string (for a
non-UTF-8 locale) and get EINVAL, simply because the filesystem, which
was mounted yesterday as NFSv3 is now mounted as NFSv4.  I would find
that intolerable and I believe many people would.  If you don't think
that translation between the current local and UTF-8 can effectively be
implemented (as opposed to believing it raises some difficult problems
that must be dealt with), then I don't see how you could have agreed
with what is in RFC3530, for ordinary path components, not getting into
symlinks.  Saying that the application should translate to UTF-8 just
doesn't fly, as you yourself argue below.  It is too late in the game to
impose such a restriction, and furthermore translation to UTF-8 is only
required on NFSv4.  Is the application going to do system calls to
determine whether the current fs is NFSv4 and then translate to UTF-8 or
not based on the result?  There are lots of problems with this stuff,
whichever way you turn, and no choice is perfect.  But some choices are
more and some choices are less evil than others and a choice which
forces the applications to be aware of the coding requirements of
individual protocols seems to me to be at the more-evil end of the
spectrum. 

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Thursday, November 29, 2007 6:46 PM
To: Noveck, Dave
Cc: Yoder, Alan; [email-masked]; [email-masked]
Subject: RE: [nfsv4] symlinks: opaque or UTF-8

> (i.e. in the protocol).  It doesn't state, whether the 
> responsibilility to make them so belongs to the client or to the 
> application, although I always assumed that the client would be doing
this translation.
> 
> A widnows client could take a position parallel to Trond's, that it 
> will not do translation from UCS-2 to UTF-8 and the application should
do it.

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 29 Nov 2007 20:54:44 -0500

> there is a precedent for how the NFS client and server can proceed.

It is a precedent, but not one that is really useful for much.

For one thing the NLS charsets that are defined in the kernel are mainly
there for converting between the more common Windows codepages and
UTF-8, for ISO8859 charsets, or for koi-8 charsets. It will allow Bruce
to export those filesystems that do define a fixed charset (i.e. BeFS,
fat, hfs, hfs+, jfs, joliet, and udf) as UTF-8.

What it won't help with is exporting those filesystems that don't define
a charset (ext[234], gfs2, reiserfs, ocfs2, xfs, .... a.k.a. the
filesystems that people actually use).
Nor is it sophisticated enough to help Dave achieve Hindu charset
compatible nirvana.

Finally, it breaks down completely in multi-user scenarios.

Trond

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 29 Nov 2007 23:58:35 -0600

> Since know Linux isn't going change, you are

Looks like you missed a pronoun or names.  I certainly don't know that
Linux isn't going to change.

> speakers will prevail. Perhaps after I'm dead.

You're reading too much into what I write.  I'm not even implying such a
prediction -- I'm betting that NFSv4.x clients will grow I18N
capabilities.

> the processes using the client ID will use the UTF-8 locale.

Eventually new versions would be rolled out that do know how and where
to do what codeset conversions.  I'm an optimist.  A just-use-8
negotiation would be useful as a migration strategy, nothing more.

> > > - get rid of UTF-8 all together.

Because we need an interoperable way to use internationalized filesystem
object names, even if not every implementation implements I18N today,
and even if it will take a few more years before all do.

That requirement is not going away.  Even if you succeeded in yanking
out the requirement for use of UTF-8 on the wire, folks would still end
up agreeing on its use anyways and would still need interop guidance.

Nico
--

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 30 Nov 2007 05:49:19 -0500

Nico is predicting that reason will prevail, eventually.

Of course he's right but the history of the past years (agreement to
I18N in NFSv4 but with fingers crossed), makes me wonder how far off
"eventually" is.  Is it in my lifetime?

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]] 
Sent: Friday, November 30, 2007 12:59 AM
To: Mike Eisler
Cc: [email-masked]
Subject: Re: [nfsv4] UTF-8 or or "just-use-8"

On Thu, Nov 29, 2007 at 04:33:02PM -0800, Mike Eisler wrote:
> > > Given the market position of the Linux client in the NFS space, 
> > > and given that it is going to send non-UTF-8 characters, NFSv4.x 
> > > is untenable in non-English-speaking locales.
> > 

> apparently predicting that an NS client on an operating system more 
> friendly to non-English speakers will prevail. Perhaps after I'm dead.

> > should allow no-I18N _options_ when it's useful for dealing with 
> > vast amounts of legacy content.
> 
> A Linux client is going to have no choice but to say: just-use-8 when 
> issuing EXCHANGE_ID, because it has no way to predict if the processes

> using the client ID will use the UTF-8 locale.

> A server that assigns locales per fsid will then store the component 
> names if the locale is not UTF-8
> 

Nico
--

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 30 Nov 2007 06:28:30 -0500

> What it won't help with is exporting those filesystems that don't 
> define a charset (ext[234], gfs2, reiserfs, ocfs2, xfs, .... a.k.a. 
> the filesystems that people actually use).

It would help in allowing non-English-speaking users to obey the NFSv4
protocol when talking to commercial servers that implement the NFSv4
protocol.  I'm a somewhat mercenary soul in caring about that, but it is
also the case that many users of Linux who have such devices might like
help in that regard.

> Nor is it sophisticated enough to help Dave achieve Hindu charset 

Perhaps not, but it can be extended and it doesn't (yet) fall into the
category of things that you have sworn never to do, so that is a plus. 

> Finally, it breaks down completely in multi-user scenarios. 

Exactly so.  In that case you would need to do different sorts of
translations for different user processes based on their, how should I
put it, locales.  When we wrote the requirement into RFC3530 (actually
into RFC3010) that path components would be presented by clients in
UTF-8 (and that if they didn't they would get NFS4ERR_INVAL and not what
they asked for) my expectation was that clients would translate
characters presented by those working in different character sets to
UTF-8 before sending them to NFSv4 servers to avoid that NFS4ERR_INVAL.

But Bruce has indicated that the probability of that being done in Linux
is effectively zero and you have gone farther in almost committing
yourself never to do such a thing.  So when we discussed this issue when
the RFC's were being written and subsequently, how did you expect the
requirements of the RFC to be met?  If you merely transcribe faithfully
the data presented across the system call interface then you are doing
exactly what was done by NFSv2 and NFSv3 clients and those do not have
any such requirement.  Did you expect all application writers to sign up
for determining when they were connected to NFSv4 and doing appropriate
translations?  That would not be a reasonable expectation, and I'm
assuming you didn't have it.  So I need to understand, why we are only
hearing about this difficulty now?  If the requirement for sending names
as UTF-8 is essentially unimplementable within Linux, why didn't we hear
much earlier that you didn't think we could support internationalization
in NFSv4 and deal with that issue then?

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Thursday, November 29, 2007 8:55 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] symlinks: opaque or UTF-8

On Thu, 2007-11-29 at 17:22 -0800, Mike Eisler wrote:
> In lieu of a system call in Linux to tell the kernel what charset the 
> process wants to use, a mount option is next best thing.
> 
> There were at least two file systems on that list that set the 
> charset. If Linux maintainers of core file name processing don't see 
> the trend, then shame on them, but until they "get it", there is a 
> precedent for how the NFS client and server can proceed.

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 30 Nov 2007 09:55:28 -0500

> in NFSv4 and deal with that issue then?

It might be nice if Linux were all about me, and what I promise to do or
not do, but it isn't. This issue keeps coming up among the kernel
community, and the concensus remains the same. If you want to see some
of Linus' ideas on the topic, then here is one example

  http://article.gmane.org/gmane.linux.kernel/182509

 (feel free to read through the entire thread starting at
http://thread.gmane.org/gmane.linux.kernel/182053/focus=182478 )

I am not proposing to 'solve' this issue by some kernel hack, because it
is current official community policy that this is an application issue.

Trond

---

From: Theodore Tso <[email-masked]>
Date: Fri, 30 Nov 2007 10:30:21 -0500

> Finally, it breaks down completely in multi-user scenarios.

If it is at all helpful, at least for ext[234], when this topic came
up last time, the answer was that in multi-user scenario, the only
sane answer was UTF-8.  If a user wants to use some other charset in
the privacy of their own system, that was fine with us, but ext[234]
was *not* going to put charset conversion into the kernel.  That way
lies complete and utter madness, and any OS engineer who needs to do
that in the kernel has my sympathies.

					- Ted

---

From: Mike Eisler <[email-masked]>
Date: Fri, 30 Nov 2007 08:15:16 -0800 (PST)

--- Trond Myklebust <[email-masked]> wrote:

> If the requirement for sending
> names

I don't think Dave's question has been answered.

> http://thread.gmane.org/gmane.linux.kernel/182053/focus=182478 )

No where in that thread did Linus say that because the Linux kernel 
does not enforce UTF-8 on inputs and outputs to system calls, that
file systems in the kernel then had a license to violate the
standard they are supposedly limiting. Nor did he say that
it it would be an evil thing if those file systems enforced the
standard they were implementing. That there are apparently file systems
in the Linux kernel that enforce character sets, reinforces this, 
because presumably he'd have something to say about it when the patch
requests were sent in.

So blaming God^WLinus doesn't work here (though with statements like:
"Think about the simple (hex) string x0A x00. That's a well-defined
UTF-8 string," its questionable if he understands UTF-8 :-). 

Yes it would be nice if he embraced the capability for processes 
to ask for character set translation and enforcement (even
if that one character set was UTF-8), but his lack of
support for a general framework does not preclude the NFS client
and server in Linux from implementing its own framework (and
perhaps sharing code with SMBfs, and other file systems; Ted
mentioned that ext had UTF-8 support for example).

> is current official community policy that this is an application

Where can I read the official policy documents?

---

From: "Yoder, Alan" <[email-masked]>
Date: Fri, 30 Nov 2007 08:27:09 -0800

Gas on the fire...

There are numerous systems out there that already
have to do charset translation, and that do it
with varying degrees of excellence or suckage.  
Netapp's product, EMC's Celerra and Samba all come 
to mind.  There are at least several others that
I don't know well (HDS, ProComm etc.).

These are all server-side products, interestingly.

Alan

> From: Theodore Tso [mailto:[email-masked]] 
> Sent: Friday, November 30, 2007 7:30 AM
> To: Trond Myklebust

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 30 Nov 2007 11:33:53 -0500

> I don't think Dave's question has been answered.

I've never been on record as stating that our client would perform utf-8
conversions, nor have I ever agreed to doing so. The RFC states that
servers expect it, which is fine by me, but I've never planned on doing
charset conversion in the kernel on behalf of applications.

> > It might be nice if Linux were all about me, and what I promise to
> do
> > or

That is not going to happen. We will not do charset conversion on behalf
of applications.

> Where can I read the official policy documents?

The mailing lists are pretty much all there is.

Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 30 Nov 2007 12:01:44 -0500

> not do, but it isn't.

So are you saying, in response to my question, that when you agreed, or
at least didn't disagree with the NFSv4 internationalization strategy,
you thought Linus could be brought around, but have since found out
otherwise?  That's not an answer I like (in the sense it the
implications for NFSv4 are not so great) but at least its an answer.
I'm not that pleased that it took a discussion of symlinks to bring this
fact out.  I would have hoped that you could have brought the issue to
the working group's attention as soon as the problem surfaced.  The
document you cite is from early 2004.

> Choice is _inherently_ good. 

I agree but we have a situation in which NFSv4 cannot choose a UTF-8
internationalization strategy and have Linux client implementations
conform to it, which seems to me to indicate that something is amiss.
It seems that some of the implementation-specifics intended to enable
choice, have quite the opposite effect.

> Trying to force a world-view is bad.

Which is kind of strange when spoken by someone who is (apparently
successfully) imposing his world-view on NFSv4.  It's not as if he
doesn't have a right to have his own world-view and to fight tenaciously
for it.  What I object to is the ex-post-facto aspect and its effect on
NFSv4.  I think we've been blind-sided, and our work has been unfairly
interfered with and hearing about how this is all in the interests of
freedom is a bit much.

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Friday, November 30, 2007 9:55 AM
To: Noveck, Dave
Cc: [email-masked]; [email-masked]
Subject: RE: [nfsv4] symlinks: opaque or UTF-8

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 30 Nov 2007 12:03:56 -0500

On Fri, 2007-11-30 at 08:15 -0800, Mike Eisler wrote:
>  Ted

BTW: Please read what Ted said. He said that UTF-8 was supported, but he
didn't say that the kernel would reject any non UTF-8 characters nor did
he say that the kernel would convert non UTF-8 characters. In fact, he
explicitly told you that idea had been rejected by the developers.

I agree that UTF-8 makes most sense for portability. Most Linux
distributions today default to UTF-8 based locales, so filename
conversion should normally not be needed.

Should, however, a user choose to use a non UTF-8 locale, then the Linux
client will put that filename unchanged on the wire, and it will be up
to the server to decide what to do. The Linux server will currently pass
that string unchanged to the filesystem, which again ensures that all
legacy filesystems can be exported without requiring any special
conversion.

Trond

---

From: Mike Eisler <[email-masked]>
Date: Fri, 30 Nov 2007 09:04:46 -0800 (PST)

--- Trond Myklebust <[email-masked]> wrote:

> > > > as UTF-8 is essentially unimplementable within Linux, why
> didn't we
> > > hear

Well this would have been good to know in 2002.
If there's anything else in v4.0 or v4.1 you won't be
complying with, especially stuff that causes the NFSv4.[01] server
to return unrecoverable errors, now would be a good time
to know.

> That is not going to happen. We will not do charset conversion on

Please state it more plainly:

"We will send non-UTF-8 to NFSv4.x servers, so tough [expletive
optional]." 

All: I recommend we remove UTF-8 from NFSv4.1.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 30 Nov 2007 12:20:05 -0500

On Fri, 2007-11-30 at 12:01 -0500, Noveck, Dave wrote:
> otherwise?

The UTF-8 requirement was apparently decided well before I got involved
with NFSv4: it is already part of RFC3010, which dates from December
2000.
I was never consulted on that issue, and have never been told that there
was some implicit assumption that client kernels would perform special
conversions on behalf of applications.

The reason why I never raised an issue is because I knew that the Linux
namespace rules are compatible with UTF-8, by which I mean that there
are no UTF-8 strings that will be misparsed by the VFS pathname
interpreter. If the server wishes to restrict itself to UTF-8 only, then
that will certainly break some userspace applications, but that is not a
problem for the kernel to solve.

> That's not an answer I like (in the sense it the

There have been earlier discussions too, but I preferred to quote the
most recent one I can find.

Trond

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 11:22:51 -0600

> "eventually" is.  Is it in my lifetime?

I think it may even be around the corner, in IETF time :^)

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 30 Nov 2007 12:33:08 -0500

> to the server to decide what to do. 

Indeed.  It can either choose to conform to the protocol and return
NFS4ERR_INVAL or it can choose to do something else and not confrom to
the protocol.

> The Linux server will currently pass

In other words, the Linux NFSv4 client and the Linux NFSv4 server both
treat path components just as they were treated in NFSv3.  They are
interoperable, with each other, but not, in non-UTF-8 locales, with
servers that implement the protocol specified in RFC3530.

Further, there is a fixed determination, not to treat this
interoperability issue as a problem and thereby confrom to the protocol.

At least we know this now.

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Friday, November 30, 2007 12:04 PM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [nfsv4] symlinks: opaque or UTF-8

Trond

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 11:36:51 -0600

> of Linus' ideas on the topic, then here is one example

I you go with the UTF-8-in-the-middle approach then you can push the
conversions to the edge, namely [g]libc syscall stubs and filesystem
modules, thus neatly bypassing kernel architecture issues.

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 11:49:04 -0600

> that in the kernel has my sympathies.

The easiest thing to do is to push codeset/encoding conversions to the
edge and use UTF-8 by convention in the middle.

On the filesystem edge very few filesystms should have to do any
conversions (e.g., an NTFS module would have to do UTF-8<->UTF-16
conversions, and some filesystems might have options for dealing with
legacy to convert to non-Unicode codesets, like ISO-8859-*).

The other edge of the middle is the system call stubs in user-land,
where knowledge of the current locale's codeset is available.

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 11:56:30 -0600

On Fri, Nov 30, 2007 at 09:04:46AM -0800, Mike Eisler wrote:
> > That is not going to happen. We will not do charset conversion on
> > behalf of applications.
> 

It's completely agreeable that his client won't do the necessary
conversions -- those belong elsewhere in the system.

> All: I recommend we remove UTF-8 from NFSv4.1.

No.  That'd be overreacting.

We may need more flexibility in the protocol to deal with legacy content
(client: "hi, do you have non-UTF-8 content in this <fsid>?  I want to
be able to warn my users about that or even disallow them access",
server: "yes I do" or "yes I do, use ISO-8859-15" or "yes I do, sorry,
but new content must be UTF-8" or "no I don't and don't accept non-UTF-8
content").

But giving up on I18N won't do for the reasons that I described earlier.

Nico
--

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 30 Nov 2007 13:08:12 -0500

Yes this can be done, and if it is important for some reason, political,
technical or otherwise, to keep it outside the kernel, then I imagine
that need can be accommodated.  The issue I see is that however you do
this, this is a lot of work, and I don't sense a willingness to
undertake that work, just because the internationalization requirements
in RFC3530 together with the needs of non-English-speaking users of
Linux, make it necessary (my opinion).

We hear a lot about what will not be done, ever.  What we don't hear is
any understanding of the fundamental problem, or acceptance that it is a
problem.  Without that as a base, design effort to work around specific
restrictions on what can be done where is beside the point.

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 12:20:01 -0600

> Linux, make it necessary (my opinion).

Actually, I believe it's not a lot of work _once_ you have the necessary
I18N infrastructure (i.e., codeset conversion tools).  Which I believe
Linux does have.  It's a SMOP in glibc, mostly, and I'm not being
sarcastic, it's really a small.  For perf reasons one might like a way
to turn this off though; getting the interface details for such a knob
right may be a bit more difficult, but hardly insurmountable.

> restrictions on what can be done where is beside the point.

You've certainly heard from me about the technical details of the
problem and its possible solutions.  I don't represent the Linux
community though, but, who does?  Trond et. al. need to talk to the
glibc folks, and it might help if they had proof of concept patches on
hand.

The rest of us shouldn't respond to the apparent unwillingness of the
Linux community to come together on this by ripping I18N out of NFSv4.
I say "apparent" because it's not clear that this has been discussed in
the right places.  I think the syscall-libc-stubs-convert approach is so
sensible (with a global/environmental on/off knob) that once it reaches
the right places (glibc community) it will be implemented.

Nico
--

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 30 Nov 2007 13:31:48 -0500

> apparent unwillingness

How about "declared unwillingness"?  

If you can make this happen, then that would be great.  You would
deserve a quite fulsome acknowledgment, but there might be difficulty in
coming up with a generally acceptable formulation.  "Got the Linux
community to see reason" might be considered unduly provocative.  I
think we'd be able to negotiate something, though.

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]] 
Sent: Friday, November 30, 2007 1:20 PM
To: Noveck, Dave
Cc: Theodore Tso; [email-masked]; [email-masked]; Trond
Myklebust

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 12:38:53 -0600

> If you can make this happen, then that would be great.  You would

Hey!  Back off!  :)  I'm not volunteering.  I work on OpenSolaris stuff
and I'm busy enough as it is.  I've taken a keen interest in fs I18N in
the past and I'm offering my technical advice, and beyond that my advice
not to give up on I18N (actually, my position on that is stronger, as
you've seen).  But that's it.

> think we'd be able to negotiate something, though.

I'll settle for getting the same thing (codeset conversions in libc fs
syscall stubs) implemented in Solaris.  We can lead by example, and
Linux can follow if the Linux community cares to.

Nico
--

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 30 Nov 2007 13:43:24 -0500

> Linux can follow if the Linux community cares to.

What does Solaris currently do?

--b.

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 12:44:00 -0600

> servers that implement the protocol specified in RFC3530.

BTW, MacOS X treats path components in NFSv3 as in NFSv4 (in fact, it
normalizes to NFD on CREATE).

> At least we know this now.

But you don't "know this."  Trond does not represent the whole Linux
community.  You, and/or at least Mike, seem desperate to find in your
representation of the Linux situation a strong enough reason to ditch
NFS I18N.  That would be most unfortunate, and definitely premature.

Nico
--

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 30 Nov 2007 13:45:48 -0500

> Linux can follow if the Linux community cares to.

So can I take it that from the above comment this isn't "just another
Linux issue", and that the current Solaris client implementation does
not perform UTF-8 encoding of filenames?

Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 30 Nov 2007 14:10:36 -0500

> What does Solaris currently do?

I thought he just told you.

It converts to UTF8 (presumably based on current locale) in libc fs
syscall stubs.  This given the client per se nothing much to do to
conform to the RFC.

I assume that the inverse conversions for data returned happen on the
way back.

One interesting question is what do the Solaris libc fs syscall stubs do
for symlinks (create and readlink).

The other issue is what the Solaris server does does if you send
non-UTF-8 path components.  I presume it obeys the RFC but getting
confirmation would be nice.

I guess a similar issue applies to symlinks.  Does it check those for
UTF-8-ness, on create, on readlink?  The more data we have on what
people actually do, the better off we are in figuring out how to deal
with our current situation in which we have a lack of interoperability.

With regards to Nico's claim that I am desperately trying to drop I18N
from NFSv4, I have not suggested that.  It may be necessary but I would
also like to avoid it.  But I'm having trouble understanding how it
would work to keep it.  Trond may not represent the entire Linux
community but I don't see how this working group can deal with the
entire Linux community on an issue like this.  I think we need to have
something that we know will work, rather than something that might work
if some set of political developments allow the Linux community to
consider this issue in a different light, at some unspecified time in
the future.  

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Friday, November 30, 2007 1:43 PM
To: Noveck, Dave; Theodore Tso; [email-masked];
[email-masked]; Trond Myklebust
Subject: Re: [nfsv4] symlinks: opaque or UTF-8

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 30 Nov 2007 14:13:27 -0500

--b.

---

From: Mike Eisler <[email-masked]>
Date: Fri, 30 Nov 2007 12:12:07 -0800 (PST)

--- Theodore Tso <[email-masked]> wrote:

> define

Trond points out I didn't read this closely.

It doesn't help fix the problem (in that respect it is useless).
It does however reinforce the intransigence among the Linux
community. 

As for complete and utter madness to put charset conversion
in the kernel, Alan Yoder pointed out that EMC and NetApp do just
on the server, and apparently smbfs does that in the client's kernel.

I don't know about smbfs, but if EMC and NetApp are
mad, it is more like being crazy like foxes, if one believes
IDC's figures on the combined share of the NAS server business 
both companies have. "Running code"

---

From: Mike Eisler <[email-masked]>
Date: Fri, 30 Nov 2007 12:20:39 -0800 (PST)

--- "Noveck, Dave" <[email-masked]> wrote:

> from NFSv4, I have not suggested that.  It may be necessary but I

Thus far I am the only one advocating that, and I've read nothing today

that changes that proposal.

A statement from senior engineers at RedHat, SUSE, and Ubuntu that
they will implement Nico's proposal would be the only thing that might
make me consider withdrawing my proposal.

---

From: Mike Eisler <[email-masked]>
Date: Fri, 30 Nov 2007 12:44:32 -0800 (PST)

> not perform UTF-8 encoding of filenames?

1. Irrelevant. With apologies to my friends at Sun, unfortunately,
   the Solaris NFS client is not the gorilla any more. Years ago,
   Linux could get away with saying that since Solaris does it one
   way, it is ok for Linux to. Now it is the other way around.

2. Solaris is a whole product: it combines the kernel, libraries, and
   commands. So even if #1 wasn't an issue, and even if Solaris did
   not have its i18n crap together, at least it could. The mantra in
   Linux land is that it is not a whole product; it is just the kernel,
   and making it a whole product is the problem of someone else. So
   it can't have its i18n crap together until the major distributors
   of the Linux kernel and compatible commands and libraries take 
   ownership.

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 14:45:04 -0600

> What does Solaris currently do?

Sadly, the Solaris NFSv4 client does nothing special beyond preferring
UTF-8 locales.  If you use UTF-8 locales (the default) then no problem.

In OpenSolaris the SMB server does perform normalization-insensitive
LOOKUPs and is normalization-preserving, it also knows how to do Unicode
case-insensitive LOOKUPs and how to preserve case (if enabled on a
filesystem).  I don't know if the OpenSolaris NFSv4 server does the same
thing, but it'd be a really small SMOP to make it do the same.  I'm not
sure if the OpenSolaris NFSv4 server rejects non-UTF-8 octet sequences
either, but again, it'd be a small SMOP to make it do so if it doesn't
already.

But ideally we should have an option to dispense with I18N for specific
filesystems where legacy requires it.

I'm wondering if Mike meant to propose removing I18N altogether or
simply utf8string.  XDR doesn't have a native UTF-8 string type, but if
it did then it would arguably be quite wrong to a) use that type in
NFSv4.x, b) allow non-UTF-8 therein for legacy filesystems.  So if Mike
meant dispensing with the utf8string type, then I could get behind that,
provided we still had required-to-implement I18N at the end of the day.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 14:46:23 -0600

> OK.  I guess I was confused by his use of the future tense ("I'll"):

No, you weren't.  Our client does not do this yet.  I'd like to see it
do that though.

---

From: Mike Eisler <[email-masked]>
Date: Fri, 30 Nov 2007 12:55:12 -0800 (PST)

> UTF-8 locales.  If you use UTF-8 locales (the default) then no

So it will send garbage too. Nice.

> thing, but it'd be a really small SMOP to make it do the same.  I'm

Gee, isn't psarc supposed to prevent these inconsistencies ? :-)

> not

I want to remove the requirement that UTF-8 be sent by the client, so
that my server can then legally accept the garbage and return the 
garbage. Believe it or not, there are customers that demand compliance
with a standards-rack specification(unless they are running 
an open/free source  operating system, then they've no one to 
make demands to). I want to comply. I cannot comply if I 
want to capture market share.

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 30 Nov 2007 14:56:16 -0600

On Nov 30, 2007, at 2:46 PM, Nicolas Williams wrote:

>>>>> I'll settle for getting the same thing (codeset conversions in  
>>>>> libc
>>> fs

And so would I so we have at least the intent.

Spencer

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 15:11:06 -0600

> And so would I so we have at least the intent.

We also now have all the infrastructure pieces.  Intent + infrastructure
-> this won't take years.  FS I18N is not just a requirement, it's
around the corner.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 30 Nov 2007 16:12:57 -0500

> want to capture market share.

The problem then becomes how to deal with case-insensitive filesystems.
For better or for worse, NFSv4 decided to support those, and I can't see
how you can do that without also defining a charset. In fact, I don't
really see how you can support case-insensitivity without defining a
full locale since the capitalised form of letters is locale-dependent
(in some countries, certain capitals lose their accents, for instance).

If you want a lecture on how complicated that stuff can be, talk to the
Samba folks...

Trond

---

From: Mike Eisler <[email-masked]>
Date: Fri, 30 Nov 2007 13:22:33 -0800 (PST)

--- Trond Myklebust <[email-masked]> wrote:

> For better or for worse, NFSv4 decided to support those, and I can't

An OPTIONAL feature of NFSv4.

> see

The point is they apparently managed to do something. And even
if it was impossible, I'm OK with sacrificing an OPTIONAL feature.
If NFSv4.1 ever goes to DRAFT standard, if no one has done it,
then it goes away anyway.

You are the one saying you will never commit to a character
set. If you don't like the consequences then make a different
commitment.

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 30 Nov 2007 16:23:07 -0500

I think case-insensitive filesystems have to be defined as UTF-8-only.
As Trond points out, there is no way to make them work otehrwise.

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Friday, November 30, 2007 4:13 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] symlinks: opaque or UTF-8

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 30 Nov 2007 16:25:33 -0500

> -> this won't take years.  FS I18N is not just a requirement, it's

I'm still not convinced this solves the problem of a mandatory UTF-8
requirement in the RFC, though. If it is true that Solaris currently
doesn't enforce UTF-8, then you will be in the exact same position as
Linux is w.r.t. the existence of filesystems that are currently in other
locales, or even ones that mix locales.
Mandating UTF-8 even at the libc level is likely to cause problems for
systems with such a legacy setup, and so you will get people who will
continue to turn that functionality off.

Trond

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 15:26:20 -0600

> how you can do that without also defining a charset. In fact, I don't

Oh yeah, if you're just-use-8 then you can't be case-insensitive.  So
what?  I don't think we'll see an IETF-sanctioned version of NFSv4 (or
v5, or whatever) without I18N.

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 30 Nov 2007 16:30:46 -0500

OK, do we have any clients that make effective provisions to send UTF-8
component names?  Given that lots don't, it wouldn't matter from a
practical point of view, but it would somehow make me feel better having
spent all this time on this, including coding the appropriate
translation on the fs side.

Come on, is there even one client out there, who does this?

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]] 
Sent: Friday, November 30, 2007 4:11 PM
To: Spencer Shepler
Cc: J. Bruce Fields; [email-masked]; Trond Myklebust; Theodore
Tso; Noveck, Dave; [email-masked]
Subject: Re: [nfsv4] symlinks: opaque or UTF-8

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 15:39:26 -0600

On Fri, Nov 30, 2007 at 12:20:39PM -0800, Mike Eisler wrote:
> > With regards to Nico's claim that I am desperately trying to drop

I'd said you and/or Mike :)

> > I18N

It's not as though your proposing it by itself creates any kind
consensus.  The rest of us don't need everyone else to agree to
*implementing* I18N in their clients in order to support retaining I18N
in the spec.  We need only consensus to keep I18N in the spec.

Actually, we don't even need WG consensus to keep I18N in the spec
because it's already there and to remove I18N we'd need much more than
WG consensus.  We'd need IESG support and IETF consensus -- that's NOT
likely to happen.  Taking the NFSv4 ball home and playing elsewhere
would only need agreement amongs the top players, but somehow I don't
see that as a likely nor practical option.

It is NOT the case that the IETF needs any "statement from senior
engineers at ...  that they will implement ..." in order to require a
major feature, like I18N (or secure protocols, for that matter).

This may sound harsh, but I think I'm describing the reality on the
ground as it is.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 15:40:59 -0600

> requirement in the RFC, though. If it is true that Solaris currently

Lets be careful with terminology.

We have an IETF requirement for support for I18N in the protocol.

I do not believe that means that we have a requirement to NOT have
non-I18N optional modes.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 15:42:03 -0600

> component names?  Given that lots don't, it wouldn't matter from a

I expect that Windows-based ones do.  And soon enough Solaris will too.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 30 Nov 2007 16:42:54 -0500

> Come on, is there even one client out there, who does this?

I seem to remember the AIX guys asking questions about stringprepping a
year or two ago. Does anybody have a copy of their Red Book to check?

Cheers
  Trond

---

From: Mike Eisler <[email-masked]>
Date: Fri, 30 Nov 2007 13:57:55 -0800 (PST)

> consensus.  The rest of us don't need everyone else to agree to

Of course.

> because it's already there and to remove I18N we'd need much more

Agreed.

> than
> likely to happen.  

I don't know. I'd like to try.

> [...] Taking the NFSv4 ball home and playing elsewhere

I have no idea what that metaphor means, and won't speculate.
For my part I want be able to comply with a standard issued by
a real open standards body and not violate it. I've never
written or said otherwise from the time Sun and ISOC signed the
agreement.

> major feature, like I18N (or secure protocols, for that matter).

Of course IETF can just mandate specfications that no one will
comply with or implement or use. That's what IETF does.  Look at IPv6
for example. DNSSEC. shttp. SPKM. Etc.

> This may sound harsh, but I think I'm describing the reality on the

Your reality is not on ground at all. It is in luxury hotel conference
room where IETF meets that is far outside the reality on ground.

If i18n stays in as is, then everyone (not just the clients,
who never said a peep in 10 years that they would fail to comply)
will not comply.

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 16:11:59 -0600

> Gee, isn't psarc supposed to prevent these inconsistencies ? :-)

You'd think, but I'm not a member ;)

> want to capture market share.

As a server vendor you can comply if we simply add a per-fsid option to
just-use-8.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 30 Nov 2007 17:12:17 -0500

> who never said a peep in 10 years that they would fail to comply)

Not completely true:

	http://playground.sun.com/pub/nfsv4/webpage/nfsv4-wg-archive-feb-03-feb-05/1176.html

	"Linux does no normalization or checking whatsoever at the VFS
	level in order to enforce UTF-8"

Admittedly that may still have been too late (2004), and not clear
enough.  (Though I'll admit I have not yet had the pleasure of
re-reading that thread in its entireity, so I may have missed some more
relevant quote.)

--b.

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 16:16:28 -0600

> As a server vendor you can comply if we simply add a per-fsid option to

I.e., no need to throw out the baby with the bathwater.

---

From: Mike Eisler <[email-masked]>
Date: Fri, 30 Nov 2007 14:24:57 -0800 (PST)

> 	level in order to enforce UTF-8"

Which is not semnantically the same as saying the Linux client
did not and will never comply. Indeed Trond's complete statement was:

``This is the whole reason why Linux does no normalization or checking
whatsoever at the VFS level in order to enforce UTF-8 (despite Linus'
official position that only UTF-8 is guaranteed to be supported by the
filesystems).''

One could infer many things but not that the Linux NFSv4 client
not do checking.

> re-reading that thread in its entireity, so I may have missed some

Too late for NFSv4.0, not too late for NFSv4.1 (around the same time,
pnfs drafts were flying by).

You Linux guys exercise your power to embrace and extend, so
tell the rest of us what you are doing.

---

From: Mike Eisler <[email-masked]>
Date: Fri, 30 Nov 2007 14:29:46 -0800 (PST)

> I.e., no need to throw out the baby with the bathwater.

Right instead just let the baby die of heat stroke in the sun.

If an attribute, per fsid or per system (I'd recommend implementing
it per system regardless) would allow us to reach consensus on
changing i18n, I'd live with it. Indeed, it would give Dave others
time to undo the beautiful UTF-8 translation code they wrote.

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 16:34:58 -0600

> then it goes away anyway.

Or if it's something that the IETF feels strongly about then it doesn't
go to Draft Standard.

But we're likely to have at least two client and two server
implementations that get I18N right.  On the client side those would be:
OpenSolaris (soon) and any Windows-based client.  On the server side
those would be MacOS X and OpenSolaris.

By "soon" I mean" before any attempt to take NFSv4.1 to Draft Standard.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 16:49:57 -0600

> Come on, is there even one client out there, who does this?

Any that are running in UTF-8 locales, yes.

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 30 Nov 2007 17:57:39 -0600

> Right instead just let the baby die of heat stroke in the sun.

I think it's time to declare that sub-thread dead.  You and I disagree
about the likelyhood of having two client and server implementations
that get I18N right.  No need to pursue that.

> time to undo the beautiful UTF-8 translation code they wrote.

I agree that having a per-fsid UTF-8/non-UTF-8 knob is desirable and
would settle the issue (NOT per-system -- we need to support migration
on a per-dataset basis).  The only other protocol-level issue is whether
the protocol should use an XDR type that might denote, to an XDR tool
chain, that values of that type can only contain valid UTF-8 sequences
-- but I think this is a non-issue today, and a warning in the text of
the RFC would suffice to avoid it being an issue in the future.

Nico
--

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 30 Nov 2007 19:45:13 -0500

> year or two ago. Does anybody have a copy of their Red Book to check?

http://www.redbooks.ibm.com/abstracts/sg246657.html

A search for the text "utf-8" has two real hits:

	The mandatory features of the NFSv4 protocol as described in RFC
	3530 are supported with the following exceptions:

	- The UTF8 requirements are not fully supported.

	- The LIPKEY and SPKM-3 security mechanisms are not supported
	  with RPCSEC_GSS authentication.

and

	NFSv4 introduces the following new tunable parameters:

	utf8
		This option allows NFSv4 to perform UTF-8 checking.
		A value of 1 turns on UTF-8 checking of file names.
		A value of 0 turns it off.

	utf8_validation
		Enables checking of file names for the NFSv4 client and
		server to ensure that they conform to the UTF-8
		specification.

(How are they different?  Who knows.)  The wording definitely makes them
sound like they just turn on some checks, rather than locale-dependent
translations.

Elsewhere it does say the both default to 1:

	http://publib.boulder.ibm.com/infocenter/pseries/v5r3/topic/com.ibm.aix.cmds/doc/aixcmds4/nfso.htm

--b.