Subject: Re: [nfsv4] work items to complete rfc3530bis (what are they?)

From: <[email-masked]>
Date: Thu, 1 Apr 2010 14:00:18 -0400

> An easy example to explain is the German ezsett character 
> (aka 'sharp s')- this character resembles a Greek beta and 
> is pronounced as if it were double-s.  The existing approach 
> is to map ezsett to double-s, and the results of that mapping 
> are *not* always easily invertible.  For example, after mapping 
> a certain German street name, "passstrasse" is the result - a 
> native speaker of German knows what the correct input string 
> was (only the last double-s was an ezsett), but computers don't 
> tend to be native speakers of German ;-).  This mapping of ezsett 
> to double-s is now wrong according to German language experts 
> (it's apparently as wrong in German as mis-spelling "receive" 
> as "recieve" would be in English), 

Or saying "for him and I" :-)

> hence ezsett will become an allowed Unicode character in domain 
> names.

OK.  That's the easy example to explain.  Thanks, David.

So they made a mistake and need to fix the mistake and it appears they
are willing to fix the mistake.  So far, so good.

> If I read Nico's note correctly that NFSv4 uses stringprep 
> for some strings, 

My understanding is that we were essentially told that we had to in
RFC3530.  Spencer can confirm or deny.  I think it was part of this
whole you-don't-want-to-deal-with-this-stuff-on-your-own line of thought
which makes perfect sense until the reality of what having others
dealing with it for you involves.

> then this change will likely have to be dealt with,

It has to be dealt with by somebody but why does it have to be dealt
with by us?  

We specified what we were told/asked to.  Our expertise regarding German
characters is limited (as it is for Greek, Arabic, Hebrew, Syriac,
Devanagari, etc. characters).

There needs to be a way to address mistakes like this found after RFC
publication that doesn't force the users of stringprep (for example) to
care or modify their specs.  We do not want to have to do RFC3530tris,
RFC3530tetrakis, etc, to address future such issues that come up.

> and not 
> doing so results in implicitly specifying that NFSv4 not deal 
> correctly with some German strings.

I understand the specification issue, but for practical purposes it
really doesn't matter.  People will do what they choose, with customer
requirements having a major role in what characters, including ezsett or
not, are accepted.  We can't even get some clients and servers to use
UTF-8 for their filenames.  People are not struggling to avoid arousing
the ire of an all-powerful All-IETF Extraordinary Commission to Combat
Specification Non-compliance.  That's a good thing in general but it
means that there is no overriding need to make the specification perfect
on matters like this, especially since perfection is moving target.  

As you indicated, we don't want to deal with this stuff ourselves, so we
should just point at the latest extant set of specs in this area, that
we can easily point to without rewriting the string-handling section,
and then say people MAY make generally accepted character mapping
changes.  They're going to do it anyway.

-----Original Message-----
From: Black, David 
Sent: Wednesday, March 31, 2010 4:13 PM
To: Noveck, David; [email-masked]'
Subject: RE: [nfsv4] work items to complete rfc3530bis (what are they?)

Dave,

> > It's also the situation that
> > at least for stringprep, there are four Unicode input characters
whose
> > on-the-wire representation changes incompatibly due to character
mapping
> > changes.
> 
> Will someone wake me up from this nightmare?
> 
> Suppose I decided that we should swap ASCII a and b, also c and d.  I
know the existing mapping has
> served us well, and it is used in many existing files, but it really
gets boring to stick to the same
> old mapping.  Perhaps that sounds a little "drbzy"?

English doesn't have character mapping issues, hence there are no good
English examples to explain what happened.  It's not a matter of
swapping characters - it's a matter of what appear (in 20/20 hindsight)
to be mapping mistakes for non-ASCII characters in languages that have
larger alphabets.

An easy example to explain is the German ezsett character (aka 'sharp
s')- this character resembles a Greek beta and is pronounced as if it
were double-s.  The existing approach is to map ezsett to double-s, and
the results of that mapping are *not* always easily invertible.  For
example, after mapping a certain German street name, "passstrasse" is
the result - a native speaker of German knows what the correct input
string was (only the last double-s was an ezsett), but computers don't
tend to be native speakers of German ;-).  This mapping of ezsett to
double-s is now wrong according to German language experts (it's
apparently as wrong in German as mis-spelling "receive" as "recieve"
would be in English), hence ezsett will become an allowed Unicode
character in domain names.

If I read Nico's note correctly that NFSv4 uses stringprep for some
strings, then this change will likely have to be dealt with, and not
doing so results in implicitly specifying that NFSv4 not deal correctly
with some German strings.

Caveat: I don't know all the NFSv4 details here, and to some extent I'm
reasoning by analogy from iSCSI where stringprep is used for on-the-wire
strings that are compared for equality (and hence these mapping changes
will have to be dealt with).

Thanks,
--David

> -----Original Message-----
> From: Noveck, David
> Sent: Wednesday, March 31, 2010 3:30 PM
> To: Black, David; [email-masked]
> Cc: Black, David
> Subject: RE: [nfsv4] work items to complete rfc3530bis (what are
they?)
> 
> > It's also the situation that
> > at least for stringprep, there are four Unicode input characters
whose
> > on-the-wire representation changes incompatibly due to character
mapping
> > changes.
> 
> Will someone wake me up from this nightmare?
> 
> Suppose I decided that we should swap ASCII a and b, also c and d.  I
know the existing mapping has
> served us well, and it is used in many existing files, but it really
gets boring to stick to the same
> old mapping.  Perhaps that sounds a little "drbzy"?
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of [email-masked]
> Sent: Wednesday, March 31, 2010 2:41 PM
> To: [email-masked]
> Cc: Black, David
> Subject: Re: [nfsv4] work items to complete rfc3530bis (what are
they?)
> 
> Nico,
> 
> > IDNAbis doesn't radically change IDNA, at least not in ways that
> matter
> > to NFSv4.  There's still A-labels and U-labels (new terminology) and
> > IDN-aware and IDN-unaware domainname slots, and we still need to
> decide
> > whether NFSv4 domainname slots are IDN-aware or not.
> 
> Unfortunately the IDNAbis mappings draft is Dead, and that's a source
of
> incompatibilities if IDNs are currently being used with NFSv4.  One
> potential problem area is that IDNAbis has no notion of case
> folding/conversion (e.g., to lower case).  It's also the situation
that
> at least for stringprep, there are four Unicode input characters whose
> on-the-wire representation changes incompatibly due to character
mapping
> changes.
> 
> > The Newprep BoF isn't relevant to the IDNAbis work, but it is
> > potentially relevant to any protocol that uses stringprep profiles
for
> > anything other than a domainname slot (that includes NFSv4).
> 
> I would suggest that *if* NFSv4 cares about Unicode issues, *then*
> it(we) should join the newprep effort.  Any decision about whether to
> adopt the stringprep-bis approach that is likely to emerge from
newprep
> can be deferred until that result is better understood.  The
alternative
> of NFSv4 tackling Unicode issues on its own is unappealing, IMHO ...
and
> I would hazard a guess that the responsible ADs will use even more
> negative language on this topic if asked ;-).
> 
> Thanks,
> --David
> 
> > -----Original Message-----
> > From: Nicolas Williams [mailto:[email-masked]]
> > Sent: Wednesday, March 31, 2010 1:51 PM
> > To: Black, David
> > Cc: [email-masked]; [email-masked];
> [email-masked]; [email-masked];
> > [email-masked]
> > Subject: Re: [nfsv4] work items to complete rfc3530bis (what are
> they?)
> >
> > On Tue, Mar 30, 2010 at 08:56:59PM -0400, [email-masked] wrote:
> > > > Search the RFC DB for "IDN" and "IDNA" and "Internationalized
> Domain
> > > Names".
> > >
> > > It gets worse ...  Anyone who believes they understand how Unicode
> works
> > > in IDN needs to go look @ the proceedings of the newprep BOF in
> Anaheim.
> > > There are IDN changes coming (approved Internet-Drafts that are
not
> yet
> > > published as RFCs).
> >
> > All of these IDNAbis I-Ds are on the RFC-Editor queue right now:
> >
> >  - draft-ietf-idnabis-protocol-18
> >  - draft-ietf-idnabis-rationale-17
> >  - draft-ietf-idnabis-defs-13
> >  - draft-ietf-idnabis-bidi-07
> >  - draft-ietf-idnabis-mappings-05
> >  - draft-ietf-idnabis-tables-09
> >
> > https://datatracker.ietf.org/doc/draft-ietf-idnabis-protocol/
> > https://datatracker.ietf.org/doc/.../
> >
> > The Newprep BoF isn't relevant to the IDNAbis work, but it is
> > potentially relevant to any protocol that uses stringprep profiles
for
> > anything other than a domainname slot (that includes NFSv4).
> >
> > IDNAbis doesn't radically change IDNA, at least not in ways that
> matter
> > to NFSv4.  There's still A-labels and U-labels (new terminology) and
> > IDN-aware and IDN-unaware domainname slots, and we still need to
> decide
> > whether NFSv4 domainname slots are IDN-aware or not.
> >
> > Nico
> > --
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 1 Apr 2010 13:06:55 -0500

> on matters like this, especially since perfection is moving target.  

This is precisely why your proposal (which I agree with) is the best way
forward: clients and servers have to be prepared to handle both,
A-labels and U-labels on the wire in NFSv4 domainname slots.

That means that for interop with non-ASCII domainnames people will need
clients and servers that either both do the same thing (send A-labels or
send U-labels, or send whatever users typed in) or that actually have
code to handle whatever the peer sends.

> changes.  They're going to do it anyway.

There's no need to re-write anything, just indicate that NFSv4's
domainname slots are (in your proposal) IDN-aware and accept both,
A-labels and U-labels, and that IDNA string prepration applies to all
U-labels in NFSv4 domainname slots.  Between that and normative
references to IDNA we're covered.

Nico
--

---

From: James Lentini <[email-masked]>
Date: Fri, 2 Apr 2010 11:41:42 -0400 (EDT)

> references to IDNA we're covered.

As noted earlier on this thread, the FedFS protocols contain RFC3490 
domain name slots. Specifically, the NSDB's LDAP attributes 
fedfsNetAddr, fedfsNsdbName, and fedfsFslHost and the ADMIN protocol's 
FedfsNsdbName type.

In our FedFS meeting yesterday, the attendees felt it would be prudent 
for FedFS to adopt the same IDN policy as NFS. That creates a 
dependency between this discussion and the IDN policy used by the 
FedFs protocols.

Is the conclusion of this thread that 3530bis will define domain name 
slots as IDN-aware? There appeared to be opinions on both sides of 
this issue when this discussion started.

>From the FedFS perspective, we have not identified a reason to prefer 
a particular IDN policy.

-james

---

From: <[email-masked]>
Date: Fri, 2 Apr 2010 11:57:37 -0400

> this issue when this discussion started. 

I don't think I've recently heard opinions "on the other side".

I would summarize what I have heard on the issue recently under the
following headings:

1) Define them as IDN-aware

2) Defining them as IDN-aware is not sufficient and we need to point
   to new specs including some not approved yet due to character mapping
   issues (the best summary I can come up with of David Black's
   position; he can correct me when he gets back from vacation).

3) Please stop it.  You're making my head hurt.

My position is basically (3) but I'd be willing to switch to (1) as part
of a compromise that involved meaningful cap-and-trade legislation :-)

-----Original Message-----
From: James Lentini [mailto:[email-masked]] 
Sent: Friday, April 02, 2010 11:42 AM
To: Nicolas Williams
Cc: Noveck, David; Black, David; [email-masked]
Subject: Re: [nfsv4] work items to complete rfc3530bis (what are they?)

a particular IDN policy.

-james

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 2 Apr 2010 19:41:27 -0400

> I don't think I've recently heard opinions "on the other side".

Well, I feel like I'm mostly repeating myself, but:

	- So far, it appears that all current implementations treat
	  domain-name slots as non-IDN-aware.  (For example, the Linux
	  client will fail to follow a referral of a non-ascii domain
	  name.)  Feel free to correct me here.

	- I'm not interested in "fixing" those implementations absent an
	  extremely clear description of what problem we're actually
	  trying to solve for our users.

	- The IDN stuff sounds complicated, frankly.  And it's changing,
	  too?  I'd rather leave it out of core filesystem code, and let
	  higher layers (applications, admin gui's, whatever) deal with
	  it.

--b.

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 2 Apr 2010 20:35:55 -0500

> 	  higher layers (applications, admin gui's, whatever) deal with

It'd be useful to know whether people are using LDAP DNs with non-ASCII,
UTF-8 in them.

In any case, I don't quite agree.  I think you'll eventually need to
implement ToASCII() and ToUnicode() regardless of whether you're on a
client or server.  One way or another, you'll find yourself needing to.

Nico
--

---

From: "Mike Eisler" <[email-masked]>
Date: Mon, 5 Apr 2010 07:23:34 -0700

> I don't think I've recently heard opinions "on the other side".

There was absolutely no notion in RFC3530 of IDN (i.e. that non ASCII characters
would be escaped). So declaring after the fact that domain names are
only IDN-aware and cannot contain non-ASCII (but UTF8) characters would be wrong.

My understanding is that IDN domains uses backslash ('\' to escape non-ASCII
characters. Since backslash is not a valid domain name character, it follows that
clients and servers can easily discern IDN from non-IDN:

- If it has both byte with the highest order bit set and a another byte equal to '/'.
  the data is invalid.
- Otherwise:
-- - If it has a byte equal to '\', then it is IDN.
-- If it does not have a byte equal to '\' then it is not IDN.

The domain name to address resolution logic on each client and server can be
easily modified to accept either forms.

> From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
> Sent: Friday, April 02, 2010 8:58 AM
> To: [email-masked]; [email-masked]
> Cc: [email-masked]; [email-masked]

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 5 Apr 2010 10:56:20 -0500

> clients and servers can easily discern IDN from non-IDN:

Your understanding is not correct.  And it sounds to me like you want to
argue that NFSv4 should be considered IDN-aware :)

IDN-aware means that A-labels and U-labels both can appear in a
domainname slot.  An A-label is a domainname label that contains
non-ASCII characters but encoded in ASCII using punycode; A-labels start
with the prefix "x--" and the characters used are those allowed for
hostnames (letters, digits and hyphens).  A U-label is a Unicode (UTF-8
in our case) string containing non-ASCII characters.

IDN-unaware, OTOH, means that you can only send A-labels, i.e., ASCII
only.

Nico
--

---

From: <[email-masked]>
Date: Mon, 5 Apr 2010 12:01:00 -0400

Yes, but I believe Mike is right about this:

> that clients and servers can easily discern IDN from non-IDN:

Or as you would say it, A-labels and U-labels.

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]] 
Sent: Monday, April 05, 2010 11:56 AM
To: Mike Eisler
Cc: Noveck, David; [email-masked]; Black, David; [email-masked]
Subject: Re: [nfsv4] work items to complete rfc3530bis (what are they?)

Nico
--

---

From: Chuck Lever <[email-masked]>
Date: Mon, 05 Apr 2010 12:06:35 -0400

> IDN-unaware, OTOH, means that you can only send A-labels, i.e., ASCII

Here's where I am a little confused.  Since punycode is ASCII-only, it 
can be conveyed via an IDN-unaware slot.  However, both ends would need 
to be capable of dealing correctly with the "x--" prefix.  Does that 
mean an IDN-unaware slot can, or cannot, contain punycode?

And what about the mixed string type?  For "user@domain" I think we are 
never interested in resolving the "domain" part.  We're just using that 
for string matching, right?  So in that case, the only real concern is 
the collation used (cs or ci).  No?

-- 
chuck[dot]lever[at]oracle[dot]com

---

From: "Mike Eisler" <[email-masked]>
Date: Mon, 5 Apr 2010 09:13:00 -0700

> An A-label is a domainname label that contains
> non-ASCII characters but encoded in ASCII using punycode; 

OK

> A-labels start
> hostnames (letters, digits and hyphens).  

Which doesn't detract from my point. We can tell if punycode is being used
or not by the presence of a backslash ... the "x--' prefix is a hint
that punycode might be present.

> A U-label is a Unicode (UTF-8

OK

> IDN-unaware, OTOH, means that you can only send A-labels, i.e., ASCII

That contradicts: " An A-label is a domainname label that contains
> non-ASCII characters"

> argue that NFSv4 should be considered IDN-aware :)

I think that NFSv4 client and server implementations MAY be IDN aware, but
I don't see why they MUST or SHOULD be. Treat a string of 8 bit characters opaquely,
and leave to resolvers to cope with whether there is pure ASCII without backslashes,
pure ASCII with backslashes, or UTF-8 with the higher bit set in some bytes.
If a site mixes puny with UTF-8-with-high-bit-set-in-some-bytes, and some of
then names map to the same domain, too bad for
the site admin and his users.

It's another case where IETF has dropped the ball on i18n, and I've
no desire to follow its lead down another dead end like string prep. When IETF
has full Standards for i18n, and 25 years of experience, then I might get
interested.

> Sent: Monday, April 05, 2010 8:56 AM
> To: Mike Eisler
> Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 5 Apr 2010 11:29:01 -0500

On Mon, Apr 05, 2010 at 12:06:35PM -0400, Chuck Lever wrote:
> Here's where I am a little confused.  Since punycode is ASCII-only,
> it can be conveyed via an IDN-unaware slot.  However, both ends
> would need to be capable of dealing correctly with the "x--" prefix.
> Does that mean an IDN-unaware slot can, or cannot, contain punycode?

An IDN-unaware slot can only contain punycode (the correct term is
'ACE', because punycode doesn't cover the "x--" prefix).  The reason for
this is that a legacy peer might not handle bytes with the 8th bit set.

For example, a caching DNS recursive server might have, and, indeed, has
no clue about IDN.  DNS on the wire is all IDN-unaware, all the time.

> And what about the mixed string type?  For "user@domain" I think we

IDN applies to "domainname slots".  Domainname slots can occur
mid-string, as in the user@domain case.  This means that the "user@" and
"domain" parts of user@domain have different rules and different
treatment.

> are never interested in resolving the "domain" part.  We're just
> using that for string matching, right?  So in that case, the only
> real concern is the collation used (cs or ci).  No?

This is OT for this thread, but since you bring it up...

Solaris most certainly resolves user@domain to "internal" identifier
forms (POSIX UIDs/GIDs and Windows SIDs).  Solaris does not store
user@domain on disk and does not do simple string matching.  This means
that Solaris splits user@domain on the '@' and does whatever is
appropriate with the two pieces.  For example, if 'domain' is a Windows
domain then Solaris will resolve the user part to the SID for that user
in that domain, if the user exists; the resolution process invloves DNS
queries and LDAP searches.

I strongly recommend that implementors NOT take the approach of storing
user@domain on disk instead of internal ID forms.  That approach works
only if you're going to implement an NFSv4-only file server, though even
then you run into or create problems (user/group renames -> bad).  If
you want to implement multi-protocol file server then storing
user@domain on disk does not relieve you of the need to perform name
resolution, nor the need to perform ID mapping (e.g., between NFSv3 and
CIFS).  Kicking the ID mapping can to a third party, say, Samba, does
not reduce system complexity -- it probably increases it.  Plus there's
the additional constraints on processes that storing user@domain on disk
imposes on users: you can't rename users/groups, and you can't reuse
user/group names, ever.

(NFSv4 should really have used SIDs instead of user@domain, and should
have provided a simple name resolution sub-protocol for resolving
server-local SIDs.  This is something we could pursue in v4.2; in fact,
I recommend we do.  Yes, we're already stuck with what we already do to
cope with name@domain on the wire, but hopefully some day we could
actually simplify the system.)

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 5 Apr 2010 11:42:20 -0500

> that punycode might be present.

Was that your point?

> That contradicts: " An A-label is a domainname label that contains

The A-label, as encoded, contains only ASCII characters, but
_decoded_ contains non-ASCII characters.

The point is that if you have a domainname label that when _displayed_
to or entered by a user contains non-ASCII characters, then to place it
in an IDN-unaware slot you must put it in A-label form.  But to place it
in an IDN-aware slot you can send the Unicode as is.

> and leave to resolvers to cope with whether there is pure ASCII without backslashes,

Resolvers are IDN-unaware.  

> pure ASCII with backslashes, or UTF-8 with the higher bit set in some bytes.

What backslashes?!  There are no backslashes to deal with.

> the site admin and his users.

That's not a useful attitude.

> has full Standards for i18n, and 25 years of experience, then I might get

The Mike I know is wiser than this; I'm surprised.  I recommend you
first get familiar with the specs and save the criticism for when you
understand the issues.  (Which isn't to say that IDNA does not deserve
criticism.)

Let's cut to the chase though.  All that developers of IDN-aware
applications need do is implement two relatively simple functions
(relative to other Unicode-related functions that they probably also
have to implement): ToASCII() and ToUnicode().  These are specified by
IDNA.  Developers of IDN-aware applications also need to know what
interfaces that they use are IDN-aware and which ones are IDN-unaware,
so that they know when to apply ToASCII()/ToUnicode().

(The hard part in implementing ToASCII() and ToUnicode() is having to
implement Unicode normalization.  There's a fair number of open source
packages that handle that.)

Nico
--

---

From: "Mike Eisler" <[email-masked]>
Date: Mon, 5 Apr 2010 11:20:22 -0700

> Resolvers are IDN-unaware.

Why should Resolvers get to be unaware and every bloody app that uses them
has to be IDN-aware to convert the names so that the resolver can cope+
That makes no sense at all. Fix gethostbyname and friends.

> What backslashes?!  There are no backslashes to deal with.

I found something yesterday that said that punycode used backslashes to escape
non-ASCII characters but today I can't find it. Instead, what I find is that there
is no way to distinguish non-Puny-encoded strings from Puny-encoded strings. Insane.
So based on that, I maintain that all domain names in NFSv4 are UTF-8, non-puny-encoded;
i.e. strings built from U-labels.

> That's not a useful attitude.

That depends on ones point of view. IETF has created a monster and it isn't my job to
slay it.

> The Mike I know is wiser than this; I'm surprised.  I recommend you

The Mike I know is tired of being burned by IESG on its latest B.S. scheme
for i18n, of which IDN is a clear example. IESG had two opportunities two
impose punycode on us (RFC3530 and RFC5661) and never uttered a word. No
surprise there.

I understand enough. From one of Nico's earlier emails:

> hostnames (letters, digits and hyphens)

Since x-- is valid non-punycode prefix of a domain name, it therefore follows
that there is no way to distinuish non-puny-encoded ASCII domain names from
puny-encoded ASCII domain names.

> packages that handle that.)

Normalization: another thing I don't care about. Few people on this list
care about normalization.

---

From: <[email-masked]>
Date: Mon, 5 Apr 2010 14:30:56 -0400

I've just noted what the title here is.

I'd like to suggest that we now know that two of the issues are:

    What we need to do about IDN? (that might mean we decide to
    do nothing but it is still an issue).

    What, if anything, we need to do about the reference to 
    stringprep, in light of how handling of i18n issues in
    the IETF has "evolved" (as biologists often inform us,
    evolution is not necessarily progressive :-).

My suggestion is that we discuss/debate these on appropriately named
threads to give people the option of tuning out issues they don't think
they can make a contribution to.  This thread can continue for people to
suggest new issues that we need to deal with.

I also think addressing these would be more efficient if we had some
concrete proposals (i.e. "of the form change the spec which now says x
to say y").  Is there a current version of the rfc3530bis xml files that
people can use a baseline?

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]] 
Sent: Monday, April 05, 2010 12:42 PM
To: Mike Eisler
Cc: Noveck, David; [email-masked]; Black, David; [email-masked]
Subject: Re: [nfsv4] work items to complete rfc3530bis (what are they?)

Was that your point?

Nico
--

---

From: <[email-masked]>
Date: Mon, 5 Apr 2010 14:34:29 -0400

> Since x-- is valid non-punycode prefix of a domain name,

I think you are wrong about that.  The Punycode encoding depends on
hyphen not being valid in domain names, at least I'm pretty sure I read
that in the Wikipedia article. 

> it therefore follows that there is no way to distinuish 
> non-puny-encoded ASCII domain names from

And so none of this could possibly work at all?  That doesn't sound like
the IETF.  The IETF has problems in this area, but they aren't of that
sort, at least that isn't my experience.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Monday, April 05, 2010 2:20 PM
To: 'Nicolas Williams'; Noveck, David; [email-masked]; Black,
David; [email-masked]

> Resolvers are IDN-unaware.

Why should Resolvers get to be unaware and every bloody app that uses
them
has to be IDN-aware to convert the names so that the resolver can cope+
That makes no sense at all. Fix gethostbyname and friends.

> What backslashes?!  There are no backslashes to deal with.

I found something yesterday that said that punycode used backslashes to
escape
non-ASCII characters but today I can't find it. Instead, what I find is
that there
is no way to distinguish non-Puny-encoded strings from Puny-encoded
strings. Insane.
So based on that, I maintain that all domain names in NFSv4 are UTF-8,
non-puny-encoded;
i.e. strings built from U-labels.

> That's not a useful attitude.

That depends on ones point of view. IETF has created a monster and it
isn't my job to
slay it.

> The Mike I know is wiser than this; I'm surprised.  I recommend you

The Mike I know is tired of being burned by IESG on its latest B.S.
scheme
for i18n, of which IDN is a clear example. IESG had two opportunities
two
impose punycode on us (RFC3530 and RFC5661) and never uttered a word. No
surprise there.

> hostnames (letters, digits and hyphens)

Since x-- is valid non-punycode prefix of a domain name, it therefore
follows
that there is no way to distinuish non-puny-encoded ASCII domain names
from
puny-encoded ASCII domain names.

---

From: "Mike Eisler" <[email-masked]>
Date: Mon, 5 Apr 2010 11:43:25 -0700

http://tools.ietf.org/html/rfc1035

The labels must follow the rules for ARPANET host names.  They must

start with a letter, end with a letter or digit, and have as interior

characters only letters, digits, and hyphen.

> From: [email-masked] [mailto:[email-masked]]

> Sent: Monday, April 05, 2010 11:34 AM

> To: [email-masked]; [email-masked]; [email-masked]; [email-masked];

> Subject: RE: [nfsv4] Dealing with IDN in rfc3530bis

> > > > Your understanding is not correct.  And it sounds to me like you

> want to

> > > I think that NFSv4 client and server implementations MAY be IDN

> aware, but

> > > I don't see why they MUST or SHOULD be. Treat a string of 8 bit

> characters opaquely,

> > > and leave to resolvers to cope with whether there is pure ASCII

> without backslashes,

> > > pure ASCII with backslashes, or UTF-8 with the higher bit set in

> some bytes.

> > > If a site mixes puny with UTF-8-with-high-bit-set-in-some-bytes, and

> some of

> > > no desire to follow its lead down another dead end like string prep.

> When IETF

> > > has full Standards for i18n, and 25 years of experience, then I

> might get

---

From: <[email-masked]>
Date: Mon, 5 Apr 2010 14:46:35 -0400

If someone points me at baseline xml files for RFC3530bis, I'll try to
make a proposal in the next few weeks about what to do about this (the
stringprep/normalization/ezsett part).  My ambitious goal is to come up
with something that satisfies Mike Eisler and Nico!  On the other hand,
I explicitly do not adopt the goal of satisfying David Black.

When David Black comes back from vacation, we can amuse the working
group, by each telling everybody why the other is completely crazy, with
our common employer paying for our time to do so.  The fact that we are
both likely stockholders/optionees will make sure that we don't let this
get completely out of hand :-) 

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Monday, April 05, 2010 2:20 PM
To: 'Nicolas Williams'; Noveck, David; [email-masked]; Black,
David; [email-masked]

want to
aware, but
characters opaquely,
without backslashes,

some bytes.

some of

When IETF
might get

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 5 Apr 2010 13:46:24 -0500

> That makes no sense at all. Fix gethostbyname and friends.

gethostbyname() is one thing.  The DNS resolver is another.  (And yes,
you can have IDN-aware resolvers, but you still need local ToASCII()/
ToUnicode() implementations.)

> impose punycode on us (RFC3530 and RFC5661) and never uttered a word. No

We've been burned?  How?  Also, NFSv4 is *HUGE*; it should not surprise
anyone that some things fell through the cracks.

> puny-encoded ASCII domain names.

A risk of collisions was unavoidable as long as the solution was
punycode; the risk is also minimal and avoidable, and therefore
acceptable.  The alternative was to break things and just send 8-bit
data hoping eventually various resolvers and servers would get fixed --
not necessarily a bad plan, but it's certainly too late for it now.

> care about normalization. 

We (Solaris developers) do care, and we do because we found that there
are popular operating systems whose input methods and what not prefer
different normalizations forms, therefore we concluded that we needed to
care about normalization.  Specifically: MacOS X seems to prefer
decomposition while Windows Solaris, Linux [and probably *BSD] prefer
composition.  (We implemented normalization-insensitive file name
comparison for LOOKUPs, but preserve normalization on CREATE.)

Nico
--

---

From: Spencer <[email-masked]>
Date: Mon, 5 Apr 2010 13:57:43 -0500

> I explicitly do not adopt the goal of satisfying David Black.

I would prefer that a sketch of that solution be provided here
on the mailing list before you spend time putting it into the final
format.  If you feel that this discussion has satisfied that, please summarize
here on the list before jumping into final formatting. 

Spencer

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 5 Apr 2010 13:56:27 -0500

> I explicitly do not adopt the goal of satisfying David Black.

It'd be nice to understand what Mike would be open to.  Right now I'm
not sure where he stands (other than his dislike of I18N in general and
the IETF's approach to it).  Perhaps we can have some clarity.

Mike, what is it that you object to:

 - punycode on the wire
 - Unicode on the wire
 - having to implement ToASCII() and ToUnicode()
 - having to have an implementation of ToASCII() and ToUnicode()
   anywhere in the stack
 - having to ... ToASCII() and ToUnicode() ... on the client
 - having to ... ToASCII() and ToUnicode() ... on the server
 - IDN no matter what form it takes or could have taken
 - something else

?

Having to implement ToASCII() and ToUnicode() in the client is
practically unavoidable.  We could avoid it on the server if we can
ensure that the server need never handle IDN-aware slots in any
protocols.  I suspect we'll not be able to avoid having to have
ToASCII() and ToUnicode() on both, the client and the server.

> get completely out of hand :-) 

<laughter/>

Nico
--

---

From: "Mike Eisler" <[email-masked]>
Date: Mon, 5 Apr 2010 12:06:23 -0700

> To: [email-masked]
> Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]
> Subject: Re: [nfsv4] What to do about stringprep for RFC3530bis
> 

Yes

>  - Unicode on the wire

Yes.

UTF-8 I am fine with.

>  - having to implement ToASCII() and ToUnicode()

Yes.

>  - having to have an implementation of ToASCII() and ToUnicode()

Yes.

>  - having to ... ToASCII() and ToUnicode() ... on the client

Yes.

>  - having to ... ToASCII() and ToUnicode() ... on the server

Yes.

>  - IDN no matter what form it takes or could have taken

I object to IDN in its current form.

> practically unavoidable.  We could avoid it on the server if we can

Not at all. The NFSv4.[01] specifications
define utf8 data types for encoding domain names
as UTF8. So it is entirely avoidable.

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 5 Apr 2010 14:18:41 -0500

> > Mike, what is it that you object to:

UTF-8 is an encoding of Unicode.  In the context of NFSv4 _obviously_ we
mean UTF-8 as the encoding when we talk about Unicode.

> I object to IDN in its current form.

Too late.  There's no other form it could take now.

> as UTF8. So it is entirely avoidable.

If you have to interface with any IDN-unaware domainname slots in other
protocols then you need ToASCII() and ToUnicode().  There are
IDN-unaware domainname slots, all over the place, starting with DNS
itself.  Thus if you need to resolve a domainname containing U-labels
(i.e., Unicode, or, for you, UTF-8), then you need an implementation of
ToASCII() somewhere in your stack.  And if you need to resolve IP
addresses (or whatever) to domainnames that you'll then put on the wire
in NFSv4 as Unicode (UTF-8) then you'll need ToUnicode() somewhere in
your stack too.

Please explain what we can say in RFC3530bis that makes it possible to
completely avoid having to have ToASCII() and ToUnicode() in the client
and server.  Me, I don't see it.  I also don't see how we can avoid IDNA
short of thumbing our noses at it and the IETF/IESG.

Nico
--

---

From: "Mike Eisler" <[email-masked]>
Date: Mon, 5 Apr 2010 12:40:26 -0700

> We've been burned?  How?  Also, NFSv4 is *HUGE*; it should not surprise

By being forced to put nameprep garbage in at the last minute for RFC3530.

> anyone that some things fell through the cracks.

Well when RFC5661 was being reviewed by IESG, it brought up the fact
that nameprep was deprecated now at IETF. I was amazingly diplomatic in
my response. The point being that IESG read botgh RFCs, said nothing about
IDN. 

> not necessarily a bad plan, but it's certainly too late for it now.

The risk exists. Had IETF picked an encoding method that used an escape
character that is not a valid DNS domain character, the problem would be
solved. Thus risks would be avoidable.

> comparison for LOOKUPs, but preserve normalization on CREATE.)

Let the clients send what they want as long as it is valid utf8.

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 5 Apr 2010 14:58:07 -0500

> my response. The point being that IESG read botgh RFCs, said nothing about

Surprises happen during IESG review, during IETF LC, and even during
AUTH48.  Sometimes you just have to let them slide -- there will be an
RFCxxxx-bis eventually.  Other times you hurry up and let them slide,
then get annoyed.  Other times you take your time and get it right.

> solved. Thus risks would be avoidable.

DNS itself allows any ASCII character other than NUL and '.'.
_Hostnames_ are restricted to LDH (letters, digits, and hyphen).
Because IDNs need to be useable where hostnames go... IDNA had to be
limited to LDH, with no possibility for escape sequences.

I don't think the authors of IDNA were stupid.  We can argue about
alternatives that might have been better.  But the approach selected was
by and large done right (there's the issues that David Black told us
about, but on the whole I think those are rather minor).

> Let the clients send what they want as long as it is valid utf8.

That's exactly what we do!

But consider a Windows client and a MacOS client trying to access the
same file named 'neé'.  If the server knows nothing about normalization
then one of those clients will fail.  Worse: the same file name could be
created twice, once in each form, and still each client would be able to
open (delete, ...) one but not the other.

The only solution is to normalize on the server, and the only question
is whether to normalize on CREATE or whether to do normalization-
insensitive matching on LOOKUP (we picked the latter).

Well, there is another solution: tell your users that they're on their
own as far as I18N is concerned, and that their best bet is to avoid
non-ASCII.  That works.  It isn't user-friendly.  Users get a vote, you
know.

All this carping boils down to one thing: "I don't want to have to
implement Unicode normalization anywhere, for anything, because it's
scary and looks like a lot of code and data".  The minute you've
implemented that you'll stop complaining.

Oh, and there's not much performance impact to doing normalization
since: a) the complex parts of the algorithm are avoided altogether for
every pair of contiguous ASCII characters, b) most strings will be all
ASCII (see (a)), c) most strings will be short, d) you'll have to
traverse most strings at least once for comparison or hashing in LOOKUP.

Nico
--

---

From: "Mike Eisler" <[email-masked]>
Date: Mon, 5 Apr 2010 13:03:12 -0700

> IDN-unaware domainname slots, all over the place, starting with DNS

I don't know what domainname slots are. Slots in this WG are session things.

> (i.e., Unicode, or, for you, UTF-8), then you need an implementation of

Since U-labels are apparently an artifact of a DNS that understands
UTF_8, it follows that because IETF has define a DNS that accepts U-labels,
we don't need to do anything else. DNS implementers need to get busy.

> short of thumbing our noses at it and the IETF/IESG.

IESG has effectively cleared RFC3530bis from IDN A-label nonsense since it approved
RFC5661. So IESG is fine with a U-label-only approach.

---

From: "Mike Eisler" <[email-masked]>
Date: Mon, 5 Apr 2010 13:23:32 -0700

> then get annoyed.  Other times you take your time and get it right.

In this case, we told IESG to stuff it. They forced nameprep onto NFSv4.0,
it is in NFSv4.1, period.

> about, but on the whole I think those are rather minor).

We can agree (disagree) to disagree. Overloading a valid label prefix was
wrong. 

> open (delete, ...) one but not the other.

We've the same issue with case sensitivity/insensitivity. So what?

Set up the apps on the clients that that UTF-8 chars in file names
to use a common normalization.

The number #1 NFSv4 client doesn't even enforce UTF-8 on requests, consequently
there's little appetite for further refining i18n in file names.

---

From: <[email-masked]>
Date: Mon, 5 Apr 2010 16:47:19 -0400

I will send out a brief summary, but I am not intending to make
proceeding to actual spec text contingent on the discussion ending.  Our
current example is that the discussion does not end, in any reasonable
time.  I think the issue we have is that this is not just a formatting
issue (in the narrow sense).  People write for the summary in a very
schematic way for those that share their approach to the matter (since
their approach is the obviously correct one, who else would they write
for).  I think that when you write for the spec, you may take more time
but the goal is to be clear to anyone reading the document, and I think
that extra effort is worth it in this case, since the more schematic
approach is not converging.  It seems to be diverging.   

-----Original Message-----
From: Spencer [mailto:[email-masked]] 
Sent: Monday, April 05, 2010 2:58 PM
To: Noveck, David
Cc: [email-masked]; [email-masked]; [email-masked];
Black, David; [email-masked]
Subject: Re: [nfsv4] What to do about stringprep for RFC3530bis

I would prefer that a sketch of that solution be provided here
on the mailing list before you spend time putting it into the final
format.  If you feel that this discussion has satisfied that, please
summarize
here on the list before jumping into final formatting. 

Spencer

they?)

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 5 Apr 2010 15:58:47 -0500

> it is in NFSv4.1, period.

Looking at the ballot it seems that the IESG folded only because v4.1
came before the updates to IDNA.

> We can agree (disagree) to disagree. Overloading a valid label prefix was

I'm not disagreeing.  The point: it doesn't matter _now_.  It happened,
it can't be undone, get over it.

> We've the same issue with case sensitivity/insensitivity. So what?

(Actually, we also have an option for case-insensitive, case-preserving
behavior.  For the same reasons.)

But it's worse for normalization than for case.  With case the user can
see the correct case and type it.  Even if the UI unhelpfully changes
the case of filenames the user can try  various combinations if need be.
Whethers with normalization there's _nothing_ that the average user can
do, and in some cases there's nothing even a power user can do, to get
at a file with a name encoded just so.  They can see it, but can't get
at it (even though they have permission to read, write, ...).

> to use a common normalization.

You _can't_.  This isn't about apps.  This is about operating system
input modes.  Also, here "apps" -> "all apps".  Requiring changes to the
apps is a non-starter.  Requiring changes to the client OSes... almost
certainly a non-starter as well.  Worse, for you, there exist servers
that get this right, so when you tell a customer to tell their client
vendors to fix their OSes, your customer's client vendors will simply
point out that servers can and do get this right and that this is
therefore _your_ problem.

> The number #1 NFSv4 client doesn't even enforce UTF-8 on requests,
> consequently there's little appetite for further refining i18n in file
> names. 

But that's not a problem because users can simply use UTF-8 locales.
But even using UTF-8 locales does not insulate you from the
normalization issues.  Servers can, and do.

Nico
--

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 05 Apr 2010 18:13:14 -0400

> therefore _your_ problem.

When you say "get it right", what do you mean?

AFAIK normalisation implicitly assumes an encoding system. What happens
then if my client's application isn't using UTF-8, but is using BIG5 or
Latin1 or some other encoding system? There are plenty of client
environments out there that don't enforce UTF-8...

I do not at all understand your distinction between implementing
normalisation-insensitive file name comparisons on LOOKUP, but
preserving normalisation on CREATE. If you are requiring (as you clearly
do above) that your UTF-8 aware application should not have to normalise
anything, then surely preserving normalisation on CREATE is insufficient
to ensure that READDIR (for instance) returns correctly normalised UTF-8
output. How does your server ensure that READDIR returns output in the
application's normalisation scheme for files that were created by a
different client?

BTW: Did MacOS ever ship an NFSv4 client, or are we ranting on about the
properties of vapourware clients here?

> normalization issues.  Servers can, and do.

That's not an argument for making normalisation a requirement in the
spec. The fact is that 7 years after the original spec was published,
almost all NFSv4 servers out there in the field don't...

Trond

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 5 Apr 2010 17:44:17 -0500

> When you say "get it right", what do you mean?

Allow both clients to access one file named 'neé' (as in my example),
with only one such file in the containing directory, and regardless of
what form of 'neé' (there's two) the client sent.

> environments out there that don't enforce UTF-8...

Actually, you _can_ treat text as 8-bit clean _and_ normalize on _LOOKUP_.
The cases where non-UTF-8 will alias as some valid UTF-8 sequence will
be few, if any, and if you implement normalization-insensitive-but-
preserving ("n-i/p" from here on) behavior then the worst that could
happen is that you'd prevent creation of two filenames that have
different non-UTF-8 names but which look two encodings of the same UTF-8
name.

What you _can't_ do is treat text as 8-bit clean and normalize on _CREATE_.
Because then if there's any codeset aliasing issues you'd be changing
the meanings of the file names created by users.

> application's normalisation scheme for files that were created by a

A file server protocol using Unicode for file naming (UTF-8, UTF-16,
UTF-32, punycode -- it doesn't matter what encoding) has these choices:

a) declare or negotiate a normalization form, client MUST normalize

b) server normalizes to its preferred normalization form on CREATE,
   LOOKUP, ...

c) server performs normalization-insensitive string comparisons (and
   hashing, if it needs to hash) but preserves form on CREATE

   A normalization-insensitive comparison of two strings is equivalent
   to normalizing both strings first then doing a memcmp() on the two
   normalized strings.  But if you're clever you can avoid allocation by
   normalizing on the fly (we did just that).

NFSv4 really only gives you choices (b) and (c).  The spec is unclear
about this, but that's really the only choices you have short of saying
"to hell with normalization".

We picked (c).

> properties of vapourware clients here?

When we decided to implement n-i/p we only tested the MacOS X NFSv3
client and server.  The MacOS X NFSv3 server normalizes to NFD on
CREATE.  The client, IIRC, normalizes to NFD always (or maybe that's
just a side effect of the MacOS X input mode).

The fact that MacOS X seemed so wedded to NFD gave us impetus to
implement n-i/p on our server.  All other OSes seem to accidentally
prefer NFC (accidentally because of their input modes; there seems to be
no additional normalization going on).  (Well, "all others" now excludes
OpenSolaris, since it does n-i/p.)

> almost all NFSv4 servers out there in the field don't...

Certainly not all.  The Sun Storage 7xxx appliance most certainly does
normalize as described earlier.  OpenSolaris also does.

I don't know what the various third-party NFSv4 client/server
implementations for MacOS do, but we suspected that the normalization-
to-NFD-on-CREATE was happening at the filesystem layer, in which case
those other implementations of NFSv4 for MacOS X will almost certainly
do the same.

Nico
--

---

From: Mike Mackovitch <[email-masked]>
Date: Mon, 5 Apr 2010 17:12:09 -0700

> > properties of vapourware clients here?

Mac OS X's NFS client implementation has had bits that are capable of
speaking NFSv4 since 10.5 (2007).  Though, because it hasn't been
considered ready for general use, we've required the use of the special
mount option "-o vers=4.0alpha" in order to gain access to it (as
documented in the mount_nfs(8) man page).

The 10.5 release didn't implement any state handling.  It was essentially
just an NFSv3 implementation that could talk NFSv4.

The 10.6 release added state management/recovery and callbacks.

Hopefully, the last remaining bits of functionality (like ID mapping,
"mirror mounts") should be coming "soon".

Unfortunately, in case it's not already obvious, NFS hasn't been
a real priority for Apple.  Not quite "vapourware" but definitely
a long time coming.

> just a side effect of the MacOS X input mode).

On Mac OS X neither the NFS client code nor the NFS server code do
normalization.

The normalization on the server happens within the file system being
exported (typically HFS+).

The normalization on the client happens above the file system in the
various system frameworks/libraries that apps use.  These all seem to
have ingrained in them that file system names should all be in the
form that HFS+ uses (NFD).

So, apps end up doing stupid things like: reading a directory to
let the user select a file; the user selects a file that has an NFC
name on the server; the framework/app converts the name to NFD and
then tries to open it; and the open fails because the server doesn't
recognize the name in NFD form.  End result: user selected an existing
file and got "no such file or directory" in return.

HTH
--macko

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 5 Apr 2010 19:24:44 -0500

> exported (typically HFS+).

That's what we figured.  It follows that an NFSv4 server on MacOS X with
HFS+ filesystems would normalize on CREATE to NFD.

> file and got "no such file or directory" in return.

Right!  Which is why we implemented normalization-insensitive (but
preserving) behavior on the server.

(If the whole world were settled on NFC, either by design or accident we
might not have bothered, though even then n-i/p would clearly have been
more correct.)

Thanks for your confirmation,

Nico
--

---

From: <[email-masked]>
Date: Tue, 6 Apr 2010 10:16:59 -0400

I think the following is Mike Eisler.
> > Let the clients send what they want as long as it is valid UTF8.

This is Nico.
> That's exactly what we do! 

So what's the problem here?  Mike can implement what he wants and Nico can implement what he wants and they are both within the current spec, aren't they.  If they were working on the same server, it would be a problem, but they aren't. 

It doesn't say anything about normalization, IIRC.  I recall Nico saying that we had to pick one normalization form and that not doing so was bad, etc.  (It was a while ago and I don't have an exact quote but my memory about that is pretty clear).

Now Nico has a solution that doesn't do that but satisfactorily addresses the normalization issue from his point of view and so does Mike.  Let's declare victory.  If we have to write some words that say Mike and Nico MAY each choose their particular solutions, let's do that.

Having Mike and Nico agree about what to implement in this area is not a work item needed to complete rfc3530bis, thank goodness. 

If there is a desire, v4.2 could have attributes (on the lines of case-sensitive and case-preserving) to tell clients what the server is doing in this area.

Mike and Nico agree that this all UTF8.  The big problem we have to deal with is with those that disagree with them (and me and RFC3530) about that.  Unfortunately deciding what to say about that issue is a work item needed to complete rfc3530bis.

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]] 
Sent: Monday, April 05, 2010 3:58 PM
To: Mike Eisler
Cc: Noveck, David; [email-masked]; Black, David; [email-masked]
Subject: Re: [nfsv4] work items to complete rfc3530bis (what are they?)

Nico
--