Subject: [I18ndir] I-D on filesystem I18N

From: Nico Williams <[email-masked]>
Date: Mon, 6 Jul 2020 17:51:40 -0500

I've submitted draft-williams-filesystem-18n-00.

The I-D covers the background of the issue, documents the existence of
the VFS, explains why I18N behaviors belong at the filesystem layer
instead of the filesystem protocol server layer, the need to expose I18N
behavior metadata to clients, and provides advice to protocol designers
and filesystem implementors.  It only deals with path components, not
any other I18N issues.

It seems unlikely that this directorate will meet "at" IETF 108.
Nonetheless, this I-D may be useful as a starting point for the issue of
filesystem I18N, and it should be useful as a follow-up to my early
review of draft-dnoveck-nfsv4-internationalization-01.

Cheers,

Nico
--

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 07 Jul 2020 07:01:42 +0200

Nico, I think this is good stuff. I must though of course(!) come with some suggestions and ideas that can make this even better.

When you talk about what context this document is about, I feel you should explicitly say that you do not deal with RTL/LTR issues. This ends up being something that is very very important as well, but display issues is definitely not within scope for this document.

https://stupid.domain.name/node/681
https://stupid.domain.name/node/682
https://stupid.domain.name/node/683

When looking at case folding, I normally succeed by explaining that the definition of case folding in fact is a function, not an attribute on the character. I.e. one can apply the function lower_case() to a character, and then with the help of that function say that "a character is said to be lower case if lower_case(s) is s. I.e. that the character is stable when applying the lower_case() function on it.

The reason for this is that it makes it easier to explain in the next step that the function might very well be (as you say) locale dependent, and I think more important that lower_case() and upper_case() are two functions that might not be inverse of each other. I.e. just because t = lower_case(s) might not imply s = upper_case(t).

When this is understood(!) one can move forward and explain why for example IDNA only look at stability of normalisation and case folding to lower case. I.e. that that is one of the main ways "stability" regarding characters is defined in the IETF, and then how/why we need that stability after storage (as you explained).

   Patrik

On 7 Jul 2020, at 0:51, Nico Williams wrote:

> The I-D covers the background of the issue, documents the existence of the VFS, explains why I18N behaviors belong at the filesystem layer instead of the filesystem protocol server layer, the need to expose I18N behavior metadata to clients, and provides advice to protocol designers and filesystem implementors.  It only deals with path components, not any other I18N issues.
>
> Nonetheless, this I-D may be useful as a starting point for the issue of filesystem I18N, and it should be useful as a follow-up to my early review of draft-dnoveck-nfsv4-internationalization-01.
>
> Cheers,
>
> Nico
> -- 
>
> -- 
> I18ndir mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i18ndir

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 02:04:57 -0500

On Tue, Jul 07, 2020 at 07:01:42AM +0200, Patrik Fältström wrote:
> Nico, I think this is good stuff. I must though of course(!) come with
> some suggestions and ideas that can make this even better.

Thanks!

> When you talk about what context this document is about, I feel you
> should explicitly say that you do not deal with RTL/LTR issues. This
> ends up being something that is very very important as well, but
> display issues is definitely not within scope for this document.

Quite true!

> https://stupid.domain.name/node/683

:)

> When looking at case folding, I normally succeed by explaining that
> the definition of case folding in fact is a function, not an attribute
> on the character. I.e. one can apply the function lower_case() to a
> character, and then with the help of that function say that "a
> character is said to be lower case if lower_case(s) is s. I.e. that
> the character is stable when applying the lower_case() function on it.
> 
> The reason for this is that it makes it easier to explain in the next
> step that the function might very well be (as you say) locale
> dependent, and I think more important that lower_case() and
> upper_case() are two functions that might not be inverse of each
> other. I.e. just because t = lower_case(s) might not imply s =
> upper_case(t).

And case folding can be designed for case-insensitive comparisons, so it
need not be the same as tolower().  That's why we speak of case folding,
not "lowering case" or such like.

> When this is understood(!) one can move forward and explain why for
> example IDNA only look at stability of normalisation and case folding
> to lower case. I.e. that that is one of the main ways "stability"
> regarding characters is defined in the IETF, and then how/why we need
> that stability after storage (as you explained).

Yes.  But it's easier for filesystems anyways, since the rules can vary
by filesystem and are applied by the filesystem (and caching clients),
so we have more locality and less in the way of global rules.  That is
as to case anyways; fortunately there's no need for local variations in
NFs!  In contrast, in DNS, the rules are not applied by the servers[*],
which complicates any attempt to vary the rules applied by clients to
domainnames under different TLDs.

That's a distinction probably worth being more explicit about.  In order
to engineer a solution, one has to know the constraints.  The
constraints in DNS are different than in filesystems.

[*] In principle, I suppose, DNS servers could pretend that all case
    variations of U-labels exist with the same content as the canonical
    A-labels.  (After all, that's what DNS servers are required to do
    for ASCII labels.)  But I suspect no one wants to implement this
    functionality in their DNS servers!  (This would be not unlike,
    e.g., Active Directory pretending that all case variations of the
    same realm name are essentially the same.)

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 02:11:39 -0500

On Tue, Jul 07, 2020 at 02:04:57AM -0500, Nico Williams wrote:
> [...]; fortunately there's no need for local variations in
> NFs!  [...]

In the current context "NFs" is probably confusing.  I meant
normalization forms / I did not typo NFS.  Ay.

---

From: John C Klensin <[email-masked]>
Date: Tue, 07 Jul 2020 08:27:36 -0400

--On Tuesday, July 7, 2020 02:04 -0500 Nico Williams
<[email-masked]> wrote:

>...
>> When you talk about what context this document is about, I
>> feel you should explicitly say that you do not deal with
>> RTL/LTR issues. This ends up being something that is very
>> very important as well, but display issues is definitely not
>> within scope for this document.
> 
> Quite true!

Except that "so not deal with RTL/LTR issues" is equivalent to
"blow off a significant number of people in the world who use
such writing systems".  There are probably ways deal with the
issues in things like file system paths, but saying "display
system are not in scope" and then moving on is probably not a
reasonable member of that set.

>...
>> The reason for this is that it makes it easier to explain in
>> the next step that the function might very well be (as you
>> say) locale dependent, and I think more important that
>> lower_case() and upper_case() are two functions that might
>> not be inverse of each other. I.e. just because t =
>> lower_case(s) might not imply s = upper_case(t).
> 
> And case folding can be designed for case-insensitive
> comparisons, so it need not be the same as tolower().  That's
> why we speak of case folding, not "lowering case" or such like.

With the understanding that it is case folding, regardless of
its other advantages, that causes some of the nasty edge cases,
e.g.,

* At least before the upper-case Eszett ("Sharp S") was
introduced as a code point (and still for backwards
compatibility reasons) 

   toLower(ß) -> ß  (already lower case)

* Similarly,

   toLower(i) -> ı (U+0131, already lower case)

Only when does case folding, mapping the former to "SS" and then
to "ss" and the latter to "I" and then to "i", do nasty edge
cases requiring locale or language-specific treatment arise.

Priority question: we have a whole series of long-outstanding
and core i18n documents more or less in queue with
draft-faltstrom-unicode12 (and the now-required
draft-faltstrom-unicode13), draft-klensin-idna-rfc5891bis,
draft-freytag-troublesome-characters, and some SMTPUTF8 ("EAI")
tweaks that have not been turned into I-Ds as obvious examples
along with, perhaps, draft-sullivan-lucid-prob-stmt, the
notorious draft-klensin-idna-5892upd-unicode70, and others.  How
do we want to prioritize those probably tedious (but important
to make/keep existing standards track specs workping] well)
specs versus work relative to Nico's much more exciting file
system proposal?

    best,
     john

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 07 Jul 2020 15:14:45 +0200

On 7 Jul 2020, at 14:27, John C Klensin wrote:

> riority question: we have a whole series of long-outstanding and core i18n documents more or less in queue with
> tweaks that have not been turned into I-Ds as obvious examples along with, perhaps, draft-sullivan-lucid-prob-stmt, the
> notorious draft-klensin-idna-5892upd-unicode70, and others.  How do we want to prioritize those probably tedious (but important to make/keep existing standards track specs workping] well)
> specs versus work relative to Nico's much more exciting file system proposal?

"My" documents will be worked on starting next week when I on Tuesday is back from vacation. I.e. 7 days from now.

   Patrik

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 10:05:43 -0500

> reasonable member of that set.

For filenames and paths I believe there are only display issues.  That
was Patrik's point.  Thus the I-D should say as much and normatively
reference Unicode.

And yes, the display is absolutely not in scope for an I-D that provides
advice to designers of filesystem protocols and implementors of those
and filesystems!  Both of those are far from the display action.  And
from user input too, for that matter.  There is nothing we can say to
them that will alter anything about input and display.  Therefore it is
appropriate and correct to leave display (and input) issues out of
scope.

> [...]

This is only for case insensitivity.  In filesystems.  I don't think
there will be too much trouble.  We can offer alternatives, but I think
we should follow what Unicode says.

> specs versus work relative to Nico's much more exciting file

My I-D is the result of my writing an early review of an external I-D
that I was assigned by a directorate chair to review.  You might even
recall pointedly telling me to write one if I wanted the issues in
question addressed (I already had written it, just not submitted it).

To answer your question, I believe the chairs can and should a) assign
individual directorate members to review each document that comes to the
group and each document by members of the group, b) cajole others to do
their own reviews as well.

As I understand it, the chairs are currently not available, so in
further answer to your question, unless we self-organize, it may be a
few more weeks before we get assignments to review the I-Ds you listed.

As to self-organization, I don't mind trading a review of some of the
above-mentioned I-Ds if that's what it takes to get more review of mine.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 10:58:56 -0500

> > reasonable member of that set.

Hmm, unless you meant that mixing of RTL and LTR scripts should be
forbidden.  But that's not a realistic proposition given that file
extensions tend to be a) ASCII (LTR) and b) required, so even within a
single directory it may not be possible.  And even where file extensions
are not required, there is the problem of "dotfile" conventions for
hiding files.  And even if all of those were non-issues and we could ban
mixing of RTL & LTR scripts in any given _filename_, we could never keep
script mixing from happening in absolute _paths_.  So we can't
effectively avoid display issues by forbidding mixing of RTL/LTR
scripts.

Display issues for mixed RTL/LTR paths affect a very large variety of
software, from terminal emulators to browsers (because paths tend to be
used as URI/IRI local parts) to native GUI apps, and much more.  What
more can we do about them than reference Unicode?

Nico
--

---

From: John C Klensin <[email-masked]>
Date: Tue, 07 Jul 2020 14:20:43 -0400

--On Tuesday, July 7, 2020 10:58 -0500 Nico Williams
<[email-masked]> wrote:

>> > Except that "so not deal with RTL/LTR issues" is equivalent
>> > to "blow off a significant number of people in the world
>> > who use such writing systems".  There are probably ways
>> > deal with the issues in things like file system paths, but
>> > saying "display system are not in scope" and then moving on
>> > is probably not a reasonable member of that set.
>> 
>> [...]
> 
> Hmm, unless you meant that mixing of RTL and LTR scripts
> should be forbidden.  But that's not a realistic proposition
> given that file extensions tend to be a) ASCII (LTR) and b)
> required, so even within a single directory it may not be
> possible.  And even where file extensions are not required,
> there is the problem of "dotfile" conventions for hiding
> files.  And even if all of those were non-issues and we could
> ban mixing of RTL & LTR scripts in any given _filename_, we
> could never keep script mixing from happening in absolute
> _paths_.  So we can't effectively avoid display issues by
> forbidding mixing of RTL/LTR scripts.
> 
> Display issues for mixed RTL/LTR paths affect a very large
> variety of software, from terminal emulators to browsers
> (because paths tend to be used as URI/IRI local parts) to
> native GUI apps, and much more.  What more can we do about
> them than reference Unicode?

Nico,

First of all, remember that, as long as numerals are expressed
in positional form (e.g., with Indo-Arabic digits and their
equivalent in other scripts), there is no such thing as a
strictly RTL script.  That is at least one reason why "bidi" is
much more appropriate as a characterization of those scripts
than RTL, even with no script mixing.   The problems you are
discussing are, however, exactly where "just referencing
Unicode" breaks down.  The Bidi algorithm was designed for, and
works really well for, ordinary running text, at least for
scripts with syntactically unambiguous "words" and word
boundaries.  As soon as something has a special syntax because
it is not running text but a special type of object expressed in
text form -- such as a filename-- unthinkingly applying the Bidi
algorithm leads to trouble.  

We've recognized that it the past.  For example, RFC 5893 does
not say "just use Unicode Bidi".  It recognizes that domain
names provide special challenges (with the most obvious example
being that DNS labels are not required to follow the rules for
"words" in relevant languages and because the fundamental syntax
for non-TLD FQDNs is Label.Label... and so provides rules for
dealing with domain names that are different from the Unicode
Bidi rules.   It seems to me that precisely the argument you've
made leads to a conclusion that file names and file paths are
special types of objects and, if they are allowed to include
code points drawn from nominally RtoL scripts, then they are
going to need their own rules.  

Possibly those rules could be drawn from RFC 5893 but, if you
have not read the discussion in Section 5.2.5 of RFC 8264
recently, I'd recommend that.  There are no there are easy
answers there either, but it very clearly doesn't say "just
reference Unicode" and move on.   It does suggest that as one
alternative and RFC 5893 as another, both preferred in most
circumstances rather than inventing more rules.  However, files
names or paths would clearly be Identifiers in the PRECIS sense
that that recommendation pushes in the RFC 5893 direction.   

Its other comments about the IDNA2008 rules being "quite
restrictive" are also one of several things that suggest that a
careful review of RFC 5893 in the light of things we have
learned in the last decade would be appropriate before we build
more on it.  It is also worth remembering that, IIR, there have
been changes in the Unicode Bidi algorithm in that same decade.
It there is any chance of those changes having an effect on what
might be written in/as a file name or path, then there is no
"reference Unicode", only "reference a specific version and
figure out what to do with transitions to and from the others".

Now, of course, you could write your spec to say "this works
only for filesystems that do not allow characters to which
Unicode assigned RTL properties and it may not be appropriate
for scripts that encounter significant issues when putting
identifiers through Case Folding or that have compatibility
equivalents to other characters either".   I think you would
have trouble getting IETF consensus (or adoption in the field by
people who already have their own systems --even if ASCII-only--
and would need to change them) with such restrictions but, if
you disagree, go for it.

Again, I'd recommend careful study of the PRECIS specs, at least
to the extent of either being able to appropriately profile them
for some of your needs or being able to explain why they are not
relevant.

And I look forward to hearing from Marc as to whether I've
gotten things completely wrong.

best,
   john

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 14:07:16 -0500

> than RTL, even with no script mixing.   The problems you are

Sure.

> algorithm leads to trouble.  

That's fair, but again, running code that displays file paths and file
names and URIs/IRIs is common and far flung.  I agree that we could
borrow from RFC 5893 to provide advice to implementors of code that
displays any of those things.  We just need to understand that we won't
be boiling any oceans, and that advice regarding display targets a very
different set of implementors than advice regarding filesystem protocols
and advice regarding filesystems themselves.

> going to need their own rules.  

Possibly.  Keep in mind that we don't and won't have a filesystem
equivalent of A-labels, and we don't have a notion of filesystem-aware
apps or protocol slots.

On the plus side, these are _filesystems_, not _the DNS_.  The problems
that arise have much much smaller impact.

(Unless we bring IRI local parts into scope, which we might do since
often IRI local parts are taken from filesystems.  But still, the impact
of bidi issues there is much smaller than for the DNS.  Mind you, I
don't want to blow up the I-D's scope too much, and would rather say
anything about IRI local parts in a subsequent document, if we do it at
all.)

> and would need to change them) with such restrictions but, if

I don't think we need to do that.  We could say that bidi can cause
problems, describe the impact, and provide implementation advice, just
as long as we understand that it's unlikely that all the relevant
software will be updated accordingly any time soon.

> for some of your needs or being able to explain why they are not

Noted.

Nico
--

---

From: John C Klensin <[email-masked]>
Date: Tue, 07 Jul 2020 15:16:13 -0400

--On Tuesday, July 7, 2020 15:14 +0200 Patrik Fältström
<[email-masked]> wrote:

>> riority question: we have a whole series of long-outstanding
>> draft-freytag-troublesome-characters, and some SMTPUTF8
>> ("EAI") tweaks that have not been turned into I-Ds as obvious
>> examples along with, perhaps, draft-sullivan-lucid-prob-stmt,
>> the notorious draft-klensin-idna-5892upd-unicode70, and
>> others.  How do we want to prioritize those probably tedious
>> (but important to make/keep existing standards track specs
>> workping] well) specs versus work relative to Nico's much
>> more exciting file system proposal?
> 
> "My" documents will be worked on starting next week when I on
> Tuesday is back from vacation. I.e. 7 days from now.

Patrik,

Thanks but not my point (and I have not been working on "my"
documents much either).  

The problem I'm concerned about is that, when you decide those
documents are ready (or other authors decided theirs are ready),
they are going to need serious review by people with significant
i18n expertise, significant IETF protocol expertise, and
significant expertise with the combination.   Assuming the
relevant document(s) go into IETF LC and that any substantive
issues raised there are addressed, the reviews I'm talking about
have to be broad enough and careful enough that Barry and Murray
(plus or minus the document shepherd) are able to convince IESG
members that they should not tie things up in DISCUSS votes
because they think things should be different based on their
personal (and obviously deep, or they wouldn't be on the IESG,
right?) understanding of i18n issues.

At least most such reviews are almost certainly going to have to
come from this directorate list because, whatever the strengths
and weaknesses of this group, it probably includes almost
everyone who has been active in the IETF on i18n work and who
has a reasonable claim to broad expertise on at least some i18n
topics (I can identify a few exceptions, but don't expect them
to be active any time soon).  And, as we have repeatedly
discovered working together, i18n documents (at least) get much
better with active collaboration and discussion, not just
reviews thrown over the wall.

However, with regard to doing that sort of collaboration
(independent of a few reviews of documents from assorted IETF
WGs), the directorate seems to be both disfunctional and lacking
any form of leadership or guidance.   And, if we can't get
enough collaboration (or some form of aggressive push-back), I
think the experience with RFC 8753 and with 5891bis strongly
suggests that more work on any of those documents may be a waste
of effort.

   best,
    john

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 14:35:16 -0500

> suggests that more work on any of those documents may be a waste

If we're failing to review I-Ds, is that because

 - assignees are failing to do them,
 - the chairs are failing to assign reviewers,

or because

 - we're talking about I-Ds that come from within this group and we're
   not providing enough energy to see them progress?

If it's the last case, note that we're not a WG, but if we're expected
(or want to) have our own documents, then maybe we should be a WG rather
than a directorate, or at least maybe we ought to meet.

IMO we should replace the I18N directorate with an I18N WG.  The main
difference is that we could have milestones involving publication of
RFCs, and we could be expected to have regular and even interim
meetings, and perhaps we could continue to function as a directorate
responsible for reviewing other WGs' I-Ds that have or should have I18N
considerations.  If we don't have the energy for a WG, then at the very
least we should meet as a directorate once or twice a year.

Regarding energy, I would say that energy can be harnessed by having a
meeting at all, or at least that having meetings would better allow us
to gauge level of energy.

(Note: it's too late to schedule a BoF about creating an I18N WG at IETF 108.)

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 14:35:57 -0500

> right?) understanding of i18n issues.

FWIW, I am not too concerned about DISCUSSes.  They happen and they are
not the end of the world.  Granted, it's nice to get one's ducks in a
row.

Also, I wouldn't assume that IESG members have deep understanding of
I18N issues, even if the vote DISCUSS on an I18N document.  I wouldn't
insist that they must be I18N experts in order to vote DISCUSS either,
nor would I have a way to make my insistence count in any way.

Nico
--

---

From: John C Klensin <[email-masked]>
Date: Tue, 07 Jul 2020 17:13:22 -0400

--On Tuesday, July 7, 2020 10:05 -0500 Nico Williams
<[email-masked]> wrote:

>...
> For filenames and paths I believe there are only display
> issues.  That was Patrik's point.  Thus the I-D should say as
> much and normatively reference Unicode.
> 
> And yes, the display is absolutely not in scope for an I-D
> that provides advice to designers of filesystem protocols and
> implementors of those and filesystems!  Both of those are far
> from the display action.  And from user input too, for that
> matter.  There is nothing we can say to them that will alter
> anything about input and display.  Therefore it is appropriate
> and correct to leave display (and input) issues out of scope.

See my earlier response to your later note.

> My I-D is the result of my writing an early review of an
> external I-D that I was assigned by a directorate chair to
> review.  You might even recall pointedly telling me to write
> one if I wanted the issues in question addressed (I already
> had written it, just not submitted it).

I am not objecting to your posting the draft.  I still think
that was a good idea.  When I will have  time to do a careful
review is a separate problem but I note that a quick reading of
parts of it turned up several things I would question.

What I'm concerned about -- and what caused that question about
priorities -- is, at an even higher level, the sort of issue
covered by the note I just sent in response to Patrik's.  With
one qualification below (which I predict you won't like), if the
IESG (via IETF LC), a WG, an AD, or maybe an individual author
who is proposing an update in line with documents that have
already achieved IETF consensus requests a review _from the
directorate_, and Pete assigns that review to someone and they
do it, then that is great.  

The qualification is that, if the review was requested from the
directorate about a document that has been written consistent
with established IETF consensus documents _and_ that does not
reopen or contradict the questions that were presumed to be
settled by that consensus, then that review ought to respond to
the i18n issues raised by the document and suggest issues with
them, not say "well, the whole concept of which the spec under
review is a {proposed) part is broken and I have a better idea".
Should the latter occur, I believe that, if we are really
functioning as a directorate, a second reviewer should
immediately be recruited and assigned to provide the review that
was requested.

Let me give two examples that are more extreme and closer to
home in the hope that they will clarify the point I'm trying to
make: 

(i) Suppose that discussions that are now underway lead to
useful results and we have an rfc5322bis in IETF Last Call.  Now
suppose someone writes a review that says "this whole effort is
a waste of time, traditional name-value pairs are outdated and
take up too much space, and the whole thing should be replaced
by a JSON structure which I am about to propose"   Now, whether
that would be a good idea or not, it would not be a useful
review of a rfc5322bis.

(ii) Even closer to home, suppose that, at some point, revised
documents are prepared to move the IDNA2008 and PRECIS documents
to Internet Standard.   At least the former is widely deployed
and, if we ignore the UTR#46 issues for a moment, demonstrated
to be interoperable.  Now suppose someone writes a review that
says "all of the problems with matching clearly show that the
decision to go to Unicode was nuts and we should go back to ISO
2022 and code pages, a decision that would allow us to encode
and process information on a per script and per language basis
and that would discourage the types of script mixing and
identification of code points as inheriting properties that have
caused so much trouble".   I'd expect such a person to be
treated as nut case, but I'd hope that a serious proposal and
write up and the rationale for it would be considered carefully
and fairly.   But, would it be a useful review of revised and
updated IDNA2008 or PRECIS documents?  Probably not.

So I think "we" owe a more focused review, perhaps modeled on
the comments Asmus and I made early on to that documents, its
author, and the NFS community.

And that brings me back to your draft and the specific issue I
was trying to raise.

>...
> As I understand it, the chairs are currently not available, so
> in further answer to your question, unless we self-organize,
> it may be a few more weeks before we get assignments to review
> the I-Ds you listed.

I'm not interested in reviews in the same sense that we are
reviewing external documents.  I'm interested in efforts closer
to what went into the discussions of draft-faltstrom-unicode11
(and 12) over a year ago, the discussions that resulted in both
RFC 8754 and the update of the IANA tables.  I think those are
fundamentally different from reviews of more external documents.

> As to self-organization, I don't mind trading a review of some
> of the above-mentioned I-Ds if that's what it takes to get
> more review of mine.

If I understand you correctly, you are suggesting that your
ideas about an i18n filesystem is sufficiently important that it
justifies pushing existing core i18n specifications, including
the reviews of Unicode 13 (and, to some extent, 12) required by
RFC 8753 aside so we can all work on your document.  I
respectfully disagree and suggest that -- or essentially turning
the directorate into a WG or design team specific to your
document and ideas -- is not a good idea.  I assume we will just
need to agree to disagree about that.

best,
   john

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 16:45:39 -0500

> immediately be recruited and assigned to provide the review that

Yours is an argument for stare decisis.  Once wrong, always wrong.  As
if consensus, once found, can never change.

I reject that argument, at least as to our documents.

It seems, too, like you're conceding that our approach to NFSv4 I18N was
wrong, but you want to keep that old concensus untouched.  Of course, I
welcome your concession even as I reject your argument that we must
never revisit consensus findings.

> fundamentally different from reviews of more external documents.

It sounds like you want us to act more like a WG and publish RFCs --
except below you say otherwise.  That seems like a contradiction, unless
you expect us to use the ISE for all our documents.  But a group that
relies on the ISE stream for publication of its documents seems like a
prime candidate for a WG, else it seems like an end-run around IETF
process that other groups have to go through.

> RFC 8753 aside so we can all work on your document.  I

How do you draw such a conclusion from what I wrote?  How would trading
reviews cause my work to push others' aside?  How would my bringing my
energy to help progress the documents you are interested in somehow push
them aside?  Are you... taking my I-D as an affront?  That's how your
responses are starting to come across to me.  I'm baffled.

> need to agree to disagree about that.

We don't have to agree to disagree.  As you well know, we have a
mechanism for making decisions.  Expect a BoF proposal from me for IETF
109 so that we can make a decision about whether we shall have an I18N
WG.

Nico
--

---

From: Asmus Freytag <[email-masked]>
Date: Tue, 7 Jul 2020 16:43:42 -0700

Here are my comments:

>     This document describes requirements for internationalization (I18N)
>     of filesystems specifically in the context of Internet protocols, the
>     architecture for filesystems in most currently popular general
>     purpose operating systems, and their implications for filesystem
>     I18N.  From the I18N requirements for filesystems and the

The first sentence doesn't scan - the constructions joined by "and" 
seemingly are not parallel enough.

>     [TBD: Add references galore.  How to reference Unicode?  How to

If you go on the Unicode site and look for the page on the latest 
version (currently https://www.unicode.org/versions/Unicode13.0.0/), you 
can navigate to suggested ways to reference. I would cite the latest 
version as of the time of drafting of the I-D and then also write: "the 
latest version is available at" and give the URL as 
http:\\www.unicode.org\versions\latest.

If you need to reference specifically the properties, the character 
database is at https://www.unicode.org/Public/13.0.0/ or  
https://www.unicode.org/Public/UCD/latest/ -- You could also link to 
UAX#44, which is the overview of the Unicode Character Database. The 
"latest" is at: https://www.unicode.org/reports/tr44/ and the current 
one, as of today, is at 
https://www.unicode.org/reports/tr44/tr44-26.html (see the "This 
version" link for the latest).

Unless you need to specifically discuss a particular topic, in which 
cases you might like to identify a specific UAX (e.g. 9 for Bidi) or 
chapter. Links to chapters are redirected correctly from the "latest" link.

If this is a bit complex, it's because Unicode really is a family of 
standards.

>   To deal with the equivalence problem, Unicode defines Normal Forms
Unicode calls these "Normalization Forms", see Section 3.11 in TUS (The 
Unicode Standard) so that is what should be used when capitalized. Ditto 
for the formal names for NFC and NFD.

I think you need to mention already here that NFC/NFD represent a 
semantic identity (and generally identical appearance) while NFKC and 
NFCD abstract away rather noticeable differences in appearance that may 
in some cases imply strong semantic differences to some users (e.g. math 
alphabets).

>   Unicode compatibility equivalence allows equivalence between
>     different representations of the same abstract character that may
>     nonetheless have different visual appearance of behavior.  There are
>     two canonical forms that support compatibility equivalence: NFKC and
>     NFKD.  Using NoCL with NFKC or NFKD may be surprising to users in a
>     visual way.  While form-insensitivity with NFKC or NFKD may surprise
>     users who might consider two file names distinct even when Unicode
>     considers them equivalent under compatibility equivalence.  The
>     latter seems less likely and less surprising, though that is an
>     entirely subjective judgement.
You really MUST not use the term "canonical" with NFKC and NFKD because 
for Unicode, the two forms NFC and NFD are considered "canonical" and 
that term is used contrastively to "compatibility".

There's an "of" that should be an "or", but it's not just behavior - 
it's also meaning. Mathematicians would object strenuously to having 
their math alphabets "normalized" to standard A-Z.

NFK(C/D) is useful in a different way: if it used to disallow code 
points that aren't stable under these normalization forms, then one 
sidesteps the issue of whether the distinct appearance etc. is 
meaningful, but without ever changing a filename (which would happen if 
it was normalized when stored). (There may be no file-systems that take 
this approach, nevertheless it's worth discussing as it is used in other 
naming schemes).

>     foldings are defined by Unicode.  Generally, case-insensitive
>     filesystems preserve original case just form-insensitive filesystems
>     preserve original form.
There's an "as" missing.

However, early case-insensitive file systems did not preserve case. Not 
sure how rare this has become.

>     listings that work the same way as on the server.  We do not specify
>     any case foldings here.  Instead we will either create a registry of
the "here" is unclear. Does it refer to recommendation in this ID 
relevant to caching clients? If so, link to section.

>    "just-use-8" or "just-use-16" (as in UTF-16 [UNICODE  <https://tools.ietf.org/html/draft-williams-filesystem-18n-00#ref-UNICODE>]), with no
>     attempt at normalization or case folding done anywhere in between.
In Unicode parlance: "just use strings of code units".

Specifically for UTF-8, this would imply that there are also no 
guarantees of well-formedness of the UTF-8 strings (likewise for 
surrogate pairs in UTF-16).

-- reached end of section 1 and out of time slot --

A./

---

From: "John Levine" <[email-masked]>
Date: 7 Jul 2020 20:42:49 -0400

>and filesystem implementors.  It only deals with path components, not

It's interesting, but if I were someone maintaining filesystem code,
my response would be "who are you and why should I pay any attention?"

You are right that NFS clients need some way to find out whether two
different names refer to the same file, but the idea of a registry
seems like a non-starter, both because nobody has any incentive to
register and because I expect there are a lot of strange corner cases
like unpaired UTF-16 surrogates that would be very hard to describe.

What clients need is a way to ask the server to map a filename to a
handle, so the client can see if it's the same handle as one it
already has. I'm pretty sure NFS already does that. Even if you had
the case folding maps, clients would still need to do this because
Unix/Linux files can have several unrelated names.

R's,
John

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 20:29:45 -0500

Thanks!

> seemingly are not parallel enough.

I think the "their" in "and their implications" is unclear as to
antecedent.  I'll wordsmith it.

> If you go on the Unicode site and look for the page on the latest version
> (currently https://www.unicode.org/versions/Unicode13.0.0/), you can
> navigate to suggested ways to reference. I would cite the latest version as
> of the time of drafting of the I-D and then also write: "the latest version
> is available at" and give the URL as http:\\www.unicode.org\versions\latest.

The question was about bibxml specifically, though I should have made
that clear.  I failed to update that comment -- I did figure out how to
reference Unicode, and did reference it.

I do still need to add a number of other references.  And for the two
books I referenced I should add section numbers.

> If you need to reference specifically the properties, the character database
> is at https://www.unicode.org/Public/13.0.0/ or 
> https://www.unicode.org/Public/UCD/latest/ -- You could also link to UAX#44,
> which is the overview of the Unicode Character Database. The "latest" is at:
> https://www.unicode.org/reports/tr44/ and the current one, as of today, is
> at https://www.unicode.org/reports/tr44/tr44-26.html (see the "This version"
> link for the latest).

In XML terms... you're saying I should use <eref>, I think.

> Unicode Standard) so that is what should be used when capitalized. Ditto for
> the formal names for NFC and NFD.
> 
> I think you need to mention already here that NFC/NFD represent a semantic
> identity (and generally identical appearance) while NFKC and NFCD abstract
> away rather noticeable differences in appearance that may in some cases
> imply strong semantic differences to some users (e.g. math alphabets).

I chose to do so one page later because I wanted to keep the intro
brief:

|   Unicode compatibility equivalence allows equivalence between
|   different representations of the same abstract character that may
|   nonetheless have different visual appearance of behavior.  There are
|   two canonical forms that support compatibility equivalence: NFKC and
|   NFKD.  Using NoCL with NFKC or NFKD may be surprising to users in a
|   visual way.  While form-insensitivity with NFKC or NFKD may surprise
|   users who might consider two file names distinct even when Unicode
|   considers them equivalent under compatibility equivalence.  The
|   latter seems less likely and less surprising, though that is an
|   entirely subjective judgement.

Here "NoCL" == normalize-on-CREATE-and-LOOKUP, defined just above the
quoted text.

Does that work for you?

> >     two canonical forms that support compatibility equivalence: NFKC and
        ^^^^^^^^^^^^^
> You really MUST not use the term "canonical" with NFKC and NFKD because for
> Unicode, the two forms NFC and NFD are considered "canonical" and that term
> is used contrastively to "compatibility".

The word 'canonical' was preceded by 'two' -- I think that's pretty
clear!  :)

> There's an "of" that should be an "or", but it's not just behavior - it's
> also meaning. Mathematicians would object strenuously to having their math
> alphabets "normalized" to standard A-Z.

Ay, yes, thanks!

> NFK(C/D) is useful in a different way: if it used to disallow code points
> that aren't stable under these normalization forms, then one sidesteps the
> issue of whether the distinct appearance etc. is meaningful, but without
> ever changing a filename (which would happen if it was normalized when
> stored). (There may be no file-systems that take this approach, nevertheless
> it's worth discussing as it is used in other naming schemes).

ZFS (in Solaris and most ports, but not OS X) form-insensitivity, and
lets the user choose a form for use for this.

See https://docs.oracle.com/cd/E71909_01/html/E71919/gpssl.html which
shows that NFC, NFD, NFKC, and NFKC.

It doesn't make much sense to offer NFC or NFKC for this given that the
behavior is form-insensitive and form-preserving (NFC is specified as
requiring canonical decomposition as a first step, which means that NFC
is very likely slower than NFD, and the same applies to NFKC and NFKD).
But whatever.

In at least the original port to OS X, before Apple abandoned it, ZFS
was made to normalize-on-CREATE (and LOOKUP) instead of being
form-preserving.  This was probably done to match HFS+'s behavior.  I
remember a post by Linus Torvalds complaining bitterly about this and
blaming ZFS in general rather than Apple, when some user complained that
Git did not handle this case correctly.  The issue was that Git looked
for the file using the name it knew from the repository, and that was in
NFC because of course the input mode used when the file was committed
produced something NFC-ish, but on HFS+ this got converted to NFD, and
Git then failed to find the file by memcmp() in the directory listing.
Ironically, that very possibility had been my motivation for pushing for
form-insensitive and form-preserving behavior in ZFS in Solaris to begin
with, but Apple engineers insisted on the HFS+ behavior in their port.

Apple's choice of NFD makes sense for performance reasons since it's
faster to normalize to form D than to form C (see above).  But
normalizing on CREATE to NFD conflicts with input modes that tend to
produce pre-composed character codepoints.  Form-insensitivity is the
obvious compromise.  Until the Git incident, I had no evidence of that
actually causing problems.

Git nowadays contains normalization code to deal with this, whereas it
could have not had any if we'd had consensus on form-insensitivity a few
years before we added it to ZFS.  Bummer.  Still, form-insensitivity is
clearly the better behavior, though I'm not proposing that we mandate
it.

> >     filesystems preserve original case just form-insensitive filesystems

Thanks :)

> However, early case-insensitive file systems did not preserve case. Not sure
> how rare this has become.

So.. sentence-case?  I refer to that later:

|   The only way to implement I18N behaviors in the VFS layer rather than
|   at the filesystem is to abandon form- and case-preserving behaviors.
|   For case-insensitivity this would require using sentence-case, or all
|   lower-case, perhaps, and all such choices would surely be surprising
|   to users.  At any rate, that approach would also render much running
|   code "non-compliant" with any Internet filesystem protocol I18N
|   specification.

> the "here" is unclear. Does it refer to recommendation in this ID relevant
> to caching clients? If so, link to section.

'here' == "this document".  I'll clarify.

> In Unicode parlance: "just use strings of code units".

"just-use-8" means strings of 8-bit words that could be... any 8-bit
codeset, or UTF-8, or whatever.

"just-use-16" is almost certainly "as in UCS-2" or "as in UTF-16",
though for all I know there may be some non-Unicode 16-bit wchar_t
codesets in use as "just-use-16".

> Specifically for UTF-8, this would imply that there are also no guarantees
> of well-formedness of the UTF-8 strings (likewise for surrogate pairs in
> UTF-16).

Well, yes.  But that wasn't what I was alluding to.  I was referring to,
e.g., using Linux (or *BSD, or Solaris/Illumos) in, say, an ISO-8859-1
locale.  In that case you might have an application all `creat(2)` with
a `pathname` that contains non-ASCII that is also not UTF-8.

The C library system call stub will *not* convert this to UTF-8 or
anything else, nor will the kernel-land system call know anything about
the locale in use in user-land.  The given `pathname` string will just
be a pointer to a zero-terminated array of `char` in user-land, and will
be copied into the kernel as such by the kernel-side of the system call,
then it will be passed to the VFS.

The VFS will interpret just two byte values specially: 0x00 (ASCII NULL)
and 0x2F (ASCII '/').  The filesystems will generally only interpret
0x00 specially, though some (e.g., ZFS) may reject strings that are not
valid UTF-8.

We call this "just-use-8" in some circles (e.g., in the KITTEN WG).  I
don't recall who coined that.

I expect something similar happens in Windows for both, the *A()
(just-use-8?) and *W() functions (just-use-16?).  But I'm not
sufficiently experienced a Windows programmer to really know.

"just-use-16" is my coinage as an obvious variation of "just-use-8".

> -- reached end of section 1 and out of time slot --

Thanks so much!

Nico
--

---

From: John C Klensin <[email-masked]>
Date: Tue, 07 Jul 2020 21:31:55 -0400

Just two quick comments - can't spend more time on this today:

--On Tuesday, July 7, 2020 16:43 -0700 Asmus Freytag
<[email-masked]> wrote:

> However, early case-insensitive file systems did not preserve
> case. Not sure how rare this has become.

Well, Unix, and every Unix-derived system I know of (definitely
including Linux, FreeBSD, and NetBSD) are case-sensitive and
getting anywhere near their file names with Case Folding or even
lower casing will cause rather interesting problems.  One the
case-insensitive side, I've heard of a system called "Windows"
that usually preserves case but does not guarantee to do so and
many operations, in practice, don't.  

I assume I don't need to comment on how rare those systems are.

In addition, Asmus caught at least one case where your
terminology is inconsistent with that of Unicode.  Such
inconsistencies (at least if not specifically identified and
justified) are an invitation to reader confusion or worse.  As
one more example, you should not be talking about case folding
and then lower casing.  They are different and have different
implications.  And, fwiw, most of my Turkish-speaking and
writing colleagues would disagree that U+0131 "could" or should
be considered equivalent to U+0069.  In my experience, that
would be considered even less correct than treating U+0061 and
U+00FC as equivalent in Swedish or German.  There are certainly
theories and models that allow that, but they are closely
related to the ones that suggest that the easy way to handle
non-English languages on the Internet is by transliteration
rather than dealing with all of those messy non-Latin, and maybe
decorated Latin, scripts and writing systems.

best,
   john

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 20:33:12 -0500

> my response would be "who are you and why should I pay any attention?"

Similarly, an NFS server implementor can't very well tell a filesystem
implementor what to do either.

"Why should I do this for NFS when SFTP and blah blah blah don't ask me
to?"

It's the same issue.

> like unpaired UTF-16 surrogates that would be very hard to describe.

Perhaps.  I was considering using some existing repository of locales,
specifically CLDR.

> Unix/Linux files can have several unrelated names.

Caching clients might support disconnected operation.  Then what?

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 20:40:55 -0500

> many operations, in practice, don't.  

At least ZFS supports case-insensitivity as an option.  ZFS is available
on just about every Unix and Unix-like OS today, and is quite popular.

That said, I doubt anyone enables case-insisitivity in ZFS datasets on
Unix unless they intend to share it with Windows clients and users
expect case-insensitivity.  Indeed, that's the reason that ZFS has this
option at all: Windows interoperability.

> terminology is inconsistent with that of Unicode.  Such

Yes yes, I beg your forgiveness.  I wrote "normal form" instead of
"normalization form".  A silly editing error.

> implications.  And, fwiw, most of my Turkish-speaking and

Did you not read that text?  I only mentioned lower case in the context
of explaining in introductory text what case-folding is.  And I did NOT
say that they are the same thing.

> be considered equivalent to U+0069.  In my experience, that

The idea was to describe how an admin might square that circle in a
case-insensitive filesystem, and then I did not endorse it.  The idea
was to describe how custom case-folding tailorings (FYI, that's the term
Unicode uses, 'tailoring') might come about.

There will be time to address all the nits.  The big idea is more
interesting.

Nico
--

---

From: Asmus Freytag <[email-masked]>
Date: Tue, 7 Jul 2020 19:16:01 -0700

> Here "NoCL" == normalize-on-CREATE-and-LOOKUP, defined just above the
The text continues to have the same problems I mentioned. While it also 
has a sentence that start with while and does not begin with a 
subordinate clause.

We agree that forcing normalization using either NFKx can surprise users 
in a bad way (although for some subset of the equivalences, some subset 
of users might like form-insensitivity: full/half-width character 
variations are often treated very similar to case variation for East 
Asian users).

Should a file system allow a form-sensitive "no break hyphen" ? That one 
is worth discussing as a downside to the "just-8" type approach. It 
would be annoying for users having to guess the invisible no-break 
nature. (This would belong in some section on i18n pitfalls).

>>>      two canonical forms that support compatibility equivalence: NFKC and
No. You MUST not use that term for any NFKx because the two NFKx are NOT 
"canonical" normalization forms as that term is defined in Unicode. 
Using Unicode defined terms in ways that conflicts with Unicode-defined 
terminology is a sure recipe for confusion.
> is very likely slower than NFD, and the same applies to NFKC and NFKD).
Can't speak to performance issues - that's a separate one from the logical
distinction better K and not-K.
> with, but Apple engineers insisted on the HFS+ behavior in their port.
I think NFD is a bad choice, because it rarely matches "raw" data. Most 
data is either unnormalized or (almost) in NFC.
> actually causing problems.
In theory it may be faster, because C logically requires performing D 
first. But as most data is entered in C, all you need to do is verify 
that, which is faster than expanding a string to do D.
>> how rare this has become.

Windows is now case-preserving, but it's predecessor (DOS) was not.
Or, more precisely, the difference is the file systems - FAT, vs FAT32, 
NTFS etc.

> |   code "non-compliant" with any Internet filesystem protocol I18N

Why would you need to do sentence case?

> 'here' == "this document".  I'll clarify.
Will "this document" create a registry? (Haven't read far enough, and 
the "either"
makes this a statement of intent rather than a description of what "this 
document"
does).
>
>>>     "just-use-8" or "just-use-16" (as in UTF-16 [UNICODE  <https://tools..ietf.org/html/draft-williams-filesystem-18n-00#ref-UNICODE>]), with no

Unicode goes to great length to provide clear definitions of code points 
(things that are interpreted as assigned to some character) and code 
units (things that are the building blocks of sequences that are matched 
by UTF-8 or UTF-16 to code points.

Check UTR#17 "Character Encoding Model".

This model works even for other character encodings, whether ISO 2022 or 
DBCS.

Now, if you want to use these less formal terms, that might be fine, but 
you should anchor them by relating them to the more formal definitions. 
"Just-8" treats filenames as strings of code units (uninterpreted bytes 
that at some other point are converted to code points that then can be 
interpreted as characters and subjected to transformations like 
case-folding and normalization).

"Just-16" is the same for the code units of UTF-16 (that being the only 
pure 16-bit format I can think of right now, but if others exist, the 
same would apply for them), but which could be big-endian or 
little-endian. Interestingly that distinction is on a level below the 
code unit; it belongs to the "serialized code unit".

So, what you are saying is that some parts of the architecture deals in 
raw strings of "serialized code units" (if that part is "in memory" 
only, then the distinction is moot and we are back at strings of code 
units. (See Section 5.1 in UTR#17).

> a `pathname` that contains non-ASCII that is also not UTF-8.

Yes, that's implied in "8-bit code unit" - it's not something that's 
specific to UTF-8. You have both: raw string of code units that may 
represent UTF-8 (well-formed or not) or that may represent any other 
encoding scheme with 8-bit code units.

> don't recall who coined that.

It's a cute name, but it needs to be defined carefully in terms like 
those of the Character Encoding Model. We need to strive to remove 
confusion, not add to it.

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 21:22:45 -0500

Regarding my slip-up, calling "NF" "normal form", I've proof that I've
called it "normalization form" correctly before.  You can find mailing
list posts from me (including my old [email-masked] address) showing that,
but then there's draft-williams-i18n-boundary-analysis-00 from 2014
showing me using it correctly.  Also draft-adamson-nfsv4-multi-domain-
federated-fs-reqs-03, from 2013, though I'm a co-author on that one.  I
can dig up more.  Or may I be forgiven with just this evidence?

Nico
--

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Tue, 7 Jul 2020 19:27:15 -0700

> "normalization form".  A silly editing error.
Not silly, it may well be that "Normal Form" would have been the better 
term. Water under the bridge.

NFC/D (no-K) use what is called "canonical de/composition" in Unicode. 
Therefore, NFKx cannot be considered "canonical" or you will confuse 
anyone who knows Unicode usage.
> say that they are the same thing.
Nico, please understand that we find your text otherwise quite readable 
and lucid - however, John's point may be that these concepts are subject 
to confusion (whether or not you share that) and that it would be useful 
to make extra-double sure that all readers get that these are different.

There was somewhere in the earlier e-mail discussion also the mention 
that is-lowercase and to-lowercase (however you name these two concepts) 
are crucially different in that there's an expectation that is-uppercase 
= ! is-lowercase (ignoring is-titlecase for the argument), but that 
to-lowercase and to-uppercase are operations that may be inverses of 
each other. German to-lowercase(to-uppercase(ß)) != ß (but "ss").

It would be good to lay the foundation by reminding everyone of these 
subtleties.
> There will be time to address all the nits.  The big idea is more

The danger of reviewing the test while reading along from the start. 
However, the only way I know how to review anything that's not in a 
change-tracked editable format.

A./

---

From: "John R Levine" <[email-masked]>
Date: 7 Jul 2020 22:27:36 -0400

> It's the same issue.

Indeed, so unless we can address it, I wouldn't put a lot of work into 
this.

> Perhaps.  I was considering using some existing repository of locales,

CLDR is great for what it does, but I would be surprised if file system 
case folding matched any of its entries.  CLDR tells the "ls" program how 
to format timestamps for display.

> [ NFS already has a way to get handles for filenames ]

> Caching clients might support disconnected operation.  Then what?

I don't think I've ever seen a disconnected NFS client, but let's assume 
one exists.  It will have to deal with all sorts of synchronization issues 
including the possibility that "foo" and "bar" are the same file.  Why is 
dealing with "foo" and "FOO" any different?

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 22:09:33 -0500

...

> We agree that forcing normalization using either NFKx can surprise users in
> a bad way (although for some subset of the equivalences, some subset of
> users might like form-insensitivity: full/half-width character variations
> are often treated very similar to case variation for East Asian users).

Yes.

> Should a file system allow a form-sensitive "no break hyphen" ? That one is
> worth discussing as a downside to the "just-8" type approach. It would be
> annoying for users having to guess the invisible no-break nature. (This
> would belong in some section on i18n pitfalls).

"just-use-8" is a reality of running code, not something I'm proposing.

String preparation on CREATE and LOOKUP, and form-insensitive/preserving
behaviors are both ways to deal with the "just-use-8" reality.

Mapping NON-BREAKING HYPHEN U+2011 to HYPHEN-MINUS U+002D is certainly
an option.

If ZFS and/or HFS+ and/or some other filesystems don't already do this
(I believe they don't), it will be difficult to make them start doing
it, since U+2011 is not considered equivalent to U+002D in any NF, and
changes like this generally can only be enabled in new filesystems.
However, it should be possible to make ZFS map it in _new_ filesystems
as an option (possibly enabled by default).

(Note: in ZFS terminology, a filesystem is known as a 'dataset'.)

That's not the only such mapping worth considering, of course.  There's
plenty of others, such as various whitespace and period characters.

> "canonical" normalization forms as that term is defined in Unicode. Using
> Unicode defined terms in ways that conflicts with Unicode-defined

My quick search in Unicode 13.0, chapters 2 and 3, finds examples that I
think support permitting my above formulation.  Still, if you insist,
I'll change that instance of 'canonical' to 'normalization'.

> distinction better K and not-K.

I _suspect_ that Apple engineers chose NFD (or something very close to
NFD) because it would perform better than NFC.  I don't have it at hand,
but I seem to remember a blog post from almost twenty years ago about
that.

> > [...]
>
> I think NFD is a bad choice, because it rarely matches "raw" data. Most data
> is either unnormalized or (almost) in NFC.

Because ZFS is form-preserving, the choice of NFC or NFD here does not
affect anything other than performance and bits-on-disk.  In particular,
applications cannot tell the difference between a ZFS dataset that uses
NFC vs one that uses NFD.

For HFS+, however, NFD does have negative effects and it was a bad
choice.

So I agree with you, and in fact, that is the reason I pushed for
form-insensitive/preserving behavior in ZFS.

> In theory it may be faster, because C logically requires performing D first.

Right.

> But as most data is entered in C, all you need to do is verify that, which
> is faster than expanding a string to do D.

I'm not sure that's always easy to do, but I imagine it can be done.

However, Apple engineers may have thought NFD would perform better.  I
suspect so.

> Or, more precisely, the difference is the file systems - FAT, vs FAT32, NTFS
> etc.

Ah yes, more evidence that in running code, I18N choices are made by the
_filesystems_.

> Why would you need to do sentence case?

I wouldn't.  That text is in a section discussing where to place I18N
behaviors.  I argue that placing them anywhere else than the filesystem
proper causes problems.  In particular, putting case-insensitivity
anywhere other than the filesystem means that in practice one cannot get
case-preserving behavior without caching entire directory listings at
whatever layer is implementing case-insensitivity.

> the "either" makes this a statement of intent rather than a
> description of what "this document" does).

I would rather NOT create a registry but use CLDR if it can be
appropriate.

I would also settle for no site- or locale-specific case-folding
tailorings.

And I might settle for letting clients fetch the set of tailorings.

There are choices to be made here.

> (things that are interpreted as assigned to some character) and code units
> (things that are the building blocks of sequences that are matched by UTF-8
> or UTF-16 to code points.
> 

Does Unicode have terminology for "a string of code units from an unkown
codeset and encoding"?

> This model works even for other character encodings, whether ISO 2022 or

Right, but in the context in question there's no knowledge of the
codesets/encodings in use.  I believe it's important to make that clear.

> Now, if you want to use these less formal terms, that might be fine, but you
> should anchor them by relating them to the more formal definitions. "Just-8"
> treats filenames as strings of code units (uninterpreted bytes that at some
> other point are converted to code points that then can be interpreted as
> characters and subjected to transformations like case-folding and
> normalization).

I might add a glossary for them.  Good idea.

> "Just-16" is the same for the code units of UTF-16 (that being the only pure
> 16-bit format I can think of right now, but if others exist, the same would
> apply for them), but which could be big-endian or little-endian.
> Interestingly that distinction is on a level below the code unit; it belongs
> to the "serialized code unit".
> 
> So, what you are saying is that some parts of the architecture deals in raw
> strings of "serialized code units" (if that part is "in memory" only, then
> the distinction is moot and we are back at strings of code units. (See
> Section 5.1 in UTR#17).

Correct, this is in memory, but with no knowledge of the codeset/
encoding in use.

For Internet filesystem protocols, of course, we do and will insist on
Unicode on the wire.  My thesis is that dealing with normalization and
case insensitivity belongs in the filesystem, not in the server (as in
NFSv4 server).  Though it _also_ belongs in the client when it is a
_caching_ client.  (Not every NFSv4 client will cache directory
contents.)

> encoding scheme with 8-bit code units.

Right.  I really want to denote "8-bit code units, codeset/encoding
unknown".

> It's a cute name, but it needs to be defined carefully in terms like those
> of the Character Encoding Model. We need to strive to remove confusion, not
> add to it.

That's fair.  I'll add a glossary.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 22:13:15 -0500

> It would be good to lay the foundation by reminding everyone of these

That's fair.  Of course, for a -00, I had only so much time before the
submission deadline for IETF 108 (for which I think there may still be
time to organize a "lunch-time" meeting of this directorate).

> change-tracked editable format.

Fair.  FYI, I did put it in a GitHub repository.  Sending PRs is not as
convenient as suggesting or making edits in a collaborative editor like
google docs though :(

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jul 2020 22:23:42 -0500

On Tue, Jul 07, 2020 at 10:27:36PM -0400, John R Levine wrote:
> > It's the same issue.
> 
> Indeed, so unless we can address it, I wouldn't put a lot of work into this.

Well, we have a spec that running code does not match.

We have two choices: leave it as-is, or find a way to make it a better
match for reality while achieving our I18N goals.  I believe we can do
the latter, and I believe that the former kinda sucks.

We in the then Solaris Engineering group chose to do the right thing,
which meant ignoring the spec.  We were able to do that by reasoning our
way through the problem, and because implementing the spec as written
leads to serious problems (described in my I-D).

So what shall we do?

And what shall we do for other protocols that deal in filenames?  E.g.,
if ever we finish SFTP, or maybe in HTTP.  RFC 7230 basically refers to
RFC 3986 regarding lookups, but it says nothing about PUT/POST and
whether new resources get their local parts normalized.

> CLDR is great for what it does, but I would be surprised if file system case
> folding matched any of its entries.  CLDR tells the "ls" program how to
> format timestamps for display.

I looked, and... I'll have to look again, but I'm pretty sure CLDR does
have case-folding tailorings.

> I don't think I've ever seen a disconnected NFS client, but let's assume one
> exists.  It will have to deal with all sorts of synchronization issues

Indeed, NFSv4 clients don't, but the whole world isn't NFSv4.  That's a
big part of what's wrong with putting I18N into the NFSv4 server: NFSv4
isn't the only protocol.

Thanks for your review!

Nico
--

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Tue, 7 Jul 2020 21:25:38 -0700

>> are often treated very similar to case variation for East Asian users).
(we agree on a lot - even if I heckle you on some wording issues :) )
> plenty of others, such as various whitespace and period characters.

I get you on running code - but the approach of pointing out small but 
crucial improvements like that is probably worthwhile - if 
interoperation with other filesystems isn't made worse. Some of these 
are so broken, like allowing U+2011, that it would be interesting to 
know if anyone ever created two filenames that relied on that distinction.

I need to read the remainder of your I-D - but probably not tonight, and 
not only because I'm still discussing this batch of feedback.

> I'll change that instance of 'canonical' to 'normalization'.

The word "canonical equivalence" and "canonical de/composition" are used 
contrastively to "compatibility equivalence" and "compatibility 
decomposition". Which is why NFC/NFD are commonly known as "canonical 
normalization" even if that term is not defined in that way.

NFKC and NFKD are, by the way, in my opinion design mistakes. Instead, 
there should have been a series of compatibility foldings. Water under 
the bridge.

> but I seem to remember a blog post from almost twenty years ago about

Probably an entirely different reasoning:

For a while, it looked like additional composed characters could be 
added to Unicode, which would have made only NFD stable. That got fixed.

Also, NFD appeals to purists, because all diacritical marks are treated 
the same way: as explicit characters.

> I'm not sure that's always easy to do, but I imagine it can be done.
Yes, there are even data tables that allow you to do that "quick check".

For the Latin script, the number of characters that require a sequence 
to encode in NFC is amazingly small (and required by some African 
writing systems). Chances are, therefore, that 100% of your filenames 
are in NFC in a form-preserving system (unless you have 
machine-generated random file names including diacritics).

Some South Asian scripts tend to have sequences that are unnormalized, 
rather than NFC or NFD. Often because diacritics are ordered the way 
fonts like them best. For those, both NFD and NFC take the hit from 
reordering (the number of 'compositions' tends to be small-ish, so the 
performance penalty for NFC cannot be that big).
> Ah yes, more evidence that in running code, I18N choices are made by the
I think you've mad that point, and well.
> whatever layer is implementing case-insensitivity.
Right, I think I've seen some apps present windows filenames using 
sentence casing and it's driven me bananas. Even if it doesn't matter in 
terms of accessing the file.
> There are choices to be made here.

I'm not that deep into your document yet to know your details.

> Does Unicode have terminology for "a string of code units from an unkown

A "code unit" is, at first, just that. A UTF-8 code unit, would be 
something more specific. Go have a look at UTR#17 - it was meant to give 
a model that would cover not only Unicode. But what is wrong about using 
your quoted phrase in a sentence like:

     We will use the term "just-8"  for string of 8-bit code units from 
an unknown codeset and encoding - with the proviso that the xxxx system 
can recognise "/", "\" and "\0"

(or whatever else it needs to recognize).

Such a sentence, corrected for what systems/protocols do need to know, 
would nicely link to the Unicode terminology.

> codesets/encodings in use.  I believe it's important to make that clear.
No argument.
> I might add a glossary for them.  Good idea.
glossaries are always good - but having glosses at first mention is 
important.
> _caching_ client.  (Not every NFSv4 client will cache directory

you know those ramifications better. If byte order is something to worry 
about, you need to address ether just-16 is 16-bit code units or 
"serialized" 16-bit code units.

> Right.  I really want to denote "8-bit code units, codeset/encoding

Code unit is defined in the Unicode Standard (glossary) as:

    /Code Unit <https://unicode.org/glossary/#code_unit>/. The minimal
    bit combination that can represent a unit of encoded text for
    processing or interchange. The Unicode Standard uses 8-bit code
    units in the UTF-8 encoding form, 16-bit code units in the UTF-16
    encoding form, and 32-bit code units in the UTF-32 encoding form.
    (See definition D77 in Section 3.9, Unicode Encoding Forms
    <http://www.unicode.org/versions/latest/ch03.pdf#G7404>.)

If you accept that this definition is meant more generically (and it is 
used in UTR#17 that way) then your definition of filenames is covered by 
"code unit sequence" for an "unspecified/unknown" character encoding.

And "just-8" is a code unit sequence of 8-bit code units for an unknown 
character encoding. Also, if you use the defined term "character 
encoding" by reference then you don't have to use alternate 
constructions like codeset/encoding.

Unicode defines terms that were needed because of the introduction of 
wide character code units:

  * CES (character encoding scheme) is the mapping from code units to
    bytes (think: serialization). Trivial for UTF-8 or ASCII, not so for
    UTF-16.
  * CEF (character encoding form) is the mapping from code points to
    code units. Again, trivial for ASCII, but in Unicode you get all the
    UTFs.
  * And "character encoding" is the umbrella term that says how the code
    points relate to characters (i.e. Unicode, JIS, whatever).

The need for all those definitions was that nobody, prior to Unicode, 
had to deal with CEFs and the serialization to bytes was more or less 
"implicit". By all means, if someone created a parallel set of IETF 
definitions for the same concepts, use those, but unless there's an RFC 
that covers the same grounds as exhaustively, I'd recommend citing 
UTR#17. (If necessary, you use the glossary to relate to other industry 
terms).

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Tue, 7 Jul 2020 21:28:41 -0700

> time to organize a "lunch-time" meeting of this directorate).
I'll move on to the rest of your document.
Nobody's blaming you - I've been often enough on the other side and 
despite all defensiveness, always end up benefiting from knowing how my 
readers actually parse my text.
> convenient as suggesting or making edits in a collaborative editor like
Indeed.

I'm not a natural with GitHub. So, I'll stick to this method.

A./

---

From: "Patrik Fältström" <[email-masked]>
Date: Wed, 08 Jul 2020 07:14:27 +0200

On 7 Jul 2020, at 21:16, John C Klensin wrote:

> (independent of a few reviews of documents from assorted IETF WGs), the directorate seems to be both disfunctional and lacking any form of leadership or guidance.   And, if we can't get
> enough collaboration (or some form of aggressive push-back), I think the experience with RFC 8753 and with 5891bis strongly suggests that more work on any of those documents may be a waste of effort.

I agree with this, but I think we also agree one of the problems is a combination of lack of time and competence.

I think we because of this have a few different (!) problems to work with:

- How do people that have competence get more time?

- How do we grow the number of people that do have competence?

- What do we do in the meantime?

Unfortunately I think this list/ad-hoc-group (I do not want to call it directorate really because of the "low quality review process we have", because although we have a somewhat dysfunctional process, we do have a process, we do have a group of people on this list, and sometimes some of us do have a few cycles to spare.

With these few cycles, we get documents and statements and review to some level of quality.

Meta questions are then:

- Is this review enough?

- How do we know the answer to the previous question is correct?

But in the mean time, we just have to work with what we have :-(

I do not like it, but...not as many people are interested in I18N as we want, and definitely not as many as IETF needs.

   Patrik

---

From: "Patrik Fältström" <[email-masked]>
Date: Wed, 08 Jul 2020 07:36:03 +0200

On 8 Jul 2020, at 5:09, Nico Williams wrote:

> So I agree with you, and in fact, that is the reason I pushed for form-insensitive/preserving behavior in ZFS.

rsync with the iconv flag is your friend.

What is bad I think is that the rsync included in MacOSX do not include that flag so working effectively in a mixed NFC/NFD environment is not easy for end users.

   Patrik

---

From: John C Klensin <[email-masked]>
Date: Wed, 08 Jul 2020 01:36:35 -0400

Nico,

We are not communicating very well or at all.   It might be my
fault, so, after this note (which I hope will help), I'm going
to withdraw from the conversation for at least a few days.

--On Tuesday, July 7, 2020 16:45 -0500 Nico Williams
<[email-masked]> wrote:

>> What I'm concerned about -- and what caused that question
>> about priorities -- is, at an even higher level, the sort of
>> issue covered by the note I just sent in response to
>> Patrik's.  With one qualification below (which I predict you
>> won't like), if the IESG (via IETF LC), a WG, an AD, or maybe
>> an individual author who is proposing an update in line with
>> documents that have already achieved IETF consensus requests
>> a review _from the directorate_, and Pete assigns that review
>> to someone and they do it, then that is great.  
>> 
>> The qualification is that, if the review was requested from
>> the directorate about a document that has been written
>> consistent with established IETF consensus documents _and_
>> that does not reopen or contradict the questions that were
>> presumed to be settled by that consensus, then that review
>> ought to respond to the i18n issues raised by the document
>> and suggest issues with them, not say "well, the whole
>> concept of which the spec under review is a {proposed) part
>> is broken and I have a better idea". Should the latter occur,
>> I believe that, if we are really functioning as a
>> directorate, a second reviewer should immediately be
>> recruited and assigned to provide the review that was
>> requested.
> 
> Yours is an argument for stare decisis.  Once wrong, always
> wrong.  As if consensus, once found, can never change.

No it isn't.  I've been arguing against positions like that, I
think consistently, for many years, including in a recent appeal
to the IESG.  I am, first, trying to make a distinction between
a review request to a directorate with expertise in particular
mechanisms and a more general review request or spontaneous
review.  If you had written exactly the same notes  (to the
authjor or to the WG list) because you were following the WG's
work and felt a desire to speak up, I think that would have been
perfectly reasonable.  For a directorate (as compared to a
review team or some other arrangements) review, I think there is
an assumption that, at least at a high level that other experts
would agree with you.  Perhaps they do, perhaps they don't, but
I don't think it is appropriate for you to use a
directorate-assigned review to push your particular ideas about
a replacement protocol approach .

I also think that a certain amount of respect is due to
established protocols that have gotten IETF consensus and broad
deployment.  That does not mean they can't be changed.  It does
not mean they are not wrong-headed either.  But it seems to me
that "I have an idea that I think is better" is not sufficient
to cause a reset to zero, treating the new idea as inherently at
least as valid as the older, established, one on an equal
comparison basis or as inherently better.  If nothing else, long
experience with trying to get people to discard something they
think works (or at least are used to) in favor of something new
should set a relatively high bar because, from a pragmatic
engineering standpoint, replacement protocols or models that
cannot be deployed are fairly useless except as learning
exercises associated with how to think about problems.  Because
this is an i18n effort, I think we have a worked example of what
I'd consider the right way (or at least a right way) to approach
a situation in which an established protocol needs changing or
replacement.  One starts with a careful analysis of what is
wrong with the old one, not just an assertion that it is broken
or that one has a better idea.  One seeks consensus about that
analysis of difficulties (see RFC 4690 as an example).  Then one
sorts out new/revised protocols (in a WG if there is any
controversy at all) that address those deficiencies and any
others that can be identified.

Now, if you think there is going to be sufficient support for
your proposal, by all means ask for a BOF.  Or ask for a mailing
list and use it to demonstrate sufficient interest and
commitment that you can go to the ADs and persuade them to get a
WG going without a BOF.  I just object to your using the
directorate as a substitute for that when the directorate has
other work to do.   And, if you think the right solution to
getting your draft (and maybe other things) the attention they
deserve, by all means go for it.  I'd expect the issue of our
not being able to get sufficient energy and commitment to move
i18n-related WGs forward to be raised (as evidenced by EAI and
PRECIS being clearly completely out of energy by the time their
work wound down) but perhaps you have a good theory as to why
the effort you are contemplating is different.

> I reject that argument, at least as to our documents.

Ok.  But the argument you are rejecting is, AFAICT, a complete
strawman... certainly nothing I was advocating.

> It seems, too, like you're conceding that our approach to
> NFSv4 I18N was wrong, but you want to keep that old concensus
> untouched.  Of course, I welcome your concession even as I
> reject your argument that we must never revisit consensus
> findings.

Someday I will tell you how I feel about NFS, how long I have
felt that way, and what I think of the additions, revisions, and
patches to it... and why.  But, from that point of view, the
situation is not a lot different from what it was when RFC 1094
was published in 1989: The thing is out there, it is widely used
and widely deployed and someone who wants to replace it with a
different model quite appropriately needs to explain not only
why their model is better but it is enough better that there is
a reasonable expectation that it will be adopted.

Enough.  I need some sleep and a break from this discussion.

    john

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jul 2020 00:49:24 -0500

On Wed, Jul 08, 2020 at 07:14:27AM +0200, Patrik Fältström wrote:
> > (independent of a few reviews of documents from assorted IETF WGs),
> > the directorate seems to be both disfunctional and lacking any form
> > of leadership or guidance.   And, if we can't get enough
> > collaboration (or some form of aggressive push-back), I think the
> > experience with RFC 8753 and with 5891bis strongly suggests that
> > more work on any of those documents may be a waste of effort.
> 
> I agree with this, but I think we also agree one of the problems is a
> combination of lack of time and competence.
> 

Not by giving up.  And not by being a directorate, which is
invitation-only.  I was asked if I wanted to join.  I joined.  I was
given an assignment, and I did it.  If the chairs are not giving out
enough assignments, then maybe we need new chairs.  If we're not
reviewing our own work, then maybe we should be a WG rather than a
directorate, which would have the main benefit of being more accessible
to newcomers.

> Unfortunately I think this list/ad-hoc-group (I do not want to call it
> directorate really because of the "low quality review process we
> have", because although we have a somewhat dysfunctional process, we
> do have a process, we do have a group of people on this list, and
> sometimes some of us do have a few cycles to spare.

How can a directorate be disfunctional?  Well, either by having chairs
or secretaries who don't give out assignments, by having members who
shirk assignments, or both.  Both problems are a matter of cajoling the
relevant people until they performed their assigned duties or quit the
chair or group.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jul 2020 00:50:20 -0500

On Wed, Jul 08, 2020 at 07:36:03AM +0200, Patrik Fältström wrote:
> What is bad I think is that the rsync included in MacOSX do not
> include that flag so working effectively in a mixed NFC/NFD
> environment is not easy for end users.

I mean, nice that rsync has that.  But rsync is just one application.

---

From: "Patrik Fältström" <[email-masked]>
Date: Wed, 08 Jul 2020 07:52:48 +0200

> I mean, nice that rsync has that.  But rsync is just one application.

It has saved me many many many many times. I.e. all the time.

   Patrik

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jul 2020 01:06:46 -0500

> that "I have an idea that I think is better" is not sufficient

That is denigrating and completely unnecessary.  I'm not saying "I have
an idea that I think is better".  I'm saying that the RUNNING CODE
differs from the damned spec.

Now, you can you not give a damn and let the spec and running code
continue to differ.  Fine.  But why even bother updating the spec at all
then?  Who cares what it says if no one bothers to implement it as
written?

We're about rough consensus, yes, but also running code damn it.

You can write fiction if you like, but I'm for writing specs that
reflect reality and have a chance in hell of improving reality.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jul 2020 01:09:21 -0500

> It has saved me many many many many times. I.e. all the time.

But it's not the only application, it's just one application.  All
applications would have to do the same thing (which, BTW, is also not
what the NFSv4 spec says).  We're not going to boil oceans.  There is a
solution.  It is implemented and widely used.  We should document it.

---

From: "Patrik Fältström" <[email-masked]>
Date: Wed, 08 Jul 2020 08:10:49 +0200

On 8 Jul 2020, at 8:09, Nico Williams wrote:

> But it's not the only application, it's just one application.  All applications would have to do the same thing (which, BTW, is also not what the NFSv4 spec says).  We're not going to boil oceans.  There is a solution.  It is implemented and widely used.  We should document it.

Completely agree.

   Paf

---

From: Asmus Freytag <[email-masked]>
Date: Wed, 8 Jul 2020 03:41:59 -0700

> reflect reality and have a chance in hell of improving reality.

This is an interesting discussion and brings to mind two examples that 
are not IETF standards. One is HTML5 where people got tired about 
unsupported and inconsistently supported features of HTML and are giving 
the RUNNING CODE a much greater weight - even when moving forward. To 
the point that some are dissatisfied that implementations are driving 
the specification. (There are millions of details here that I don't care 
to tease apart here, I'm after a very rough analogy for the purpose of 
this discussion.)

The other is the early Unicode. Which was an absolute departure from 
existing character encodings and a bit of a clean break. As conceived 
initially, it would require a rewrite of all code, with (lossy) mapping 
to legacy encodings (but basically loss-less round trip ones) as the 
only means of interacting. Here, the inadequacies of the existing 
non-solution (character set independence) was so overwhelming that there 
was a critical mass of implementations that lined up behind the new, 
incompatible scheme.

Until someone invented the "universal adapter plate" in the form of 
UTF-8, which neatly allowed Unicode to co-exist in most character set 
independent environments and, without rewriting that code, to become 
dominant. (In doing so, they "fixed" a number of shortcomings of 
existing multi-byte encodings, not least of which are stateless and 
self-synchronizing nature of the mapping from UTF-8 to Unicode characters).

Crucially, the UTF-8 adaptation brings all the other elements of the 
Unicode departure from existing character set encodings, to wit, a fixed 
set of character properties and behaviors (such as renderings) that come 
into force any time a system treats UTF-8 as Unicode (instead of 
pass-through bytes).

Later Unicode had it's share of "fiction"; good ideas that remained 
that, like "paragraph separator" and "line separator" characters, or 
language tag characters. Some were later deprecated, which is good, some 
repurposed (for emoji flag sequences).

But Unicode also always allowed the definition of "what is a character?" 
to be broad enough to track "de-facto characters" that were defined and 
used in competing standards (compatibility characters, but also emoji 
and many other sets). And by absorbing these, preventing competing 
standards based on competing ideas of character-hood.

Now, as rough as these examples are (and I'm not suggesting we refine 
them), they indicate to me that it's not always possible to define 
a-priori what course is the best one to take. A clean break that 
requires new code to fix something may be the right answer if the 
problem is bad enough and the fix enables new capabilities.

In other cases, if a "good idea" leads to issues with interoperability 
so that nobody can rely on it, because either RUNNING CODE can't be 
changed, or the code that could adapt lacks critical mass, or 
implementers don't see the cost/benefit of change then it's a bad idea - 
even if it's been on paper for a long time. Sometimes "breaking the 
spec" to make it better match what's on the ground is the wiser approach 
(like Unicode's occasional deprecation of characters).

Trying to argue this in the abstract isn't going to resolve this issue. 
Each case has to be looked at based on its merits and the full details 
of the situation. And, like UTF-8, never count out the possibility that 
someone figures out the universal adapter plate.

A./

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jul 2020 10:47:34 -0500

On Wed, Jul 08, 2020 at 03:41:59AM -0700, Asmus Freytag wrote:
> This is an interesting discussion and brings to mind two examples that are
> not IETF standards. One is HTML5 where people got tired about unsupported
> and inconsistently supported features of HTML and are giving the RUNNING
> CODE a much greater weight - even when moving forward. To the point that
> some are dissatisfied that implementations are driving the specification.
> (There are millions of details here that I don't care to tease apart here,
> I'm after a very rough analogy for the purpose of this discussion.)
> 
> [...]

Noted.  The NFSv4 I18N mistake is a 17 year mistake.  We do have a
chance to improve real-world outcomes, but not if we persist in doing
something we know all have and will continue to ignore.  If there was no
chance to improve things, I wouldn't bother, and we could let a 17 year
mistake become a 117 year mistake.

I'm more convinced now than earlier that this should have been a WG, not
a directorate.  Again, expect a BoF proposal on that for IETF 109.

Nico
--

---

From: "John R Levine" <[email-masked]>
Date: 8 Jul 2020 11:50:40 -0400

> the latter, and I believe that the former kinda sucks.

If I wasn't clear enough, my point is that any remote file scheme has to 
deal with situations where two different names refer to the same file.

Case folding and Unicode normalization are just examples of that, and as 
far as I can see, not particularly interesting ones.  There is no problem 
to be solved here.

I suppose we could offer advice saying it would be nice if file systems 
used UTF-8 names and consistent normalization but I don't see why anyone 
would pay attention.

Regards,

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jul 2020 11:16:45 -0500

On Wed, Jul 08, 2020 at 11:50:40AM -0400, John R Levine wrote:
> I suppose we could offer advice saying it would be nice if file systems used
> UTF-8 names and consistent normalization but I don't see why anyone would
> pay attention.

Several filesystems do just that.  No one implements what NFSv4 says to
do for I18N, so clearly persisting in that approach won't help.

Having a document specifically targeted at filesystem implementors is
much more likely to help, even if we have no authority over them.  After
all, we don't have a protocol compliance police either, yet we write
protocol documents all the time.

Nico
--

---

From: "John R Levine" <[email-masked]>
Date: 8 Jul 2020 12:23:53 -0400

> protocol documents all the time.

So long as we make it clear what kind of advice it is, and we acknowledge 
the issues in migrating from what they do now to UTF-8 nirvana, that could 
be somewhat useful.

Regards,

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jul 2020 11:50:49 -0500

> the issues in migrating from what they do now to UTF-8 nirvana, that could

Right!  Thank you for seeing that.  This work sure can't hurt.

Migration is tough of course, but for _new_ filesystems it is a
non-issue (unless the intent is to restore existing contents that needs
migration).  Stopping the rot is often the first step towards repairing
it.

Two schemes are described in my I-D: the HFS+ scheme (normalize on
create and lookup) and the ZFS scheme (form-insensitive/preserving).
(As you can see, there is nothing terribly original in this I-D...)

The former is by far the simplest to retrofit onto existing filesystem
code that doesn't index directory contents.  Conversely, the latter is
equally easy to retrofit into filesystems that do index directory
contents.

The latter yields superior user experience and interoperability, but the
former would be good enough if it normalized to NFC (sadly, Apple went
with NFD, but others could choose to use NFC).

Those two schemes were arrived at by Apple and Sun engineers between 2
and 1.5 decades ago, respectively.  To my knowledge, in all that time no
better approach has been found.

We should document this and take the opportunity to make small
improvements, such as mapping U+2011 to U+002D and such.  In the worst
case, it will do as much good as the NFSv4 spec's I18N provisions
(none), though also no harm.  In the best case it will help make all
sorts of filesystems better.

Lastly, I believe this approach has wider applicability, thus
documenting it may help in areas other than filesystems.

Nico
--

---

From: "John Levine" <[email-masked]>
Date: 8 Jul 2020 16:38:11 -0400

In article <9044C737C36C0787B9EAE190@PSB> you write:
>lower casing will cause rather interesting problems. 

FWIW MacOS is Unix underneath and its native file system is case
folding.  I've been surprised how little trouble it's caused when
I'm moving code back and forth to FreeBSD and even NFS mounting
my MacOS filesystem on FreeBSD virtual machines.

The point about ZFS is also a good one, since it's gotten very popular
on BSD and Solaris due to its flexible disk management. (Less so on
most linux distros due to poor kernel integration.) It gives you a
per-filesystem choice of case sensitifity, and also optionally doing
any of the four Unicode normalizations on filenames.

R's,
John

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jul 2020 16:18:27 -0500

> any of the four Unicode normalizations on filenames.

Yeah.

In ZFS the NF is about form-insensitivity, so there's no observable
difference between picking NFC and NFD, or NFKC and NFKD.

BTW, I found this thread where Git users on OS X using ZFS were bitten
by normalization to NFD on create.  It took some doing to find it!

http://public-inbox.org/git/[email-masked]/

and here Linus blames ZFS (though others correctly blame Apple)

http://public-inbox.org/git/[email-masked]/

saying that

| UFS is a traditional Unix filesystem, and will not mangle your filenames.
| 
| ZFS apparently acts like a case-sensitive HFS+: it still tries to 
| normalize to UTF-8 (and does it badly, at that - picking an Apple-specific 
| almost-NFD form of normalization rather than the more sensible NFC form).
| 
| So ZFS may not corrupt cases, but it still corrupts UTF-8 filenames.

But again, the fault was Apple's: they chose to normalize to NFD on
create.

Normalizing to NFC would have been better because that would have
aligned better with input modes' output.  Input modes don't necessarily
produce NFC though.  This is why form-insensitive and form-preserving
behavior is a better choice.

As it is, important apps with squeaky users get the normalization grease
(rsync, Git, as mentioned).  But if you're using ZFS on anything other
than OS X, no apps need any grease.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jul 2020 17:08:45 -0500

Apparently, with APFS Apple switched from normalize-on-create to
form-insensitivity:

https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html#//apple_ref/doc/uid/TP40016999-CH6-DontLinkElementID_3

| APFS accepts only valid UTF-8 encoded filenames for creation, and
| preserves both case and normalization of the filename on disk in all
| variants. APFS, like HFS+, is case-sensitive on iOS and is available in
| case-sensitive and case-insensitive variants on macOS, with
| case-insensitive being the default.
| 
| In macOS High Sierra, APFS is normalization-insensitive in both the
| case-insensitive and case-sensitive variants, using a hash-based native
| normalization scheme. In iOS 11, APFS is normalization-insensitive as
| well, using either a native normalization scheme (erase restores only)
| or runtime normalization scheme (upgrades from previous versions).
| Runtime normalization will also be available in iOS 10.3.3 and macOS
| Sierra 10.12.6. Being normalization-insensitive ensures that
| normalization variants of a filename cannot be created in the same
| directory, and that a filename can be found with any of its
| normalization variants. This means that you don’t need to do any
| additional work to ensure correct normalization behavior in these
| versions of macOS and iOS.

With this I think it's fair to say then that form-insensitivity is now
the best common practice in filesystems!

(It'd be nice to have a survey of Unicode support in the various Linux
and *BSD filesystems, like BTRFS, HAMMER, HAMMER2, etc.
https://en.wikipedia.org/wiki/Comparison_of_file_systems does not
mention this stuff.)

Earlier versions of APFS though did not do any normalization or form-
insensitivity:

| In iOS 10.3 and in the case-sensitive variant of the developer preview
| of APFS in macOS Sierra, APFS is normalization-sensitive ...

which yielded this complaint:

https://eclecticlight.co/2017/04/06/apfs-is-currently-unusable-with-most-non-english-languages/

Ooops.  So Apple says:

| To avoid introducing bugs in your code with mismatched Unicode
| normalization (for iOS 10.3.0, 10.3.1 and 10.3.2) in filenames, do the
| following:
|
|  o Use high-level Foundation APIs such as NSFileManager and NSURL when
|    interacting with the filesystem.
|
|  o Use the fileSystemRepresentation property of NSURL objects when
|    creating and opening files with lower-level filesystem APIs such
|    as POSIX open(2), or when storing filenames externally from the
|    filesystem.

So... like Git and rsync as mentioned here earlier.  But Apple did fix
APFS.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jul 2020 19:15:25 -0500

> https://en.wikipedia.org/wiki/Comparison_of_file_systems does not

Well, I took a look at Linux.  f2fs and ext4 use utf8_strncasecmp() for
case-insensitivity, and IIUC utf8_strncasecmp() is also
form-insensitive (normalizing as needed as it goes).

There is also a utf8_strncmp() function that does case-sensitive but
form-insensitive comparison.  Nothing uses it.

It may be a relatively simple change to add an option for form-
insensitive, case-sensitive behavior in some Linux filesystems, though
in all cases how easy it might be will depend on how directory contents
are encoded.  Maybe we should file a bug about this.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 9 Jul 2020 17:16:30 -0500

FWIW, Linux's utf8_strncmp() normalizes to NFD as it goes.  That's
because the underlying normalization functionality (which is also
available to FS code) only implements NFD.