Subject: [urn] Case normalization (CREF 6)

From: [email-masked] (Henry S. Thompson)
Date: Thu, 19 Mar 2015 11:04:39 +0000

I support the changes in section 4.1.  I am agnostic about the
p-component issue.

Wrt https://github.com/MelindaShore/urnbis/issues/3

 1) I agree with Melinda that the reference to 3986 6.2.2.1 is not
    really sufficient to explain what is actually _meant_ by "case
    normalization", so suggest that each of the three numbered items
    have a clause of the form ", by conversion to {upper,lower} case",
    as appropriate;

 2) On the other hand, her concerns that suggest the whole vexed
    question of case normalization for Unicode is involved here are,
    in my view, mistaken.  URIs, and hence URNs, are constrained to be
    ASCII, for which upper- and lower-casing _are_ well-defined.

    In this I disagree with what I _think_ Peter [1] is implying: I
    don't see any interaction here with the p-component question.
    p-components are no different from any other part of URNs: they
    have to be expressed in ASCII, including %-encoding as necessary.

ht

[1] http://www.ietf.org/mail-archive/web/urn/current/msg02797.html
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Mon, 23 Mar 2015 21:36:28 -0600

Hi Henry, thanks for your continued engagement in these issues.

>      have a clause of the form ", by conversion to {upper,lower} case",

That seems fine to me, and more readable for sure.

>      ASCII, for which upper- and lower-casing _are_ well-defined.

Agreed.

>      have to be expressed in ASCII, including %-encoding as necessary.

I think that what you propose here is fine (since we're talking about 
URIs), but as always in matters of internationalization I prefer to be 
especially careful, so I was hedging my bets in the email that you 
referenced.

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 24 Mar 2015 07:08:29 +0100

> On 24 mar 2015, at 04:36, Peter Saint-Andre - &yet <[email-masked]> wrote:
> 

Small detail...the question is not so much wether upper- and lower-casing (the functions) are well-defined. It is whether the two functions are inverse to each other or not, and how they are applied in combination with whatever normalization function that might also be applied. I.e. for each string one can absolutely apply a lower case function and get a well defined result.

   Patrik

---

From: John C Klensin <[email-masked]>
Date: Tue, 24 Mar 2015 03:48:48 -0400

> Hi Henry, thanks for your continued engagement in these issues.

Yes.

>...
>>   1) I agree with Melinda that the reference to 3986 6.2.2.1
>>   is not really sufficient to explain what is actually
>>      _meant_ by "case normalization", so suggest that each of
>>      the three numbered items have a clause of the form ", by
>>      conversion to {upper,lower} case", as appropriate;
> 

First of all, I thought we decided that p-components were to be
case-sensitive (see Section 4.1 of 2141bis-10, including CREF7,
and elsewhere).

Second, while this is a non-issue as long as the NID and NSS
(less any p-components) are strictly ASCII (see below),
evolution in other areas probably makes it unwise to assume that
URIs with the "urn" scheme will really be constructed and
treated that way long term (this is part of the downside of
"generic URI" assumptions).  The rules in 3986 are actually
fairly complicated, hand waving with "as appropriate" is no
better, and might be worse, than the reference.   The more
important part of this is that the Unicode doctrine about
case-independent matching involves CaseFold (to a first
approximation, "convert to upper, then convert to lower") and
not either one of upper or lower.  I've got a lot of issues with
case folding, especially wrt local variations (see recent note
to the LUCID list), but, if anything generic is applied, it is
likely what will be used.

>>   2) On the other hand, her concerns that suggest the whole
>>   vexed question of case normalization for Unicode is
>>      involved here are, in my view, mistaken.  URIs, and
>>      hence URNs, are constrained to be ASCII, for which
>>      upper- and lower-casing _are_ well-defined.
> 
> Agreed.

Blame me, the stress some people want to put on "generic URI
resolution", and a number of other conversations, not Melinda.

Let me try to be clear about where I personally stand on this:
my preference is precisely "constrained to ASCII".  I have some
doubts that it is, and will remain, practical in the light of
changes going on in WHATWG and W3C because, if a new "URI" spec
is established that intermixes URI and IRI concepts (including a
revised definition of the latter), the IETF semi-formally
abandons the generic URI (as well as IRI definition business (as
seems very likely at this stage), and people continue to focus
on generic URI processing, we can reference 3986 all we like but
the libraries, etc., will almost certainly be built around that
new spec.

That doesn't mean that we shouldn't say whatever we need to say
(and on which the WG can reach rough consensus).  It does mean
that we need to be very clear about what we want to say and that
the document needs to then say it very quickly, not reference
something else and hope that the reader will figure out, e.g.,
that URNs expect strict enforcement of the "URIs are entirely
ASCII" principle.

>>      In this I disagree with what I _think_ Peter [1] is
>>      implying: I don't see any interaction here with the
>>      p-component question. p-components are no different from
>>      any other part of URNs: they have to be expressed in
>>      ASCII, including %-encoding as necessary.
> 
> I think that what you propose here is fine (since we're
> talking about URIs), but as always in matters of
> internationalization I prefer to be especially careful, so I
> was hedging my bets in the email that you referenced.

See above.  If we are relying on "have to be expressed in ASCII"
that is fine with me but I think the document needs to remind
people of that at critical points.

best,
   john

---

From: John C Klensin <[email-masked]>
Date: Tue, 24 Mar 2015 03:57:02 -0400

--On Tuesday, March 24, 2015 07:08 +0100 Patrik Fältström
<[email-masked]> wrote:

> Small detail...the question is not so much wether upper- and
> lower-casing (the functions) are well-defined. It is whether
> the two functions are inverse to each other or not, and how
> they are applied in combination with whatever normalization
> function that might also be applied. I.e. for each string one
> can absolutely apply a lower case function and get a well
> defined result.

To be a bit more specific,

(1) Outside the ASCII range, ToUpper(string) is generally not
equal to ToUpper(ToLower(string)) and similarly for the
lower-case variation.  The operations are not only not "inverse"
to each other, for particular characters, they many-one rather
than one-one too).  That is, more or less, where the preference
for case folding comes from.   Again, if the string in question
is limited strictly to ASCII, these are non-issues.
  
(2) Outside the ASCII range, the statement "absolutely apply a
lower case function and get a well defined result" is strictly
true only if one knows the locale (or possibly just the
language) or has a rule that a "locale-insensitive" (really
"locale-ignoring and some locales lose") operation is being used.

    john

---

From: [email-masked] (Henry S. Thompson)
Date: Tue, 24 Mar 2015 08:19:40 +0000

> case-sensitive (see Section 4.1 of 2141bis-10, including CREF7,

Sure, none of the clauses I am referring to applhy to the p-component.

> "generic URI" assumptions).

See our previous failure to communicate wrt what I have called the
distinction between human-appropriate and protocol-appropriate forms.
We really do have to come back to that once this document is sorted.

> The rules in 3986 are actually fairly complicated, hand waving with
> "as appropriate" is no better, and might be worse, than the
> reference.

Sorry I was too brief and hence confusing -- that 'as appropriate' is
_outside_ the quote marks, i.e. _not_ meant to go in the spec.  Rather
I meant that for the three clauses, in some cases the text should read
'upper-case' and in others 'lower-case'.

> The more important part of this is that the Unicode
> doctrine about case-independent matching involves CaseFold (to a
> first approximation, "convert to upper, then convert to lower") and
> case folding, especially wrt local variations (see recent note to
> the LUCID list), but, if anything generic is applied, it is likely
> what will be used.

Not if we don't call it case-folding, which 3986 doesn't either, IIRC.

> resolution", and a number of other conversations, not Melinda.

Noted :-).

>> [concerns about creeping loss of consensus wrt ASCII-only]

Agreed that the discourse around this topic is in a
less-than-wonderful place.  And that staying out of that debate is the
right thing _for this document, at this time_.

> people of that at critical points.

For sure.

ht

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Tue, 24 Mar 2015 16:58:10 -0600

On 3/24/15 1:48 AM, John C Klensin wrote:
> my preference is precisely "constrained to ASCII".

For now that is, I think, the only reasonable path toward completing 
this work soon.

> I have some

I wish good luck to the folks who work on 2141ter. :-)

> people of that at critical points.

Yes, I will look for some places in the document to say this loud and clear.

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: "Svensson, Lars" <[email-masked]>
Date: Wed, 25 Mar 2015 15:51:35 +0000

On Tuesday, March 24, 2015 11:58 PM, Peter wrote:
[...]
> Yes, I will look for some places in the document to say this loud and clear.

+1

/Lars