Subject: [Lucid] Please clarify the i's in Appendix A

From: Shawn Steele <[email-masked]>
Date: Fri, 20 Mar 2015 00:01:58 +0000

https://datatracker.ietf.org/doc/draft-sullivan-lucid-prob-stmt/?include_text=1 that seems odd to me, where does Unicode document this?

   U+069 vs \u'0069'\u'0307'  LATIN SMALL LETTER I followed by COMBINING
      DOT ABOVE by definition, renders exactly the same as LATIN SMALL
      LETTER I by itself and does so in practice for any good font.  The
      same would be true if "i" was replaced with any of the other
      Soft_Dotted characters defined in Unicode.  The character sequence
      \u'0069'\u'0307' (followed by no other combining mark) is
      reportedly rather common on the Internet.  Because base character
      and stand-alone code point are the same in this case, and the code
      points affected have the Soft_Dotted property already, this could
      be mitigated separately via a context rule affecting U+0307.

IDNA makes this worse by mapping İ U+0130 to U+0069 + U+0307, apparently to get a visual effect.  It seems peculiar to me that there would need to be an exception to the existing exception.

I’d like an attribution for the remark “\u'0069'\u'0307' (followed by no other combining mark) is reportedly rather common on the Internet”.  I’d also be interested in what contexts.  I’d also be curious about how those came into existence.  (NFD followed by lower case?)

-Shawn

 
http://blogs.msdn.com/shawnste
http://L3-G0.blogspot.com

---

From: Patrik Fältström <[email-masked]>
Date: Fri, 20 Mar 2015 06:55:06 +0100

> On 20 mar 2015, at 01:01, Shawn Steele <[email-masked]> wrote:
> 

Can you please clarify?

IDNA do not include mapping, so I presume you mean "IDNA would make this worse IF it mapped..."?

   Patrik

---

From: Shawn Steele <[email-masked]>
Date: Fri, 20 Mar 2015 06:51:52 +0000

U+0130 is disallowed, however most implementations do mapping for lookup, in which case U+0130 changes to U+0069 + U+0307, due to the application of NFKC + Casefolding.  http://www.unicode.org/Public/idna/8.0.0/IdnaMappingTable.txt 

Or is lucid only concerned with the case where no mapping is applied?

-Shawn

-----Original Message-----
From: Patrik Fältström [mailto:[email-masked]] 
Sent: Thursday, March 19, 2015 10:55 PM
To: Shawn Steele
Cc: [email-masked]
Subject: Re: [Lucid] Please clarify the i's in Appendix A

Can you please clarify?

   Patrik

---

From: Patrik Fältström <[email-masked]>
Date: Fri, 20 Mar 2015 08:02:58 +0100

> On 20 mar 2015, at 07:51, Shawn Steele <[email-masked]> wrote:
> 

Ok, then I understand.

That is though for me very different from IDNA including mapping, which it does not.

> Or is lucid only concerned with the case where no mapping is applied?

Not from my perspective.

But one of the ways IDNA2008 was able to move forward was to, as we all know, ensure mapping was not part of IDNA, just because different contexts required and needed different mappings. And because we really needed in IDNA strict rules for 1:1 mappings between A-label and U-label, and whether codepoints where allowed or not.

So we maybe unfortunately must be a bit precise when we look at solutions, or I am afraid it will be very hard to come to a conclusion.

   Patrik

---

From: Andrew Sullivan <[email-masked]>
Date: Sat, 21 Mar 2015 12:45:17 -0500

Hi,

On Fri, Mar 20, 2015 at 06:51:52AM +0000, Shawn Steele wrote:
> U+0130 is disallowed,

Ok.

> however most implementations do mapping for lookup,

Yes.

> in which case U+0130 changes to U+0069 + U+0307, due to the application of NFKC + Casefolding.

> http://www.unicode.org/Public/idna/8.0.0/IdnaMappingTable.txt 

What you're saying, actually, is that "most implementations" do
casefolding of upper-case input strings prior to IDNA processing
according to an approach that Unicode has (in this case pre-?)
published?  I'm just trying to understand.

> Or is lucid only concerned with the case where no mapping is applied?

I think we have to worry about cases where input strings are the
result of mapping, yes, but _before_ we tackle the really hard cases
can we sort out the merely hard cases?

Thanks,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Shawn Steele <[email-masked]>
Date: Sat, 21 Mar 2015 19:40:23 +0000

> What you're saying, actually, is that "most implementations" do casefolding of upper-case input strings prior to IDNA processing according to an approach that Unicode has (in this case pre-?) published?  I'm just trying to understand.

In IDNA 2008 it was recognized that mappings existed in IDNA2003 and that implementations such as browser address bars would find them useful.  It was also recognized that mapping is a complex problem, so IDNA 2008 didn't tackle the mapping problem, suggesting that different folks could implement their own appropriate mappings.

Clearly it doesn't help much to have everyone doing their own different mapping, so Unicode provided a mapping which was influenced heavily by the mappings of IDNA 2003 for compatibility.  I'm not quite sure how U+0130 ended up with its peculiar mapping, which seems like a quirk or the order the rules are applied.  Though I find that particular mapping confusing, it is consistent with IDNA2003.

> I think we have to worry about cases where input strings are the result of mapping, yes, but _before_ we tackle the really hard cases can we sort out the merely hard cases?

Hmm

---

From: Andrew Sullivan <[email-masked]>
Date: Sat, 21 Mar 2015 16:32:42 -0500

On Sat, Mar 21, 2015 at 07:40:23PM +0000, Shawn Steele wrote:

> In IDNA 2008 it was recognized that mappings existed in IDNA2003 and
> that implementations such as browser address bars would find them
> useful.  It was also recognized that mapping is a complex problem,
> so IDNA 2008 didn't tackle the mapping problem, suggesting that
> different folks could implement their own appropriate mappings.

> Clearly it doesn't help much to have everyone doing their own
> different mapping,

My memory is different about this case (no pun intended).  What I
recall about IDNA2008 was that we realized, though experience, that a
single set of mappings would not work reliably universally.  The
Turkic I-set you're raising with this example is actually quite a good
example of this, because whereas the case folding of I is mostly
straightforward, Turkic causes trouble.  Moreover, different
conventions for i-accent handling cause trouble, too.  What we
concluded in the WG was that, if localization were to be taken
seriously, then you could not come up with a universal handler for
this, and registration rules would have to suffice to avoid serious
problems where the community was Internet- (or even
significant-chunk-of-Internet-) scale.  (This is part of why Asmus's
and my draft discusses the network scope as a factor; we seem to have
forgotten this part of the original assumptions of i3.)

The Unicode single-mapping approach basically just does away with that
assumption of the IDNA2008 work (on which PRECIS is founded).
Presumably, Unicode thinks this is a stupid idea, but it was one of
the original founding notions in i3 and if the IETF is to abandon it
we ought to do so with our eyes open.

Anyway, it still seems to me that we have a problem with things that
are straightforwardly PVALID, and we ought to try to understand that
case before we work out what to do with strings that are DISALLOWED
but that have some other transformation on the path to i3.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]