Subject: [I18nrp] Mappings for IDNA2008 ?

From: "John Levine" <[email-masked]>
Date: 4 Feb 2019 17:50:46 -0500

The biggest difference between IDNA2003 and IDNA2008 is that the
latter separated out the preliminary mapping from the IDNA algorithms.
See RFC 5895 which describes a mapping and says don't use it, because
a useful mapping reflects the expectations of users, notably the
language they're using.

Are there any published IDNA2008 mappings?  As far as I can tell,
everyone uses one from UTS46 by default, and it's not very good.

-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Asmus Freytag <[email-masked]>
Date: Mon, 4 Feb 2019 15:27:49 -0800

> everyone uses one from UTS46 by default, and it's not very good.
Examples of its badness, please.

A./

---

From: "John Levine" <[email-masked]>
Date: 4 Feb 2019 19:25:54 -0500

>Examples of its badness, please.

As I understand it:

If you speak Turkish, the case folding is wrong.

If you speak Persian, the joiners are wrong.

If you speak Arabic, the mapping or lack thereof between
ASCII and Arabic digits is often wrong.

If you speak Chinese, the whole thing is wrong because Chinese users
expect their ASCII pinyin to be turned into Chinese.

IDNA 2008 said very clearly that good mappings depend on the user's
context, with the language being a large part of that context.

R's,
John

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 04 Feb 2019 19:40:50 -0500

And some of this, of course, is the result of attempts to be backward 
compatible with IDNA2003, when the WG consensus was to have a transition 
that fixed the problems in 2003.  The decision of UTS46 is really to have 
that transition last essentially forever. This is, I suspect, a difference 
in taste between the ietf and UTC communities about how to make those 
trade-offs, but it has created part of the problem.

A
--
Andrew Sullivan
Please excuse my clumbsy thums.

On February 4, 2019 19:26:21 "John Levine" <[email-masked]> wrote:

> _______________________________________________
> i18nRP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i18nrp

---

From: "John R Levine" <[email-masked]>
Date: 4 Feb 2019 19:43:46 -0500

> compatible with IDNA2003, when the WG consensus was to have a transition that 
> fixed the problems in 2003.  The decision of UTS46 is really to have that 
> transition last essentially forever. This is, I suspect, a difference in 
> taste between the ietf and UTC communities about how to make those 

This started with discussions with the people maintaining the python IDNA 
libraries.  They clearly had no idea that any mapping other than the 
ones in UTS46 is even possible.

I don't think it's so much a matter of taste as that most people doing 
this stuff have a very thin understanding of what's going on.  When I 
point out that UTS46 has Western European language assumptions baked into 
it, oh, huh, how about that.

R's,
John

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Mon, 4 Feb 2019 16:51:36 -0800

> If you speak Turkish, the case folding is wrong.

The mapping says to take CAPITAL I WITH DOT to 0069 + 0307 (which is 
lowercase i + dot).

This is formally correct, because the sequence looks precisely like "i", 
but probably not helpful unless Turkish IDNs use that convention over 
just using lowercase 'i'.

In my personal view no registries should support any soft-dotted letter 
followed by dot above as that leads to an immediate homoglyph issue. So, 
yes, that mapping is indeed questionable.

> If you speak Persian, the joiners are wrong.

The Root Zone will be like IDNA2003 here, disallowing the joiners. Yes, 
stuff looks "wrong" without in some languages - worse for Sinhala (used 
in Sri Lanka) than for Persian, by the way.

This one is a no-win situation. Some zones may support these, others 
won't - what is a generic mapping to do?

> expect their ASCII pinyin to be turned into Chinese.

That can't be a unique mapping. I imagine it would be more like the T9 
phone input.

Many, many Chinese characters share the same pronunciation (and 
therefore same pinyin).

And pinyin isn't ASCII - it has accents.

> context, with the language being a large part of that context.

None of these examples, except the first one using U+0307 strike me as 
"bad" -- only as not tailored.

I was hoping you had more examples where the mapping choices are 
inappropriate for most (or even all) languages.

A./

---

From: "John R Levine" <[email-masked]>
Date: 4 Feb 2019 19:58:24 -0500

> The Root Zone will be like IDNA2003 here, disallowing the joiners. Yes, stuff 
> looks "wrong" without in some languages - worse for Sinhala (used in Sri 
> Lanka) than for Persian, by the way.
>
> This one is a no-win situation. Some zones may support these, others won't - 
> what is a generic mapping to do?

Well, you know, the whole point of this inquiry is to find mappings that 
are NOT generic.  If you know what language your user speaks, why not give 
her a mapping that makes sense for that language?  That's why IDNA2008 
doesn't have a generic mapping.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Nico Williams <[email-masked]>
Date: Mon, 4 Feb 2019 18:59:17 -0600

> If you speak Turkish, the case folding is wrong.

So, IDNA says not to case-fold.  So users need to get case right -- oof.
Now, mostly users are accustomed to seeing domainnames in lower-case,
and type them in that way.  Sadly, mobile UIs stupidly up-case the first
letter (yay!), so if we don't at least down-case that first letter, we
have a problem for IDNs (ASCII labels are matched case-insensitively, so
they're OK; it's only IDNs that have this problem).

Now, if you never up-case, and only down-case, and if the user enters
the i or dotless i in lower-case, then there's no case folding issue for
Turkish.

The Turkish issue is academic, not a real issue.

Now, IF we could get all the broken mobile URI entry UIs in the world to
stop up-casing the first letters domainnames, then we could just train
users to enter domainnames in lower-case (or whatever case mix is
appropriate) and we'd have no problem...

...except users are not going to remember correct mix-case domainnames,
so case folding is basically necessary.  That there are a few ambiguous
cases like the Turkish dotless i seems like one of those "tough, deal
with it" moments.

Nico
--

---

From: "John R Levine" <[email-masked]>
Date: 4 Feb 2019 20:01:46 -0500

> So, IDNA says not to case-fold.

No, really, that is not what it says.  See RFC 5895.

Regards,

---

From: Asmus Freytag <[email-masked]>
Date: Mon, 4 Feb 2019 17:11:24 -0800

On 2/4/2019 4:40 PM, Andrew Sullivan wrote:
> compatible with IDNA2003, when the WG consensus was to have a 
> transition that fixed the problems in 2003.  The decision of UTS46 is 
> really to have that transition last essentially forever. This is, I 
> suspect, a difference in taste between the ietf and UTC communities 
> about how to make those trade-offs, but it has created part of the 
> problem.

To my reading the document calls out the "transition" vs. the "not 
transition" parts; that would seem to allow it in principle to 
prioritize the not-transition over time.

In the summary the goal is stated as "/allowing client software to 
access domains that are valid under either system./"

Now, this is a numbers game; if most IDNs are IDNA 2008, then "client 
software" doesn't need to presume the presence of IDN2003 labels.

Maybe something has changed on the ground since 2008 (the year this was 
first draft) and the UTC may feel differently about the "transitional" 
aspect of it. Time for a new foray?

Also, looking at the "differences from" section, it is expressed as 
differences from IDNA2003, and seems to follow IDNA2008. Not sure if 
that emphasis is new, but without some more concrete examples I somehow 
don't see the issue as big as it is being stated.

Has UTS#46 changed over time, perhaps?

A./

PS: from the dates of this, it appears all of this post-dates my active 
membership in the UTC.

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Mon, 4 Feb 2019 17:13:23 -0800

On 2/4/2019 4:58 PM, John R Levine wrote:
>> The Root Zone will be like IDNA2003 here, disallowing the joiners. 
>> Yes, stuff looks "wrong" without in some languages - worse for 
>> Sinhala (used in Sri Lanka) than for Persian, by the way.
>>
> Well, you know, the whole point of this inquiry is to find mappings 
> that are NOT generic.  If you know what language your user speaks, why 
> not give her a mapping that makes sense for that language?  That's why 
> IDNA2008 doesn't have a generic mapping.
>
> Regards,
Now you are going in circles.

Yes, give the user a tailored mapping, but nobody is served by everyone 
having to reinvent the wheel for the generic case.

So, some generic mapping (not built into the protocol, so it can, in 
fact be tailored) is clearly useful.

That people are too lazy / don' t know enough to tailor it, is  not the 
fault of a generic mapping.

A./

---

From: Asmus Freytag <[email-masked]>
Date: Mon, 4 Feb 2019 17:14:53 -0800

On 2/4/2019 4:43 PM, John R Levine wrote:
> most people doing this stuff have a very thin understanding of what's 
> going on. 

That we are in full agreement on.

A./

---

From: "John Levine" <[email-masked]>
Date: 4 Feb 2019 20:24:56 -0500

>Now you are going in circles.

No, this is the third time I'm asking for mappings other than the generic one.

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Mon, 4 Feb 2019 20:24:46 -0800

> No, this is the third time I'm asking for mappings other than the generic one.
Who would you think would be the one provide free and ready-to-use 
mappings that are specific to some target context (country, language, zone)?

A./

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 05 Feb 2019 08:24:55 +0100

> Examples of its badness, please.

Mapping of characters, including case folding, is LOCALE specific.

   Patrik

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Tue, 5 Feb 2019 00:29:59 -0800

> Mapping of characters, including case folding, is LOCALE specific.

That's not in contradiction to using a generic mapping for the 90% case 
and applying tailoring (on a per locale or other basis).

It may explain why it's bad to use a default WITHOUT adding tailoring, 
but does not explain why a particular generic mapping represents a "not 
very good" starting point for such tailoring.

A./

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 5 Feb 2019 14:00:14 +0100

> On 5 Feb 2019, at 09:29, Asmus Freytag (c) <[email-masked]> wrote:
> 
> That's not in contradiction to using a generic mapping for the 90% case and applying tailoring (on a per locale or other basis).
> 
> It may explain why it's bad to use a default WITHOUT adding tailoring, but does not explain why a particular generic mapping represents a "not very good" starting point for such tailoring.
> 

Let me try with a different explanation...

Depending on context (writing direction, locale, where the data in received etc) the application must for IDNA2008 FIRST make a conscious choice whether any mapping should happen (or just validation), and IF mapping is to happen, whether for example only NFC should happen, or case folding (locale dependent) etc.

One can not (IETF found when discussing IDNA2003) do everything “in one go”.

1. Mapping
2. Validation

   Patrik

---

From: "John R Levine" <[email-masked]>
Date: 5 Feb 2019 11:51:23 -0500

> Who would you think would be the one provide free and ready-to-use mappings 
> that are specific to some target context (country, language, zone)?

I don't know.  That's why I'm asking.

Regards,

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Tue, 5 Feb 2019 09:04:55 -0800

> I don't know.  That's why I'm asking.
I wouldn't know anyone who would do this kind of work, and would be very 
surprised.

Unicode's CLDR provides tailorings for some common issues like sorting.

A./

---

From: "John R Levine" <[email-masked]>
Date: 5 Feb 2019 12:09:23 -0500

> I wouldn't know anyone who would do this kind of work, and would be very 

Not astonished but I figured it can't hurt to ask.

> Unicode's CLDR provides tailorings for some common issues like sorting.

I've been looking at CLDR and as far as I can tell it doesn't describe 
case folding.  Did I miss it or is it really not there?

Regards,

---

From: Asmus Freytag <[email-masked]>
Date: Tue, 5 Feb 2019 09:09:28 -0800

On 2/5/2019 5:00 AM, Patrik Fältström wrote:
>
>
> On 5 Feb 2019, at 09:29, Asmus Freytag (c) <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>> That's not in contradiction to using a generic mapping for the 90% 
>> case and applying tailoring (on a per locale or other basis).
>>
>> It may explain why it's bad to use a default WITHOUT adding 
>> tailoring, but does not explain why a particular generic mapping 
>> represents a "not very good" starting point for such tailoring.
>>
>
> Let me try with a different explanation....
>
> Depending on context (writing direction, locale, where the data in 
> received etc) the application must for IDNA2008 FIRST make a conscious 
> choice whether any mapping should happen (or just validation), and IF 
> mapping is to happen, whether for example only NFC should happen, or 
> case folding (locale dependent) etc.

OK, if you make the choice that all of this should happen, then what?

You'll have to use some tables / algorithm to do the mapping. Even when 
locale dependent (notionally) you have 90%+ of all cases that are 
unchanged and some small subset would get adjusted (tailored).

That makes providing generic mapping tables useful.

> One can not (IETF found when discussing IDNA2003) do everything “in 
You've lost me here. I would need a bit more background to understand 
this statement.

I assume that "mapping" is something you do when prepping a 
user-supplied string for lookup.

But not sure what "validation" refers to.

A./

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Tue, 5 Feb 2019 09:26:31 -0800

On 2/5/2019 9:09 AM, John R Levine wrote:
>> I wouldn't know anyone who would do this kind of work, and would be 
>> very surprised.
>

Case __folding  __may be seen as universal, but case mapping has a few 
locale specific tweaks (very few).

The FAQ states:

"To make case mapping language sensitive, the Unicode Standard 
specificially allows—but does not provide the necessary 
data—implementations to tailor the mappings for each language. The file 
SpecialCasing.txt 
<ftp://ftp.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt> is 
included in the Standard as a guide to a few of the more important 
individual character mappings needed for specific languages, notably the 
Greek script and the Turkic languages. However, for most 
language-specific mappings and tailoring, users should refer to CLDR 
<http://cldr.unicode.org> and other resources."

So you'd expect to find some data in CLDR.

A./

---

From: Martin J. Dürst <[email-masked]>
Date: Wed, 6 Feb 2019 11:05:29 +0000

> mappings that are specific to some target context (country, language, 

Just some wild (but hopefully ultimately productive) guesses:

- The ICANN Universal Acceptance effort might if they think it will
   help with adoption of IDNs.

- Some national registry (e.g. for Turkey or Iran) might have the
   expertise to recommend what's appropriate for the major language(s)
   in that country.

- Somebody may be able to explain what data in CLDR would be helpful
   to create such mappings (rather than pointing out that some text
   in a spec has a pointer to CLDR). I don't know to what extent this
   might work or not, though.

- There may be some clever way to derive (part of some) mappings from
   LGR files that are (being) developed at ICANN.

Some of the above ideas in combination may also help. In addition, as 
Asmus is continuing to argue lower down in the thread, the generic 
mapping is a very good start; the examples that have come up so far all 
are about a few little (but of course important) tweaks at most.

Regards,    Martin.

---

From: "Patrik Fältström" <[email-masked]>
Date: Fri, 08 Feb 2019 08:56:29 +0100

On 5 Feb 2019, at 18:09, Asmus Freytag wrote:

> You'll have to use some tables / algorithm to do the mapping. Even when locale dependent (notionally) you have 90%+ of all cases that are unchanged and some small subset would get adjusted (tailored).
>

I can as well say it is completely useless for the cases where the mapping *IS* special.

> You've lost me here. I would need a bit more background to understand this statement.
>
> I assume that "mapping" is something you do when prepping a user-supplied string for lookup.

Yes, case folding (LOCALE-dependent), normalization and what not.

> But not sure what "validation" refers to.

Validation that a label really is an A-LABEL/U-LABEL, i.e. valid according to IDNA2008 (or not), including (when you do call some library) conversion between A-LABEL and U-LABEL.

I.e. this discussion comes up when programmers ask how they should "convert a random string to punycode".

Some people think you can call convertToPunycode($randomString) and what I say is that you have to call mapAndNormalize($randomString) and then toALabel($randomstring) and toULabel($randomString).

   Patrik

---

From: Martin J. Dürst <[email-masked]>
Date: Fri, 8 Feb 2019 08:20:44 +0000

Hello Patrik, others,

> I can as well say it is completely useless for the cases where the mapping *IS* special.

I don't understand the last sentence. If we take the case of Turkish, 
then the generic mapping tables are useful because once we have them, a 
suitable Turkish mapping can be defined as a small delta (e.g. a small 
preprocessing step) to the generic mapping tables.

Regards,    Martin.

---

From: Patrik Fältström <[email-masked]>
Date: Fri, 8 Feb 2019 09:31:20 +0100

> On 8 Feb 2019, at 09:20, Martin J. Dürst <[email-masked]> wrote:
> 
> Hello Patrik, others,
> 

lowerCase($string) delivers different result for some strings depending on LOCALE.

That’s what I mean.

   Patrik

---

From: "John Levine" <[email-masked]>
Date: 8 Feb 2019 12:58:13 -0500

>preprocessing step) to the generic mapping tables.

That would be great if the small delta tables actually exist.  But
they don't.  It's been a decade since IDNA2008 and I have yet to see
any language-specific mappings at all.

I agree that the mapping tables for languages written in Latin
characters will mostly be the same, but it's the differences that
cause problems.  Software that works most of the time not the same as
software that works.

---

From: Asmus Freytag <[email-masked]>
Date: Fri, 8 Feb 2019 11:02:24 -0800

> any language-specific mappings at all.

Meaning you hoped to get something for free and find you have to pay for it.

> cause problems.  Software that works most of the time not the same as

Curious, how many Turkish IDNs exist (i.e. those with dotless i)?

A./

---

From: "Patrik Fältström" <[email-masked]>
Date: Fri, 08 Feb 2019 20:16:39 +0100

On 8 Feb 2019, at 9:20, Martin J. Dürst wrote:

> I don't understand the last sentence. If we take the case of Turkish, then the generic mapping tables are useful because once we have them, a suitable Turkish mapping can be defined as a small delta (e.g. a small preprocessing step) to the generic mapping tables.

Ok, let me try again. It must be my lack of proper knowledge of english language that do not make me be able to make my point.

As you need this extra local delta, JUST using the plain mapping is wrong, include errors, and can not be used.

You MUST use the generic mapping (as you name it) PLUS the specific exceptions.

And it is definitely not the case that you can do f(x) and then f'(x), i.e. f(f'(x)) but instead you need to use y(x) as the function where y(x) is as you write f(x) modified with the delta that the LOCALE defines.

If one could do f(f'(x)) then it would be easy.

   Patrik

---

From: "Patrik Fältström" <[email-masked]>
Date: Fri, 08 Feb 2019 20:37:25 +0100

On 8 Feb 2019, at 18:58, John Levine wrote:

> It's been a decade since IDNA2008 and I have yet to see any language-specific mappings at all.

Hmm...do you say that you have not for example seen any LOCALE specific case folding functions?

I.e. this is completely separated from IDNA-discussions.

   Patrik

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 8 Feb 2019 14:38:29 -0500

On Fri, Feb 08, 2019 at 08:16:39PM +0100, Patrik Fältström wrote:
> 
>  modified with the delta that the LOCALE defines.

I'm still trying to understand how you know what the locale of the
various network identifiers are.  People might recall that this was
one of the problems we have identified as a serious issue in the past.
You can know what the locale is, maybe, at lookup time, but since the
information doesn't travel around with the identifier you can't know
what the locale is at creation time.  And that, of course, is how you
get a security problem out of this.

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Patrik Fältström" <[email-masked]>
Date: Fri, 08 Feb 2019 20:47:24 +0100

On 8 Feb 2019, at 20:38, Andrew Sullivan wrote:

> I'm still trying to understand how you know what the locale of the various network identifiers are.  People might recall that this was one of the problems we have identified as a serious issue in the past.
> You can know what the locale is, maybe, at lookup time, but since the information doesn't travel around with the identifier you can't know what the locale is at creation time.  And that, of course, is how you get a security problem out of this.

For example, if I as a user with $LC_CTYPE == sv_SE.UTF-8 do type or view a string of characters, then I am pretty sure I am interested in having case folding and others things mapped according to that LOCALE. But only when/if the input/output should include such mappings. Most of them should probably not have any mappings at all.

And what you point out about security is once again exactly why IDNA2008 is not including any mappings at all. It gives the ability for the application to do the mapping of characters in such a way that the rendered glyphs are such that the end user is not surprised. And that might be very different for me than you. Depending on for example LC_CTYPE.

   Patrik

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 8 Feb 2019 14:54:49 -0500

> For example, if I as a user with $LC_CTYPE == sv_SE.UTF-8 do type or view a string of characters, then I am pretty sure I am interested in having case folding and others things mapped according to…

Yes.  And the way to mess with you is to (e.g.) send you something
that has various foldings or mappings in _your_ locale that are not
the same as the various foldings and mappings in the locale with which
the string was generated.  These are blessedly rare, but they're not
"never", which is John's point upthread about working irregularly vs
working.  This gets even worse when the mappings vary according to
application or whatever, which is _also_ a potential feature of IDNA.

UTS#46 doesn't do anything to help this _either_ because it is no more
able to know the locale-at-identifier-generation time than anything
else is, because that information is simply not carried around with
the identifier.

This problem is, I should note, well understood among network protocol
people, who spent a lot of time talking about it during PRECIS.  But
it keeps coming back up because people keep forgetting that these
protocols are locale-free even if you think you know the locale at
lookup time.

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Patrik Fältström" <[email-masked]>
Date: Fri, 08 Feb 2019 21:06:25 +0100

On 8 Feb 2019, at 20:54, Andrew Sullivan wrote:

> Yes.  And the way to mess with you is to (e.g.) send you something that has various foldings or mappings in _your_ locale that are not the same as the various foldings and mappings in the locale with which the string was generated.  These are blessedly rare, but they're not "never", which is John's point upthread about working irregularly vs working.  This gets even worse when the mappings vary according to application or whatever, which is _also_ a potential feature of IDNA.

Correct.

> UTS#46 doesn't do anything to help this _either_ because it is no more able to know the locale-at-identifier-generation time than anything else is, because that information is simply not carried around with the identifier.
>
> This problem is, I should note, well understood among network protocol people, who spent a lot of time talking about it during PRECIS.  But it keeps coming back up because people keep forgetting that these protocols are locale-free even if you think you know the locale at lookup time.

And this is why I am stronger and stronger a believer that we should in protocols and what not only allow strings of characters which really are allowed in the protocols.

It is the mix between "displayed string" and "allowed in protocol string" which is problematic, specifically as we seems to allow a larger set of characters (and combination of characters) in the "displayed string", EVEN IF we are really and intentionally exposing the protocol element (like a domain name) to the end user.

That is when things breaks down I claim.

And this is why the mapping, if any, must be done as close to when the string enters "the system" as possible, and after that stays as only the allowed characters in the allowed combinations in the protocol and because of that as display.

Real protocols (whatever that is -- but lets take email as an example) should separate display strings from protocol elements, and this is why I thing the deal with local part of email addresses is a so called Big Mistake. The protocol element is ascii only. The display part is different. But this is easy for me to say that uses latin characters. I do understand people using other scripts (specifically different directionality).

I.e. my email address is:

  "Patrik H:son Fältström" <[email-masked]>

The whole thing! First the display string, and then the protocol element.

A security issue of course, like HTML:

<A HREF="security-error">display</A>

But that is at the same time the best we can get...

/me thinks

   Patrik

---

From: Asmus Freytag <[email-masked]>
Date: Fri, 8 Feb 2019 16:26:28 -0800

On 2/7/2019 11:56 PM, Patrik Fältström wrote:
> I can as well say it is completely useless for the cases where the mapping*IS*  special.
>
Narrowly defined it matters only for *labels* that happen to have a code 
point with a locale-sensitive mapping.

Most *labels*, even all the other ones for the same locale, are unaffected.

A./

PS: I'm still wondering to what extent applying locale-based mappings is 
even useful. I can see it when typing in an address bar, but already 
when accepting a URL from a document, I'm not sure it helps 
inter0perability.

---

From: Martin J. Dürst <[email-masked]>
Date: Wed, 13 Feb 2019 07:18:06 +0000

Hello Patrick, others,

>> If we take the case of Turkish, then the generic mapping tables are useful because once we have them, a suitable Turkish mapping can be defined as a small delta (e.g. a small preprocessing step) to the generic mapping tables.

> If one could do f(f'(x)) then it would be easy.

If f(x) is the generic mapping, and f'(x) is the mapping of the 
exceptions, then f(f'(x)), which applies f' *before* f, should do the job.

Proof: If it doesn't, just change f' to include the cases that were not 
taken into account correctly. You may have to do this several times. In 
the end, you may end up with f' doing all the work, and f no longer 
having to do anything, but there should not be any such cases in 
practice (not the least because each locale will leave mappings for 
scripts and characters not used in that locale to the generic mapping).

As an example, if f' for Turkish is defined as follows:
f'(I) → ı
f'(İ) → i
f'(_x_) → _x_     # leave everything else alone

then things should work for Turkish.

Regards,   Martin.

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 13 Feb 2019 08:46:42 +0100

> On 13 Feb 2019, at 08:18, Martin J. Dürst <[email-masked]> wrote:
> 

This works in some cases and not in other cases. All depending on what the functions do. Sure, might be able to find f’(x) that works in Turkish, or French or German, but it also sets a requirement on f(x).

But this is what I mean by the need for whatever see a string the first time should do mapping according to locale and after that just validation that strings are valid.

   Patrik

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Wed, 13 Feb 2019 00:19:56 -0800

On 2/12/2019 11:46 PM, Patrik Fältström wrote:
> This works in some cases and not in other cases.
Examples?
>   All depending on what the functions do. Sure, might be able to find f’(x) that works in Turkish, or French or German, but it also sets a requirement on f(x).
>

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 13 Feb 2019 09:25:38 +0100

> On 13 Feb 2019, at 09:19, Asmus Freytag (c) <[email-masked]> wrote:
> 

This is math. :-)

If f(x)=x’ and in the locale used f(x) should be x and not x’, you have dependencies between f(x) and f’(x) if you want f(f’(x)) to be x’ and f(f”(x)) to be x.

   Patrik

---

From: Martin J. Dürst <[email-masked]>
Date: Wed, 13 Feb 2019 10:31:14 +0000

Hello Patrik, others,

> If f(x)=x’ and in the locale used f(x) should be x and not x’, you have dependencies between f(x) and f’(x) if you want f(f’(x)) to be x’ and f(f”(x)) to be x.

Well, I have to admit that I didn't explicitly talk about that case in 
my 'proof'. For functions in general (e.g. a function such as f(x) = 
(x+7) mod 15), you are right.

But f(x) (the general mapping function) is idempotent (i.e. f(x) = 
f(f(x)), which means that you can map as many times as you want, you 
always get the same result). This is also true for any locale-dependent 
mapping function. If it were not the case, human users would be very 
surprised: a maps to A, but then A maps to a: That wouldn't make any sense.

Also, and even more important, for the general mapping function, if x is 
allowed in IDNs, then f(x) = x. This may not be true for a 
locale-dependent mapping function (i.e. it would be possible for a 
German mapping function to map (Hungarian) ȁ to ä, although that's not 
needed because German keyboards won't produce ȁ in the first place).

So if f(x)=x', that would mean that x is not allowed in IDNs. That would 
mean that it is impossible that in a specific locale, the desired result 
of f(f'(x)) is x. It may be different from x', i.e. y, but we can easily 
get this by defining that f'(x) = y.

So applying the (in most if not all cases very small) locale-specific 
mapping before the general mapping always will work.

Regards,   Martin.

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 13 Feb 2019 05:56:15 -0500

Surely the assumption fails because case folding and mapping of characters 
I  IDNA2003 is not always idempotent -- the very thing 2008 set out to fix?

A

On February 13, 2019 05:31:25 "Martin J. Dürst" <[email-masked]> wrote:

>> If f(x)=x’ and in the locale used f(x) should be x and not x’, you have 
>> dependencies between f(x) and f’(x) if you want f(f’(x)) to be x’ and 
>> f(f”(x)) to be x.
>

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 13 Feb 2019 12:02:58 +0100

> On 13 Feb 2019, at 11:31, Martin J. Dürst <[email-masked]> wrote:
> 
> Hello Patrik, others,
> 
> mapping function.

I can not say it is. It might be a requirement that you set on the function. I don’t know enough about various languages in the world, bidirectionality and punctuation. See latest discussions about punctuation in Armenian language and script and it’s impact on registration.

> If it were not the case, human users would be very 

No, we talk about f(a)=b and f(b)=c as one thing and f(a)=b and f(b)=a as a different one.

> allowed in IDNs, then f(x) = x.

That’s another requirement. Maybe in one locale you DO map from one to another?

> This may not be true for a 

It might produce it via copy and paste.

> mapping before the general mapping always will work.

I don’t understand what your goal is.

What I am saying is that what mapping is done when a string enters the system the first time, mapping is to happen and it is context dependent where context can for example be locale.

What is wrong with that statement?

   Patrik

---

From: Martin J. Dürst <[email-masked]>
Date: Wed, 13 Feb 2019 11:07:09 +0000

Hello Andrew, others,

On 2019/02/13 19:56, Andrew Sullivan wrote:
> Surely the assumption fails because case folding and mapping of 
> characters I  IDNA2003 is not always idempotent -- the very thing 2008 
> set out to fix?

If the general mapping is not idempotent, then indeed it has to be 
fixed. Given the purpose of the mapping, it seems obvious (at least in 
hindsight) that it should be idempotent.

My general direction of arguing is that if things are reasonably 
designed (i.e. idempotent,...), then it makes a lot of sense to split 
the mapping into a general one and some locale-specific ones, and the 
locale-specific ones will go before the general one. Also, each 
locale-specific one should be rather small (the Turkish one, often 
brought as an example, essentially consists of mappings for 2 letters).

I understand John's (Levine) point that locale-specific mapping data 
might not be available to some Python programmer writing some U-label -> 
A-label function or some such. I'm not very familiar with Python, but 
very familiar with Ruby, where it would be the same.

But it wouldn't be too difficult to get the necessary data if one really 
wanted; it's not that this is something needing big bucks (as it seems 
to me Asmus is suggesting).

Regards,   Martin.

>>> If f(x)=x’ and in the locale used f(x) should be x and not x’, you 
>>> have dependencies between f(x) and f’(x) if you want f(f’(x)) to be 
>>> x’ and f(f”(x)) to be x.
>>

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 13 Feb 2019 06:17:17 -0500

I get your general point. My general point is that 2003 was not idempotent 
and therefore any backward compatible transition mechanism necessarily 
fails the same precondition, QED.

A

---

From: Martin J. Dürst <[email-masked]>
Date: Thu, 14 Feb 2019 06:20:03 +0000

Hello Patrik,

> I can not say it is. It might be a requirement that you set on the function. I don’t know enough about various languages in the world, bidirectionality and punctuation. See latest discussions about…

Do you have a pointer?

> It might produce it via copy and paste.

Yes, that would be an issue. But in that case, it was probably intended 
to be a Hungarian ȁ, not a mistyped ä, and therefore mapping would be 
doubtful.

> I don’t understand what your goal is.

I tried to explain it in general in my mail to Andrew, and will repeat 
it here:

> What is wrong with that statement?

Nothing. It's perfectly possible that two statements (yours and mine) 
are both true :-). Your statement is about the point in time of where to 
apply a mapping. My statement is about how mappings can be split into a 
locale-specific and a general part.

Regards,   Martin.