Subject: Re: [I18nrp] Conservatism principle doesn't go far enough

From: Larry Masinter <[email-masked]>
Date: Sat, 2 Feb 2019 20:01:03 -0800

I was just pointing out an interesting document (the Chrome rules for
displaying URLs ) that covers material in more detail than RFC 3987 and
isn't currently part of the WHATWG "living standard" for URLs.
As far as it goes, people choosing names don't know how the world's
population with their raft of OS'es and browsers will behave. They might not
be idiots, just not informed.

One rule in the document calls for normalizing the Unicode as a first step. 
https://chromium.googlesource.com/chromium/src/+/master/docs/security/url_di
splay_guidelines/url_display_guidelines.md#display-urls-in-canonical-form 

But I think this is counter-productive and wrong. If you're showing someone
a URL which is not in normal form, normalization will lose this critical
information. Better to just display the punicode for any unnormalized
domains.

I'd submit an issue but I can't figure out where.

---

From: Adam Roach <[email-masked]>
Date: Sat, 2 Feb 2019 22:49:25 -0600

> I'd submit an issue but I can't figure out where.

If you want to pursue this with the Chromium team through a bug tracker, 
the entry point is here: https://chromiumbugs.appspot.com/

There may be more productive ways of engaging, though. I suspect that 
Mathias Bynens <[email-masked]> could have a useful discussion about 
this -- try sending him an email.

/a

---

From: Asmus Freytag <[email-masked]>
Date: Sat, 2 Feb 2019 23:52:45 -0800

> information. Better to just display the punicode for any unnormalized

I'm missing something here. What "critical" information (other than the 
normalization state) is it that gets lost?

There are some scripts where ordinary text is most likely not normalized 
and where it's also not necessarily trivial for users to enter 
normalized text. Should all of those URLs get displayed as punycode?

A./

---

From: "John R Levine" <[email-masked]>
Date: 3 Feb 2019 17:43:33 -0500

> isn't currently part of the WHATWG "living standard" for URLs.

This M3AAWG doc about displaying Unicode in URLs and other places was 
largely written by Mark Risher from Google.  How does it compare?

https://www.m3aawg.org/sites/default/files/m3aawg-unicode-best-practices-2016-02.pdf

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Nico Williams <[email-masked]>
Date: Sun, 3 Feb 2019 17:48:47 -0600

> normalization state) is it that gets lost?

There's a confusable issue, isn't there?  If you show me a URI which if
I were to type in would take me elsewhere...  That is an important bit
of information.  If DNS servers just had had to implement normalization-
insensitive matching, that wouldn't be a problem a tall, but obviously
we couldn't get DNS servers to do anything other than ASCII case
insensitivity.

Whether the punycoded name is displayed, or perhaps U-labels are
displayed but in some fashion as to make the danger clear (in a loud red
font, say, modulo accessibility), is another story.  I think I'd prefer
to see the Unicode and some danger warning, or perhaps hide the URI and
when I try to see it tell me it's dangerous and why.

> There are some scripts where ordinary text is most likely not normalized and

Ordinary text, as in prose from some document, yes, but security-
relevant identifiers generally need to be normalized.

> where it's also not necessarily trivial for users to enter normalized text.
> Should all of those URLs get displayed as punycode?

Users cannot be expected to enter normalized text, naturally (how could
they?).  Normalization has to be a software function, and a non-
negotiable part of the interface; somewhere, security-relevant
identifiers must be normalized.

Again, in some systems one could just normalization-insensitive
matching, but DNS cannot be such a system, therefore for DNS we must
normalize on create.

Nico
--

---

From: "John Levine" <[email-masked]>
Date: 3 Feb 2019 20:45:50 -0500

>when I try to see it tell me it's dangerous and why.

You might well, but I doubt that most users would.

I've seen examples of mixed direction labels which overlay so that it
looks like microsoft.com although it's really
microsoft.something.something.something.com.  I doubt that any
explanatory text would persuade the generic grandma that it's not
really microsoft.com and the computer is just being silly again.

If there's nothing useful the normal user can do with it, don't
display it.

---

From: "John Levine" <[email-masked]>
Date: 3 Feb 2019 20:48:13 -0500

In article <[email-masked]> you write:
>-=-=-=-=-=-

It seems to me that by the time text makes it into a URL that comes
from a web page or similar source, if it's not normalized, somthing
rather peculiar is going on.

>normalized text. Should all of those URLs get displayed as punycode?

I believe that normalizing user input is a separate issue.  Of course
you normalize what the user typed before using it.

R's,
John

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Sun, 3 Feb 2019 18:52:38 -0800

On 2/3/2019 3:48 PM, Nico Williams wrote:
> of information.

I see this issue in case mapping, but can't come up with a normalization 
example.
Hence my question.

>   If DNS servers just had had to implement normalization-

The user doesn't care what's on the server. The user cares what's 
displayed in the URL / address bar.

> when I try to see it tell me it's dangerous and why.

Still like to see the specific scenario you guys are thinking about.
> relevant identifiers generally need to be normalized.
Right, but that means they are not normalized when type as URL into HTML 
source or into an address field on a form.
> identifiers must be normalized.
No disagreement here. However, the question was, should all string that 
are entered by users as unnormalized be displayed as A-labels?
> matching, but DNS cannot be such a system, therefore for DNS we must

No disagreement here; we were talking about how to "display" something - 
that's not a function of the DNS.

A./

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 04 Feb 2019 04:53:47 +0100

On 4 Feb 2019, at 3:52, Asmus Freytag (c) wrote:

> No disagreement here; we were talking about how to "display" something - that's not a function of the DNS.

You also have to look at the larger picture.

1. The string is coming from "somewhere". Where is that?

2. The string is passed to the browser. How?

3. The browser want to display something in the address bar. What?

4. What is to be displayed is to be rendered. What is rendered?

5. The string is also to be acted upon. What is the process?

6. As part of the action, some DNS lookups might be done on a substring. Which one?

I think the question here is conflating too many things. Including statements like "case mapping". As we know the mapping in reality is either "to lower case" or "to upper case" and they are not isomorphic functions, we have to be specific even there.

   paf

---

From: Nico Williams <[email-masked]>
Date: Sun, 3 Feb 2019 22:33:30 -0600

> If there's nothing useful the normal user can do with it, don't

I mean, don't display the page at all either then.

---

From: Nico Williams <[email-masked]>
Date: Sun, 3 Feb 2019 22:34:50 -0600

On Sun, Feb 03, 2019 at 06:52:38PM -0800, Asmus Freytag (c) wrote:
> No disagreement here. However, the question was, should all string that are
> entered by users as unnormalized be displayed as A-labels?

No!  If it was entered by the user (and it's not punycoded), then
normalize.

---

From: Anne van Kesteren <[email-masked]>
Date: Mon, 4 Feb 2019 10:06:02 +0100

Hey Larry,

Glad to see you're still involved. I believe Emily and Eric (when
still at Google) have been largely responsible for this document. I
took the liberty of copying them. I suspect the document here is not
talking about Unicode normalization, but rather how parsing a URL will
normalize various aspects of it. (Having said that, ToASCII as defined
at https://www.unicode.org/reports/tr46/#ToASCII and used by the URL
Standard will do some Unicode normalization for the domain name. I
suspect that cannot be changed at this point.)

Kind regards,

Anne

On Sun, Feb 3, 2019 at 5:01 AM Larry Masinter <[email-masked]> wrote:
> https://chromium.googlesource.com/chromium/src/+/master/docs/security/url_display_guidelines/url_display_guidelines.md#display-urls-in-canonical-form
>

---

From: Larry Masinter <[email-masked]>
Date: Mon, 4 Feb 2019 12:10:15 -0800

Let me be a little more careful. The document 

https://chromium.googlesource.com/chromium/src/+/master/docs/security/url_display_guidelines/url_display_guidelines.md

seems to be primarily about one use case:  The user has somehow navigated to a site (through links, typing in, search, whatever), and is now looking at a web page; what should the UA show (in the address bar or any other security context) to validate “who is this coming from? Which site sent me this data? It says it’s my bank, did it come from them?”

The spec is only about the visual display and doesn’t cover copy/paste or save-as-bookmark or other operations which work with a different form.

Insofar as

*	normalization doesn’t change the visual display (?)
*	The URL that was actually used for DNS was normalized before use anyway

then at worst the advice to normalize is harmless. So I take back my critique.

is about different use cases, with similar but not identical advice and requirements.

The original point, though, stands: the “Conservatism” principle doesn’t go far enough in warning those registering domains to avoid those labels which downstream processors will reject or display poorly.

Is it possible to get any statistics on DNS requests that include unnormalized strings? And some hits about where they come from?

I know at one time Scott Hollenbeck gathered some data on use of %xx encoding in DNS which I will forward.

Subject: RE: Universal Acceptance of IDN TLDs
From: "Hollenbeck, Scott" <[email-masked]>
To: "Larry Masinter" <[email-masked]>
Date: Fri, 14 Mar 2014 09:26:40 -0800

I haven’t been following the discussion. You can post this info if you wish and identify me and Verisign as the source.

Scott

From: Larry Masinter [mailto:[email-masked]] 
Sent: Friday, March 14, 2014 12:21 PM
To: Hollenbeck, Scott
Subject: RE: Universal Acceptance of IDN TLDs

maybe send this stuff out to the URI list?

From: Hollenbeck, Scott [mailto:[email-masked]] 
Sent: Friday, March 14, 2014 4:34 AM
To: Larry Masinter
Subject: RE: Universal Acceptance of IDN TLDs

Larry, my researcher also studied the geolocation attributes of the source IP addresses responsible for queries containing “%xx”, the networks from which those queries originated, and the most commonly encoded characters.

Geolocation: 33% from China, 21% from the USA, long tail distribution on the rest.

ASNs: More than 6,000 unique ASN sources.

Top 10 most common characters (in descending order): %20, %2A, %2C, %2F, %40, %7C, %5c, %24, %25, %29

Scott

From: Larry Masinter [mailto:[email-masked]] 
Sent: Wednesday, March 05, 2014 1:06 PM
To: Hollenbeck, Scott
Subject: FW: Universal Acceptance of IDN TLDs

What happens when DNS gets queries for %xx-encoded names?

How difficult would it be to deploy %xx-utf8 queries to the equivalent punycode?

Of the IDNs in use, how many would exceed the DNS length limit when encoded as %xx (3 bytes / UTF-8 byte)?

 

From: Larry Masinter 
Sent: Wednesday, March 05, 2014 2:40 PM
To: 'Mark Davis ☕'
Cc: Andre Schappo; www International; Don Hollander;  <mailto:[email-masked]> [email-masked]
Subject: RE: Universal Acceptance of IDN TLDs

The handling of %xx-encoded domain names in DNS servers would be a fallback for use in legacy systems that are not IDN-aware.

So the length limit argument doesn’t carry a lot of weight – it is strictly a transitional deployment enhancement for working around legacy components which extract domain names from URIs but rcan only process 7-bit URIs and not 8-bit IRIs.

You can deploy IDNs when all of  the applications you care about will work for the users you care about for the DNS names you want to use.

Components that handle IRIs directly and pull out domain names for future processing shouldn’t ever need the %xx encoding, although decoding it is also a good idea.

From: [email-masked] <mailto:[email-masked]>  [mailto:[email-masked]] On Behalf Of Mark Davis ?
Sent: Wednesday, March 05, 2014 2:16 PM
To: Larry Masinter
Cc: Andre Schappo; www International; Don Hollander;  <mailto:[email-masked]> [email-masked]
Subject: Re: Universal Acceptance of IDN TLDs

If you mean having the DNS system natively accept %xx for domain labels as well as Punycode, I suspect that that ship has long since sailed. (That was one of the options discussed, but was turned down because of the length limitations.)

If on the other hand, you mean that client software should accept %xx notation as well as straight Unicode and punycode, that is another story. That can be handled by a client-side mapping, permitted by either IDNA2008 or UTS46. (And I agree that it's a good idea.)

With that, I could type in my address bar any of:

1.	xn--idna--x-l6c.blogspot.com <http://xn--idna--x-l6c.blogspot.com> 
2.	IDNA-ȿ-x.blogspot.com <http://xn--idna--x-l6c.blogspot.com> 
3.	IDNA-%C8%BF-x.blogspot.com <http://BF-x.blogspot.com> 
4.	IDNA-Ȿ-X.BLOGSPOT.COM <http://xn--idna--x-lt7e.BLOGSPOT.COM> 
5.	IDNA-%E2%B1%BE-X.BLOGSPOT.COM <http://BE-X.BLOGSPOT.COM> 

And they'd all resolve to  <http://xn--idna--x-l6c.blogspot.com> xn--idna--x-l6c.blogspot.com.

1.	I just checked on Chrome, and all of these work.
2.	Firefox is a bit odd: if I type in the #3, it fails; *but* it converts it in the address bar, so a subsequent enter goes to the right place. #4/#5 just fail.
3.	Don't know about other browsers.

Mark <https://google.com/+MarkDavis> 

— Il meglio è l’inimico del bene —

On Wed, Mar 5, 2014 at 2:32 PM, Larry Masinter <[email-masked] <mailto:[email-masked]> > wrote:

there’s a gap between IDN and URI in that IRI -> URI would prefer to use the %xx percent-hex URL encoding in general.

What would be preferable would be to insure that DNS requests for %xx encoded names is an acceptable alternative to punycode.

From: Andre Schappo [mailto:[email-masked] <mailto:[email-masked]> ] 
Sent: Tuesday, March 04, 2014 3:51 PM
To: www International
Cc: Don Hollander; [email-masked] <mailto:[email-masked]> 
Subject: Re: Universal Acceptance of IDN TLDs

① Is this document available online? I have looked round http://aptld.org but cannot find it.

② There are indeed barriers to the effective, real world use of IDNs. A fundamental problem is that IDNs, in general, are not properly catered for and not properly integrated into systems. One reason often quoted for treating IDNs differently is "Security". Well, I posit that any IDN security issues pale in comparison to the ubiquitous "… for further information please click here."

Here are some examples from Social Media:

Twitter

If the Unicode form is entered —

#test  http:// <http://北大.中国> 北大.中国

It is not recognised as a Domain Name & not displayed as clickable link

If the punycode form is entered —

#test http://xn--djry4l.xn--fiqs8s

It is now recognised as a Domain Name and displayed as a clickable link but displayed as punycode instead of Unicode

Sina Weibo

Same results —
#test# http:// <http://北大.中国> 北大.中国
#test# http://xn--djry4l.xn--fiqs8s

There is also the related issue of having to Percent Encode the Unicode pathname components of a URL.

③ In my experience, another fundamental problem is the lack of IT Internationalization teaching in Schools and Universities. Certainly in England, IT Internationalisation has not yet become an accepted part of the curriculum. We need to produce students that have an appreciation/understanding of IT Internationalisation in order to, amongst other goals, properly integrate IDNs into systems/apps/websites …etc…

For several years I have been teaching a module entitled "International Computing" which covers several aspects of IT i18n. One of the topics I cover is IDNs :) And I am keeping my students up to date with the idn new gTLDs as they are delegated to DNS Root :)

During my years teaching this module I have found few students (regardless of which country they come from) with even a basic appreciation of IT Internationalization because it is a topic that was never discussed/raised in their prior studies.

So, any initiative in "to improve the use of IDN TLDs in the real world" should get Universities onboard and encourage Universities/Schools to teach "IT Internationalization"

André
http://schappo.blogspot.co.uk

On 4 Mar 2014, at 12:14, Richard Ishida wrote:

I was contacted last week by Don Hollander, General Manager of the Asia Pacific Top Level Domain Association, who is trying to improve the use of IDN TLDs in the real world, and looking for support.

See the attached PDF (from him) outlining what are the barriers to the effective use of IDN TLDs and who can help address these issues.

He's hoping to create a community of interested stakeholders. He expects this community to include ICANN, many ccTLDs, ISOC, and hopefully commercial developers. He is also looking to set up some opportunities to meet and discuss how to move things forward.

If you are interested in getting involved, please raise your voice.

Don says "There is a HUGE population with interest in this - but it is not really the current 2Billion, but the next 2 Billion - those who aren’t yet connected."

RI
<Addressing the issue of Universal Acceptance of IDN TLDs-1.pdf>

马馬骉驫马馬骉驫马馬骉驫马馬骉驫
http://twitter.com/andreschappo
http://schappo.blogspot.co.uk
http://weibo.com/andreschappo
http://blog.sina.com.cn/andreschappo

Subject: RE: Universal Acceptance of IDN TLDs
From: "Larry Masinter" </O=EXCHANGELABS/OU=EXCHANGE ADMINISTRATIVE GROUP (FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=C72AF1C9D1504700BD8BD7A2CD33F15D-MASINTER>
To: "Hollenbeck, Scott" <[email-masked]>
Date: Fri, 14 Mar 2014 09:21:04 -0800

maybe send this stuff out to the URI list?

From: Hollenbeck, Scott [mailto:[email-masked]] 
Sent: Friday, March 14, 2014 4:34 AM
To: Larry Masinter
Subject: RE: Universal Acceptance of IDN TLDs
 
Larry, my researcher also studied the geolocation attributes of the source IP addresses responsible for queries containing “%xx”, the networks from which those queries originated, and the most commonly encoded characters.
 
Geolocation: 33% from China, 21% from the USA, long tail distribution on the rest.
 
ASNs: More than 6,000 unique ASN sources.
 
Top 10 most common characters (in descending order): %20, %2A, %2C, %2F, %40, %7C, %5c, %24, %25, %29
 
Scott
 
From: Larry Masinter [mailto:[email-masked]] 
Sent: Wednesday, March 05, 2014 1:06 PM
To: Hollenbeck, Scott
Subject: FW: Universal Acceptance of IDN TLDs
 
What happens when DNS gets queries for %xx-encoded names?
How difficult would it be to deploy %xx-utf8 queries to the equivalent punycode?
Of the IDNs in use, how many would exceed the DNS length limit when encoded as %xx (3 bytes / UTF-8 byte)?

 
From: Larry Masinter 
Sent: Wednesday, March 05, 2014 2:40 PM
To: 'Mark Davis ☕'
Cc: Andre Schappo; www International; Don Hollander; [email-masked] <mailto:[email-masked]> 
Subject: RE: Universal Acceptance of IDN TLDs
 
The handling of %xx-encoded domain names in DNS servers would be a fallback for use in legacy systems that are not IDN-aware.
 
So the length limit argument doesn’t carry a lot of weight – it is strictly a transitional deployment enhancement for working around legacy components which extract domain names from URIs but rcan only process 7-bit URIs and not 8-bit IRIs.
 
You can deploy IDNs when all of  the applications you care about will work for the users you care about for the DNS names you want to use.
 
Components that handle IRIs directly and pull out domain names for future processing shouldn’t ever need the %xx encoding, although decoding it is also a good idea.

From: [email-masked] <mailto:[email-masked]>  [mailto:[email-masked]] On Behalf Of Mark Davis ?
Sent: Wednesday, March 05, 2014 2:16 PM
To: Larry Masinter
Cc: Andre Schappo; www International; Don Hollander; [email-masked] <mailto:[email-masked]> 
Subject: Re: Universal Acceptance of IDN TLDs
 
If you mean having the DNS system natively accept %xx for domain labels as well as Punycode, I suspect that that ship has long since sailed. (That was one of the options discussed, but was turned down because of the length limitations.)
 
If on the other hand, you mean that client software should accept %xx notation as well as straight Unicode and punycode, that is another story. That can be handled by a client-side mapping, permitted by either IDNA2008 or UTS46. (And I agree that it's a good idea.)
 
With that, I could type in my address bar any of:
1.	xn--idna--x-l6c.blogspot.com <http://xn--idna--x-l6c.blogspot.com> 
2.	IDNA-ȿ-x.blogspot.com <http://xn--idna--x-l6c.blogspot.com> 
3.	IDNA-%C8%BF-x.blogspot.com <http://BF-x.blogspot.com> 
4.	IDNA-Ȿ-X.BLOGSPOT.COM <http://xn--idna--x-lt7e.BLOGSPOT.COM> 
5.	IDNA-%E2%B1%BE-X.BLOGSPOT.COM <http://BE-X.BLOGSPOT.COM> 
And they'd all resolve to xn--idna--x-l6c.blogspot.com <http://xn--idna--x-l6c.blogspot.com> .
1.	I just checked on Chrome, and all of these work.
2.	Firefox is a bit odd: if I type in the #3, it fails; *but* it converts it in the address bar, so a subsequent enter goes to the right place. #4/#5 just fail.
3.	Don't know about other browsers.

Mark <https://google.com/+MarkDavis> 
 
— Il meglio è l’inimico del bene —
 
On Wed, Mar 5, 2014 at 2:32 PM, Larry Masinter <[email-masked] <mailto:[email-masked]> > wrote:
there’s a gap between IDN and URI in that IRI -> URI would prefer to use the %xx percent-hex URL encoding in general.
 
What would be preferable would be to insure that DNS requests for %xx encoded names is an acceptable alternative to punycode.

From: Andre Schappo [mailto:[email-masked] <mailto:[email-masked]> ] 
Sent: Tuesday, March 04, 2014 3:51 PM
To: www International
Cc: Don Hollander; [email-masked] <mailto:[email-masked]> 
Subject: Re: Universal Acceptance of IDN TLDs
 
① Is this document available online? I have looked round http://aptld.org but cannot find it.

② There are indeed barriers to the effective, real world use of IDNs. A fundamental problem is that IDNs, in general, are not properly catered for and not properly integrated into systems. One reason often quoted for treating IDNs differently is "Security". Well, I posit that any IDN security issues pale in comparison to the ubiquitous "… for further information please click here."

Here are some examples from Social Media:

Twitter

If the Unicode form is entered —

#test  http:// <http://北大.中国> 北大.中国

It is not recognised as a Domain Name & not displayed as clickable link

If the punycode form is entered —

#test http://xn--djry4l.xn--fiqs8s

It is now recognised as a Domain Name and displayed as a clickable link but displayed as punycode instead of Unicode

Sina Weibo

Same results —
#test# http:// <http://北大.中国> 北大.中国
#test# http://xn--djry4l.xn--fiqs8s

There is also the related issue of having to Percent Encode the Unicode pathname components of a URL.

③ In my experience, another fundamental problem is the lack of IT Internationalization teaching in Schools and Universities. Certainly in England, IT Internationalisation has not yet become an accepted part of the curriculum. We need to produce students that have an appreciation/understanding of IT Internationalisation in order to, amongst other goals, properly integrate IDNs into systems/apps/websites …etc…

For several years I have been teaching a module entitled "International Computing" which covers several aspects of IT i18n. One of the topics I cover is IDNs :) And I am keeping my students up to date with the idn new gTLDs as they are delegated to DNS Root :)

During my years teaching this module I have found few students (regardless of which country they come from) with even a basic appreciation of IT Internationalization because it is a topic that was never discussed/raised in their prior studies.

So, any initiative in "to improve the use of IDN TLDs in the real world" should get Universities onboard and encourage Universities/Schools to teach "IT Internationalization"

André
http://schappo.blogspot.co.uk
 
On 4 Mar 2014, at 12:14, Richard Ishida wrote:
 
I was contacted last week by Don Hollander, General Manager of the Asia Pacific Top Level Domain Association, who is trying to improve the use of IDN TLDs in the real world, and looking for support.

See the attached PDF (from him) outlining what are the barriers to the effective use of IDN TLDs and who can help address these issues.

He's hoping to create a community of interested stakeholders. He expects this community to include ICANN, many ccTLDs, ISOC, and hopefully commercial developers. He is also looking to set up some opportunities to meet and discuss how to move things forward.

If you are interested in getting involved, please raise your voice.

Don says "There is a HUGE population with interest in this - but it is not really the current 2Billion, but the next 2 Billion - those who aren’t yet connected."

RI
<Addressing the issue of Universal Acceptance of IDN TLDs-1.pdf>
 
马馬骉驫马馬骉驫马馬骉驫马馬骉驫
http://twitter.com/andreschappo
http://schappo.blogspot.co.uk
http://weibo.com/andreschappo
http://blog.sina.com.cn/andreschappo

Subject: RE: Universal Acceptance of IDN TLDs
From: "Hollenbeck, Scott" <[email-masked]>
To: "Larry Masinter" <[email-masked]>
Date: Fri, 14 Mar 2014 04:34:11 -0800

Larry, my researcher also studied the geolocation attributes of the source IP addresses responsible for queries containing “%xx”, the networks from which those queries originated, and the most commonly encoded characters.

Geolocation: 33% from China, 21% from the USA, long tail distribution on the rest.

ASNs: More than 6,000 unique ASN sources.

Top 10 most common characters (in descending order): %20, %2A, %2C, %2F, %40, %7C, %5c, %24, %25, %29

Scott

From: Larry Masinter [mailto:[email-masked]] 
Sent: Wednesday, March 05, 2014 1:06 PM
To: Hollenbeck, Scott
Subject: FW: Universal Acceptance of IDN TLDs

What happens when DNS gets queries for %xx-encoded names?

How difficult would it be to deploy %xx-utf8 queries to the equivalent punycode?

Of the IDNs in use, how many would exceed the DNS length limit when encoded as %xx (3 bytes / UTF-8 byte)?

 

From: Larry Masinter 
Sent: Wednesday, March 05, 2014 2:40 PM
To: 'Mark Davis ☕'
Cc: Andre Schappo; www International; Don Hollander;  <mailto:[email-masked]> [email-masked]
Subject: RE: Universal Acceptance of IDN TLDs

The handling of %xx-encoded domain names in DNS servers would be a fallback for use in legacy systems that are not IDN-aware.

So the length limit argument doesn’t carry a lot of weight – it is strictly a transitional deployment enhancement for working around legacy components which extract domain names from URIs but rcan only process 7-bit URIs and not 8-bit IRIs.

You can deploy IDNs when all of  the applications you care about will work for the users you care about for the DNS names you want to use.

Components that handle IRIs directly and pull out domain names for future processing shouldn’t ever need the %xx encoding, although decoding it is also a good idea.

From: [email-masked] <mailto:[email-masked]>  [mailto:[email-masked]] On Behalf Of Mark Davis ?
Sent: Wednesday, March 05, 2014 2:16 PM
To: Larry Masinter
Cc: Andre Schappo; www International; Don Hollander;  <mailto:[email-masked]> [email-masked]
Subject: Re: Universal Acceptance of IDN TLDs

If you mean having the DNS system natively accept %xx for domain labels as well as Punycode, I suspect that that ship has long since sailed. (That was one of the options discussed, but was turned down because of the length limitations.)

If on the other hand, you mean that client software should accept %xx notation as well as straight Unicode and punycode, that is another story. That can be handled by a client-side mapping, permitted by either IDNA2008 or UTS46. (And I agree that it's a good idea.)

With that, I could type in my address bar any of:

1.	xn--idna--x-l6c.blogspot.com <http://xn--idna--x-l6c.blogspot.com> 
2.	IDNA-ȿ-x.blogspot.com <http://xn--idna--x-l6c.blogspot.com> 
3.	IDNA-%C8%BF-x.blogspot.com <http://BF-x.blogspot.com> 
4.	IDNA-Ȿ-X.BLOGSPOT.COM <http://xn--idna--x-lt7e.BLOGSPOT.COM> 
5.	IDNA-%E2%B1%BE-X.BLOGSPOT.COM <http://BE-X.BLOGSPOT.COM> 

And they'd all resolve to  <http://xn--idna--x-l6c.blogspot.com> xn--idna--x-l6c.blogspot.com.

1.	I just checked on Chrome, and all of these work.
2.	Firefox is a bit odd: if I type in the #3, it fails; *but* it converts it in the address bar, so a subsequent enter goes to the right place. #4/#5 just fail.
3.	Don't know about other browsers.

Mark <https://google.com/+MarkDavis> 

— Il meglio è l’inimico del bene —

On Wed, Mar 5, 2014 at 2:32 PM, Larry Masinter <[email-masked] <mailto:[email-masked]> > wrote:

there’s a gap between IDN and URI in that IRI -> URI would prefer to use the %xx percent-hex URL encoding in general.

What would be preferable would be to insure that DNS requests for %xx encoded names is an acceptable alternative to punycode.

From: Andre Schappo [mailto:[email-masked] <mailto:[email-masked]> ] 
Sent: Tuesday, March 04, 2014 3:51 PM
To: www International
Cc: Don Hollander; [email-masked] <mailto:[email-masked]> 
Subject: Re: Universal Acceptance of IDN TLDs

① Is this document available online? I have looked round http://aptld.org but cannot find it.

② There are indeed barriers to the effective, real world use of IDNs. A fundamental problem is that IDNs, in general, are not properly catered for and not properly integrated into systems. One reason often quoted for treating IDNs differently is "Security". Well, I posit that any IDN security issues pale in comparison to the ubiquitous "… for further information please click here."

Here are some examples from Social Media:

Twitter

If the Unicode form is entered —

#test  http:// <http://北大.中国> 北大.中国

It is not recognised as a Domain Name & not displayed as clickable link

If the punycode form is entered —

#test http://xn--djry4l.xn--fiqs8s

It is now recognised as a Domain Name and displayed as a clickable link but displayed as punycode instead of Unicode

Sina Weibo

Same results —
#test# http:// <http://北大.中国> 北大.中国
#test# http://xn--djry4l.xn--fiqs8s

There is also the related issue of having to Percent Encode the Unicode pathname components of a URL.

③ In my experience, another fundamental problem is the lack of IT Internationalization teaching in Schools and Universities. Certainly in England, IT Internationalisation has not yet become an accepted part of the curriculum. We need to produce students that have an appreciation/understanding of IT Internationalisation in order to, amongst other goals, properly integrate IDNs into systems/apps/websites …etc…

For several years I have been teaching a module entitled "International Computing" which covers several aspects of IT i18n. One of the topics I cover is IDNs :) And I am keeping my students up to date with the idn new gTLDs as they are delegated to DNS Root :)

During my years teaching this module I have found few students (regardless of which country they come from) with even a basic appreciation of IT Internationalization because it is a topic that was never discussed/raised in their prior studies.

So, any initiative in "to improve the use of IDN TLDs in the real world" should get Universities onboard and encourage Universities/Schools to teach "IT Internationalization"

André
http://schappo.blogspot.co.uk

On 4 Mar 2014, at 12:14, Richard Ishida wrote:

I was contacted last week by Don Hollander, General Manager of the Asia Pacific Top Level Domain Association, who is trying to improve the use of IDN TLDs in the real world, and looking for support.

See the attached PDF (from him) outlining what are the barriers to the effective use of IDN TLDs and who can help address these issues.

He's hoping to create a community of interested stakeholders. He expects this community to include ICANN, many ccTLDs, ISOC, and hopefully commercial developers. He is also looking to set up some opportunities to meet and discuss how to move things forward.

If you are interested in getting involved, please raise your voice.

Don says "There is a HUGE population with interest in this - but it is not really the current 2Billion, but the next 2 Billion - those who aren’t yet connected."

RI
<Addressing the issue of Universal Acceptance of IDN TLDs-1.pdf>

马馬骉驫马馬骉驫马馬骉驫马馬骉驫
http://twitter.com/andreschappo
http://schappo.blogspot.co.uk
http://weibo.com/andreschappo
http://blog.sina.com.cn/andreschappo

Subject: RE: Universal Acceptance of IDN TLDs
From: "Hollenbeck, Scott" <[email-masked]>
To: "Larry Masinter" <[email-masked]>
Date: Thu, 13 Mar 2014 16:39:47 -0800

OK, more research done: I just did some testing with all of the xn-- domain names registered in .com. Unfortunately I can’t provide absolute numbers without getting clearance from my Legal department. What I can say is that approximately 14.5% of the registered names would produce “%xx” labels longer than 63 characters. Here’s what I did using PHP:

Started with a list of all xn-- names.

Stripped the .com characters from the end of the name.

Decoded the resulting a-labels using a punycode decoder, producing u-labels.

Used the PHP urlencode() function to convert Unicode characters to %xx characters.

Measured the length of the resulting character string and incremented counters appropriately.

Scott

From: Larry Masinter [mailto:[email-masked]] 
Sent: Wednesday, March 05, 2014 1:06 PM
To: Hollenbeck, Scott
Subject: FW: Universal Acceptance of IDN TLDs

What happens when DNS gets queries for %xx-encoded names?

How difficult would it be to deploy %xx-utf8 queries to the equivalent punycode?

Of the IDNs in use, how many would exceed the DNS length limit when encoded as %xx (3 bytes / UTF-8 byte)?

 

From: Larry Masinter 
Sent: Wednesday, March 05, 2014 2:40 PM
To: 'Mark Davis ☕'
Cc: Andre Schappo; www International; Don Hollander;  <mailto:[email-masked]> [email-masked]
Subject: RE: Universal Acceptance of IDN TLDs

The handling of %xx-encoded domain names in DNS servers would be a fallback for use in legacy systems that are not IDN-aware.

So the length limit argument doesn’t carry a lot of weight – it is strictly a transitional deployment enhancement for working around legacy components which extract domain names from URIs but rcan only process 7-bit URIs and not 8-bit IRIs.

You can deploy IDNs when all of  the applications you care about will work for the users you care about for the DNS names you want to use.

Components that handle IRIs directly and pull out domain names for future processing shouldn’t ever need the %xx encoding, although decoding it is also a good idea.

From: [email-masked] <mailto:[email-masked]>  [mailto:[email-masked]] On Behalf Of Mark Davis ?
Sent: Wednesday, March 05, 2014 2:16 PM
To: Larry Masinter
Cc: Andre Schappo; www International; Don Hollander;  <mailto:[email-masked]> [email-masked]
Subject: Re: Universal Acceptance of IDN TLDs

If you mean having the DNS system natively accept %xx for domain labels as well as Punycode, I suspect that that ship has long since sailed. (That was one of the options discussed, but was turned down because of the length limitations.)

If on the other hand, you mean that client software should accept %xx notation as well as straight Unicode and punycode, that is another story. That can be handled by a client-side mapping, permitted by either IDNA2008 or UTS46. (And I agree that it's a good idea.)

With that, I could type in my address bar any of:

1.	xn--idna--x-l6c.blogspot.com <http://xn--idna--x-l6c.blogspot.com> 
2.	IDNA-ȿ-x.blogspot.com <http://xn--idna--x-l6c.blogspot.com> 
3.	IDNA-%C8%BF-x.blogspot.com <http://BF-x.blogspot.com> 
4.	IDNA-Ȿ-X.BLOGSPOT.COM <http://xn--idna--x-lt7e.BLOGSPOT.COM> 
5.	IDNA-%E2%B1%BE-X.BLOGSPOT.COM <http://BE-X.BLOGSPOT.COM> 

And they'd all resolve to  <http://xn--idna--x-l6c.blogspot.com> xn--idna--x-l6c.blogspot.com.

1.	I just checked on Chrome, and all of these work.
2.	Firefox is a bit odd: if I type in the #3, it fails; *but* it converts it in the address bar, so a subsequent enter goes to the right place. #4/#5 just fail.
3.	Don't know about other browsers.

Mark <https://google.com/+MarkDavis> 

— Il meglio è l’inimico del bene —

On Wed, Mar 5, 2014 at 2:32 PM, Larry Masinter <[email-masked] <mailto:[email-masked]> > wrote:

there’s a gap between IDN and URI in that IRI -> URI would prefer to use the %xx percent-hex URL encoding in general.

What would be preferable would be to insure that DNS requests for %xx encoded names is an acceptable alternative to punycode.

From: Andre Schappo [mailto:[email-masked] <mailto:[email-masked]> ] 
Sent: Tuesday, March 04, 2014 3:51 PM
To: www International
Cc: Don Hollander; [email-masked] <mailto:[email-masked]> 
Subject: Re: Universal Acceptance of IDN TLDs

① Is this document available online? I have looked round http://aptld.org but cannot find it.

② There are indeed barriers to the effective, real world use of IDNs. A fundamental problem is that IDNs, in general, are not properly catered for and not properly integrated into systems. One reason often quoted for treating IDNs differently is "Security". Well, I posit that any IDN security issues pale in comparison to the ubiquitous "… for further information please click here."

Here are some examples from Social Media:

Twitter

If the Unicode form is entered —

#test  http:// <http://北大.中国> 北大.中国

It is not recognised as a Domain Name & not displayed as clickable link

If the punycode form is entered —

#test http://xn--djry4l.xn--fiqs8s

It is now recognised as a Domain Name and displayed as a clickable link but displayed as punycode instead of Unicode

Sina Weibo

Same results —
#test# http:// <http://北大.中国> 北大.中国
#test# http://xn--djry4l.xn--fiqs8s

There is also the related issue of having to Percent Encode the Unicode pathname components of a URL.

③ In my experience, another fundamental problem is the lack of IT Internationalization teaching in Schools and Universities. Certainly in England, IT Internationalisation has not yet become an accepted part of the curriculum. We need to produce students that have an appreciation/understanding of IT Internationalisation in order to, amongst other goals, properly integrate IDNs into systems/apps/websites …etc…

For several years I have been teaching a module entitled "International Computing" which covers several aspects of IT i18n. One of the topics I cover is IDNs :) And I am keeping my students up to date with the idn new gTLDs as they are delegated to DNS Root :)

During my years teaching this module I have found few students (regardless of which country they come from) with even a basic appreciation of IT Internationalization because it is a topic that was never discussed/raised in their prior studies.

So, any initiative in "to improve the use of IDN TLDs in the real world" should get Universities onboard and encourage Universities/Schools to teach "IT Internationalization"

André
http://schappo.blogspot.co.uk

On 4 Mar 2014, at 12:14, Richard Ishida wrote:

I was contacted last week by Don Hollander, General Manager of the Asia Pacific Top Level Domain Association, who is trying to improve the use of IDN TLDs in the real world, and looking for support.

See the attached PDF (from him) outlining what are the barriers to the effective use of IDN TLDs and who can help address these issues.

He's hoping to create a community of interested stakeholders. He expects this community to include ICANN, many ccTLDs, ISOC, and hopefully commercial developers. He is also looking to set up some opportunities to meet and discuss how to move things forward.

If you are interested in getting involved, please raise your voice.

Don says "There is a HUGE population with interest in this - but it is not really the current 2Billion, but the next 2 Billion - those who aren’t yet connected."

RI
<Addressing the issue of Universal Acceptance of IDN TLDs-1.pdf>

马馬骉驫马馬骉驫马馬骉驫马馬骉驫
http://twitter.com/andreschappo
http://schappo.blogspot.co.uk
http://weibo.com/andreschappo
http://blog.sina.com.cn/andreschappo

Subject: RE: Universal Acceptance of IDN TLDs
From: "Hollenbeck, Scott" <[email-masked]>
To: "Larry Masinter" <[email-masked]>
Date: Thu, 13 Mar 2014 06:10:23 -0800

Larry,

Right now we return rcode 3 when we receive a query for %xx-encoded names. I had one of my engineers do a little research into the DNS queries we received for com/net domain names that contain a “%” character in the qname. He looked at queries received from January 1, 2014 through March 4, 2014. The average number of unique qnames was 279,566 with a high of 337,498 and a low of 221,130. With repeated queries included we saw an average of 1,574,793 queries per day with a high of 1,817,813 and a low of 1,199,300.

I’m still looking into question #3. The answer to question #2 is probably academic because it would be impossible for us to deploy anything that isn’t part of the standards identified in our contracts with ICANN.

Scott

From: Larry Masinter [mailto:[email-masked]] 
Sent: Wednesday, March 05, 2014 1:06 PM
To: Hollenbeck, Scott
Subject: FW: Universal Acceptance of IDN TLDs

What happens when DNS gets queries for %xx-encoded names?

How difficult would it be to deploy %xx-utf8 queries to the equivalent punycode?

Of the IDNs in use, how many would exceed the DNS length limit when encoded as %xx (3 bytes / UTF-8 byte)?

 

From: Larry Masinter 
Sent: Wednesday, March 05, 2014 2:40 PM
To: 'Mark Davis ☕'
Cc: Andre Schappo; www International; Don Hollander;  <mailto:[email-masked]> [email-masked]
Subject: RE: Universal Acceptance of IDN TLDs

The handling of %xx-encoded domain names in DNS servers would be a fallback for use in legacy systems that are not IDN-aware.

So the length limit argument doesn’t carry a lot of weight – it is strictly a transitional deployment enhancement for working around legacy components which extract domain names from URIs but rcan only process 7-bit URIs and not 8-bit IRIs.

You can deploy IDNs when all of  the applications you care about will work for the users you care about for the DNS names you want to use.

Components that handle IRIs directly and pull out domain names for future processing shouldn’t ever need the %xx encoding, although decoding it is also a good idea.

From:  <mailto:[email-masked]> [email-masked] [ <mailto:[email-masked]> mailto:[email-masked]] On Behalf Of Mark Davis ?
Sent: Wednesday, March 05, 2014 2:16 PM
To: Larry Masinter
Cc: Andre Schappo; www International; Don Hollander;  <mailto:[email-masked]> [email-masked]
Subject: Re: Universal Acceptance of IDN TLDs

If you mean having the DNS system natively accept %xx for domain labels as well as Punycode, I suspect that that ship has long since sailed. (That was one of the options discussed, but was turned down because of the length limitations.)

If on the other hand, you mean that client software should accept %xx notation as well as straight Unicode and punycode, that is another story. That can be handled by a client-side mapping, permitted by either IDNA2008 or UTS46. (And I agree that it's a good idea.)

With that, I could type in my address bar any of:

1.	xn--idna--x-l6c.blogspot.com <http://xn--idna--x-l6c.blogspot.com> 
2.	IDNA-ȿ-x.blogspot.com <http://xn--idna--x-l6c.blogspot.com> 
3.	IDNA-%C8%BF-x.blogspot.com <http://BF-x.blogspot.com> 
4.	IDNA-Ȿ-X.BLOGSPOT.COM <http://xn--idna--x-lt7e.BLOGSPOT.COM> 
5.	IDNA-%E2%B1%BE-X.BLOGSPOT.COM <http://BE-X.BLOGSPOT.COM> 

And they'd all resolve to  <http://xn--idna--x-l6c.blogspot.com> xn--idna--x-l6c.blogspot.com.

1.	I just checked on Chrome, and all of these work.
2.	Firefox is a bit odd: if I type in the #3, it fails; *but* it converts it in the address bar, so a subsequent enter goes to the right place. #4/#5 just fail.
3.	Don't know about other browsers.

Mark <https://google.com/+MarkDavis> 

— Il meglio è l’inimico del bene —

On Wed, Mar 5, 2014 at 2:32 PM, Larry Masinter <[email-masked] <mailto:[email-masked]> > wrote:

there’s a gap between IDN and URI in that IRI -> URI would prefer to use the %xx percent-hex URL encoding in general.

What would be preferable would be to insure that DNS requests for %xx encoded names is an acceptable alternative to punycode.

From: Andre Schappo [mailto: <mailto:[email-masked]> [email-masked]] 
Sent: Tuesday, March 04, 2014 3:51 PM
To: www International
Cc: Don Hollander;  <mailto:[email-masked]> [email-masked]
Subject: Re: Universal Acceptance of IDN TLDs

① Is this document available online? I have looked round http://aptld.org but cannot find it.

② There are indeed barriers to the effective, real world use of IDNs. A fundamental problem is that IDNs, in general, are not properly catered for and not properly integrated into systems. One reason often quoted for treating IDNs differently is "Security". Well, I posit that any IDN security issues pale in comparison to the ubiquitous "… for further information please click here."

Here are some examples from Social Media:

Twitter

If the Unicode form is entered —

#test  http:// <http://北大.中国> 北大.中国

It is not recognised as a Domain Name & not displayed as clickable link

If the punycode form is entered —

#test http://xn--djry4l.xn--fiqs8s

It is now recognised as a Domain Name and displayed as a clickable link but displayed as punycode instead of Unicode

Sina Weibo

Same results —
#test# http:// <http://北大.中国> 北大.中国
#test# http://xn--djry4l.xn--fiqs8s

There is also the related issue of having to Percent Encode the Unicode pathname components of a URL.

③ In my experience, another fundamental problem is the lack of IT Internationalization teaching in Schools and Universities. Certainly in England, IT Internationalisation has not yet become an accepted part of the curriculum. We need to produce students that have an appreciation/understanding of IT Internationalisation in order to, amongst other goals, properly integrate IDNs into systems/apps/websites …etc…

For several years I have been teaching a module entitled "International Computing" which covers several aspects of IT i18n. One of the topics I cover is IDNs :) And I am keeping my students up to date with the idn new gTLDs as they are delegated to DNS Root :)

During my years teaching this module I have found few students (regardless of which country they come from) with even a basic appreciation of IT Internationalization because it is a topic that was never discussed/raised in their prior studies.

So, any initiative in "to improve the use of IDN TLDs in the real world" should get Universities onboard and encourage Universities/Schools to teach "IT Internationalization"

André
http://schappo.blogspot.co.uk

On 4 Mar 2014, at 12:14, Richard Ishida wrote:

I was contacted last week by Don Hollander, General Manager of the Asia Pacific Top Level Domain Association, who is trying to improve the use of IDN TLDs in the real world, and looking for support.

See the attached PDF (from him) outlining what are the barriers to the effective use of IDN TLDs and who can help address these issues.

He's hoping to create a community of interested stakeholders. He expects this community to include ICANN, many ccTLDs, ISOC, and hopefully commercial developers. He is also looking to set up some opportunities to meet and discuss how to move things forward.

If you are interested in getting involved, please raise your voice.

Don says "There is a HUGE population with interest in this - but it is not really the current 2Billion, but the next 2 Billion - those who aren’t yet connected."

RI
<Addressing the issue of Universal Acceptance of IDN TLDs-1.pdf>

马馬骉驫马馬骉驫马馬骉驫马馬骉驫
 <http://twitter.com/andreschappo> http://twitter.com/andreschappo
 <http://schappo.blogspot.co.uk> http://schappo.blogspot.co.uk
 <http://weibo.com/andreschappo> http://weibo.com/andreschappo
 <http://blog.sina.com.cn/andreschappo> http://blog.sina.com.cn/andreschappo

Subject: FW: Universal Acceptance of IDN TLDs
From: "Larry Masinter" </O=EXCHANGELABS/OU=EXCHANGE ADMINISTRATIVE GROUP (FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=C72AF1C9D1504700BD8BD7A2CD33F15D-MASINTER>
To: "Hollenbeck, Scott" <[email-masked]>
Date: Wed, 5 Mar 2014 10:05:58 -0800

What happens when DNS gets queries for %xx-encoded names?
How difficult would it be to deploy %xx-utf8 queries to the equivalent punycode?
Of the IDNs in use, how many would exceed the DNS length limit when encoded as %xx (3 bytes / UTF-8 byte)?

 
From: Larry Masinter 
Sent: Wednesday, March 05, 2014 2:40 PM
To: 'Mark Davis ☕'
Cc: Andre Schappo; www International; Don Hollander; [email-masked]
Subject: RE: Universal Acceptance of IDN TLDs
 
The handling of %xx-encoded domain names in DNS servers would be a fallback for use in legacy systems that are not IDN-aware.
 
So the length limit argument doesn’t carry a lot of weight – it is strictly a transitional deployment enhancement for working around legacy components which extract domain names from URIs but rcan only process 7-bit URIs and not 8-bit IRIs.
 
You can deploy IDNs when all of  the applications you care about will work for the users you care about for the DNS names you want to use.
 
Components that handle IRIs directly and pull out domain names for future processing shouldn’t ever need the %xx encoding, although decoding it is also a good idea.

From: [email-masked] <mailto:[email-masked]>  [mailto:[email-masked]] On Behalf Of Mark Davis ?
Sent: Wednesday, March 05, 2014 2:16 PM
To: Larry Masinter
Cc: Andre Schappo; www International; Don Hollander; [email-masked] <mailto:[email-masked]> 
Subject: Re: Universal Acceptance of IDN TLDs
 
If you mean having the DNS system natively accept %xx for domain labels as well as Punycode, I suspect that that ship has long since sailed. (That was one of the options discussed, but was turned down because of the length limitations.)
 
If on the other hand, you mean that client software should accept %xx notation as well as straight Unicode and punycode, that is another story. That can be handled by a client-side mapping, permitted by either IDNA2008 or UTS46. (And I agree that it's a good idea.)
 
With that, I could type in my address bar any of:
1.	xn--idna--x-l6c.blogspot.com <http://xn--idna--x-l6c.blogspot.com> 
2.	IDNA-ȿ-x.blogspot.com <http://xn--idna--x-l6c.blogspot.com> 
3.	IDNA-%C8%BF-x.blogspot.com <http://BF-x.blogspot.com> 
4.	IDNA-Ȿ-X.BLOGSPOT.COM <http://xn--idna--x-lt7e.BLOGSPOT.COM> 
5.	IDNA-%E2%B1%BE-X.BLOGSPOT.COM <http://BE-X.BLOGSPOT.COM> 
And they'd all resolve to xn--idna--x-l6c.blogspot.com <http://xn--idna--x-l6c.blogspot.com> .
1.	I just checked on Chrome, and all of these work.
2.	Firefox is a bit odd: if I type in the #3, it fails; *but* it converts it in the address bar, so a subsequent enter goes to the right place. #4/#5 just fail.
3.	Don't know about other browsers.

Mark <https://google.com/+MarkDavis> 
 
— Il meglio è l’inimico del bene —
 
On Wed, Mar 5, 2014 at 2:32 PM, Larry Masinter <[email-masked] <mailto:[email-masked]> > wrote:
there’s a gap between IDN and URI in that IRI -> URI would prefer to use the %xx percent-hex URL encoding in general.
 
What would be preferable would be to insure that DNS requests for %xx encoded names is an acceptable alternative to punycode.

From: Andre Schappo [mailto:[email-masked] <mailto:[email-masked]> ] 
Sent: Tuesday, March 04, 2014 3:51 PM
To: www International
Cc: Don Hollander; [email-masked] <mailto:[email-masked]> 
Subject: Re: Universal Acceptance of IDN TLDs
 
① Is this document available online? I have looked round http://aptld.org but cannot find it.

② There are indeed barriers to the effective, real world use of IDNs. A fundamental problem is that IDNs, in general, are not properly catered for and not properly integrated into systems. One reason often quoted for treating IDNs differently is "Security". Well, I posit that any IDN security issues pale in comparison to the ubiquitous "… for further information please click here."

Here are some examples from Social Media:

Twitter

If the Unicode form is entered —

#test  http:// <http://北大.中国> 北大.中国

It is not recognised as a Domain Name & not displayed as clickable link

If the punycode form is entered —

#test http://xn--djry4l.xn--fiqs8s

It is now recognised as a Domain Name and displayed as a clickable link but displayed as punycode instead of Unicode

Sina Weibo

Same results —
#test# http:// <http://北大.中国> 北大.中国
#test# http://xn--djry4l.xn--fiqs8s

There is also the related issue of having to Percent Encode the Unicode pathname components of a URL.

③ In my experience, another fundamental problem is the lack of IT Internationalization teaching in Schools and Universities. Certainly in England, IT Internationalisation has not yet become an accepted part of the curriculum. We need to produce students that have an appreciation/understanding of IT Internationalisation in order to, amongst other goals, properly integrate IDNs into systems/apps/websites …etc…

For several years I have been teaching a module entitled "International Computing" which covers several aspects of IT i18n. One of the topics I cover is IDNs :) And I am keeping my students up to date with the idn new gTLDs as they are delegated to DNS Root :)

During my years teaching this module I have found few students (regardless of which country they come from) with even a basic appreciation of IT Internationalization because it is a topic that was never discussed/raised in their prior studies.

So, any initiative in "to improve the use of IDN TLDs in the real world" should get Universities onboard and encourage Universities/Schools to teach "IT Internationalization"

André
http://schappo.blogspot.co.uk
 
On 4 Mar 2014, at 12:14, Richard Ishida wrote:
 
I was contacted last week by Don Hollander, General Manager of the Asia Pacific Top Level Domain Association, who is trying to improve the use of IDN TLDs in the real world, and looking for support.

See the attached PDF (from him) outlining what are the barriers to the effective use of IDN TLDs and who can help address these issues.

He's hoping to create a community of interested stakeholders. He expects this community to include ICANN, many ccTLDs, ISOC, and hopefully commercial developers. He is also looking to set up some opportunities to meet and discuss how to move things forward.

If you are interested in getting involved, please raise your voice.

Don says "There is a HUGE population with interest in this - but it is not really the current 2Billion, but the next 2 Billion - those who aren’t yet connected."

RI
<Addressing the issue of Universal Acceptance of IDN TLDs-1.pdf>
 
马馬骉驫马馬骉驫马馬骉驫马馬骉驫
http://twitter.com/andreschappo
http://schappo.blogspot.co.uk
http://weibo.com/andreschappo
http://blog.sina.com.cn/andreschappo

---

From: Nico Williams <[email-masked]>
Date: Mon, 4 Feb 2019 14:51:50 -0600

> Let me be a little more careful. The document 

Thanks.

> https://chromium.googlesource.com/chromium/src/+/master/docs/security/url_display_guidelines/url_display_guidelines.md

| In general, the web platform does not have limits on the length of
| URLs (although 2^31 is a common limit). Chrome limits URLs to a
| maximum length of 2MB for practical reasons and to avoid causing
| denial-of-service problems in inter-process communication.

Ha.  I've seen stacks that fail to handle more than 4KB or 8KB request
header sizes including the start-line.

| On most platforms, Chrome’s omnibox limits URL display to 32kB
| (kMaxURLDisplayChars) although a 1kB limit is used on VR platforms.

I'm not sure how I could understand a 32KB URI in a status bar.  Or even
1KB.  And we're not talking about my parents here.

That document seems reasonable otherwise :)

> seems to be primarily about one use case:  The user has somehow
> navigated to a site (through links, typing in, search, whatever), and

I think we really have just these cases:

 - the user followed a link
 - the user pasted a link
 - the user typed something in exactly
 - the user typed something in and selected a partial match from a
   search engine

The specific case matters because if the user types it in, the UA must
normalize and so on, but if it's followed then...

If the user followed or pasted a link, then that needs to be sanitized
to some degree, and if it's malicious, then simply fail to follow --
don't just follow and attempt to indicate the maliciousness through the
status bar.

If the user picked an incremental result while typing... we should treat
that a lot like following a link (even if we trust the search engine).

Following a link in a page of search results is still following a link,
so search in any context other than typing in the status bar is not
special.

> is now looking at a web page; what should the UA show (in the address
> bar or any other security context) to validate “who is this coming
> from? Which site sent me this data? It says it’s my bank, did it come
> from them?”

The original sin here is the username:password@ optional prefix to the
authority component of the URI: it confuses many users (probably most).

So we should start by removing the username:password from the status /
origin bar!  (The Chromium doc says as much.)

If the URI was typed in, prep it as usual and use it.

If a link was followed, no prep is needed, but perhaps we should apply
it anyways since it's idempotent, applying ToUnicode() and ToASCII()
again to normalize.

The linked doc's proposal for how to handle bi-di scripts and "sneaky"
codepoints and characters is a good start.

It would be useful to know what scripts the user can read, that way the
UA can be a bit more dramatic about URIs whose authorities contain
characters from scripts foreign to the user.

Zalgo detection would be nice :)

> The spec is only about the visual display and doesn’t cover copy/paste
> or save-as-bookmark or other operations which work with a different
> form.

Copy/paste seems no different that following a link -- the UA won't know
the origin of the link, but we must not assume the user typed it in
somewhere.

> *	normalization doesn’t change the visual display (?)

Normalization indeed does not change the display.  It cannot.  That's
part of the definition of Unicode equivalence forms.

(I suppose there may be implementations where form artifacts leak to the
rendering, but that would be a bug.)

> then at worst the advice to normalize is harmless. So I take back my

It is harmless to normalize again and again.  Normalization is
idempotent.

> The original point, though, stands: the “Conservatism” principle
> doesn’t go far enough in warning those registering domains to avoid
> those labels which downstream processors will reject or display
> poorly.

I mean, a registrant can test how browsers would handle their domains.

But browsers can change how they handle the registrant's domains...

> Is it possible to get any statistics on DNS requests that include
> unnormalized strings? And some hits about where they come from?

A-labels that decode into non-canonical Unicode are busted.  I have no
numbers about them.

Nico
--

---

From: Emily Stark <[email-masked]>
Date: Mon, 4 Feb 2019 15:58:16 -0800

Hi,

I might be missing some context here (thread might be cut off?) -- is there
a suggested update to the URL display guidelines? Displaying punycode for
unnormalized domains sounds user-unfriendly to me but I feel like I'm not
understanding the suggestion.

Thanks,
Emily