Subject: [I18n-discuss] Comments on "troublesome-characters" from Arabic script

From: Abdulaziz Al-Zoman <[email-masked]>
Date: Mon, 24 Jul 2017 04:41:34 +0000

Dear Sir/Mam (i18n-discuss)

Please find my comments concerning the draft-freytag-troublesome-characters-01.txt: “Those Troublesome Characters: A Registry of Unicode Code Points Needing Special Consideration When Used in Network Identifiers”

As the aim of this Internet draft was to create a registry of code points that need special consideration when used as identifiers so that it would guide system administrators in setting parameters for allowable code points in an  identifier system, and to aid applications in creating security aids for users. I have substantial concern on what code points should be part of this registry and hence how the recipients (software developer, registries, registrars, etc.) interpret the inclusion of a code point in the repository.

For example, the registry includes some essential characters (letters) that may result at the end useless identifiers if these characters are restricted or blocked (because they are part of the repository).   For instance, with respect to the Arabic language, the registry consists of a large portion of the Arabic basic alphabet that may result to a limited character set for creating identifiers (e.g., the Arabic language consists of 28 essential alphabet characters, while the repository suggests that out of these 28 characters there are 22 troublesome-characters; i.e. more than 78% of the Arabic language characters can't be used if they are blocked).  This will cause impractical use of the Arabic language in network identifier. While the actual "troublesome-characters" are the non-spacing marks.

Therefore,  we would suggest that the registry includes only the problematic code points such as non-spacing marks but not the basic characters (they can be in the comment field or the reason for including such a non-spacing mark). As adding basic characters to the repository  may lead to preventing their use in identifiers while the origin of the problem is due to the misuse of non-spacing marks. Thus, the repository should cover only the non-spacing marks indicating their risk-free usage, and their harmful usage which should be blocked.

Sincerely  Yours,
Abdulaziz Al-Zoman

From: [email-masked] [mailto:[email-masked]] On Behalf Of Sarmad Hussain
Sent: 4/Jul/2017 12:15 PM
To: TF-AIDN
Subject: [TF-AIDN] "troublesome-characters" from Arabic script

Dear All,

A new version of draft-freytag-troublesome-characters-01.txt has been posted to the IETF repository. This standards track internet-draft aims to address the IAB Statement<https://www.iab.org/documents/correspondence-reports-documents/2015-2/iab-statement-on-identifiers-and-unicode-7-0-0/> and more.

The draft lists many code points from the Arabic script, which may need the review of the task force.  You can send comments to [email-masked]<mailto:[email-masked]>.

Name:                  draft-freytag-troublesome-characters

Revision:              01

Title:                     Those Troublesome Characters: A Registry of Unicode Code Points Needing Special Consideration When Used in Network Identifiers

Document date:               2017-06-30

Group:                  Individual Submission

Pages:                  42

URL: https://www.ietf.org/internet-drafts/draft-freytag-troublesome-characters-01.txt<URL:https://www.ietf.org/internet-drafts/draft-freytag-troublesome-characters-01.txt>

Status:https://datatracker.ietf.org/doc/draft-freytag-troublesome-characters/

Htmlized:https://tools.ietf.org/html/draft-freytag-troublesome-characters-01

Htmlized:https://datatracker.ietf.org/doc/html/draft-freytag-troublesome-characters-01

Diff:https://www.ietf.org/rfcdiff?url2=draft-freytag-troublesome-characters-01

Abstract:

    Unicode's design goal is to be the universal character set for all

    applications.  The goal entails the inclusion of very large numbers

    of characters.  It is also focused on written language; special

    provisions have always been needed for identifiers.  The sheer size

    of the repertoire increases the possibility of accidental or

    intentional use of characters that can cause confusion among users,

    particularly where linguistic context is ambiguous, unavailable, or

    impossible to determine.  A registry of code points that can be

    sometimes especially problematic may be useful to guide system

    administrators in setting parameters for allowable code points in an

    identifier system, and to aid applications in creating security aids

    for users.

regards,
Sarmad

---

From: Raed Al-Fayez <[email-masked]>
Date: Tue, 25 Jul 2017 08:20:28 +0000

Dear Authors of " draft-freytag-troublesome-characters-01.txt",

I would like to share with you my comments on the internet draft: draft-freytag-troublesome-characters-01.txt, titled: “Those Troublesome Characters: A Registry of Unicode Code Points Needing Special Consideration When Used in Network Identifiers”. My comments are drawn up based on my knowledge of the Arabic Script and from the experiences accumulated and gained through my heavy involvements in creating and running an IDN ccTLD and through my active participation as a member of the TF-AIDN (Task Force on Arabic Script Internationalized Domain Names<https://community.icann.org/display/MES/Task+Force+on+Arabic+Script+IDNs>), the ASIWG (Arabic Script IDN Working Group), and the Arabic Domain Name Team.

I regret to inform you that I completely opposing of including essential code points of the Arabic language (which are also used by many other languages in the Arabic script) to the repository tables of this standards track internet-draft. As a reader of this (later-to-be) RFC may consider them as "Troublesome Characters" while they are not!

The majority of the problematic cases listed in the internet-draft's table (part of the Arabic code points) were due to the misuses of non-spacing marks. Also, most of the cases are not used by any language in the Arabic script and some of the others do not make any sense. Therefore, it is not wise and not practical to risk essential code points because of not solid cases.

Finally, with respect to code points that have some similarities (e.g. 0629 and 06C3), I would encourage you to adopt the variant concepts that was done in IDNs to solve the identifiers issue (for example IANA's Repository of IDN Practices & RFC 7940).

With best regards,

Raed Alfayez
______________
Director of SaudiNIC, CITC
Tel: +966-11-4618216  -  Fax: +966-11-4619497
www.nic.sa<http://www.nic.sa>

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 27 Jul 2017 17:13:20 -0400

Greetings, and thank you both for the comments on the draft.

I am keen to understand something in what you are both saying, because
it suggests to me that our intent in this draft is not coming across clearly.

On Mon, Jul 24, 2017 at 04:41:34AM +0000, Abdulaziz Al-Zoman wrote:

> For example, the registry includes some essential characters
> (letters) that may result at the end useless identifiers if these
> characters are restricted or blocked (because they are part of the
> repository).   For instance, with respect to the Arabic language,
> the registry consists of a large portion of the Arabic basic
> alphabet that may result to a limited character set for creating
> identifiers

On Tue, Jul 25, 2017 at 08:20:28AM +0000, Raed Al-Fayez wrote:
 
> I regret to inform you that I completely opposing of including
> essential code points of the Arabic language (which are also used by
> many other languages in the Arabic script) to the repository tables
> of this standards track internet-draft. As a reader of this
> (later-to-be) RFC may consider them as "Troublesome Characters"
> while they are not!
> 
> The majority of the problematic cases listed in the internet-draft's
> table (part of the Arabic code points) were due to the misuses of
> non-spacing marks. Also, most of the cases are not used by any
> language in the Arabic script and some of the others do not make any
> sense. Therefore, it is not wise and not practical to risk essential
> code points because of not solid cases.
> 

Do you both agree that the characters to which you are referring can
properly be used for identifies only by people who actually understand
how they relate to each other, and not without substantial care and
understanding on the part of whoever is permitting the use or
registering the identifier?  If not, do you think instead that the
characters can be used without any policies at all?

The aim of the draft is to create the conditions for guidance for
operators and applications, so that it is possible for (for instance)
my user agent to work globally, even when some parts of the global
linguistic environment is foreign to me and when there are no
in-protocol clues about the language I'm facing.  (Examples of this
sort of identifier are things like DNS names, mail names, XMPP
chatroot names, and so on.  Web pages and the like have the ability to
negotiate language, which makes the problem somewhat less acute.)  One
thing we can do to make those conditions a little bit safer, I think,
is to have evidence that whoever is in charge of the registration
permissions actually has some policy or set of rules; and so my user
agent could check to see that, even if I can't read the identifier
reliably, the person responsible for its creation has followed some
set of rules that prevents serious confusion from arising.  In other
words, the registry is not intended to "block" all of the included
code points, but rather to indicate that these are code points for
which it is even more important than usual that extra care has been
taken.

I am a little worried that the above does not seem to be the
conclusion you drew from the draft, which suggests to me that we have
not made ourselves clear enough.  Is that a fair assessment, or are
you opposed to the inclusion of these code points instead because you
do not think they warrant special attention?

Thanks and best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Abdulaziz H. Al-Zoman" <[email-masked]>
Date: Sun, 30 Jul 2017 08:37:52 +0000

Good day Andrew,

Thanks for your follow-up email to my feedback.

I do understand the goal of the internet draft as you've stated:
     "to create the conditions for guidance
      for operators and applications, so that
      it is possible for (for instance) my
      user agent to work globally, even when
      some parts of the global linguistic
      environment is foreign to me and when
      there are no in-protocol clues about
      the language I'm facing."

but I'm worried by the inclusion of "basic" code points (22 out of 28 Arabic language alphabet) to the repository instead of only the problematic code points (non-spacing marks). This would make the Arabic language unusable (or troublesome) to operators and developers. While, I would like to see some encouragement to support Arabic IDNs rather than shun them away from it.

Therefore,  I would suggest that the registry includes only the problematic sequence of code points that incorporate non-spacing marks but not the basic (and essential) characters. 

So I would suggest that the repository table looks like the following table (for example) where: Column 1 represent the problematic (sequence of) code points and Column 2 contains the Reasons and Comments. The Column 1 should NOT contain a basic code point alone without non-spacing mark causing the problem.

Column 1             | Column 2
---------------------+------------------
062F, 065C           | Identical in appearance to ...
---------------------+------------------
062F, 06EC           | Identical in appearance to ...
---------------------+------------------
0631, 06EC           | Identical in appearance to ...
---------------------+------------------
0633, 06DB           | Identical in appearance to ...

Yours,
Abdulaziz Al-Zoman

> -----Original Message-----
> From: Andrew Sullivan [mailto:[email-masked]]
> Sent: 28/Jul/2017 12:13 AM
> To: Abdulaziz H. Al-Zoman; Raed Al-Fayez
> Cc: [email-masked]'
> Subject: Re: [I18n-discuss] Comments on "troublesome-characters" from
> Arabic script
> 
> I am keen to understand something in what you are both saying, because it
> suggests to me that our intent in this draft is not coming across clearly.
> 
> > the registry consists of a large portion of the Arabic basic alphabet
> > that may result to a limited character set for creating identifiers
> 
> > many other languages in the Arabic script) to the repository tables of
> > this standards track internet-draft. As a reader of this
> properly be used for identifies only by people who actually understand how
> they relate to each other, and not without substantial care and
> understanding on the part of whoever is permitting the use or registering
> the identifier?  If not, do you think instead that the characters can be
> used without any policies at all?
> 
> operators and applications, so that it is possible for (for instance) my
> user agent to work globally, even when some parts of the global linguistic
> environment is foreign to me and when there are no in-protocol clues about
> the language I'm facing.  (Examples of this sort of identifier are things
> like DNS names, mail names, XMPP chatroot names, and so on.  Web pages and
> the like have the ability to negotiate language, which makes the problem
> somewhat less acute.)  One thing we can do to make those conditions a
> little bit safer, I think, is to have evidence that whoever is in charge
> of the registration permissions actually has some policy or set of rules;
> and so my user agent could check to see that, even if I can't read the
> identifier reliably, the person responsible for its creation has followed
> some set of rules that prevents serious confusion from arising.  In other
> words, the registry is not intended to "block" all of the included code
> points, but rather to indicate that these are code points for which it is
> even more important than usual that extra care has been taken.
> 
> I am a little worried that the above does not seem to be the conclusion
> you drew from the draft, which suggests to me that we have not made
> ourselves clear enough.  Is that a fair assessment, or are you opposed to
> the inclusion of these code points instead because you do not think they
> warrant special attention?
> 
> Thanks and best regards,
> 
> A
> 
> --
> Andrew Sullivan
> [email-masked]

---

From: John C Klensin <[email-masked]>
Date: Sun, 30 Jul 2017 11:46:57 -0400

Andrew, Dr. Al-Zoman,

Let me suggest what may be a middle ground (a variation of one
I've already suggested off-list to Andrew and Asmus) to see if
it works for you.   I think the key here is an expansion of
things Andrew has already said, which is that this particular
draft should not (in IETF-speak, MUST NOT) be interpreted out of
context with other documents.  Partially for that reason, it may
need additional text to explain those connections.

FWYW, my hypothesis is that Arabic script is not the only one
that would be adversely affected if "troublesome" were
interpreted as "bad character" or "don't allow this".  We were
just lucky that you and your colleagues spotted the problems
early, mentioned it, and were able to clearly explain the issue.

Would you agree with the following:

(1) The use of combining characters with Arabic script is
problematic, to the point of requiring special attention and
knowledge whether they should be prohibited or not.  AFAICT,
combining characters are not allowed by the analysis and
recommendations of RFC 5564 and, by adding more precombined
forms, Unicode has decided to not encourage their use.

(2) Arabic script as defined by Unicode is unique in having two
sets of code points for digits within the same script.  The
observation that European digits are often used with the Arabic
language in some places further complicates that problem.

(3) At least for the part of the world's population that does
not use them every day, scripts that are normally written with
the characters connected (a category that is certainly not
limited to Arabic) are more of a problem than scripts written
with more obvious character boundaries.  This does not reflect
badly on those scripts in any way, but it does make it harder
for those who are ignorant about the scripts to look characters
up in tables, transcribe written forms into computer input, etc.

(4) Again, at least for those who do not use them every day,
scripts that are normally written right to left, especially
those that also use numerals that are written left to right
(i.e., with the highest-valued digit on the left), are more
difficult than left to right ones.  While this is not a
significant problem with the scripts themselves, it can, unless
care is exercise by people who understand what is going on,
result in strange behavior in some circumstances.
Fully-qualified domain names where some labels start or end with
digits have proven to be particularly difficult examples. 

Now, if we agree on that much, I suggest that the document
should probably say the following (in different words) and say
it very clearly and strongly:

(i) Arabic script (because of the first two characteristics
above) and _any_ script with either or both of the other two
characteristics make the requirements that a registry understand
the script(s) it allows, that it carefully design a plan for
working with each of those scripts, and that it consider itself
accountable for the results, even more important than those
requirements usually are.  Code points that are, or are not,
listed by this document may be useful as input to such registry
understanding and policy development efforts, but are not a
substitute for them.  Again, those principles apply to all
registries and all scripts, but the ones identified by (1)-(4)
above require extra attention (so, by the way, do registries
allowing more than one of Greek, Latin, or Cyrillic scripts and
several other combinations -- perhaps the document should
include a section of "troublesome" combinations of scripts).

(ii) The rule of numeral homogeneity (RFC 5564, Section 2.3.1),
expanded to also prohibit mixing with "Eastern" Arabic-Indic
digits, is far more important and useful than trying to call out
any of those digit code points as problematic.  

(iii) Combining marks are extremely problematic with Arabic
script.  If combining marks are prohibited by the registry, then
code points that are listed in the table as problematic because
of possible combining sequences become non-problematic.  While,
if I understand RFC 5564 and your comments correctly, simply
prohibiting all of them is reasonable for the Arabic language, I
note that the latest version of the ASIWG recommendations that I
can find (June 2008,
http://arabic-domains.org/docs/jun2010/IDNA200x_Arabic_Script_Disallowed_Codepoints.pdf)
[1] allows a large number of combining characters.

(iv) If recommendations for domain name use (or identifier use
more generally) have been developed by groups reflecting a broad
range of expertise, including experts on the relevant
linguistics and writing systems (not just, e.g.,
name-marketing), for a particular script or language and those
recommendations are very specific about what they cover and are
applied carefully, they are most likely to be a better guide to
what is appropriate and/or troublesome than a list, including
this "troublesome" one, developed for general use.  For the
Arabic language written in Arabic script and a domain registry
that does not allow labels based on other languages written in
Arabic script, that implies that RFC 5564 is a better guide than
this document, no matter what this one says.  By contrast, if
one had a registry that was exclusively registering labels based
on, e.g., Persian writing conventions, this document (perhaps in
combination with the generic recommendations for Arabic script
in the ASIWG work) would be more appropriate than trying to
apply 5564 (if specific and carefully-developed recommendations
for Persian existed, that would, of course, be better yet).  

Given an explanation of that type, the vast majority of the
Arabic code points could either be removed from the Internet
Draft or turned into cross-references to the explanation or more
authoritative documents.

Would that be a reasonable direction to take?

best regards,
    john

[1] Were the ASIWG recommendations ever completed and formally
approved by ESCWA and the relevant countries?   The table
mentioned above is the latest I can find but it is dated even
earlier than the 2009 meeting I attended, and the asiwg.org site
to which several references point appears to now be in Chinese
script.  An up-to-date reference, if it exists, would be
appreciated.

--On Sunday, July 30, 2017 08:37 +0000 "Abdulaziz H. Al-Zoman"
<[email-masked]> wrote:

> I do understand the goal of the internet draft as you've
> stated:      "to create the conditions for guidance
> but I'm worried by the inclusion of "basic" code points (22
> out of 28 Arabic language alphabet) to the repository instead
> of only the problematic code points (non-spacing marks). This
> would make the Arabic language unusable (or troublesome) to
> operators and developers. While, I would like to see some
> encouragement to support Arabic IDNs rather than shun them
> away from it.
> 
> Therefore,  I would suggest that the registry includes only
> the problematic sequence of code points that incorporate
> non-spacing marks but not the basic (and essential)
> characters. 
> 
> So I would suggest that the repository table looks like the
> following table (for example) where: Column 1 represent the
> problematic (sequence of) code points and Column 2 contains
> the Reasons and Comments. The Column 1 should NOT contain a
> basic code point alone without non-spacing mark causing the
> problem.
> 
> Column 1             | Column 2