Subject: [I18n-discuss] comments on draft-freytag-troublesome-characters-01

From: "Marc Blanchet" <[email-masked]>
Date: Sun, 09 Jul 2017 17:02:12 -0400

Hello,
  comments embedded with <MB>. only relevant text is shown.

Marc.

======

Intended status: Standards Track

<MB>Given that this document is proposing an « informal » registry 
of code points and is not normative as in protocol, I wonder if 
Standards Track is the best intended status. I would suggest either 
Informational or Best Current Practice.
</MB>

Those Troublesome Characters: A Registry of Unicode Code Points Needing
          Special Consideration When Used in Network Identifiers
                 draft-freytag-troublesome-characters-01

1.  Unicode code points and identifiers

    Identifiers that are used in a network or, especially, an Internet
    context present several special problems because of the above 
feature
    of Unicode:

    1.  In many (perhaps most) uses of identifiers, it is either
…

    4.  For some scripts the repertoire of shapes is shared, so that
        there are cases of two strings in which all the code points in
        one script in the first string, and all the code points in
        another script in the second string, are respectively confusable
        with one another.  In that case, the strings cannot be
        distinguished by a reader, and the whole string is confusable.

<MB>this paragraph above makes sense but was somewhat not clear at first 
reading, at least to me. I wonder if some punctuation or added precision 
would help. A suggestion:
For some scripts, each of their repertoire of shapes is shared. 
Therefore, a string in one script may be visually identical to another 
string in another script, while the code points are different.
</MB>

    5.  For some scripts, both users and rendering systems do not expect
        to encounter code points in arbitrary sequence.  Most code 
points
        normally occur only in specific locations within a syllable.  If
        random labels were permitted, some would not display as expected
        (including having some features misplaced or not displayed) 
while
        others would present recognition problems to users experienced
        with the script.  Some devices may also not support arbitrary
        input.

<MB>It seems to me that there should be a specific point on the 
rendering systems in this list, along the lines that: - rendering 
systems are updated at a different pace than code points, - use 
different font mechanisms - are sometimes substituting fonts or shares 
when they don’t have the exact code point match, etc…
</MB>

    Operators of systems that create identifiers (whether through a
    registry or through a peer-to-peer identifier negotiation system)
    need to make policies for characters they will permit.  Operators of
    registries, for instance, can help by adopting good registration
    policies: "Users will benefit if registries only permit characters
    from scripts that are well-understood by the registry or its
    advisers."[RFC5894]

<MB>Well, that idea has always been in my mind wishful thinking. Good 
that we are now saying this is not working. However, the words below to 
me are just not enough strong. But I guess that does not matter too 
much. I would have said something along the lines of « The Unicode 
repertoire, or a subset of it, is just too large to be carefully looked 
at for an operator of an identifier system, therefore, these operators 
rely only on third party tables. »
</MB>

The difficulty for many operators, however, is
    that they do not have the writing system expertise to claim any
    character is "well-understood", and they do not really have the time
    to develop that expertise.  Such operators should in fact not use or
    register such characters.  Unfortunately, in many cases the 
operators
    are stewards of systems where the user population demands 
identifiers
    useful to them in their local languages.  In other cases, operators
    may proceed without a proper understanding owing to financial or
    market share incentives.  The risk for Internet identifiers in such
    cases is obviously that ill-understood and potentially exploitable
    gaps in registration policies will open. To help mitigate such

<MB>I think a new paragraph is needed here</MB>

    issues, a registry of Unicode code points that present special 
issues
    for network identifiers can help guide protocol and operating
    decisions about whether to permit a given code point or sequence of
    code points.  This will not completely protect against poor
    registration or use, but it may provide operational guidance
    necessary for people who are responsible for creating policies.

    Note that the registry defined herein does not address any of the
    issues created by whole-string confusables where each of the
    identifiers is of a different script.  A common workaround, limiting
    a registry to identifiers of only a single script, would mitigate
    this issue.

    For some of the code points (or code point sequences listed hat

<MB>s/hat/that</MB>

    present issues for identifiers, it may be most expeditious to simply
    not include them, even though they are valid according to the
    protocol.  Sometimes, one of a pair of identical code points (or 
code
    point sequences) may be deemed preferable over the other for
    practical reasons.

    In the case of registries, it is not always necessary or desirable 
to
    exclude characters.  Sometimes, it is merely necessary to ensure 
that
    for two otherwise identical identifiers, only one of a set of

<MB>I guess here it meant : otherwise _visually_ identical 
identifiers?</MB>

    mutually exclusive characters (or sequences of characters) is used,
    while preventing the later registration of the the label containing

<MB>s/the the/the/<MB>

    the other one in order to avoid ambiguity.  This way the operator
    does not need to make a choice.  In certain cases, where both of
    these identifiers mean the same thing,

<MB>might need to define what « mean the same thing » mean… maybe 
« has the same language meaning »?</MB>

  an operator may decide to
    allow both labels to be registered simultaneously, but only to the
    same entity.

    In every case, the registry here defined includes code points that
    require special attention when they are to be used in identifiers.

<MB>well, the reader has not seen the registry, so it is confusing. I 
would suggest:

s/In every case, the registry here defined…/This specification defines 
a registry of code points that
require special attention when used in identifiers/
</MB>

    An administrator who does not have the time or inclination to 
develop
    the requisite understanding would be well-advised simply not to
    permit these code points at all.

4.  A registry of code points

4.1.  Discussion

    The registry contains three fields.  The first field, called "Code
    Point(s)", is a code point or sequence of code points.  The second,
    "Cross Reference", contains zero or more cross references to related
    code points.  The third, called "Explanation", is a free form text
    field that briefly describes the issue.  The explanation field also
    contains one or more references to documents defining the code point
    and the reason why it presents an issue.  These reference may be to
    documents external to the registry, so long as the reference is
    stable.

<MB>I would suggest to put the references in a separate column. easier 
to follow through, to parse, to read, to update, …</MB>

    The registry is not intended as an alternative to normal operational
    policies that are used for protocols under normal administrative
    scope.  For instance, zone operators that support IDNA are expected
    to create policies governing the code points that they will permit
    (see [RFC5894] and [I-D.rfc5891bis]).  The registry herein defined 
is
    intended to highlight particularly troublesome code points or code
    point sequences for the benefit of administrators creating such
    policies.  It is also intended to highlight characters that may
    create identifier ambiguities and thereby create security
    vulnerabilities.

<MB> the table seems to imply a class of issues as first token in the 
explanation column: i.e. Identical, Restricted Context, Not Recommended, 
… I would suggest to put them in a separate column for readability and 
sorting purposes (the IANA XML registries can be sorted by any column), 
labelling the column « Rationale »  or « Class » or  
« Type » …

    If a character appears in the registry, that does not automatically
    mean that it is a bad candidate for use in identifiers generally.
    Absent a well-defined and verifiable policy, however, such a code
    point or sequence might well be treated with suspicion by users and
    by tools.

    The registry is updated by Expert Review.

<MB> »Expert Review », as is, I disagree. As we all know and 
discussed since the early days of idn wg, sometimes a single code point 
may raise a lot of different opinions and perspectives. Therefore, a 
single expert to me is not enough to approve changes to this registry. I 
would suggest something in the form of the mime process, where the 
proposal is sent to a specific mailing list for discussion and maybe a 
reviewer (or two) will inform IANA of the conclusion.  Or any public 
process that includes more than one person isolated talking to IANA.
</MB>

It ought to contain only
    code points that are significant in identifiers and that need 
special
    policies (including policies of exclusion).  Only code points that
    are eligible for use in identifiers (i.e. that are not DISALLOWED)
    ought to be included.  Code points that are CONTEXTJ or CONTEXTO
    ought to only be included if concerns are identified that are not
    mitigated by the existing IDNA context rules.

<MB>I think this policy is not enough precise for the Expert Review (or 
whatever process) to identify which one should be included or not. 
Therefore, I would highly suggest that the authors define a set of 
criteria to help the review process
</MB>

<MB>I wonder if there should be some words about managing the changes in 
this table. Specially, if later, things are found that make an entry not 
« backward » compatible with an earlier version of the table. I 
think assuming that the table will only add stuff may be risky… And I 
think we can assume that this registry will be parsed by software 
programs in registries, registrars, etc… Therefore, some guidelines 
for both the expert review or whatever process and for the 
implementor/reader about non-compatible changes is necessary.
</MB>

4.2.  Registry initial contents

4.2.1.  Code Point Table

<MB>This table should be parsed by software. I don’t care about the 
RFC itself,  but I care about precision. In the code point and « cross 
reference » columns, there are list of code points. It should be clear 
to IANA and the reader which are sequences of code points that should 
belong together, which are separate list of code points/sequences. 
Therefore, at least an explanation of the format is needed and also 
instructions to IANA to make sure these are properly set in the XML 
registry so they can be parsed automatically.

Some additional information might be appropriate, may be as a glossary 
in the beginning of the document. I know this document says that the 
reader should be aware of Unicode, IDNA, etc… But as a good writing 
style, I think it would be good to have a glossary that identifies where 
the UPPERCASE keywords come from. I would see a glossary such as:

DISALLOWED, CONTEXTJ, CONTEXTO, PVALID, … : defined in IDNA [RFC…]
LETTER … : are text descriptions of code points from Unicode []
…
</MB>

<MB>I, for myself and likely for others, would love to get a machine 
readable format of this table so I can do my own validation/parsing/QA 
of this table before it goes published. I wonder if for the duration of 
the discussion, a github file of the table itself could be referenced in 
the draft and an above note to the RFC editor to delete the pointer at 
the time of publication
</MB>

    +----------+-----------+--------------------------------------------+
    | Code     | Cross     | Explanation                                
|
    | Point or | Reference |                                            
|
    | Sequence |           |                                            
|
    +----------+-----------+--------------------------------------------+
    | 0307     |           | Restricted Context: By definition, LATIN   
|
    |          |           | SMALL LETTER I plus combining DOT ABOVE    
|
    |          |           | renders exactly the same as LATIN SMALL    
|
    |          |           | LETTER I by itself and does so in practice 
|
    |          |           | for any good font. The same is true for    
|
    |          |           | all Unicode characters with the            
|
    |          |           | soft_dotted property; they lose their dot  
|
    |          |           | if followed by a combining mark. DOT ABOVE 
|
    |          |           | should be excluded, or restricted to       
|
    |          |           | contexts where it does not follow a        
|

…

    |          |           | mark, problematic for identifiers [120]    
|
    | 10486    | 104A0     | Identical: Identical in appearance U+104A0 
|
    |          |           | OSMANYA DEEL [115] [150]                   
|
    | 104A0    | 10486     | Identical: Identical in appearance to      
|
    |          |           | U+10486 OSMANYA DIGIT ZERO [115] [150]     
|
    +----------+-----------+--------------------------------------------+

         Table 1: Registry of Unicode Code Points Requiring Special
                    Consideration in Network Identifiers

4.2.2.  References for Registry

    [99]  The Unicode Consortium, "The Unicode Standard", (latest
       version) http:www.unicode.org/versions/latest (Multiple, or 
latest
       version)

Appendix A.  Additional Background

A.1.  The Theory of Inclusion

    The mechanism that the IETF has come to prefer for
    internationalization of identifiers may be called "inclusion-based
    identifier internationalization", or "inclusion" for short.  Under

    Inclusion depends on the assumption that such strings as will be 
used

<MB> « as will »  ? </MB>

    in identifiers will not have any ambiguous matching to other 
strings.
    In practice, this means that input strings to the protocol are
    expected to be in Normalization Form C.

<MB>as you know, precis and precis profiles is doing NFKC. So it might 
be appropriate to stick or assume NFC here</MB>

   This way, any alternative
    sequences of code points for the same characters will be normalized
    to a single form.  If all the characters in the string are also
    included for the protocol's candidate identifiers, then the string 
is
    eligible to be an identifier under the protocol.

…

    Also, operators should only ever use the smallest repertoire of code
    points possible for their environment.  So, for example, if there is
    a code point that is sometimes used but is perhaps a little obscure,
    it is better to leave it out and gain some experience with other
    cases first.  In particular, code points used only in a language 
with
    which the administrator is not familiar should probably be excluded.
    The same applies to code points used in specialized contexts, such 
as
    those only found in historic or sacred documents, or only used for
    phonetic transcription or poetry.

<MB>It seems that this discussion should reference the Integrated Issues 
Report</MB>

  In the case of IDNA, some client
    programs restrict display of U-labels to top-level domains known to
    have policies about single-script labels.

    None of these policies or convention, other than ensuring mutual
    exclusion, will do anything to help strict homoglyphs of each other
    in the same script (see Appendix B for some example cases.)

    Finally, there are some writing systems where characters do not
    normally occur in arbitrary locations in the context of each
    syllable.  Neither users nor rendering systems for such scripts are
    adept at handling arbitrary sequences of such characters.  While 
some
    latitude beyond strict spelling rules may be accommodated, policies
    that enforce a minimal set of structural rules are required to 
ensure
    that users can identify the identifier and systems can render them
    predictably.

A.2.2.  Not everything can be solved

    As noted in Section 1, it is not possible to solve all the problems
    with identifier systems, particularly when human factors are taken
    into account.

Appendix B.  Examples

    There are a number of cases that illustrate the combining sequence 
or
    digraph issue:
…

---

From: Asmus Freytag <[email-masked]>
Date: Sun, 9 Jul 2017 22:42:48 -0700

> comments embedded with <MB>. only relevant text is shown.
Thanks for the insightful comments. Have added replies with <AF><AF>. 
Where possible I've already updated my private copy with new language 
and am showing it here.

A./

> <MB>Given that this document is proposing an « informal » registry of 
> code points and is not normative as in protocol, I wonder if Standards 
> Track is the best intended status. I would suggest either 
> 1.
>     Unicode code points and identifiers
>
>     context present several special problems because of the above feature
>     of Unicode:
>
>      1. In many (perhaps most) uses of identifiers, it is either …
>         another script in the second string, are respectively
>         confusable with one another. In that case, the strings cannot
>         be distinguished by a reader, and the whole string is confusable.
>
> <MB>this paragraph above makes sense but was somewhat not clear at 
> first reading, at least to me. I wonder if some punctuation or added 
> precision would help. A suggestion:
<AF> -- Does this accommodate your concern?

     <t>For some scripts their repertoire of shapes overlaps with one
     or more other scripts, so that there are cases where two strings
     look identical to each other, even though all the code points in
     the first string are of one script, and all the code points in
     the second string  are of another script.  In these cases, the 
strings cannot be
     distinguished by a reader, and the whole strings are confusable.</t>

</AF>
>     to encounter code points in arbitrary sequence. Most code points
>     (including having some features misplaced or not displayed) while
>     with the script. Some devices may also not support arbitrary input.
>
<AF>
The point of these list items is to set up the reader for the various 
reasons that specific characters, specific sequences (or, in point 5, 
also generic patterns) are individually "troublesome".

While it is true that rendering engines (and fonts) in general may take 
liberties with rendering any identifier, that general problem cannot be 
addressed by taking precaution based on specific characters (e.g. those 
from the registry we are trying to set up).

If you meant this as a generic issue, it should perhaps go into the "not 
all problems can be solved" section of the text. If I misunderstood what 
you were aiming at, please set me straight.
</AF>
> that we are now saying this is not working. However, the words below 
> to me are just not enough strong. But I guess that does not matter too 
> repertoire, or a subset of it, is just too large to be carefully 
> looked at for an operator of an identifier system, therefore, these 
> operators rely only on third party tables. »
> </MB>
>

<AF>
The first level would be for registries to be aware that "anything goes" 
will lead to specific, identifiable problems. The registry will help 
identify what those problems are and give some suggestions on how to 
avoid them.

We are clearly taking the (implied) position that it's not workable for 
everyone to do original research in this arena. However, everyone will 
need to set policy. "Let's stay away from troublesome characters" is a 
good defensive move if you don't have access to expertise that would 
allow you to implement (and administer) something more finely tailored.

Don't know how to express that best. Below is some added wording,
</AF>
> register such characters. Unfortunately, in many cases the operators
> are stewards of systems where the user population demands identifiers
<AF> break the paragraph here, check.</AF>
>
> issues, a registry of Unicode code points that present special issues
<AF> -- perhaps modify last sentence and continue as suggested here--
      By itself, such a registry will not completely
       protect against poor registration or use, but it may provide
       operational guidance necessary for people who are responsible
       for creating policies.  It also obviates the need for everyone to
     repeat basic investigation into the behavior of Unicode characters.
     Instead, scarce expertise can be focused on ways to mitigate
     issues, perhaps caused by user requirements for a specific character.<
</AF>
> For some of the code points (or code point sequences listed hat
<AF>check</AF>
> protocol. Sometimes, one of a pair of identical code points (or code
> In the case of registries, it is not always necessary or desirable to
> exclude characters. Sometimes, it is merely necessary to ensure that
> <MB>I guess here it meant : otherwise /visually/ identical 
> identifiers?</MB>
>
<AF>Actually, no. Literally the same code point. Case is something like 
two strings,
one uses reverse E the other uses SCHWA. Both are identical in all other 
code points.

Let me know if I can fix the text to help make that clear</AF>
> while preventing the later registration of the the label containing
<AF> check </AF>
> « has the same language meaning »?</MB>
<AF>I see no difference between "meaning" and "language meaning", other 
than that the latter doesn't sound terribly good in English. Perhaps we 
can write: "mean the same thing to the native reader"</AF>
> An administrator who does not have the time or inclination to develop
<AF>Currently, the data resides in a separate database and what's 
included here is just a report from that database. Therefore, such 
changes can be easily implemented. For the actual registry, we might 
want to use a format other than a table in an RFC. I tried several 
different ways. If others concur, I can add a column for the 
references.</AF>
> (see [RFC5894] and [I-D.rfc5891bis]). The registry herein defined is
> explanation column: i.e. Identical, Restricted Context, Not 
> Recommended, … I would suggest to put them in a separate column for 
> readability and sorting purposes (the IANA XML registries can be 
> sorted by any column), labelling the column « Rationale » or « Class » 
> or  « Type » …
>
<AF>This category token is currently autogenerated by the database 
report based on other fields. Again, changing the table format at this 
point means no more or less than a few lines of report generating code, 
so I can try out some in the next draft and we can make further changes 
or revert as needed.</AF>
> <MB> »Expert Review », as is, I disagree. As we all know and discussed 
> since the early days of idn wg, sometimes a single code point may 
> raise a lot of different opinions and perspectives. Therefore, a 
> single expert to me is not enough to approve changes to this registry. 
> I would suggest something in the form of the mime process, where the 
<AF>
I agree that we are not there yet in defining the principles that should 
be applied in updating this registry. What criteria do we expect this 
registry to embody? The sample initial collection tries to answer that 
implicitly, but we will still need to discuss this some more and come to 
an explicit description of "what kind of trouble" gets a character 
listed here.

I do think that we should require that ALL entries carry a reference, 
and strongly encourage the use of external references as a means to 
ensure that the registry is not a collection of personal prejudice, but 
an attempt at collecting the shared experience and expertise of a wider 
community of experts.

(For external references that are not stable, or do not have a stable 
URL, an internal reference that cites (quotes) the relevant information 
from the original document may be acceptable).

In other words, structure the process to favor a mostly objective 
approach to the issue, so that the nature of the reviewers becomes less 
important.

Nevertheless, something like your suggestion, if workable, might be 
appropriate.
</AF>
>
> It ought to contain only
> code points that are significant in identifiers and that need special
> <MB>I think this policy is not enough precise for the Expert Review 
> (or whatever process) to identify which one should be included or not. 
<AF>totally agree - this is a TBD

One of the criteria ought to apply to newly encoded code points: it 
should explicitly suggest that if a new code point is "analogous" to an 
existing troublesome character in the way it is used, that it be 
considered as troublesome in its own right, whether or not anyone 
outside the reviewers has identified any issues with it in a form that 
can be referenced.
</AF>
>
> <MB>I wonder if there should be some words about managing the changes 
> in this table. Specially, if later, things are found that make an 
> entry not « backward » compatible with an earlier version of the 
> table. I think assuming that the table will only add stuff may be risky…
>
<AF>There's a difference between "Troublesome" and "DISALLOWED". If 
there is a code point that isn't labeled "troublesome" here, it does not 
mean that it is white-listed. It just means that there's no issues been 
identified with it. Assuming a new Unicode code point creates a new 
confusable pair of identical appearance with an existing Unicode 
character. There would be every reason for the existing code point to be 
listed as troublesome from that point on forward. Registries may decide 
that the new code point should be the one to be excluded, which would 
keep their policies backward compatible. Or they could decide to make 
these blocked variants of each other. This still maintains the registry, 
because before the new code point was included, no label with it could 
be registered, now that it is available, users may find some possible 
labels already blocked. That's normal first-come-first-serve.

A more difficult issue arises is some other issue is newly found with 
some code point. Registries that do not include that code point already 
have no issues excluding that code point. Other registries may need to 
evaluate whether to disallow future registrations. But from the point of 
the "troublesome character registry", it's not clear that it benefits 
from any rules that would prevent making known any issues, no matter how 
late they arise.
</AF>
>
> And I think we can assume that this registry will be parsed by 
> software programs in registries, registrars, etc… Therefore, some 
> guidelines for both the expert review or whatever process and for the 
<AF>I think that's reasonable. I'll need to go over the draft text and 
put some placeholders for all that, as I can't think of suitable draft 
language right now.</AF>
> registry so they can be parsed automatically.
<AF>No disagreement here. As I mentioned, the underlying data is not 
kept in this RFC format, so it would be possible to (a) make some 
adjustments to the presentation in the RFC, and (b) export the data in 
any XML format suitable for IANA - the underlying data is specific on 
what are singleton code points and what are sequences.</AF>
> style, I think it would be good to have a glossary that identifies 
> where the UPPERCASE keywords come from. I would see a glossary such as:
>
> of this table before it goes published. I wonder if for the duration 
> of the discussion, a github file of the table itself could be 
> referenced in the draft and an above note to the RFC editor to delete 
> the pointer at the time of publication
>
<AF>Let's figure out the logistics. The underlying data is currently 
kept in an XML database that conforms to RFC 7940 -- I know that you and 
others have tools to read it.</AF>
> | Code | Cross | Explanation |
> | Point or | Reference | |
> | Sequence | | |
> | 0307 | | Restricted Context: By definition, LATIN |
> | | | SMALL LETTER I plus combining DOT ABOVE |
> | | | renders exactly the same as LATIN SMALL |
> | | | LETTER I by itself and does so in practice |
> | | | for any good font. The same is true for |
> | | | all Unicode characters with the |
> | | | soft_dotted property; they lose their dot |
> | | | if followed by a combining mark. DOT ABOVE |
> | | | should be excluded, or restricted to |
> | | | contexts where it does not follow a |
>
> …
>
> | | | mark, problematic for identifiers [120] |
> | 10486 | 104A0 | Identical: Identical in appearance U+104A0 |
> | | | OSMANYA DEEL [115] [150] |
> | 104A0 | 10486 | Identical: Identical in appearance to |
> | | | U+10486 OSMANYA DIGIT ZERO [115] [150] |
> |Table 1: Registry of Unicode Code Points Requiring Special 
> Consideration in Network Identifiers |
>
> version) http:www.unicode.org/versions/latest 
> <http://www.unicode.org/versions/latest> (Multiple, or latest
> version)
>
> Inclusion depends on the assumption that such strings as will be used
>
> <MB> « as will » ? </MB>
>
<AF>Not my phrasing, but as far as I can tell, possible in English; but 
what do I know?</AF>
>
> in identifiers will not have any ambiguous matching to other strings.
<AF>so no change?</AF>
> included for the protocol's candidate identifiers, then the string is
> cases first. In particular, code points used only in a language with
> The same applies to code points used in specialized contexts, such as
> <MB>It seems that this discussion should reference the Integrated 
> Issues Report</MB>
>
> adept at handling arbitrary sequences of such characters. While some
> that enforce a minimal set of structural rules are required to ensure
> There are a number of cases that illustrate the combining sequence or
> digraph issue:
> …
>

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 10 Jul 2017 12:53:08 -0400

Hi,

Thanks for the comments.

On Sun, Jul 09, 2017 at 05:02:12PM -0400, Marc Blanchet wrote:
> <MB>Given that this document is proposing an « informal » registry of code
> points and is not normative as in protocol, I wonder if Standards Track is
> the best intended status. I would suggest either Informational or Best
> Current Practice.
> </MB>

I think we concluded it had to be standards track to create an IANA
registry and create the rules for updating it.  What is an "informal"
registry?

> For some scripts, each of their repertoire of shapes is shared. Therefore, a
> string in one script may be visually identical to another string in another
> script, while the code points are different.
> </MB>

Thanks.  

> <MB>It seems to me that there should be a specific point on the rendering
> systems in this list, along the lines that: - rendering systems are updated
> at a different pace than code points, - use different font mechanisms - are
> sometimes substituting fonts or shares when they don’t have the exact code
> point match, etc…
> </MB>

Are you suggesting a little background section (maybe ¶ or 2) about
rendering systems and why they're problematic for this case in a way
that they usually aren't for IETF concerns?

> <MB> « The Unicode repertoire, or a subset
> of it, is just too large to be carefully looked at for an operator of an
> identifier system, therefore, these operators rely only on third party
> tables. »
> </MB>

Hmm, I like this.  Will try to do something with it.  Thanks.

> <MB>I guess here it meant : otherwise _visually_ identical identifiers?</MB>

I think we're trying to keep open the possibility that they are
somehow logically the same as well, but I'll keep a flag here.

> <MB>might need to define what « mean the same thing » mean… maybe « has the
> same language meaning »?</MB>

Unfortunately, that doesn't make the problem any better.  The basic
issue is that humans interpret labels like natural language, even
though they're not.  

> <MB>I would suggest to put the references in a separate column. easier to
> follow through, to parse, to read, to update, …</MB>

I don't understand: it is a separate column, no?

> explanation column: i.e. Identical, Restricted Context, Not Recommended, … I
> would suggest to put them in a separate column for readability and sorting
> purposes (the IANA XML registries can be sorted by any column), labelling
> the column « Rationale »  or « Class » or  « Type » …

This is an extra column, then?

> <MB> »Expert Review », as is, I disagree. As we all know and discussed since
> the early days of idn wg, sometimes a single code point may raise a lot of
> different opinions and perspectives. Therefore, a single expert to me is not
> enough to approve changes to this registry. I would suggest something in the
> form of the mime process, where the proposal is sent to a specific mailing
> list for discussion and maybe a reviewer (or two) will inform IANA of the
> conclusion.  Or any public process that includes more than one person
> isolated talking to IANA.
> </MB>

One of the things we're having trouble with in this document is
figuring out where the experts are going to come from.  The i18n
expertise around the IETF is pretty thin as it is.  Why do we think we
have extra people who will participate, beyond the appointed expert?

> Therefore, I would highly suggest that the authors define a set of criteria
> to help the review process
> </MB>

Do you have views about what those criteria should be?  So far, we
haven't come to agreement about them.

> « backward » compatible with an earlier version of the table. I think
> assuming that the table will only add stuff may be risky

Why?  Surely we could define it that way?

> <MB>This table should be parsed by software. I don’t care about the RFC
> itself,  but I care about precision. In the code point and « cross
> reference » columns, there are list of code points. It should be clear to
> IANA and the reader which are sequences of code points that should belong
> together, which are separate list of code points/sequences. Therefore, at
> least an explanation of the format is needed and also instructions to IANA
> to make sure these are properly set in the XML registry so they can be
> parsed automatically.

I think this is correct.

>    Inclusion depends on the assumption that such strings as will be used

Yes, "such strings as will be used."  It's correct English.

> <MB>as you know, precis and precis profiles is doing NFKC. So it might be
> appropriate to stick or assume NFC here</MB>

John has argued that we ought to give up making this general-purpose,
and start with IDNA only, precisely because of the precis/IDNA
differences.  I've been gradually coming around to his view, and this
is one more reason.

> <MB>It seems that this discussion should reference the Integrated Issues

Why?

Thanks very much for the discussion.

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Marc Blanchet" <[email-masked]>
Date: Mon, 10 Jul 2017 13:07:40 -0400

>> Informational or Best Current Practice.

you haven’t responded on this one.

>> Therefore, a string in one script may be visually identical to 
>> another string in another script, while the code points are 
>> different.
>> </MB>
>>

much better to me.

>> different font mechanisms - are sometimes substituting fonts or 
>> shares when they don’t have the exact code point match, etc…
>> </MB>
>>
> <AF>
> While it is true that rendering engines (and fonts) in general may 
> take liberties with rendering any identifier, that general problem 
> cannot be addressed by taking precaution based on specific characters 
> (e.g. those from the registry we are trying to set up).

sure. but they are known code points that are badly rendered. I think 
the « generic » issue should be pointed out. Again, reading the 
document, that issue was really missing. I’m leaving to the authors to 
find the right place and right wording. (And yes, we shall not put all 
i18n problems back into the document.)

>
> If you meant this as a generic issue, it should perhaps go into the 
> "not all problems can be solved" section of the text. If I 
> misunderstood what you were aiming at, please set me straight.
> </AF>
>>
>> to me are just not enough strong. But I guess that does not matter 
>> too much. I would have said something along the lines of « The 
>> Unicode repertoire, or a subset of it, is just too large to be 
>> carefully looked at for an operator of an identifier system, 
>> therefore, these operators rely only on third party tables. »
>> </MB>
>>
>
> <AF>
> The first level would be for registries to be aware that "anything 
> goes" will lead to specific, identifiable problems. The registry will 
> help identify what those problems are and give some suggestions on how 
> to avoid them.
>
> We are clearly taking the (implied) position that it's not workable 
> for everyone to do original research in this arena. However, everyone 
> will need to set policy. "Let's stay away from troublesome characters" 
> is a good defensive move if you don't have access to expertise that 
> would allow you to implement (and administer) something more finely 
> tailored.
>

I’m not sure it gets my point, but anyway. I was trying to say that 
for the large majority, they have to rely on third party tables, such as 
this registry, to create their registration policy. (None of them (not 
only in the domain business) has the time, expertise, incentive to do a 
code point by code point analysis, even for a single « simple » 
script. )

> <AF>Actually, no. Literally the same code point. Case is something 
> like two strings,
> one uses reverse E the other uses SCHWA. Both are identical in all 
> other code points.
>

well, even rereading with your explanation, it was not clear to me. 
maybe it is my french biased reading.

> <AF>I see no difference between "meaning" and "language meaning", 
> other than that the latter doesn't sound terribly good in English. 
> Perhaps we can write: "mean the same thing to the native reader"</AF>

I think I was « confused » because it could be related to protocol 
or else. Language was added to clarify. anyway.

>> s/In every case, the registry here defined…/This specification 
>> defines a registry of code points that

not responded to this one.

>> <MB>I would suggest to put the references in a separate column. 
>> easier to follow through, to parse, to read, to update, …</MB>
>>
>> sorted by any column), labelling the column « Rationale » or « 
>> Class » or  « Type » …
>>
> point means no more or less than a few lines of report generating 
> code, so I can try out some in the next draft and we can make further 
> changes or revert as needed.</AF>
>>
>> discussed since the early days of idn wg, sometimes a single code 
>> point may raise a lot of different opinions and perspectives. 
>> Therefore, a single expert to me is not enough to approve changes to 
>> this registry. I would suggest something in the form of the mime 
>> process, where the proposal is sent to a specific mailing list for 
>> discussion and maybe a reviewer (or two) will inform IANA of the 
> I agree that we are not there yet in defining the principles that 
> should be applied in updating this registry. What criteria do we 
> expect this registry to embody? The sample initial collection tries to 
> answer that implicitly, but we will still need to discuss this some 
> more and come to an explicit description of "what kind of trouble" 
> gets a character listed here.

yes. that is what I was trying to say.

> ensure that the registry is not a collection of personal prejudice, 
> but an attempt at collecting the shared experience and expertise of a 
> wider community of experts.
>
> URL, an internal reference that cites (quotes) the relevant 
> information from the original document may be acceptable).
>
> approach to the issue, so that the nature of the reviewers becomes 
> less important.
>
>> (or whatever process) to identify which one should be included or 
>> not. Therefore, I would highly suggest that the authors define a set 
>> of criteria to help the review process
>> </MB>
>>
> should explicitly suggest that if a new code point is "analogous" to 
> an existing troublesome character in the way it is used, that it be 
> there is a code point that isn't labeled "troublesome" here, it does 
> not mean that it is white-listed. It just means that there's no issues 
> been identified with it. Assuming a new Unicode code point creates a 
> new confusable pair of identical appearance with an existing Unicode 
> character. There would be every reason for the existing code point to 
> be listed as troublesome from that point on forward. Registries may 
> decide that the new code point should be the one to be excluded, which 
> would keep their policies backward compatible. Or they could decide to 
> make these blocked variants of each other. This still maintains the 
> registry, because before the new code point was included, no label 
> with it could be registered, now that it is available, users may find 
> some possible labels already blocked. That's normal 
> first-come-first-serve.
>
> some code point. Registries that do not include that code point 
> already have no issues excluding that code point. Other registries may 
> need to evaluate whether to disallow future registrations. But from 
> the point of the "troublesome character registry", it's not clear that 
> it benefits from any rules that would prevent making known any issues, 
> no matter how late they arise.

I know this is difficult, specially for a non-normative registry. But I 
think the document should:
- discuss the potential issue
- provide if possible some advice
- and at least clearly state that this is likely to happen.

I’m saying all this because most likely, like it or not, people will 
parse the machine-readable format and just put them as black-listed. One 
day, the new version of the registry will have a « non-compatible » 
change, and then they will be « screwed up ». By adding some text, 
at least, they are aware (at least if they read the document…) that it 
might happen.

>> RFC itself, but I care about precision. In the code point and « 
>> cross reference » columns, there are list of code points. It should 
>> be clear to IANA and the reader which are sequences of code points 
>> that should belong together, which are separate list of code 
>> points/sequences. Therefore, at least an explanation of the format is 
>> needed and also instructions to IANA to make sure these are properly 
>> set in the XML registry so they can be parsed automatically.
>>
>> Some additional information might be appropriate, may be as a 
>> glossary in the beginning of the document. I know this document says 
>> that the reader should be aware of Unicode, IDNA, etc… But as a 
>> good writing style, I think it would be good to have a glossary that 
>> identifies where the UPPERCASE keywords come from. I would see a 
>> glossary such as:
>>

you haven’t responded on this one above.

>> readable format of this table so I can do my own 
>> validation/parsing/QA of this table before it goes published. I 
>> wonder if for the duration of the discussion, a github file of the 
>> table itself could be referenced in the draft and an above note to 
>> the RFC editor to delete the pointer at the time of publication
>>
> <AF>Let's figure out the logistics.

github is easy, available, used often in ietf, has versioning 
« free », … I don’t think we need to think a lot about this, 
just do it, so we can look at it.

> The underlying data is currently kept in an XML database that conforms 
> to RFC 7940 -- I know that you and others have tools to read it.</AF>
>>
>> </MB>
>>
> <AF>Not my phrasing, but as far as I can tell, possible in English; 
> but what do I know?</AF>
>>
>> <MB>as you know, precis and precis profiles is doing NFKC. So it 
>> might be appropriate to stick or assume NFC here</MB>
>>
> <AF>so no change?</AF>

oups. wanted to say « appropriate to _not_ stick …

Marc.

> _______________________________________________
> I18n-discuss mailing list
> [email-masked]
> https://www.iab.org/mailman/listinfo/i18n-discuss

---

From: "Marc Blanchet" <[email-masked]>
Date: Mon, 10 Jul 2017 13:16:20 -0400

On 10 Jul 2017, at 12:53, Andrew Sullivan wrote:

> registry and create the rules for updating it.

you don’t have to have a standards track document to create an IANA 
registry. I wrote an RFC to create IANA registries for DTN, from the 
DTNRG, which were all experimental RFC.

> What is an "informal"

there is no such thing. I was talking about the status of the document, 
not the registry.

>> first
>> Therefore, a
>> rendering

yeap. I was.

> issue is that humans interpret labels like natural language, even

see my reply to Asmus.

>> easier to

not it is not, at least in the ascii draft.

>> Recommended, … I

yes.

>> discussed since

not about participation, but if you have a single appointed expert, then 
all the discussions between IANA and the appointed expert are not public 
(just by design, not as a wanted feature) and therefore do not have any 
scrutiny.

>> (or

;-) I think we can’t find some.

well, then, it goes strongly to my point above: given that the criteria 
are potentially « vague », then the best way to make sure the 
process is strong is to have multiple reviewers and in public way.

The mime process is to me pretty good. Same as i18n, not that many are 
experts and are there, but at least, you have a mailing list of a few 
people that care about it and then chime in when they think. Also you 
have a public archive to come back later to understand the rationale of 
the request.

>> in
>> think

see my response to Asmus.

>> RFC

ok.

>> might be
>> Issues

because it is talking about many issues discussed in that report, so 
reader can get an even more detailed perspective.

Marc.

---

From: "Marc Blanchet" <[email-masked]>
Date: Mon, 10 Jul 2017 13:21:13 -0400

> DTNRG, which were all experimental RFC.

and that RFC (RFC6255) was informational, which is to me the right 
status: i.e. the document, itself, is not a protocol specification that 
you implement, therefore it can’t be a standards track. It is a 
request to create a registry.

Marc.