Subject: [EAI] Another subject: left-to-right languages

From: "Daniel Taharlev" <[email-masked]>
Date: Sun, 18 Nov 2007 21:07:21 +0000

I've been reading the threads here, and hope I didn't miss this being discussed yet.

Adding non-ascii characters is great for left-to-right languages; with all of the issues, there will probably be a solution including ascii-only secondary addresses, or something similar.

However, what about right-to-left languages, such as Hebrew? If both the address and the domain are right-to-left, the address may look like this:
[email-masked]

But it could also look like this:
moc.niamod@resu

See, when an smtp connection starts speaking, it sends the characters in the correct order, i.e no matter the language, the first char will be "u", then "s", then "e" and so on; however, when looked up internally, when forwarded, etc., the address' display must be reversed!

If anyone ever tried just setting up a simple Active Directory environment with usernames, passwords or even OUs using right-to-left names, I'm sure you got a taste of the problem; with world-wide smtp, it could be ten times worse!

Would love to hear what you have to say.

DT

[email-masked]

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Sun, 18 Nov 2007 16:51:22 -0500

--On Sunday, 18 November, 2007 21:07 +0000 Daniel Taharlev
<[email-masked]> wrote:

> Adding non-ascii characters is great for left-to-right
> languages; with all of the issues, there will probably be a
> solution including ascii-only secondary addresses, or
> something similar.
> 
> However, what about right-to-left languages, such as Hebrew?
> If both the address and the domain are right-to-left, the
> address may look like this: [email-masked]
> 
> See, when an smtp connection starts speaking, it sends the
> characters in the correct order, i.e no matter the language,
> the first char will be "u", then "s", then "e" and so on;
> however, when looked up internally, when forwarded, etc., the
> address' display must be reversed!
> 
> If anyone ever tried just setting up a simple Active Directory
> environment with usernames, passwords or even OUs using
> right-to-left names, I'm sure you got a taste of the problem;
> with world-wide smtp, it could be ten times worse!
> 

Daniel,

We have been wrestling with these issues in several contexts,
most notably with IDNAbis (see, in particular,
draft-alvestrand-idnabis-bidi-01.txt).  

The short answer is that this sort of thing has to be treated as
a human interface issue.  The protocol --which is all these
documents specify -- has to be to send the material in network
order, which is what you suggest ("u", then "s", etc.).  The
display routines (part of the user interface) will need to
understand how to correctly render the characters.  That implies
not only right-to-left for appropriate scripts, but specific
rendering actions for scripts (even left-to-right ones) that
require special rendering actions depending on context and
embedded format-effecting characters.

The other problem involves the question of whether users can get
from what is written on a business card to the right address for
typing into a computer.  One can only hope that those users who
are familiar with the script and language in question will get
it right.  Those who are not may need help from ASCII
alternative addresses or some other type of hint: as I said
several times at IGF last week, those transcript problems are
issues that are direct consequences of ones that have been with
us for around 5000 or 6000 years.  Neither the Internet, nor
some small piece of protocol work, is going to fix them.

I think our advice to user interface designers should be "be
careful, as there are additional traps in this area".   If you
would like to propose specific text along those lines and where
to put it --for either the EAI documents or for
draft-klensin-idnabis-issues, since there is considerable
overlap in interest and editors-- I think I can speak for my
co-authors and co-editors in saying that such contributions
would be very welcome.

In the longer term, and for the broader issues, I think the IETF
community and others would welcome one or more informational
tutorials on how to address the many interface problems
associated with right-to-left characters and scripts, especially
when the relevant language is unknown (e.g., the issues are
slightly different with the Hebrew script for Yiddish and
Hebrew, and for the Arabic script with Persian, Urdu, Arabic,
and other languages).

   john

---

From: Martin Duerst <[email-masked]>
Date: Mon, 19 Nov 2007 08:49:12 +0900

>> But it could also look like this:

It not only could look like this, if all the components are
RTL, it should look like this. An alternative might be
[email-masked], but that would make it extremely difficult
to carry such addresses in plain text and to display and
later keyboard them again in a consistent way.

>draft-alvestrand-idnabis-bidi-01.txt).  

That draft deals with an important detail that had been
forgotten in previous specs, namely the issue of combining
marks on RTL letters, in particular at the end of an RTL
component. My understanding is that everybody involved
agrees that the proposal in the above draft is a non-brainer,
it's clear that a fix is needed. But the above draft, although
related to the problem of RTL and Bidi, doesn't deal with the
core of the issue at all.

For much more on this issue, the best place to look at at
the moment is section 4 of RFC 3987, or its update, currently
at http://www.ietf.org/internet-drafts/draft-duerst-iri-bis-01.txt.
(which does not yet integrate all the necessary fixes from
the issues list, including some related to bidi).
This is about IRIs, the international extension of URIs,
which have to deal not only with mail addresses, but with
a huge number of other identifier schemes.

>embedded format-effecting characters.

Yes. Other rendering issues are also important, but their
effect is always local (within a few codepoints).

>it right.

Yes. This is a particularly important and tricky point for
RTL and bidi. What we (actually Mati Allouche from IBM Israel)
came up with were some simple conventions that should make
the mappings from graphic (visual) to electronic (logical)
deterministic. For that, some restrictions were needed on
combinations of letters of different directionality in
different components. These restrictions were also carried
over into IDNA. And these restrictions were where we forgot
about combining marks, which we have to fix. 

>co-authors and co-editors in saying that such contributions

I think just saying 'be careful' isn't enough. But putting
in specific text runs the risk of creating different solutions
for IDNs, for non-ASCII emails, for IRIs, and for other kinds
of identifiers. One way to go would be to point to section
4 of RFC 3987 (or its successor) and specify what the components
of an email LHS are.

>Hebrew, and for the Arabic script with Persian, Urdu, Arabic,

John, are you saying that e.g. there are differences between
the Hebrew and Yiddish languages? There are definitely differences
in the character repertoire and in the use and importance of
vowel marks, but I don't know about any differences with respect
to directionality.

Please note that the W3C already has quite a bit of material
on Bidi issues (see
http://www.w3.org/International/resource-index.php?topic=bidi),
although there isn't much on identifiers.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: John C Klensin <[email-masked]>
Date: Sun, 18 Nov 2007 19:00:06 -0500

--On Monday, 19 November, 2007 08:49 +0900 Martin Duerst
<[email-masked]> wrote:

>...
>> In the longer term, and for the broader issues, I think the
>> IETF community and others would welcome one or more
>> informational tutorials on how to address the many interface
>> problems associated with right-to-left characters and
>> scripts, especially when the relevant language is unknown
>> (e.g., the issues are slightly different with the Hebrew
>> script for Yiddish and Hebrew, and for the Arabic script with
>> Persian, Urdu, Arabic, and other languages).
> 
> the Hebrew and Yiddish languages? There are definitely
> differences in the character repertoire and in the use and
> importance of vowel marks, but I don't know about any
> differences with respect to directionality.

Only in the "vowel mark" issue, e.g., all of the combining mark
issues that you describe in your note can be avoided for Hebrew,
but not for Yiddish.

>...

   john

---

From: "Daniel Taharlev" <[email-masked]>
Date: Mon, 19 Nov 2007 00:49:41 +0000

With regards to what John said, let's make it even more interesting. In Hebrew, the letter "Beit" can sound as a "B" or as a "V" -- depending on if it has a "dagesh" -- a little dot -- in it. That dot is part of the letter; however, when typing, one of the common solutions is to actually add the "dagesh" character right after the letter, where the system font basically has a setting saying "this special character needs to be embedded into the previous letter." The dotted letter appears in the charset as a unique ascii code, but is never used -- you'll always use the font trick.

Now, if an smtp connection is created, it transmits the characters one by one; the server can know how to embed the character into the display, but will that make "[email-masked]" a different email address than "b(with a [email-masked]"? For Hebrew speakers, that's like having cursive-written addresses different from printed addresses. One can easily take advantage of that, posing as someone else, by adding a small dot into their username -- which is even harder to detect that a phishing site taking advantage mistyping "google" as "googel".

[email-masked]

-----Original Message-----
From: John C Klensin <[email-masked]>

Date: Sun, 18 Nov 2007 19:00:06 
To:Martin Duerst <[email-masked]>, [email-masked],eai list <[email-masked]>
Cc:Richard Ishida <[email-masked]>, Felix Sasaki <[email-masked]>
Subject: Re: [EAI] Another subject: left-to-right languages

   john

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 19 Nov 2007 09:43:44 +0100

> Now, if an smtp connection is created, it transmits the characters one by one; the server can know how to embed the character into the display, but will that make "[email-masked]" a different email…
We're happily leaving all those details to the gentle hands of the 
Unicode Consortium, which has normatively stated that HEBREW LETTER BET 
WITH DAGESH is a single character with codepoint 0xFB31, but this 
character is excluded from being normalized to, so the correct 
representation for anyone, whether they use NFC, NFKC or NFD, is 0x05D1 
(BET) followed by 0x05BC (DAGESH OR MAPIQ).

So, as far as the standards are concerned, BET+DAGESH is 2 characters, 
and BET WITH DAGESH is one character. It is up to the sysadmin to decide 
whether BET, BET+DAGESH and BET WITH DAGESH, when occuring on the 
left-hand side, all get delivered to the same mailbox; it is up to the 
administrators of the zones allowing Hebrew character registration to 
decide whether or not to allow the registration of BET and BET+DAGESH to 
different entities or not.

One interesting aspect is that we've studiously avoided having the 
standards require that normalization be applied to the left hand side of 
the email address, while the current IDNA specs require the use of NFKC, 
so BET WITH DAGESH is allowed by the standards on the left hand side, 
but not on the right hand side.

Any sysadmin who consciously creates unnormalized mailboxes is, in my 
opinion, making a really stupid decision, but our job is not to try to 
forbid stupidity. (ASCII spaces in mailbox names are stupid too, but the 
base standards have always allowed them, if quoted.)

                  Harald

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 19 Nov 2007 09:52:18 +0100

There is one issue I don't understand, the cs.ox.ac.uk/uk.ac.ox.cs thing.

Suppose two domains exist, which are rendered as asdf.gh anf qwer.ty. 
(Pretend asdf and qwer is right-to-left and gh and ty left-to-right.) 
Now a user at qwer.ty gets a localpart of gh.asdf and a user at qwer.yu 
a localpart of gh.asdf.gh.

IIRC the @ doesn't have directionality, so @ the unicode bidi algorithm 
will render those in the same way.

(I have a cold and can't think quite straight. Maybe the localparts have 
to be rewq.ty or ty.qwer or ty.rewq or something. But I'm sure you can 
see the problem.)

Arnt

---

From: Martin Duerst <[email-masked]>
Date: Mon, 19 Nov 2007 18:23:25 +0900

At 17:52 07/11/19, Arnt Gulbrandsen wrote:
>Suppose two domains exist, which are rendered as asdf.gh anf qwer.ty. (Pretend asdf and qwer is right-to-left and gh and ty left-to-right.)

To make things clearer, the convention among Bidi experts is to use
upper-case for RTL, so that would be ASDF.gh and QUER.ty, yes?
Can we assume that the order written here is logical, given that
these are the sequences as we find them on a keyboard?

>Now a user at qwer.ty gets a localpart of gh.asdf and a user at qwer.yu a localpart of gh.asdf.gh.

Okay, so we have:

Local part:     Domain:
gh.ASDF         QUER.ty
gh.ASDF.gh      QUER.yu

Still assuming these are logical.

>IIRC the @ doesn't have directionality, so @ the unicode bidi algorithm will render those in the same way.

Seems you may have intended a sligthly different example:
A user at QUER.ty gets a local part of gh.ASDF, and a user at
ASDF.gh gets a local part of ty.QUER.

Logically, this would produce the following two addresses:
a) [email-masked]
b) [email-masked]

If we apply Section 4 of the IRI spec (RFC 3987) to this, we would
get visual displays reading:
a) [email-masked]
b) [email-masked]

So we definitely get two different things.

>(I have a cold and can't think quite straight.

Zigzag thinking is much more important for Bidi problems than
straight thinking :-).

>Maybe the localparts have to be rewq.ty or ty.qwer or ty.rewq or something. But I'm sure you can see the problem.)

You mean that there is a problem that two different logical sequences
of characters can render in the same way? Such situations are possible,
but they have to involve components starting or ending with digits or
mixed components (components containing both RTL and LTR characters).
That's the reason why IDNA prohibits the mixing of RTL and LTR characters
in a single label (the IRI spec does the same but uses the term component
to be more general). That's also why both the IDNA spec and the IRI
spec prohibit digits at the start or the end of an RTL compontent/label.
Unfortunately, it was not possible to prohibit digits at the start or
the end of an LTR component/label, because these already existed.
So there is still a chance to create havoc. For the update of the IRI
spec, I have been asked to add another example to explain this case
and say that it's a bad idea to have e.g. a digit at the end of an
LTR component next to a RTL component.

Regards,    Martin.s

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 19 Nov 2007 13:12:27 +0100

> LTR component next to a RTL component.
The first interesting example I found when working on my bidi draft was 
this:

<logical> a.12-34.b   -> <visual> a.12-34.b
<logical> A.12-34.B  -> <visual> B.34-12.A

again using the "uppercase is RTL" convention.

                            Harald

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 19 Nov 2007 13:18:02 +0100

Martin Duerst writes:
> of characters can render in the same way?

Yes. Actually that an attacker might construct an address which is 
rendered like the victim's.

> Such situations are possible, but they have to involve components 
> starting or ending with digits or mixed components (components 
> containing both RTL and LTR characters).

Could you elaborate? (I'm really dense today, I'm afraid.) Which 
characteristics must the victim's address have in order to be 
attackable?

Arnt