Subject: [Sip] i18n issue: composed characters

From: Adam Roach <[email-masked]>
Date: Tue, 26 Nov 2002 00:21:10 -0600

While digging through the confctrl archives, circa 08/99, I
came across a thread that probably shouldn't be lost now that
we have started giving internationalization more serious thought.

With Unicode, there are two different ways to encode most
accented Latin characters. I beleive the same issue occurs
in several other character sets (e.g. composition of Korean
Hangul versus use of precomposed characters).

For example, the character "Å" (an "A" with a loop over it)
can be encoded in Unicode as U+00C5 or as U+212B or as the
combination U+0041 U+030A.

Now, with multiple possible encodings for single characters,
we have the problem of, for example, someone registering as
"sip:[email-masked]", and someone else inviting
"sip:[email-masked]". In both cases, the interface
would render the user portion as "Åse" to the users, but
matching is difficult.

Of course, we *could* just leave this as a server implementation
issue. This is a bit of a pain, since it means that *all* servers
need to perform normalization of unicode characters before
doing any processing. Bleah.

Alternately, taking the suggestion Kevin Payne made so many years
ago, we could specify that any user agent that has the ability
to accept characters that might have multiple Unicode encodings
is responsible for normalization of that input to a single, canonical
form.

My understanding is that Unicode defines at least two methods
of normalization (Canonical and Compatibility), and that a proposal
was floated to define two *additional* methods by performing the
decomposition described by Unicode, followed by a canonical
composition. This has the advantage of taking less space. It's not
clear whether this method [1] made it into any Unicode standards.

So, do we want to address this issue at all, or so we just let
things break?

/a

[1] http://www.unicode.org/unicode/reports/tr15/tr15-10.html
_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol
Use [email-masked] for questions on current sip
Use [email-masked] for new developments on the application of sip

---

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 26 Nov 2002 11:06:51 -0500

inline.

> would render the user portion as "Åse" to the users, but

Actually, a first, more troubling point. THe above escape encoding 
requires that the SIP URI explicit point out that it uses UTF-8 for 
conversion of the octet sequence generated from the URI, to the original 
character string. Although SIP generally uses UTF-8, nowhere does the 
specification point out this property of the SIP URI. I recently logged 
this in bugzilla, as Bug 678. We'll need to clarify that in the next 
release.

Of course, the main issue once we specify that is canonicalization, as 
you point out below.

> doing any processing. Bleah.

We really don't want to do that. I suspect that this is a nightmare 
procedure as a general rule. I'd rather distribute it to clients which 
support those particular characters. Not all UI will support entry of 
all unicode characters, so the UA would only need to know about 
canonicalization of those characters it supports. THis is compared to a 
proxy, which would need to canonicalize every potential unicode sequence 
(shudder).

> is responsible for normalization of that input to a single, canonical

This seems like a better approach.

Note that, we do currently specify that the proxy is supposed to do 
canonicalization:

> 5. The registrar extracts the address-of-record from the To header
>          field of the request.  If the address-of-record is not valid
>          for the domain in the Request-URI, the registrar MUST send a
>          404 (Not Found) response and skip the remaining steps.  The URI
>          MUST then be converted to a canonical form.  To do that, all
>          URI parameters MUST be removed (including the user-param), and
>          any escaped characters MUST be converted to their unescaped
>          form.  The result serves as an index into the list of bindings.

this is reasonably well defined for US-ASCII characters, but broken for 
UTF-8 as you point out. As such, would we split the canoncicalization, 
with clients using the canonical escape encodings, and then the proxy 
un-escaping? If we specify a canonical escape encoding, and FURTHER 
mandate that clients never escape characters which do not require 
escaping, we may not need the proxy to unescape at all. Its not clear if 
there is an advantage to splitting the canonicalization between the two 
elements.

> So, do we want to address this issue at all, or so we just let

Yes, we definitely want to deal with this. I don't think this is part of 
the requirements work that Fujimoto-san is working on; this is really 
part of the core specification, addressing a hole we currently have.

I am surprised that the method of canonicalization is not addressed by 
Unicode. After all, don't they specify rules for equality checking (that 
is, after all, what we are after, just using canonicalization for 
equality checking)? Don't those rules include such encoding variations? 
I'm not a unicode expert, but this seems like a very basic issue.

-Jonathan R.
-- 
Jonathan D. Rosenberg, Ph.D.                72 Eagle Rock Ave.
Chief Scientist                             First Floor
dynamicsoft                                 East Hanover, NJ 07936
[email-masked]                     FAX:   (973) 952-5050
http://www.jdrosen.net                      PHONE: (973) 952-5000
http://www.dynamicsoft.com

---

From: Rohan Mahy <[email-masked]>
Date: Tue, 26 Nov 2002 13:25:50 -0800

On Monday, November 25, 2002, at 10:21 PM, Adam Roach wrote:

> of normalization (Canonical and Compatibility),

A SIP entity which adds or modifies a URI must insure that any 
characters it supports which have multiple representations are 
represented in canonical form. (this should also cover needlessly 
escaping characters which don't need to be escaped).

Now do we want the same requirement for quoted strings?

thanks,
-rohan

> and that a proposal

---

From: Sean Olson <[email-masked]>
Date: Tue, 26 Nov 2002 13:34:08 -0800 (PST)

> Now do we want the same requirement for quoted

The requirement for canonical form makes sense in
any string where a software may need to make a
comparison/collation such as in a URI. I think 
extending this to quoted strings in general may be
overkill.

> -rohan

__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol
Use [email-masked] for questions on current sip
Use [email-masked] for new developments on the application of sip