Subject: accents!

From: Kyle McNamara <[email-masked]>
Date: Sun, 25 Jul 1999 08:52:10 -0400

Hello-

Does anyone have info on how the discussion has evolved around accents in the URN?

As the Net becomes less US-centric, it is obviously imperative to address this limitation... anecdotal evidence:

I am a programmer and own a company called Espanol.com... obiously I cannot include the tilda (~) above the n because it is not supported... everwhere I advertise I always have to make sure that we point that out... not only that, 100's of thousands of our target audience have the accents built right into their keyboards and will unassumingly never be able to access the site because they are able to type in the n /wtih/ the tilda above it...

How can I make a motion, write a proposal or make enough noise to help us move towards a system that takes into account that english is only one of many others prominent languages?

Thank you!

Kyle

---

From: Dirk-Willem van Gulik <[email-masked]>
Date: Sun, 25 Jul 1999 15:08:06 +0200

> Does anyone have info on how the discussion has evolved around accents in the URN?

You might want to consult the archives of this mailing list, and review
the various requirments documents as well as the final documents on URNs.

Both localization and internationalization conderations have most
certainly been taken into account during the design.

> How can I make a motion, write a proposal or make enough noise to help
> us move towards a system that takes into account that english is only
> one of many others prominent languages?

Write code :-) write lots of code which does allow users to both enter and
view information in their favourite cast; which does know how to deal with
UTF8 or whatever encoding is relevant and which does understand how to
transport such as octed stream safely.

As where URNs are concerned; one example application of URNs is in a
project called CILS a collaboration of the Commitee on Earth Observing
Sattelites and the United Nations Environmental Programme; where the
distributed information pointed at by the URN contains information in
English, Russian, Chinese, Spanish.. and some other languages in various
encodings.

I can assure you that where the URNs are concerned there are no problem;
though an end user might be challenged to understand the information.
After all we do not all read the worlds most commonly read language; see
for example the document urn:inet:cils.ceo.org:612-6612-1261 on exhaust
gasses of the Li-Chi-Teng factory near Xi-Pen.

Dw

---

From: Justin Couch <[email-masked]>
Date: Sun, 25 Jul 1999 21:29:01 +0900

> Kyle McNamara wrote:

> Does anyone have info on how the discussion has evolved around accents

[snip]

> one of many others prominent languages?

URNs are a basic naming convention and an accompanying resolution
system. The basic premise of URNs is that they are _not_ meant to be
human readable - ie it is unlikely that a user would type it into a
Location box like they might in your conventional web browser.

That said, there is nothing stopping the use of non-ascii characters in
a URN scheme. All you need to do is encode it into an escape sequence
(eg %20 is the equivalent of the tilde character IIRC). As with standard
URLs, you need to present escaped chars for anything not ASCII.  (Java
includes URLEncode and URLDecode classes specifically for this purpose.
I don't know what is available in C/C++ in standard form)

It is only a programming convenience that Apache et al handle
/~username/ inside the passed HTTP request. In reality, that is an
illegal URL. What should be passed to the web server is /%20username/
for a correctly formed URL. What should happen is that the user can type
in anything in the text field and the program goes through and encodes
any nasty bits and generates a correctly formed URL/URN. This is then
passed to the resolver system and off you go.

Tilde is only one of many standard ascii characters that cannot be used
unencoded inside a URN string (and obviously even less in the form that
you want it over the top of a character, not a standalone char). If you
read the specs carefully, what can be used in URNs is extremely limited
- basically limited to alphanumerics and a couple of the punctuation
characters. Offically, you can't even use '/' to denote a heirarchy in a
URN namespace specific string unless it is escaped first!

Adding support for full internationalised strings as standard, unencoded
character streams would be pretty close to impossible these days. It is
not just URN support, but also URLs, the HTTP, FTP and any other
protocols. You would also need to change DNS as well if using NAPTR
records for resolution (I couldn't imagine what sort of problems this
would bring to the regular expression parsers used by DNS. You'd
probably want to write your DNS server in PERL or something that could
handle it reasonably - yuck!).

So, in summary, it can't be done as part of the standard syntax. There
are known ways of including this information (encoded text) so there is
no need to try to modify the existing bases to accomodate it.

--
Justin Couch                                   Author, Java Hacker
Snr Software Engineer                     [email-masked]
ADI Ltd, Systems Group              http://www.vlc.com.au/~justin/
Java3D FAQ:       http://tintoy.ncsa.uiuc.edu/~srp/java3d/faq.html
-------------------------------------------------------------------
"Look through the lens, and the light breaks down into many lights.
 Turn it or move it, and a new set of arrangements appears... is it
 a single light or many lights, lights that one must know how to
 distinguish, recognise and appreciate? Is it one light with many
 frames or one frame for many lights?"      -Subcomandante Marcos
-------------------------------------------------------------------

---

From: "David G. Durand" <[email-masked]>
Date: Sun, 25 Jul 1999 10:22:56 -0400

>Location box like they might in your conventional web browser.

This has been a point of contention for a long time. I don't think that
human unreadibilit was ever made a design goal of URNs, since not everyone
takes that view of URNs. There are certainly URN-style namespaces that are
human-readable (like SGML's Formal Public Identifiers).

  -- David
_________________________________________
David Durand              [email-masked]  \  [email-masked]
http://www.cs.bu.edu/students/grads/dgd/  \  Director of Development
    Graduate Student no more!              \  Dynamic Diagrams
--------------------------------------------\  http://www.dynamicDiagrams.com/
MAPA: mapping for the WWW                    \__________________________

---

From: Larry Masinter <[email-masked]>
Date: Sun, 25 Jul 1999 09:29:54 PDT

I suggest you look at

   draft-masinter-url-i18n-04.txt

for a discussion of some of the issues involved in using
non-ASCII characters in URIs. This has been languishing
because it's not clear where it should go, but perhaps
we should just Last Call it as a Proposed Standard?

Larry
--
http://www.parc.xerox.com/masinter

---

From: Dirk-Willem van Gulik <[email-masked]>
Date: Sun, 25 Jul 1999 19:34:20 +0200

Of course, in my previous message I glossed over the reason why full
normative decomposition is not a solution IMHO; the canonical ordering
behaviour for non spacing marks which do not interact; which is rather
commin,m is just not defined in the spec. Though we could say that this
needs to by sorted by code point or somthing like this.

Dw.

---

From: Dirk-Willem van Gulik <[email-masked]>
Date: Sun, 25 Jul 1999 19:32:17 +0200

> we should just Last Call it as a Proposed Standard?

Though I must admit I certainly have no _solutions_ I am still very wary
of the way 'normalization' is glossed over in that document. It is sad
that the Unicode spec just does not solve it. Given that one does not
nessesarily know the language, tone of speach, dialect, scriptur etc, at
the time of transcoding there is either a lack of information or a lossy
compression happening.

Looking explictly at the unicde 2 spec; even outside the normative
sections; equivalence is _not_ dealt with; at best there is a reference in
2.2 to loose implementation guidelines in 5.9. Same goes for the accuracy
statement made.

The only thing I can see is using the maximal decompositions as a way to
unify the spelling in a standard way. But this is might not be user
friendly enough; i.e. the normative spelling is rarely used. And it
certainly makes the live of a casual coder hell. The precomposed chars are
so much easier to deal with. Though this is what the 2.0 spec suggest.

Dw.

---

From: Dirk-Willem van Gulik <[email-masked]>
Date: Sun, 25 Jul 1999 19:17:59 +0200

> human-readable (like SGML's Formal Public Identifiers).

Whilst deploying I most certainly have found it usefull to allow encoded
UTF8 presentations in the opaque part of the URN; it greatly eases legacy
applications. In particular a number of chinese archives where much
simpler to open-up by simply utf8 encoding followed by the usual URI %
escaping of the actual (already fundamentally unique) document qualifiers.

Sure, they might look like a bunch of silly digits in my ascii pine email
tool; but they do make sense in say a ZH version of netscape.

Dw.

---

From: Larry Masinter <[email-masked]>
Date: Sun, 25 Jul 1999 21:16:55 PDT

> the time of transcoding there is either a lack of information or a lossy

I thought that http://www.unicode.org/unicode/reports/tr15/  was our
reference for normalization.

Language-sensitive normalization cannot be done for URLs;
we can no more deal with tone, dialect or other elements than
we can automatically mandate that URLs spelled "colour" and "color"
should be equivalent.

On the other hand, we wanted to deal with the artifacts of the
choice of Unicode/UTF8 and its own coding ambiguities, by
using the Unicode specification.

I'm hoping that CNRP will allow the use of actual 'friendly' names
to identify/locate resources, and allow for language/tone/dialect
sensitive matching on the server, but it is no longer a simple
equality.

---

From: Larry Masinter <[email-masked]>
Date: Sun, 25 Jul 1999 23:17:41 PDT

> needs to by sorted by code point or somthing like this.

I'm confused by this; I thought we were talking about equivalence
rather than ordering (collating).  Could you perhaps explain more
about what your problem is? I'd think this is something we should
address in the URL I18N draft if it is indeed a problem.

Larry

---

From: "Martin J. Duerst" <[email-masked]>
Date: Mon, 26 Jul 1999 12:39:14 +0900

> > we should just Last Call it as a Proposed Standard?

Some more background can be found at:

http://www.w3.org/International/O-URL-and-ident.html

Any comments welcome!

> so much easier to deal with. Though this is what the 2.0 spec suggest.

This is being addressed. Please have a look at http://www.w3.org/TR/WD-charmod,
in particular http://www.w3.org/TR/WD-charmod#TextNormalization,
http://www.w3.org/TR/WD-charreq#3, http://www.unicode.org/unicode/reports/tr15/tr15-10.html, and
http://www.w3.org/International/charlint/ for an implementation.

This concerns the problem of precomposed/decomposed equivalences.
There are a lot of other (more-or-less) equivalences; for most
cases, they are similar to those that can already appear now, i.e.
0/O, 8/B, 1/l/I,...

Regards,   Martin.

#-#-#  Martin J. Du"rst, World Wide Web Consortium
#-#-#  mailto:[email-masked]   http://www.w3.org

---

From: Dirk-Willem van Gulik <[email-masked]>
Date: Mon, 26 Jul 1999 10:52:01 +0200

> > On Sun, 25 Jul 1999, Larry Masinter wrote:
...
> http://www.w3.org/International/O-URL-and-ident.html

Yes; perhaps such background needs to be folded into the draft before it
goes near any standard.

> This is being addressed. Please have a look at
...

Martin, you are absolutely right; but it is being _addressed_ (and it is
being addressed well IMHO) , but it is not 'solved' and part of the IETF
draft (yet) or anywhere IMHO in a state where it should go onto an IETF
standards track.

Any hint at a solution (and the unicode standard does give us a unique
unambigious decomposition for each and every entry) should be part of that
draft before it goes anywhere.

Or the draft should be rewritten as a requirments document. But not
something in between.

> This concerns the problem of precomposed/decomposed equivalences.

Yes; this is the core of the issue; I've been running into these a lot
with the Thai language or even as simple with germans entering swedish
URNs. To me it means that right now; the draft does not allow a third
party to reliably enter the URN from, say, an advert in its international
form (as opposed to in its URI % escape form).

Normalizing the string to fully decomposed would help.

> cases, they are similar to those that can already appear now, i.e.

Though some of these glyphy equivalence we will always have to live with.
And it does not concern me nearly as much. As say, the issue surrounding
for example european accented chars where one nation treats it as a
separate char, and the other just as a char with some extra tacked on.

The thing which does concern me is that I'd like to have something which
meets the requirments you and Larry have set out in the draft; that you
can transcrcibe the internationalized urn reliable, enter it and get the
correct record back. That I've seen failing far to often in the actual
world; when using the loose approach described in the draft.

Dw

---

From: Larry Masinter <[email-masked]>
Date: Mon, 26 Jul 1999 20:18:18 PDT

re http://www.w3.org/International/O-URL-and-ident.html;

Also, for another approach to the problem, see:

 http://lists.internic.net/archives/cnrp-ietf.html

 http://www.ietf.org/internet-drafts/draft-popp-cnrp-goals-00.txt

This is a very different approach to the problem ("cannot put accents
 in URLs"), namely "how to avoid using URLs for the things that
people type".

Larry