Subject: RE: [Ltru] Re: Language transformation

From: "M.T. Carrasco Benitez" <[email-masked]>
Date: Fri, 1 Apr 2005 08:20:01 +0100 (BST)

This is attempt to distill the previous discussions and a proposal to go forward:

1) Transliteration
Use this term as it is the one most people associate with this mechanisms.
Define the term in a way the includes transcription and similar nuances.
I will write a short definition for approval.

2) Semantics
The important issue is to be able to tag transliteration; it should be specifically
mentioned and explained in the RFC.

3) Syntaxis
The syntaxis must follow the current situation and avoid unwarranted complications.
As per Addison proposal "don't distinguish [transliteration] per-se, just register
variants for the different ones". Assuming that "widegile" (8 characters) is a
registered variant, I assume that one could have the following language tags:

 zh-widegile
 zh-TW-widegile

4) Transliteration Registry
Transliterations must be first registered in the proposed "Transliteration Registry".
Then following the appropiate procedure, there would be an application for inclusion
as a variant. The Transliteration Registry is out of scope of LTRU.

Regards
Tomas

Send instant messages to your online friends http://uk.messenger.yahoo.com 

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 31 Mar 2005 23:35:48 -0800

M.T. Carrasco Benitez <mtcarrascob at yahoo dot com> wrote:

> From the experience of the last time, having the target language
> caused confusion: it must be very clear that the text tagged by
> "ru-t-foo" is Russian. So having "ru-t-fr" lead to believe that there
> was some French in the text.

There actually isn't a way to construct a single language tag that means
"Russian with some French mixed in."

In a tag that has an extension singleton such as 't', everything
following the singleton is part of the extended meaning (until you get
to the next singleton, or the end of the tag).  So in the tag "ru-t-fr",
you know that the 'fr' is part of the extension because it comes after
the singleton 't'.

The draft explains this in Section 2.2.6, with another example:

"All subtags following the singleton and before another singleton are
part of the extension. Example: In the tag 'fr-a-Latn', the subtag
'Latn' does not represent the script subtag 'Latn' defined in the IANA
Language Subtag Registry. Its meaning is defined by the extension 'a'."

> Also, I suspect (I am not certain) that transformation schemes would
> typically target more than one language; except in languages like
> Armenian with a very specific script.

That is why it would be good to be able to indicate both (a) schemes
that are language-dependent and (b) schemes that are not.

-Doug Ewell
 Fullerton, California
 http://users.adelphia.net/~dewell/

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 31 Mar 2005 23:43:38 -0800

JFC (Jefsey) Morfin <jefsey at jefsey dot com> wrote:

> What should be considered is the result to the user. A recording or a
> transcoding resulting in the same voice-text, look the same to me? I
> do not see script difference between a voice, in Russian and a voice
> in English speaking the same way? Questions can be about the sex,
> accent, timber, age of the voice and about the speaking style? Same a
> short-hand or phonetic or icons or music or menu figures (as long as
> there is a common understanding of what means what - what is a
> script?)

Spoken content should not be tagged as to script, transcription method,
or transliteration method.

Conversely, if a system were developed for tagging spoken accents, that
system should not be applied to written content.

This should be apparent.

-Doug Ewell
 Fullerton, California

---

From: "M.T. Carrasco Benitez" <[email-masked]>
Date: Fri, 1 Apr 2005 09:27:26 +0100 (BST)

>[Carrasco]
>> By the way, Armenians are celebrating this year the 1600th anniversary
>> of their script. It would be nice to go to the "Targmanchats Ton"
>> (translators day). This is a case of very successful standardisation:
>> would LTRU be here in 1600 years ?  -:)
>
> [Ewell] 
> Well, certainly not the working group.  We're only chartered to do 2
> documents, and the second is scheduled to go into Last Call this August.

Even if the Last Call was scheduled for August 2105, the Last Call of the
members would come before -:), though members could be replaced ...
 
> If you mean, "Will the revised language tagging standard exist and be
> relevant in 1600 years?" the answer is probably no.  But then, most
> technology standards don't last as long as most writing systems, so that
> seems an odd comparison.

Lets be optimistic and go for a "yes".

<div class="-:)">
About 1610 years ago, King Vramshapuh Arshakouni and Catholicos Sahak Partev
(Areas Directors) started a working group with Saint Mesrop Mashtots as Chair.

He travelled (mailing list) in Armenia to collect data on the different dialects;
went to Alexandria and delivered the alphabet (RFC) in 405 AD. The working group
did a quite good job as the 36 characters alphabet did not have to be updated
for about 500 years (RFCbis). Not due to a mistake in their part: the were a
couple of new sounds in the language and their principle is to have a direct
correletion between the phonetics and the ortography.

Yes, a very successful standardisation.
</div>

Regards
Tomas

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Fri, 01 Apr 2005 13:40:44 +0200

At 00:12 01/04/2005, M.T. Carrasco Benitez wrote:
>--- "JFC (Jefsey) Morfin" <[email-masked]> wrote:
> > What should be considered is the result to the user.
>
>Yes. This is the point.
>
> > A recording or a
> > transcoding resulting in the same voice-text, look the same to me? I do 
> not
> > see script difference between a voice, in Russian and a voice in English
> > speaking the same way? Questions can be about the sex, accent, timber, age
> > of the voice and about the speaking style? Same a short-hand or 
> phonetic or
> > icons or music or menu figures (as long as there is a common understanding
> > of what means what - what is a script?)
> >
> > Describing the process would be like documenting the version of Apache
> > being used to show an HTML page?
>
>There is real need for tagging transliterations and the requirements are as
>discussed before.

I accept that there might be a need. I accept the requirements are along 
the line you discussed. But I would need to understand what is the need. 
The way to address a need should start from the description of the need, 
not from a partial way to address it.

>Some of your comments above might well fall in a similar model to
>transliteration; so any volunteer ? Other comments looks more like metadata,
>though the language is also metadata.

True. IMHO we have two possibilities:
- either to define them all and we will take a huge amount of time for 
nothing, as you say elsewhere the users communities will decide.
- or we structure an open concept able to support several semantic and 
syntaxes and any user/standard addition in the proper place and 
understandable to the participants to a relation.

I obviously favor the second one, but legacy, transition and legitimate 
needs of some applications designers (such as seemingly some Unicode and 
W3C persons) should also be supported. This should be easy in having the 
first solution supported as one of the possibilities of the second one.

As a general comment, after a debate of three months I still fail to 
understand the need. Which means the basics who, when, why, how. This 
affects the whole proposition credibility. Harald said four years ago: we 
will store a few data in the IANA through a privately organized list. This 
solution turned out to be too limited to some and experimented huge delays 
in updating the IANA server. The first question is therefore to know if RFC 
3066 made good choices. Errare humanum est (RFC 3066), diabolicum 
perseverare (current idea of dramatically increasing the load on a lame 
solution).

Until I have a clear description of the need(s), I do not think a response 
can be described and quantified. Until then I do not think we can say if 
the IANA is the appropriate location and for what.

For example, I suppose that your voice reading support (not to enter the 
debate of the trans-  faux-amis) concerns primarily the mobile and mail 
reading in market terms. There are 3 billions phones and 1.3 billions 
mobiles. This makes 5.3 billions of probable users. We all understand that 
we need to discuss better to know how many times a year, a month, a day 
they may have to download 300 or 600.000 bytes?

Thank you.
jfc

---

From: "Addison Phillips" <[email-masked]>
Date: Fri, 1 Apr 2005 10:05:41 -0800

Some comments below.

Addison P. Phillips
Globalization Architect, Quest Software
Chair, W3C Internationalization Core Working Group

Internationalization is not a feature.
It is an architecture. 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of M.T. Carrasco Benitez
> Sent: jeudi 31 mars 2005 23:20
> To: [email-masked]
> Subject: RE: [Ltru] Re: Language transformation
> 
[Addison Phillips] 

(wearing the editor's hat for a second)

How should it be explained? Variants are already explained and cover orthographic variations and other things like transliteration. Specifically, section 2.2.9 defines what can be registered as a variant:

<q>
Dialect or other divisions or variations within a language, its orthography, writing system, regional variation, or historical usage may be registered as variant subtags. An example is the 'scouse' subtag (the Scouse dialect of English).
</q>

Would it be sufficient to modify this to say:

<q>
Dialect or other divisions or variations within a language, its orthography, writing system, regional or historical usage, transliteration, or other transformation or distinguishing variation may be registered as variant subtags. An example is the 'scouse' subtag (the Scouse dialect of English).
</q>

(Returning to private commentary)

Phrased another way, why should the RFC draft attempt to explain transmogrification any more than some other language variation? The draft as it sits is not very specific precisely to allow for a wide range of variation to be considered or proposed for registration.
> registered variant, I assume that one could have the following language
[Addison Phillips] 

Yes. Exactly so.

> as a variant. 
[Addison Phillips] 

Why? More specifically: what distinguishes a transliteration from some other variant registration request? How does the Subtag Reviewer or the ietf-languages list tell if it should or should not entertain a specific request? I understand the desire for a vetting process, but this RFC will need to explain what the rules are for registering variants clearly. If you go down the variant path, then someone quite clearly could (at least attempt to) register a transliteration variant without it being in the Transliteration Registry, perhaps by not calling it a transliteration. Conversely, whatever the process is for creating entries in the Transliteration Registry would affect the actions of the ietf-languages list and the Subtag Reviewer who are maintaining LTRU's registry. 

Since your work is in its very early stages, I don't think it mature enough to enshrine in the LTRU draft except as a possibility for registration of variants (as above). Perhaps if it is ready for 3066ter that would be an appropriate time to consider incorporating more formal, normative language related to transliterations and/or a registry thereof. For now you'll be better off if we DON'T mention it because your work will not be constrained by an existing RFC (and LTRU will be better off by not constraining its delivery date or actions artificially).

I should add that the ietf-languages community is pretty responsible. They have actively avoided, for example, creating subtags that would be grandfathered in 3066bis for nearly two years now. As long as progress on and the syntax of transliteration variants is clear, I would suspect that the registry would act 'de facto' according to the rules expected to be formalized in the near future regarding transliterations. The key is to define the structure, syntax, and other formalisms required for transliterations well enough so that no registrations are performed that cause problems for a transliteration registry in the future.

[Addison Phillips] > The Transliteration Registry is out of scope of LTRU.
[Addison Phillips] 

Yes, I agree, although the (non-)existence of that registry affects LTRU's ability to refer to it in a normative sense.

---

From: "Addison Phillips" <[email-masked]>
Date: Fri, 1 Apr 2005 10:16:08 -0800

Hmm...

Transmogrifications not mentioned in the Language Subtag Registry cannot be used in (valid) language tags. Having a separate registry is, indeed, useful. But it isn't a requirement for registering a variant and cannot practically be made into one until there is an RFC and registry for the other.

I suspect, based on past conversations with Mark and others, that transliteration may not be well-defined by a single, simple tag. It might be that this is wrong: I'm no expert. But it might not (and end up needing an extension after all). Therefore I think that trans*tions need some bake time to ensure we know what makes the most sense.

In my last email I proposed some language allowing for the possibility of registering trans*tion variants, but I don't think we're in a position to go beyond that in 3066bis. I don't see an ironclad requirement for handling these in language tags currently (to date NO registered values have even been proposed for indicating them), although I don't dismiss the need either. 

Is there something else we SHOULD do now? Or is my proposal enough, giving Tomas time to pursue his work?

Addison

Addison P. Phillips

> Behalf Of John Cowan
> Sent: vendredi 1 avril 2005 09:52
> To: Doug Ewell
> Cc: [email-masked]
> Subject: Re: [Ltru] Re: Language transformation
> 
> Doug Ewell scripsit:
> 
> > On the other hand, if these are implemented as variant subtags, there is
> > no need for a separate registry to provide further explanation or
> > description of the variants.  They are listed in the registry, with a
> > recommended prefix, and that is that.
> 
> This overstates the case.  In order to make practical use of such
> orthographies,
> they must be defined somewhere, and if the users of trans*tions find it
> useful to have a registry of these things, then LTRU has nothing to say
> about it.
> 
> After all, if you want to know what "nn" codes for, the registry will tell
> you "Nynorsk", but if you want to know what Nynorsk is, or how to
> recognize
> it, or how to spell-check it, you must look elsewhere.  At the moment
> there
> is no single "elsewhere" to look: this may be accounted good or bad.
> 
> --
> John Cowan  [email-masked]  www.ccil.org/~cowan
> www.reutershealth.com
> The competent programmer is fully aware of the strictly limited size of
> his own
> skull; therefore he approaches the programming task in full humility, and
> among
> other things he avoids clever tricks like the plague.  --Edsger Dijkstra
>

---

From: "Peter Constable" <[email-masked]>
Date: Fri, 1 Apr 2005 10:45:56 -0800

> From: [email-masked] [mailto:[email-masked]]
On Behalf Of
> Addison Phillips

> > The important issue is to be able to tag transliteration; it should
be
> > specifically

> Phrased another way, why should the RFC draft attempt to explain
transmogrification
> any more than some other language variation? The draft as it sits is
not very specific
> precisely to allow for a wide range of variation to be considered or
proposed for
> registration.

I agree that it would be good to mention transliteration, but not to
explain it, both because other comparable things aren't explained, and
because -- as Tomas as pointed out several times -- trying to come up
with an explanation that everyone would agree to is inviting polemic
debates.

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Fri, 1 Apr 2005 10:48:00 -0800

On Behalf Of

> Is there something else we SHOULD do now? Or is my proposal enough,
giving Tomas
> time to pursue his work?

I think your suggestion to mention transliteration as a possible kind of
variant, and perhaps something like zh-wadegile as a hypothetical
example is adequate for now.

Peter Constable

---

From: "M.T. Carrasco Benitez" <[email-masked]>
Date: Tue, 5 Apr 2005 10:29:46 +0100 (BST)

Back from a long weekend ... 

Comments to posting by Cowan, Phillips, Constable and Ewell

-----> 1) In short

- Mentioning
It is enough just to mention transliteration with an example. No need to explain it.

- Transliteration Registry
A separated "thing" (RFC, orgazition, extension, whatever) should take care of
registering the transliteration; so one can find meaning to the tag.
The "thing" will have to follow the rules in 3066bis; e.g., max size, availability
of the subtag, variant registration procedure, etc. Hopefully (no guarantee), a
transliteration tag already in the "thing" should be easier to register in 3066bis.
I am working in this.

-----> 2) In long

>[Cowan]
>This overstates the case.  In order to make practical use of such orthographies,

Yes.

>[Phillips]

><q>
>Dialect or other divisions or variations within a language, its orthography, writing system, 
>regional or historical usage, transliteration, or other transformation or distinguishing
>variation may be registered as variant subtags. An example is the 'scouse' subtag
>(the Scouse dialect of  English).
></q>

Yes. With example as suggested by Phillips below.

>Phrased another way, why should the RFC draft attempt to explain transmogrification any more than

>some other language variation? The draft as it sits is not very specific precisely to allow for a

>wide range of variation to be considered or proposed for registration.

It should not try to explain it. Just to mention it, so people know how to mark transliteration. 

[Constable]
>with an explanation that everyone would agree to is inviting polemic

Yes.

[Phillips]
>example is adequate for now.

Yes.

[Ewell]
>What concerns me is the idea of a Transliteration Board that maintains
>its own registry of transliterations, and tells the Language Subtag
>Reviewer and ietf-languages which variant subtags to register, based on
>what they have already approved for their registry.
>
>If Tomas wants a separate organization to be responsible for vetting
>these, he should pursue the extension-RFC path, which does exactly that,
>but requires the use of a singleton such as -t-.  If he wants to be able
>to write "zh-wadegile" without the -t-, then he is talking about
>variants, and those will be proposed and debated and approved (or
>rejected) using the same process that will apply to all other variants.

Yes.

Regards
Tomas

---

From: "Addison Phillips" <[email-masked]>
Date: Tue, 5 Apr 2005 09:03:52 -0700

These changes are going into the next editor's copy right now, along with record-jar format and should be visible shortly for review.

Addison

Addison P. Phillips

> Sent: mardi 5 avril 2005 02:30
> To: [email-masked]
> transmogrification any more than
>

---

From: "Addison Phillips" <[email-masked]>
Date: Wed, 6 Apr 2005 09:27:47 -0700

Apparently this wasn't explained simply enough the last few times?

XML, HTML, CSS, CLDR, LDAP, XSLT, J2EE, .NET are all applications that use RFC 3066 language tags. This is not an exhaustive list. Some of the items are proprietary in nature and others are open standards widely implemented.

NONE of them need to be referenced in RFC 3066bis: they are examples of (potential) consumers of that document, not the other way around.

Addison

Addison P. Phillips

> Behalf Of JFC (Jefsey) Morfin
> Sent: mardi 5 avril 2005 21:07
> To: Randy Presuhn; [email-masked]
> At 05:51 06/04/2005, Randy Presuhn wrote:
> >This kind of remark obviously make me react. Whose plan: IETF, IESG, IAB?
> >
> >Obviously not, since CLDR is a Unicode project.
> >http://www.unicode.org/cldr/
> >
> > > 1. this CLDR plan should be documented in the Draft as per Charter. At
> > > least to explain why it is not supported now.
> >...
> >
> >There is no such requirement in the charter.  The charter mentions CLDR
> as
> >one of the users of RFC 3066.
> 
> I dont think this is correct. The CLDR is not documented by any RFC. This
> is a copyrighted private project. There are other projects with comparable
> aims and different needs. The needs of the CLDR are parts of the needs
> calling a the BPC 047 review. In not quoting CLDR the current draft
> implies
> that the CLDR needs are satisfied what puts it ad advantage over other
> projects or projects studies the needs of which are not satisfied.
> 
> I fully understand that CLDR IPRs should be respected, the same as other
> projects. But the Internet standard process calls for protected solutions
> to be considered after open ones.
> jfc
> 
>