Subject: [Ltru] Re: Identifying script (or global) variants

From: "Doug Ewell" <[email-masked]>
Date: Sat, 17 Feb 2007 10:27:14 -0800

Martin Hosken <martin underscore hosken at sil dot org> wrote:

> In my unstinting desire to be able to separate out script and language 
> from a language tag, it would be really nice if there was some 
> non-semantic mechanism by which a variant that is being used to modify 
> a script (and therefore has no constraining language in its 
> specification) may be identified over a language specific variant 
> (which may be to do with script, but is langauge specific).

For reference, here's a brief summary of the existing variant subtags 
and the type of variation indicated by each.

Difference in writing system (major):
    Language-independent:  fonipa, fonupa
    Language-dependent:  monoton, polyton

Difference in orthography (minor):
    1901, 1996

Dialect:
    arevela, arevmda. boont, nedis, rozaj, scouse

Note that "major" and "minor" are intended to be vaguely descriptive, 
not value judgments.   I considered the difference between "ß" and "ss" 
to be less major than the difference between traditional alphabets and 
purpose-built phonetic alphabets.  I really, really don't want to reopen 
the question of whether IPA is a script!  These categories may be 
merged, or divided along different lines.

> There are numerous ways this could be done, for example by restricting 
> them to start with a specific letter or letters, or somesuch. 
> Whichever way that may result I see no problem in having to change 
> standards since it is merely a constraint on variant naming that can 
> be added to the standard after the fact if necessary. No change to the 
> regexp or anything that has gone before.

I see two problems.  First, it would be very difficult to assume that 
all future variants can be neatly boxed into one of these categories. 
It's possible that one dialect of a language is commonly used with one 
writing system or orthographic convention, and another is used with 
another, and for practical purposes the dialect issue and the 
writing-system issue cannot be separated.

Second, we already have the 12 variants listed above, and in order to 
institute a syntactic distinction (assuming such is practical) we would 
have to deprecate at least some of them and replace them with new, 
conformant subtags.  That brings with it the usual problem of 
deprecation—the old subtags still exist and must still be supported by 
software.

> Is there any hope that I may be able to write code that can truly 
> separate language and script on this basis or should I give up now and 
> resign myself to having to track the latest repository and rerelease 
> my code every so often to reflect that?

My opinion is that variants by their nature are a bit of a catch-all 
category, and it will always be necessary to update software to some 
extent to reflect changes in the Registry.  You will need to do that 
anyway if you wish to make semantic use of the subtags (for example, 
switching to an IPA-enabled font upon seeing the "fonipa" subtag).

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Martin Hosken <[email-masked]>
Date: Tue, 20 Feb 2007 08:53:36 +0700

Dear Doug,

> Difference in orthography (minor):

Thanks for this list. Hmm. Yes I probably do need to include the
language specific orthography/script variants, but not the dialect ones.
> not value judgments.   I considered the difference between "ß" and
> "ss" to be less major than the difference between traditional
> alphabets and purpose-built phonetic alphabets.  I really, really
> don't want to reopen the question of whether IPA is a script!  These
> categories may be merged, or divided along different lines.

Neither do I want to reopen that debate. But I do have some basic
questions that I hope you can help with. Let's take the example of
scouse English in IPA. How would I tag this?

en-Latn-scouse-fonipa
en-Latn-fonipa-scouse
en-fonipa-scouse
en-scouse-fonipa

Now if I were to want to get hold of any scouse text in IPA, what should
my RFC 4647 filter be?

en-*-fonipa-scouse
en-*-scouse-fonipa

do I really need to have two filters for this?

> switching to an IPA-enabled font upon seeing the "fonipa" subtag).

Given the number of orthography variants in languages around the world,
I wonder what the best way to handle them all is. Do we (I'm thinking
SIL with its involvement in so many languages and linguists with their
test orthographies, etc.) need to submit every single one, or would it
help you guys if we were to internally register them (perhaps using a
singleton based extension) and then forward the interesting ones to you?

Yours,
Martin

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 19 Feb 2007 17:59:11 -0800

That's a good question. One option, not sure about it thought, is to have
the canonical form put variants in alphabetical order. That way at least the
result would be predictable.

Mark

-- 
Mark

---

From: Martin Duerst <[email-masked]>
Date: Tue, 20 Feb 2007 12:13:52 +0900

>do I really need to have two filters for this?

As Mark said, an interesting question, and one that we haven't
yet addressed, probably mostly just because we didn't have that
many variants yet.

Using alphabetical order would be one solution, but the results
may look a bit too unnatural. Using 'major before minor' would
work in general, but in many cases, there will be different opinions.

Should we change RFC 4647 to say that the order of variants doesn't
count? Are there other aspects of our syntax where the same problem
arises? It seems to happen at least for extensions.

>singleton based extension) and then forward the interesting ones to you?

If you want to use a singleton extension, then you have to apply
for it, andReceived: from [10.91.34.44] (helo=ietf-mx.ietf.org)
	by megatron.ietf.org with esmtp (Exim 4.43) id 1HJLZs-0001CE-9R
	for [email-masked]; Mon, 19 Feb 2007 22:22:04 -0500
Date: Tue, 20 Feb 2007 12:13:52 +0900
To: Martin Hosken <[email-masked]>, Doug Ewell <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: [Ltru] Re: Identifying script (or global) variants
Cc: LTRU Working Group <[email-masked]>

>do I really need to have two filters for this?

As Mark said, an interesting question, and one that we haven't
yet addressed, probably mostly just because we didn't have that
many variants yet.

Using alphabetical order would be one solution, but the results
may look a bit too unnatural. Using 'major before minor' would
work in general, but in many cases, there will be different opinions.

Should we change RFC 4647 to say that the order of variants doesn't
count? Are there other aspects of our syntax where the same problem
arises? It seems to happen at least for extensions.

>singleton based extension) and then forward the interesting ones to you?

If you want to use a singleton extension, then you have to apply
for it, and once you applied, you would keep you own registry.
There would not be a need to forward interesting ones to the
general registry, as everybody can take the variants from your
registry if they need them. The problem might be that the
singleton extensions were imagined for 'by-topic' extensions,
and your registry could end up with all kinds of extensions.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]     

 once you applied, you would keep you own registry.
There would not be a need to forward interesting ones to the
general registry, as everybody can take the variants from your
registry if they need them. The problem might be that the
singleton extensions were imagined for 'by-topic' extensions,
and your registry could end up with all kinds of extensions.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Martin Hosken <[email-masked]>
Date: Tue, 20 Feb 2007 13:32:40 +0700

Dear Martin,

> work in general, but in many cases, there will be different opinions.

Thinking a little more about it. I think part of the issue arises from
the different domains that variants can fall into. In the scouse in IPA
example, we are dealing with two variants from two different domains:
dialect and script. I don't think it matters what the relative order of
the domains is and there is no way to do major vs minor between domains.
So sorting alphabetically is as good as any. This should probably apply
to extensions too.

Even if you went for a domain based approach where do you draw the line:

de-1901-fonipa

Language tags describe 3 things: language, script and region. Therefore
variants can modify one or more of those parameters. Given that regional
variants are handled via other regions (AFAIK). That leaves us with
language, script, language+script (=orthography). But I'm not sure if
that helps us any.

Yours,
Martin

---

From: Martin Hosken <[email-masked]>
Date: Tue, 20 Feb 2007 13:34:42 +0700

Dear All,

I realise I may have slipped in 2 questions here that I was after
asking. In addition to knowing which order variants can occur, I'm also
not sure whether Latn is required since en has a Latn suppress script on
it. Does adding fonipa require that the Latn reappear or can it remain
suppressed?
>     scouse English in IPA. How would I tag this?

TIA,
Martin

---

From: Martin Duerst <[email-masked]>
Date: Tue, 20 Feb 2007 15:52:31 +0900

>it. Does adding fonipa require that the Latn reappear or can it remain

Thinking purely semantically, I think Latn can be even more supressed
once fonipa is added :-). In other words, Latn might be suppressed, even
on lanugages that otherwise don't suppress it, when fonipa is added.
As an example, sr (assuming it means Serbian) would definitely need
sr-Latn to say it's written in Latin, but sr-fonipa would be as good
as sr-Latn-fonipa because fonipa implies Latn.

Regards,     Martin.

---

From: Peter Constable <[email-masked]>
Date: Mon, 19 Feb 2007 22:53:53 -0800

The way things are now, strictly speaking, both cho-fonipa and cho-Latn-fonipa (intentionally picking a non en example for a moment) are equally valid. I was a little surprised -- I guess I was thinking fonipa would have specified a prefix of Latn. Mind you, if Latn was specified as a prefix for fonipa, then there'd be a bit of confusion in the case of some languages, like en, since they specify "Suppress-script: Latn". So, at present, it seems like the recommendation implied by the current sub-tag registry and RFC 4646 (for good or bad) is *not* to include Latn.

Peter Constable

-----Original Message-----
From: Martin Hosken [mailto:[email-masked]]
Sent: Monday, February 19, 2007 10:35 PM
To: LTRU Working Group
Subject: Re: [Ltru] Re: Identifying script (or global) variants

Dear All,

TIA,
Martin

---

From: "Doug Ewell" <[email-masked]>
Date: Mon, 19 Feb 2007 23:02:05 -0800

Martin Duerst <duerst at it dot aoyama dot ac dot jp> wrote:

> Using alphabetical order would be one solution, but the results may 
> look a bit too unnatural. Using 'major before minor' would work in 
> general, but in many cases, there will be different opinions.

People will differ on which variants are more "major" than others. 
That's a certainty.

I don't think it's reasonable to talk about "canonicalizing" tags by 
putting the variants in alphabetical order.  That would put "boont" 
before "fonipa", bot "fonipa" before "scouse", which is completely 
arbitrary.

It could also separate prefixes that include other variants.  Let's say 
you have a variant "socal" that identifies the Southern California 
dialect of English (like, y'know, fer sure dude) and suppose you have 
another variant "orange" that identifies the Orange County variant of 
"en-socal" and has "en-socal" as its only Prefix.  (I don't know of any 
uniquely Orange County dialect, but play along.)  "Canonicalizing" the 
variants in alphabetical order would give "en-orange-socal", destroying 
the prefix relationship.  Setting the rules of canonicalization to take 
prefixes into account, on the other hand, would be confusing in the 
extreme, and nobody outside this list would implement it correctly.

> arises? It seems to happen at least for extensions.

Yes, tag matching should allow variants to appear in any order.

Unlike variants, canonicalizing extension singletons by alphabetical 
order makes perfect sense, because extension singletons are 
semi-arbitrary one-letter identifiers and different classes of 
extensions (identified by different singletons) aren't expected to be 
interrelated.

> If you want to use a singleton extension, then you have to apply for 
> it, and once you applied, you would keep you own registry. There would 
> not be a need to forward interesting ones to the general registry, as 
> everybody can take the variants from your registry if they need them. 
> The problem might be that the singleton extensions were imagined for 
> 'by-topic' extensions, and your registry could end up with all kinds 
> of extensions.

I was a longtime advocate of creating an extension for transcriptions 
and transliterations, including phonetic alphabets.  Now that we have 
assigned variants for that purpose, however, it would be unwise to go 
back and create the extension and deprecate the subtags we just created. 
That is the complete opposite of stability, and it makes us look like we 
are just shooting in the dark.

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 20 Feb 2007 07:24:01 -0800

I disagree. We don't have any provision for the ordering of variants meaning
something different in 4646, and until such time as we do, there is no
purpose to allowing arbitrary distinctions by ordering according to the whim
of the composer.

Now, we should be willing to consider a non-arbitrary ordering, but we would
have to build it in to the registry. A simple model would be:

In the record for the variant, have a Order field, with values from 0 to
999. If Order does not occur, then the value is defaulted to 500. The
canonical order for variants is by ascending Order.

Then the "more significant differences" (like polyton) could be given
precedence.

Mark

---

From: John Cowan <[email-masked]>
Date: Tue, 20 Feb 2007 10:52:01 -0500

> canonical order for variants is by ascending Order.

Way way overkill.

> Then the "more significant differences" (like polyton) could be given

Y'know, I deny that polytonic is a "more significant" difference at all.
The difference in legibility is nearly nil; you can train someone who can
read one mode to read the other in about five minutes.  Writing polytonic
if you only know monotonic is harder, but that's because polytonic has
not been functional in Greek for millennia; at the moment it was invented,
indeed, it was already obsolete.

There are minor technological issues around polytonic rendering, and of
course the political significance is huge.  If it weren't for the latter
point, it would be about as worthwhile giving it a subtag as coding the
difference between "Romanian with 's with comma below'" and "Romanian with
's with cedilla'".

(All this is a specimen worm from the can that you open if you try to
decide in any formal way the relative significance of variant subtags.)

-- 
It was dreary and wearisome.  Cold clammy winter still held way in this
forsaken country.  The only green was the scum of livid weed on the dark
greasy surfaces of the sullen waters.  Dead grasses and rotting reeds loomed
up in the mists like ragged shadows of long-forgotten summers.
        --"The Passage of the Marshes"          http://www.ccil.org/~cowan

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 20 Feb 2007 08:02:28 -0800

Overkill compared to what? Please be more explicit.

My main point was that we have a problem with multiple variants, because the
ordering doesn't make a difference in 4646, but we don't canonicalize to any
order. The simplest way to do that is alphabetical, which I think is fine.
Doug wanted something different, so I sketched what something could look
like. But I'm fine with alphabetical.

Mark

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 20 Feb 2007 17:29:36 +0100

Mark Davis wrote:

> I'm fine with alphabetical.

I'm not for the reason stated by Doug:  There can be a variant of a
variant.  Constructed example:  The 'coastal' dialect of 'valencia',
with prefix ca-valencia, results in ca-valencia-coastal.  You can't
swap that saying ca-coastal-valencia without giving up on the prefix.

The concept of "variant of variant" isn't used yet, but it could be.

For independent variants let folks just pick an order reflecting their
idea of importance, tags are truncated right to left if necessary.

Frank

---

From: John Cowan <[email-masked]>
Date: Tue, 20 Feb 2007 11:36:16 -0500

> Overkill compared to what? Please be more explicit.

Compared with any other option, including doing nothing.

> My main point was that we have a problem with multiple variants,

I'm willing to be convinced that we have a problem, but I don't see
any problem yet.

> because the ordering doesn't make a difference in 4646,

There is no statement to that effect in 4646, which simply says that
multiple variants are allowed, without saying that order matters
or doesn't matter semantically.  (Order does, of course, matter
to the algorithms in 4647.)

> but we don't canonicalize to any order.

If order matters in some particular case, we provide for it by encouraging
the use of a Prefix: field on the secondary variant that specifies the
appropriate primary variant(s).  Otherwise, I don't see any reason to
fix what isn't broken.

-- 
Values of beeta will give rise to dom!          John Cowan
(5th/6th edition 'mv' said this if you tried    http://www.ccil.org/~cowan
to rename '.' or '..' entries; see              [email-masked]
http://cm.bell-labs.com/cm/cs/who/dmr/odd.html)

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 20 Feb 2007 10:48:43 -0800

Since people's understanding of the text differs from yours, we have at
least a problem in clarity of the text.

I think we do have a problem. Not a huge one, since the variants are little
used, but insofar as they are used, someone looking at the text of RFC and
the registry should be able to tell whether
en-VARIANT1-VARIANTA
is the same as
en-VARIANTA-VARIANT1
or not.

   - If they are not intended to be the same, then what is the difference
   between them? Which one should be used where?
   - If they are intended to be different, then the canonicalization
   should be the same, so that spurious differences can be erased in
   comparison.

If the standard is silent on these issues, then we just have a muddle.

Mark

---

From: Addison Phillips <[email-masked]>
Date: Tue, 20 Feb 2007 12:02:14 -0800

Mark Davis wrote:
 > I think we do have a problem. Not a huge one, since the variants are
 > little used, but insofar as they are used, someone looking at the text
 > of RFC and the registry should be able to tell whether
 > en-VARIANT1-VARIANTA
 > is the same as
 > en-VARIANTA-VARIANT1
 > or not.
 >

This is a reason why general variants were hotly debated in the 
"3066bis" era. Specific variants, such as the German orthographic 
variations, tend to have clearly defined interactions. In fact, they are 
defined in ways that prevent usage together and Section 2.2.5 says:

"Most variants that share a prefix are mutually exclusive. ... A variant 
that can meaningfully be used in combination with another variant SHOULD 
include a 'Prefix' field in its registry record that lists that other 
variant."

The general purpose variants are problematic because they introduce this 
sort of ordering problem where none exists with specific variants.

The problem in this thread is that it is unclear if a given general 
variant is "wider" or "narrower" than a given specific variant. RFC 4647 
basically says: put your subtags in order from widest to narrowest. For 
some, IPA seems wider than scouse. For others, the reverse is true.

One solution would be to specify that general variants "SHOULD" appear 
before specific variants:

    en-fonipa-scouse

While it is true that multiple general variations might occur in a 
single tag, that tag will border on incomprehensibility. Personally, I 
think we should have banned general variants and should have sent the 
transcription folks off to write an extension (so I could be free to 
ignore it).

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 20 Feb 2007 14:10:40 -0800

In that case, I'd make canonicalization put all general variants before all
specific variants, and within those two groups, put them in alphabetical
order.

>Personally, I

Personally, I think that the transcriptions are reasonable. Had we known
that the Script RA was going to  exclude them (in my opinion, for no good
reason, given such goofiness as Latf), we would have reserved some subtags
for registering them in the script position, like 4 characters starting with
a letter and ending with a digit.

Mark

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 21 Feb 2007 16:16:41 +0100

Addison Phillips wrote:

> Personally, I think we should have banned general variants and should
> have sent the transcription folks off to write an extension (so I could
> be free to ignore it).

+1