Subject: [Ltru] Re: Macrolanguages, countries & orthographies

From: "Mark Davis" <[email-masked]>
Date: Tue, 13 Feb 2007 10:23:54 -0800

> I wonder if there is, or is going to be, a public forum to discuss ISO
> 639-3 and its structure and policies.  This list really isn't it; we
> take what the external standards give us, and synthesize and arrange.
> We don't directly influence what the external standards do, though there
> are list members who can do so indirectly.

That would be very useful. We are very dependent on the interpretation of
the codes. In particular, there is a key item that we need a response for,
and that is the temporal scope of ISO language codes. For example, we have
french broken up into fr, frm, and fro, plus various creoles. So  tagging
old French data with fr is incorrect, and requesting a variant subtag to
apply to fr to indicate French of say (1100-1200) is out of scope and will
be legitimately rejected.

For most other languages, on the other hand, we do not have that granularity.
So that means what a document that is tagged with cs (Czech) represents is
completely unspecified. We do not know what the !#$@ to do with a request.
We are faced with an unknown situation which the ISO committee doesn't
respond to.

Assume that old Czech is as different from modern as fro is from fr. There
are the following two alternatives:

   1. The subtag 'cs' means any Czech, modern or old, and therefore
      1. cs can legitimately be used to tag a document with old Czech.
      2. a request for a variant-subtag applying to cs indicating old
      Czech is perfectly reasonable, and SHOULD be granted.
      3. ISO can only add another subtag that means old Czech if cs is
      treated as a metalanguage, since otherwise that would invalidate old
      subtags.
      2. The subtag 'cs mean only modern Czech, and therefore
      1. cs cannot be correctly used to tag a document with old Czech,
      so the only recourse is to use a private use code or 'und'.
      2. a request for a variant-subtag applying to cs indicating old
      Czech is illegitimate, and MUST not be granted.
      3. ISO can and should add a different subtag that means old
      Czech

I've requested a response from the JAC (through Peter Constable on Jan 11)
to clarify this, but have gotten nothing. His opinion was number 2, which
seems reasonable, but we really have to know what the situation is for RFC
4646bis!

Mark

-- 
Mark
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "CE Whitehead" <[email-masked]>
Date: Tue, 13 Feb 2007 15:22:29 -0500

>>are list members who can do so indirectly.
+1

_________________________________________________________________
Search for grocery stores. Find gratitude. Turn a simple search into 
something more. 
http://click4thecause.live.com/search/charity/default.aspx?source=hmemtagline_gratitude&FORM=WLMTAG

---

From: "Don Osborn" <[email-masked]>
Date: Tue, 13 Feb 2007 16:48:09 -0500

If a new forum on ISO-639 is in the offing it might be worth giving it the scope of all of ISO-639 and not just part 3, though that is where a lot of the discussion is at the moment. If 639 is a system of several parts then inevitably discussion will involve the others; all the more with the eventual arrival of 5 & 6 and recalibration (or reinvention?) of 2.

On the other hand, I have to say that as one who only relatively recently joined ietf-languages & ltru that I would almost prefer recasting one of those to cover ISO-639. Or even to have a new larger language coding list be mandated to replace the set (hopefully that is not too heretical). *Another* list to follow is, well, yet another list to follow. And inevitably it will be cc’d on many messages to one or more of the other lists anyway. 

Just thought… 

Don

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Tuesday, February 13, 2007 1:24 PM
To: Doug Ewell
Cc: [email-masked]; [email-masked]
Subject: Re: Macrolanguages, countries & orthographies

That would be very useful. We are very dependent on the interpretation of the codes. In particular, there is a key item that we need a response for, and that is the temporal scope of ISO language codes. For example, we have french broken up into fr, frm, and fro, plus various creoles. So  tagging old French data with fr is incorrect, and requesting a variant subtag to apply to fr to indicate French of say (1100-1200) is out of scope and will be legitimately rejected. 

For most other languages, on the other hand, we do not have that granularity. So that means what a document that is tagged with cs (Czech) represents is completely unspecified. We do not know what the !#$@ to do with a request. We are faced with an unknown situation which the ISO committee doesn't respond to. 

. . .

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 13 Feb 2007 16:22:00 -0800

Saying that it is not as important is, I agree, your prejudice. Importance
is in the eye of the beholder, and ISO 639-3 has 7,500 languages, which make
distinctions that to people concerned with Czech will be far less important
than the difference between old Czech and modern Czech.

Moreover, one cannot fixate on the exact example used. There are plenty of
others, because very few languages have "Old" variants in 639-3. The
principle is the same for any other language: do we presume that the code
means only the modern variant, or covers all historical variations? We need
to get an answer for that; without that answer, we can't know whether to
accept or reject historic variant proposals.

Mark

On 2/13/07, Lars Aronsson <[email-masked]> wrote:
>
> Mark Davis wrote:
>
> > Assume that old Czech is as different from modern as fro is from fr.
>
> But is this a real problem?  How much total literature is written
> and available in different variations of Czech?  My prejudice says
> that as a nation with a language and literature of its own, Czech
> is about as young as Finnish, Norwegian or Serbian, i.e. 19th
> century.  Can you give any concrete examples when not having a
> separate *code* for pre-renaissance Czech is a practical problem?
>
> Linguists of course have *names* for Swedish of all ages, but I
> see no real use for having ISO or the IETF specify language
> *codes*.  I could be wrong, but if so please enlighten and correct
> me.  Nobody is going to translate OpenOffice or Mozilla to the
> language spoken by vikings (Old Norse) or the Swedish used during
> the Lutheran reformation (called New Swedish, ironically).
>
> Yes, there is now a branch of Wikipedia in Old English
> (ang.wikipedia.org), but that is a rare exception.  I don't expect
> this to happen in other languages.  Ang has now 744 articles,
> compared to the 11,000 articles of the Latin Wikipedia.
>
> I'm scanning old books, and I'm starting to see a practical
> problem with different orthographies and spelling reforms, similar
> to those addressed with the IETF defined codes for German de-1901
> and de-1996.  Analogous to these codes, we could perhaps find use
> for sv-1801, sv-1889, sv-1906, da-1775, da-1892 and da-1948,
> because we now have *significant amounts* of text online in each
> of these language versions. But before 1775/1801 the orthography
> of Swedish and Danish varies so heavily with each work, that it
> becomes pretty much useless to try to identify more versions.
> And before that time, there is also so small amounts of literature
> available, that any automatic processing becomes insignificant.
>
>
>
> --
>   Lars Aronsson ([email-masked])
>   Aronsson Datateknik - http://aronsson.se
> Ietf-languages mailing list
> [email-masked]
> http://www.alvestrand.no/mailman/listinfo/ietf-languages
>

---

From: "Debbie Garside" <[email-masked]>
Date: Wed, 14 Feb 2007 02:17:32 -0000

Mark wrote:
 
>> The principle is the same for any other language: do we presume that the
code means only the modern variant, or covers all historical variations? We
need to get an answer for that; without that answer, we can't know whether
to accept or reject historic variant proposals. 
 
My personal opinion is that ISO 639-3 subtags cover the "whole language" as
described; all of the language, every part of the language, written and
spoken and... historical.  Even when there is an ISO 639-3 historical subtag
that covers part of it.
 
My advice, accept urgent proposals for historic variants on the basis that
they will be deprecated when ISO 639-6 comes into being - assuming it is
incorporated within RFC4646bis or ter.  Inform proposers of such variants
that ISO 639-6 is currently being designed and if the need is not urgent
delay until ISO 639-6 is published.
 
I think this group needs to make a decision wrt ISO 639-6.  
 
Best regards
 
Debbie

  _____  

From: Mark Davis [mailto:[email-masked]] 
Sent: 14 February 2007 00:22
To: Lars Aronsson
Cc: [email-masked]; LTRU Working Group
Subject: [Ltru] Re: Macrolanguages, countries & orthographies

Mark

Yes, there is now a branch of Wikipedia in Old English
( ang.wikipedia.org <http://ang.wikipedia.org> ), but that is a rare
exception.  I don't expect
this to happen in other languages.  Ang has now 744 articles,
compared to the 11,000 articles of the Latin Wikipedia.

--
  Lars Aronsson ([email-masked]  <mailto:[email-masked]> )
  Aronsson Datateknik - http://aronsson.se
_______________________________________________
Ietf-languages mailing list
[email-masked]  <mailto:[email-masked]> 
http://www.alvestrand.no/mailman/listinfo/ietf-languages

-- 
Mark

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 13 Feb 2007 18:50:33 -0800

1. Your personal opinion contradicts what I heard as a rough consensus
earlier, that fr is not a superset of fro or frm. The highest priority is
that the interpretation be well-defined and consistent; I think your opinion
would be reasonable, but it doesn't seem to be the consensus.

2. If you asked me right now up-or-down on ISO 649-6, I'd say absolutely
not, since (a) it would introduce all kinds of duplicate encodings, and (b)
there has been no clear rationale given that the other information is worth
adding.

For example, the second two of these, from your example, would be
duplicates.
kca     obgc    Khanty
kcal    kcaw    Khanty Written Latin Script
kcac    kcaw    Khanty Written Cyrillic Script

The only justification I've seen is that " Needless to say, I think
there is a need for ISO 639-6 within this RFC.  There is a need for
variants.  There is a need for a hierarchical system that will facilitate
the creation of pick lists; 7,200 entities are more than a little unwieldy
for the end user when setting language preferences!"

But as per earlier messages, end users have very little idea of language
taxonomies. A far better interface for narrowing choices would be one that
lets them pick a country, and then languages in use in the country. But all
of that is beside the point -- UI is *not* the goal of BCP 47.

A hierarchy of languages may well be useful in some circumstances, but it is
orthogonal to the requirements of BCP 47.

Mark

On 2/13/07, Debbie Garside <[email-masked]> wrote:
>
>  Mark wrote:
>
> >> The principle is the same for any other language: do we presume that
> the code means only the modern variant, or covers all historical variations?
> We need to get an answer for that; without that answer, we can't know
> whether to accept or reject historic variant proposals.
>
> My personal opinion is that ISO 639-3 subtags cover the "whole language"
> as described; all of the language, every part of the language, written and
>  ------------------------------
> *From:* Mark Davis [mailto:[email-masked]]
> *Sent:* 14 February 2007 00:22
> *To:* Lars Aronsson
> *Cc:* [email-masked]; LTRU Working Group
> *Subject:* [Ltru] Re: Macrolanguages, countries & orthographies
>
> > ( ang.wikipedia.org), but that is a rare exception.  I don't expect
> >   Lars Aronsson ([email-masked] )

---

From: "Debbie Garside" <[email-masked]>
Date: Wed, 14 Feb 2007 03:13:18 -0000

Mark wrote:

Thank you.  May be we should work on making it consensus.
 
2. If you asked me right now up-or-down on ISO 649-6, I'd say absolutely
not, since (a) it would introduce all kinds of duplicate encodings, 
 
I believe we worked out a system of flagging the ISO 639-6 data in order
that duplicate encodings would not be introduced within the Registry; a
simple procedure which is currently being actioned by GeoLang (compilers of
the ISO 639-6 data).
 
and (b) there has been no clear rationale given that the other information
is worth adding. 
 
Hmmm... I thought we were having a discussion on variants and historical
variants, something that is facilitated by the proposed ISO 639-6 code.

[snip]But as per earlier messages, end users have very little idea of
language taxonomies. A far better interface for narrowing choices would be
one that lets them pick a country, and then languages in use in the country.
But all of that is beside the point -- UI is *not* the goal of BCP 47.
 
But if one can have both... a faceted approach... what a winner :-)  BTW
FYI, ISO 639-6 data is also linked to ISO 3166-1 and ISO 15924.  I believe
we have had the discussion wrt numbers of language selections in some
countries but I can certainly see that in some cases this would be
preferable.  Tell me, hypothetical scenario, what would you recommend a user
to do when their language is not listed under their country of residence as
there are only two speakers of Khanty residing in the UK?  Just a thought.  
 
The biggest problem on the score of pick lists is that language names and
country names in the users language are required; linked to the ISO
639/3166-1 code. 
 
Best 
 
Debbie

---

From: John Cowan <[email-masked]>
Date: Tue, 13 Feb 2007 22:36:37 -0500

Mark Davis scripsit:

> 2. If you asked me right now up-or-down on ISO 639-6, I'd say absolutely
> not, since (a) it would introduce all kinds of duplicate encodings, and
> (b) there has been no clear rationale given that the other information
> is worth adding.

Let me reiterate my proposal on 693-6: discard all the codes except
those in the following two classes:

A) Code elements which represent collections and are not subordinate to
any 639-2 collection code element.

B) Code elements which represent variants that are subordinate to code
elements that are equivalent to some language-script-region 4646 tag.

Class A code elements are made into 4-letter initial subtags (currently
reserved).  Class B code elements are made into variant subtags by
prepending a '6' (not '6-') to the code element.  This is artificial,
but it satisfies the 4646 syntax.

> kcac    kcaw    Khanty Written Cyrillic Script

Indeed, and none would be added under my proposal, since they are
equivalent to "kca", "kca-Latn", and "kca-Cyrl" respectively.

> A hierarchy of languages may well be useful in some circumstances,
> but it is orthogonal to the requirements of BCP 47.

Indeed.  I don't see any need for the hierarchical information
in RFC 4646bis or -ter.

-- 
Even a refrigerator can conform to the XML      John Cowan
Infoset, as long as it has a door sticker       [email-masked]
saying "No information items inside".           http://www.ccil.org/~cowan
        --Eve Maler

---

From: "David Starner" <[email-masked]>
Date: Wed, 14 Feb 2007 07:06:05 -0600

> spoken and... historical.  Even when there is an ISO 639-3 historical subtag

As a user of en, enm and ang, I don't like that one bit. fr and en are
more mutually intelligible then ang and en, and I don't see any use in
labelling ang as en. Furthermore, if ang can validly be labeled en, it
can also be validly labeled sco, adding another layer of complexity.
Can la really be tagged as any Romance language? Can dum (Middle
Dutch) be tagged as af (Afrikaans)? Can 17th century Dutch be tagged
as af (Afrikaans)?

> Inform proposers of such variants

It's one thing to wait for ISO 639-3, which is clearly available in a
late draft, but something that is "currently being designed" is not
something I feel it's reasonable to expect people to wait for.

---

From: "Debbie Garside" <[email-masked]>
Date: Wed, 14 Feb 2007 15:27:43 -0000

David Starner wrote:

> As a user of en, enm and ang, I don't like that one bit. fr 
> and en are more mutually intelligible then ang and en, and I 
> don't see any use in labelling ang as en.

But I see people who are looking for a language subtag to denote Old English
using English as a starting point in a hierarchical system such as ISO
639-6; makes sense to me.

> Furthermore, if ang 
> can validly be labeled en, it can also be validly labeled 
> sco, adding another layer of complexity.

I note Ethnologue have classified sco under English!  But you are right, to
be able to link languages by mutual intelligibility requires a
multi-parent/child relationship system.   That's not what I am advocating at
the moment (emphasis on the word moment :-)).  

> It's one thing to wait for ISO 639-3, which is clearly 
> available in a late draft, but something that is "currently 
> being designed" is not something I feel it's reasonable to 
> expect people to wait for.

We have waited some 5 years for ISO 639-3.  I think for the last year people
have been holding off on registration requests because of it.  ISO 639-6 is
due for publication in early 2008 but I don't think people should have to
wait if their need is urgent; register variants via RFC4646.  I have no
problem with that.

Best regards

Debbie

---

From: "David Starner" <[email-masked]>
Date: Wed, 14 Feb 2007 11:45:42 -0600

> the moment (emphasis on the word moment :-)).

My comment had nothing to do with mutual intelligibility. There is no
theoretic reason to prioritize English (en) over Scots (sco) as
including ang and enm. They both descended from enm and ultimately
ang. Likewise, there's no reason to prioritize modern Dutch over
Afrikaans as covering Middle Dutch (dum); it is equally valid to
consider Middle Dutch a historical form of one as the other.

> wait if their need is urgent; register variants via RFC4646.  I have no

Until I can see the draft standard, I'm not going to worry about it.
I'll hold off on ISO 639-3 only because I can have it in my hands
right now. I don't think it's reasonable to expect people to hold off
for a standard that's expected to be published in a year and that they
can't check today to see if it fits their needs; I don't even know if
ISO 639-6 is going to be considered suitable and useful for the
language tagging RFC.

---

From: Peter Constable <[email-masked]>
Date: Wed, 14 Feb 2007 10:29:54 -0800

From: Debbie Garside [mailto:[email-masked]]

> But I see people who are looking for a language subtag to denote
> Old English using English as a starting point in a hierarchical
> system such as ISO 639-6; makes sense to me.

It's by no means obvious to me that it makes sense. That's using the ID that has very widely been associated with the modern language as the root for some extended set of historical connections of unclear scope. David has clearly demonstrated how that can lead to an absolute mess. If anything is appropriate as the root of some historical hierarchy, it is a protolanguage, or the concept of a collection based on historical ("genetic") associations. IDs for collection exist and capture such concepts. If you really want a hierarchy, something along the lines of Indo-European/Germanic/Middle-English might makes sense.

Peter

---

From: "Debbie Garside" <[email-masked]>
Date: Wed, 14 Feb 2007 21:17:01 -0000

IMHO, the hierarchy should look something like this (missing a few bits
out):

Indo-European
- Germanic
-- West Germanic
--- Anglo-Frisian/North Sea Germanic
---- Anglic 
----- Olde English/Anglo Saxon
----- Middle English
------ Early Northern Middle English
------- Early Scots Northern Middle English
-------- Middle Scots
--------- Modern Scots
------ Early Southern and S Western Middle English
------ Early Midland and S Eastern Middle English
------- Early Modern English
-------- Modern Standard English

Has the ISO 639 ID been used specifically for just modern? How have people
tagged early scots? sco? Historically (in language tagging terms) has en
always been used for modern?   I am not saying you are wrong, but decisions
need to be made wrt these issues.  Do  all ISO 639-1/2/3 ID's represent the
modern language (written and spoken or just written?) unless historic is
specifically stated?   Questions questions questions... A veritable can of
worms.

Best regards 

Debbie

> -----Original Message-----
> From: Peter Constable [mailto:[email-masked]] 
> Sent: 14 February 2007 18:30
> To: [email-masked]; 'LTRU Working Group'
> Subject: RE: [Ltru] Re: Macrolanguages, countries & orthographies
> 
> >> As a user of en, enm and ang, I don't like that one bit. fr and en 
> >> are more mutually intelligible then ang and en, and I 
> don't see any 
> > But I see people who are looking for a language subtag to 
> denote Old 
> > English using English as a starting point in a hierarchical system 
> > such as ISO 639-6; makes sense to me.
> 
> It's by no means obvious to me that it makes sense. That's 
> using the ID that has very widely been associated with the 
> modern language as the root for some extended set of 
> historical connections of unclear scope. David has clearly 
> demonstrated how that can lead to an absolute mess. If 
> anything is appropriate as the root of some historical 
> hierarchy, it is a protolanguage, or the concept of a 
> collection based on historical ("genetic") associations. IDs 
> for collection exist and capture such concepts. If you really 
> want a hierarchy, something along the lines of 
> Indo-European/Germanic/Middle-English might makes sense.
> 
> 
> 
> Peter
>