Subject: [Ietf-languages] Suppress-Script, assumptions, and guesses (was: RE: Suggestion to update Urdu Script Designation in the subtag registry)

From: "Doug Ewell" <[email-masked]>
Date: Fri, 14 Aug 2020 10:32:07 -0600

Richard Wordingham wrote:

>> But it does not follow from this that all English text written in the
>> Latin script should be tagged "en-Latn". Most of the time, the
>> assumption is correct and adequate. This is why the word SHOULD is
>> used, instead of MUST.
>
> That seems to answer the issue.  If the language tag is describing
> something that already exists, then if one may need to know the script
> from just the tag, one should not heed suppress-script - which is why
> the word 'SHOULD' is used.

If the content already exists, and one has retrieved it, and the language tag doesn't reveal the script, one can figure out what script it is written in by inspecting the content. This is just like the mechanic inspecting the car to figure out how many wheels it has.

I confess that I don't understand this exercise.

> I was coming to the conclusion that the ambiguity between "en" for
> "en-Latn" and "en" for "English, but I won't tell you the script", was
> resolved by the tags Zxxx, Zyyy and Zzzz.  Then the latter meaning for
> text gets translated to en-Zyyy, and then the abbreviation of en-Latn
> to en is resolved by the suppress-script.

'Zxxx' is a different animal: "Code for unwritten documents" from ISO 15924 is interpreted here to mean that this content is not written. It doesn't necessarily mean that the language of this content is never written.

'Zzzz' is "Code for uncoded script," and that's pretty clearly intended by 15924 to apply to things like Voynich or Utopian that are, well, not coded.

Let me try again with the "undetermined" use case:

If I tag textual content as "en", I am not saying anything about the script. I am leaving it up to you, the recipient of the content, to figure out what the script is. Because it is English, the great likelihood is that the script is Latin. One way to find out would be to inspect the content.

Alternatively, you COULD go to the Registry and look at the Suppress-Script value of 'Latn' to give you a strong hint that the text is in Latin script. But that is not a guarantee; "en" is also a perfectly valid tag for English written in Runic. 

If I tag it as "en-Runr", I am explicitly saying the script is Runic.

If I tag it as "en-Latn", I am explicitly saying the script is Latin. I would hope that I am doing so for a reason, and not just adding a script subtag automatically because I didn't read Section 4.1. Maybe I am anticipating that readers will call into question whether the text is in Latin script, and need to know without looking at the content.

If I tag it as "en-Zyyy", that implies that I, the tagger, don't know what script it is in. It seems unlikely to me that I would know a text is in English but don't know the script. For some historical languages, this is more plausible.

Does this help?

Languages are very often not a precise thing lending themselves to concrete rules. I know some have bemoaned the fact that BCP 47 has so many SHOULDs and so few MUSTs, but there are really very few bright lines when dealing with languages and language usage. Trying to attach guarantees to language tag selection is a Hard Problem. We haven't even solved the problem of people writing in I-Ds and on Stack Overflow that a language tag contains a 2-letter 639 code and a 2-letter 3166 code.

>> In your everyday life, it is not necessary under normal circumstances
>> to point out that your car has four wheels, because the overwhelming
>> majority of cars do.
>
> But this information could be vital for positioning it over an
> inspection pit.

Exactly so. Thank you for proving my point about "under normal circumstances."

>>> Would a rule that the script must be indicated somehow make a
>>> difference, e.g. by making plain "en" or "ur" imply that the script
>>> subtag had been suppressed?
>>
>> A rule that the script must be indicated would be the exact opposite
>> of what we were trying to accomplish in 2005, which is backward
>> compatibility with the huge volume of existing language-tagged data.
>
> I was thinking of a rule applying to that index of mine, not one for
> universal use.

For your personal use, where the probability that English text is in Latin script is much lower than normal, I might suggest using "en-Latn". That way you don't need to consult the Registry to guess the script.

I'm skipping over the remainder, which continues to claim that not using a script subtag for languages overwhelmingly written in a single script creates problematic ambiguity. We have heard very little in the past 15 years that either Suppress-Script or the overall philosophy of "tag content wisely" has greatly compromised the usefulness of BCP 47 tags. Maybe people just aren't telling us.

--
Doug Ewell | Thornton, CO, US | ewellic.org

---

From: Richard Wordingham <[email-masked]>
Date: Fri, 14 Aug 2020 20:26:51 +0100

On Fri, 14 Aug 2020 10:32:07 -0600

> If the content already exists, and one has retrieved it, and the
> language tag doesn't reveal the script, one can figure out what
> script it is written in by inspecting the content. This is just like
> the mechanic inspecting the car to figure out how many wheels it has.

The "various texts cluttering my house" aren't all in the same room
as one another.  To inspect the text, I would have to get it out from
its location, which could require a sequential search through a
document wallet.  (Perhaps random access would be possible after the
indexing exercise.)

> > "en-Latn" and "en" for "English, but I won't tell you the script",
> > was resolved by the tags Zxxx, Zyyy and Zzzz.  Then the latter
> > meaning for text gets translated to en-Zyyy, and then the
> > abbreviation of en-Latn to en is resolved by the suppress-script.  
> 
> 'Zxxx' is a different animal: "Code for unwritten documents" from ISO
> 15924 is interpreted here to mean that this content is not written.
> It doesn't necessarily mean that the language of this content is
> never written.

Yes, I'd figured that this would handle audio CDs if I were including
them in the index.  It would also cover pure pictures.

> If I tag textual content as "en", I am not saying anything about the
> script. I am leaving it up to you, the recipient of the content, to
> figure out what the script is. Because it is English, the great
> likelihood is that the script is Latin. One way to find out would be
> to inspect the content.

Not easy.  And even if the material were online, determining the script
of a bit map is not always straightforward.

> Alternatively, you COULD go to the Registry and look at the
> Suppress-Script value of 'Latn' to give you a strong hint that the
> text is in Latin script. But that is not a guarantee; "en" is also a
> perfectly valid tag for English written in Runic. 

I do have some en-Runr-NZ stuff!

> If I tag it as "en-Latn", I am explicitly saying the script is Latin.
> I would hope that I am doing so for a reason, and not just adding a
> script subtag automatically because I didn't read Section 4.1. Maybe
> I am anticipating that readers will call into question whether the
> text is in Latin script, and need to know without looking at the
> content.

I thought we'd already agreed that this was the correct solution in my
case.

> If I tag it as "en-Zyyy", that implies that I, the tagger, don't know
> what script it is in. It seems unlikely to me that I would know a
> text is in English but don't know the script.

The mechanism I had in mind is to read the cover and not look at the
content.  It's not very likely, but not impossible.  I also
deliberately didn't mention the complication of different paragraphs
having different language-script combinations.

> I'm skipping over the remainder, which continues to claim that not
> using a script subtag for languages overwhelmingly written in a
> single script creates problematic ambiguity. We have heard very
> little in the past 15 years that either Suppress-Script or the
> overall philosophy of "tag content wisely" has greatly compromised
> the usefulness of BCP 47 tags. Maybe people just aren't telling us.

I did point that usually whatever process needs to know the script of
some text can look at the text itself, and wouldn't bother about any
label.  That knowledge is usually required at a lower granularity than
the labelling would naturally be applied at.

However, there may be cases where 'likely tags' are being used, so that
'en' just gets translated to 'en-Latn'.  If that process goes wrong,
the most likely response is advice to specify the script, implying that
'en' on its own would be wrong for 'en-Thai'.
  
Richard.