Subject: [Ltru] Re: Remove extlang from ABNF?

From: "Doug Ewell" <[email-masked]>
Date: Thu, 6 Dec 2007 07:00:06 -0800

Peter Constable <petercon at microsoft dot com> wrote:

>> That's my whole point - the danger that specs writers might look at 
>> the dropped extlang and say "they are dropping features between 
>> versions of  BCP 47, so we better refer to an RFC *only* and even 
>> leave 'or its successor' out".
>
> But this is a key: we are *not* dropping any features. We are dropping 
> the possibility of a future feature. The change to the ABNF (whether 
> by removing the extlang subtag entirely or by renaming and/or 
> comments) is to clean it up so that implementers do not implement for 
> non-and-never-to-be-features.

I think what Felix meant was not that we are dropping features, but that 
it may appear to the outside observer that we are dropping features.

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://home.roadrunner.com/~dewell
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Thu, 6 Dec 2007 08:16:31 -0800

> From: Doug Ewell [mailto:[email-masked]]

> it may appear to the outside observer that we are dropping features.

I understood that. But I think it's significant that we are *not* removing any features, and it seems to me that could be explained easily enough.

The only real stability concern is that some existing parsers would recognize certain hypothetical tags they might encounter as being well formed when in fact they are not well formed. What possible malicious effect could this have? Is this a tag in a query statement? It very likely won't match any content. Is this a tag attributed to content? There likely won't be any requests for it. Only in some closed system in which somebody decided to use proprietary extlang subtags are both going to exist by design. If the parser is encountering a tag somehow constructed such that it has a not-by-design extlang, it won't be catching an condition that could be considered an error, but the error will still be reflected in a failure to match anything, and the essential fix must be in the process *constructing* that tag rather than in the parser.

Peter

---

From: "Felix Sasaki" <[email-masked]>
Date: Fri, 7 Dec 2007 08:13:07 +0900 (JST)

> it may appear to the outside observer that we are dropping features.

thank you for helping with clarifying, that is what I wanted to say. The
"outside observer" will probably look at the spec and see

1) there is a conformance type "wellformed"
2) the ABNF is a part of that conformance type
3) the ABNF has changed, i.e. a production has been dropped, ergo
4) a feature has been dropped which was part of this conformance type.

Felix

---

From: Martin Duerst <[email-masked]>
Date: Tue, 11 Dec 2007 16:48:42 +0900

At 01:16 07/12/07, Peter Constable wrote:
>Content-Language: en-US
>Content-Type: text/plain; charset="utf-8"
>
>I understood that. But I think it's significant that we are *not* removing 
>any features, and it seems to me that could be explained easily enough.

[chairs hat OFF]

It looks like this could be explained easily enough, but I'm quite
sure that this won't be the case. XML, and XML Schema, are very
strictly defined languages. Fortunately, we managed to get XML
away from including a grammar for language tags in an early erratum/
corrigendum. But if XML Schema has indeed used RFC 4646 for defining
the syntactic range of language tags, then we should not remove
some productions. It is clear to us as well as to them that there
cannot be any valid language tags of the form ab-cde-fgh. But
these tags, in RFC 4646 and therefore in XML Schema version foo,
are well-formed. If they change to be non-wellformed in a new
version of XML Schema, that essentially means that there are
potentially some documents that conform to a schema in one
version of XML Schema, but not in a newer one. Even if these
language tags have not been valid, and don't make sense, that's
a very bad idea. As an extreme example, such a tag may by accident
be part of a document used for managing a nuclear plant. With
a software update, the document could suddently become non-conforming,
triggering some error.

Peter, I recommend that you talk to some XML Schema folks inside
Microsoft to understand some of their thinking. I'm sure Felix
can provide some points of reference if needed.

As for implementation effort, I agree that leaving the extlang
production in increases implementation effort. But quite a bit
of it can be saved by knowing that these aren't really used anymore
(i.e. implementations only have to do the parsing for them,
they don't have to provide any additional functionality).
Also, we already have a long list of tests and some regular
expressions out there which cover extlangs, which both can help
implementers.

Regards,    Martin.

>The only real stability concern is that some existing parsers would 
>recognize certain hypothetical tags they might encounter as being well 
>formed when in fact they are not well formed. What possible malicious 
>effect could this have? Is this a tag in a query statement? It very likely 
>won't match any content. Is this a tag attributed to content? There likely 
>won't be any requests for it. Only in some closed system in which somebody 
>decided to use proprietary extlang subtags are both going to exist by 
>design. If the parser is encountering a tag somehow constructed such that 
>it has a not-by-design extlang, it won't be catching an condition that 
>could be considered an error, but the error will still be reflected in a 
>failure to match anything, and the essential fix must be in the process 
>*constructing* that tag rather than in the parser.
>
>
>Peter

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Felix Sasaki <[email-masked]>
Date: Tue, 11 Dec 2007 21:32:02 +0900

Martin Duerst wrote:
>>>> But this is a key: we are *not* dropping any features. We are
>>>>         
>>> dropping
>>>       
> some productions. 
This is a repetition, but maybe useful in this long thread.

The W3C i18n core Working Group asked the XML Schema Working Group to 
use the RFC 4646 syntactic range of language tags for XML Schema 1.1 
(datatypes), see
http://www.w3.org/Bugs/Public/show_bug.cgi?id=3079#c2 (search for "here 
RFC 4646 grammar")
We assumed that the RFC 4646 ABNF will be stable, and since XML Schema 
1.0 (datatypes) uses the ABNF from RFC 3066
http://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#language
it seems natural to propose for XML Schema 1..1 to have the RFC 4646 
ABNF for XML Schema 1.1.

Felix

---

From: Peter Constable <[email-masked]>
Date: Tue, 11 Dec 2007 07:21:18 -0800

If XML Schema 1.1 is still a work in progress, and assuming 4646bis wraps up quickly (which I hope will be the case), why would it not be possible (and, indeed, why would it not be welcomed) to suggest that XML Schema 1.1 use ABNF from 4646bis that differs from 4646 by removing a never-to-be-used construct and hence is simpler?

Peter

> -----Original Message-----
> From: Felix Sasaki [mailto:[email-masked]]
> Sent: Tuesday, December 11, 2007 4:32 AM
> To: Martin Duerst
> Cc: Peter Constable; LTRU Working Group
> Subject: Re: [Ltru] Re: Remove extlang from ABNF?
>
> Martin Duerst wrote:

---

From: Peter Constable <[email-masked]>
Date: Tue, 11 Dec 2007 07:24:33 -0800

> From: Martin Duerst [mailto:[email-masked]]

> can provide some points of reference if needed.

I can do that. Is Paul Cotton still involved in that, or is there someone else?

Peter

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 11 Dec 2007 07:45:29 -0800

I agree with Peter that it'd be better to remove extlang. However, I think
we can also live with option #2: leave the extlang in the ABNF, but in
normative text make it absolutely clear that that part of the ABNF is
OBSOLETE; no valid tag will ever have extlangs. That makes the people who
want stability of the ABNF (no matter how that relates to valid tags) happy,
while allowing the rest of us to remove extlang from our code and APIs.

Mark

-- 
Mark

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 11 Dec 2007 18:37:52 +0100

Martin Duerst wrote:

> As an extreme example, such a tag may by accident be
> part of a document used for managing a nuclear plant.

Something's wrong with this installation if it depends
on the syntactical well-formedness of <obs-extlang>.

And you could use the same example with a "malicious"
<obs-extlang>, when it survives a syntax check still
allowing non-existent <obs-extlang>s as "well formed".

> With a software update, the document could suddently
> become non-conforming, triggering some error.

It might as well trigger an error in lazy 4646bis code
not prepared to support any <obs-extlang> because they
don't exist.

4646bis code with the new minimal "most perverse tag"
buffer size might correctly identify the non-existing
well-formed <obs-extlang> construct as too long, and
follow a code path not tested for the nuclear plant -
this example isn't compelling, it cuts on both sides.

 Frank

---

From: Felix Sasaki <[email-masked]>
Date: Wed, 12 Dec 2007 08:58:49 +0900

> If XML Schema 1.1 is still a work in progress, and assuming 4646bis wraps up quickly (which I hope will be the case), why would it not be possible (and, indeed, why would it not be welcomed) to sug…

As I said before, the I18N core Working Group within W3C proposed to the 
XML Schema Working Group to refer to the ABNF from RFC 4646. I had a 
call with someone from the XML Schema working group this morning, and it 
seems they are various opinions in the XML Schema Working Group about 
how to refer to language tag specs:

1) refer to a specific spec, e.g. RFC 3066 (this is what is currently at 
http://www.w3.org/TR/xmlschema-2/#language , i.e. a reference to RFC 3066)
2) make no type constraint, just refer to the (ISO) language codes
3) say that a validator needs to check current version of language code 
spec if it wants to do the right thing (where "needs" doesn't have any 
normative power), and providing the necessary reference for  that.

The current tendency within the XML Schema Working Group is that they 
might go for 3) and replace the reference to 3066 with a reference to 
BCP 47.
However, the pattern for validation of the language data type will 
probably not be changed, it will stay as RFC 3066 like

[a-zA-Z]{1,8}(-[a-zA-Z0-9]{1,8})*

So for this thread I think this mainly means good news (an "old" 
reference to the RFC 4646 ABNF in XML Schema is not a problem anymore, 
since they will not have it anyway ) and some bad news (there will be 
the old RFC 3066 ABNF also in XML Schema 1.1). I think the later news 
should not prevent people from in LTRU moving forward.

Felix|
|

---

From: Martin Duerst <[email-masked]>
Date: Wed, 12 Dec 2007 15:50:28 +0900

At 08:58 07/12/12, Felix Sasaki wrote:

>As I said before, the I18N core Working Group within W3C proposed to the XML Schema Working Group to refer to the ABNF from RFC 4646. I had a call with someone from the XML Schema working group this morning, and it seems they are various opinions in the XML Schema Working Group about how to refer to language tag specs:
>
>1) refer to a specific spec, e.g. RFC 3066 (this is what is currently at http://www.w3.org/TR/xmlschema-2/#language , i.e. a reference to RFC 3066)
>3) say that a validator needs to check current version of language code spec if it wants to do the right thing (where "needs" doesn't have any normative power), and providing the necessary reference for  that.
>
>The current tendency within the XML Schema Working Group is that they might go for 3) and replace the reference to 3066 with a reference to BCP 47.
>However, the pattern for validation of the language data type will probably not be changed, it will stay as RFC 3066 like
>

This is very much the right way to go. There is no point for a
using spec to be too detailled in terms of the syntax. And
for the semantics, pointing to BCP 47 is the best thing to do.

But I don't think this solves the problems for us.

In RFC 4646, we defined some tags as well-formed. In RFC 4646bis, we suddenly
say that some of these tags are not well-formed. We never would do this for
valid tags, so I'm really not sure it is a good thing to do for well-formed tags.

Regards,    Martin.

---

From: Felix Sasaki <[email-masked]>
Date: Wed, 12 Dec 2007 22:25:21 +0900

> valid tags, so I'm really not sure it is a good thing to do for well-formed tags.
+1

Felix

---

From: "Frank Ellermann" <[email-masked]>
Date: Wed, 12 Dec 2007 17:20:59 +0100

Martin Duerst wrote:

>> The current tendency within the XML Schema Working Group is
>> that they might go for 3) and replace the reference to 3066
>> with a reference to BCP 47.
>> However, the pattern for validation of the language data type
>> will probably not be changed, it will stay as RFC 3066 like
>>
 
> This is very much the right way to go.

+1, that could be also a plan for 2616bis (their 2616 pattern
    is definitely wrong).

> There is no point for a using spec to be too detailled in
> terms of the syntax. And for the semantics, pointing to BCP 47
> is the best thing to do.

+1 (however a validator should go to the trouble to check what
    it can check, the 4646bis ABNF is no exercise in futility)

 [end of the +1, now for the details:] 
 
> In RFC 4646, we defined some tags as well-formed. In RFC 
> 4646bis, we suddenly say that some of these tags are not 
> well-formed.

Some of these well-formed tags were _reserved_ for 4646bis.

They were never _valid_ tags.  Similar example, after some
fights and flamewars 2822upd will admit that there never was
a domain literal using NO-WS-CTL.  It's not yet clear if that
goes straight to /dev/null, or gets an <obs-domain-literal>.

Lately I think the promotion or revision of a standard should
never be done by the original authors, they tend to "know"
too many things that were never spelled out in their document.

> I'm really not sure it is a good thing to do for well-formed
> tags.

Let's ask our ADs, they know the similar issue with 2822upd.

Our <obs-extlang> is a border case, not a simple "MUST NOT
generate, MUST accept", it's "MUST NOT generate, MUST NOT
accept as valid, MUST accept as well-formed" (and the third
MUST could degenerate into a MAY or a SHOULD in practice,
and end up as security issue)

 Frank

---

From: Addison Phillips <[email-masked]>
Date: Wed, 12 Dec 2007 08:22:44 -0800

>> valid tags, so I'm really not sure it is a good thing to do for 

I agree.... except....

This is exactly what we did in RFC 4646. We made a vast array of 
"well-formed" but invalid tags illegal (by narrowing the ABNF). In 
4646bis, one could say that we were doing the same thing--making a (much 
smaller) array of tags (which were never valid) illegal. And we *have* 
changed the ABNF in a manner that narrows it in 4646bis already. We have 
narrowed the grandfathered production substantially.

I have no problem with XML Schema or others referencing the 4646 ANBF 
instead of the 4646bis ABNF for well-formness checking. It won't 
introduce anything particularly bad. And I am somewhat allergic to 
changing the ABNF because I have personally felt like we should resist 
tampering.

As an implementer, though, I really hate supporting extlang, now that it 
does nothing. So I'd propose:

1. Remove extlang from the ABNF in 4646bis. That would make the language 
production:

language      = (2*3ALPHA)             ; shortest ISO 639 code
               / 4ALPHA                 ; reserved for future use
               / 5*8ALPHA               ; registered language subtag

2. In the section on conformance, permit 4646bis well-formedness to 
reference either the current ABNF or an "obs-language" production that 
looks like:

obs-language  = (2*3ALPHA [ extlang ]) ; shortest ISO 639 code
               / 4ALPHA                 ; reserved for future use
               / 5*8ALPHA               ; registered language subtag

extlang       = *3("-" 3ALPHA)         ; removed in this version

3. Add a note saying that no tags were ever valid under obs-language, 
but that some processors may permit them. Also note that 3066 
well-formedness differed substantially from 4646/4646bis well-formedness 
and provide the 3066 production in that section for completeness.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 12 Dec 2007 09:18:58 -0800

I like that approach; clever.

---

From: "Frank Ellermann" <[email-masked]>
Date: Wed, 12 Dec 2007 18:34:36 +0100

> I like that approach; clever.

<abbr style="mark-davis"> LGTM </abbr>

---

From: "Ira McDonald" <[email-masked]>
Date: Wed, 12 Dec 2007 15:53:28 -0500

Hi,

Addison - well done!

Nice and simple - not obscure to any implementor or designer.

Cheers,
- Ira

-- 
Ira McDonald (Musician / Software Architect)
Chair - Linux Foundation Open Printing WG
Blue Roof Music / High North Inc
PO Box 221  Grand Marais, MI  49839
work: +1-906-494-2434
home: +1-906-494-2697
email: [email-masked]

---

From: Felix Sasaki <[email-masked]>
Date: Thu, 13 Dec 2007 06:06:02 +0900

+1

Felix

---

From: Peter Constable <[email-masked]>
Date: Wed, 12 Dec 2007 14:51:55 -0800

Works for me.

(I assume no one will complain that I didn't use "+1" ;-)
Peter.

> Sent: Wednesday, December 12, 2007 1:06 PM
> To: Ira McDonald
> Cc: [email-masked]