Subject: [Ltru] Test suite for language tags?

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 1 Aug 2006 22:33:51 +0200

I just wrote a non-validating parser for language tags and I'm looking
for test data. I want to test bizarre tags to see if the parser does
classify them properly.

I'm specially interested in badly-formed tags: the I-D contains mostly
well-formed tags.

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Addison Phillips <[email-masked]>
Date: Tue, 01 Aug 2006 14:06:02 -0700

> for test data. I want to test bizarre tags to see if the parser does

Good for you!

 > I'm specially interested in badly-formed tags: the I-D contains mostly

Your best bet is probably to generate subtag sequences based on the 
ABNF. Some particular problem cases to check would be:

- singletons in the first position (except for 'x' and the grandfathered 
list)
- overlong subtags (longer than 8 characters)
- more than three extlangs
- misplaced extlang (3ALPHA in the third or later position following any 
of these: 4ALPHA, 2ALPHA, 3DIGIT, 5*8alphanum, DIGIT 3alpha)[note: stop 
at singleton]
- misplaced script (4ALPHA following any of these: 2ALPHA, 3DIGIT, 
5*8alphanum, DIGIT 3alphanum)[note: stop at singleton]
- misplaced variant (five or more characters, or four or more starting 
with a digit; either occurring before an extlang/script/region is an error).
- non-x singleton followed immediately by a singleton (including 'x')
- missing subtag ("--")
- a dangling hyphen ("foo-bar-baz-") or initial hyphen ("-foo-bar-baz")
- digits in the primary (first) subtag
- repeated singleton (note case insensitivity)

Thus, these are all errors:

"a-foo"
"abcdefghi-012345678"
"ab-abc-abc-abc-abc"
"ab-abcd-abc"
"ab-ab-abc"
"ab-123-abc"
"ab-abcde-abc"
"ab-1abc-abc"
"ab-ab-abcd"
"ab-123-abcd"
"ab-abcde-abcd"
"ab-1abc-abcd"
"ab-a-b"
"ab-a-x"
"ab--ab"
"ab-abc-"
"-ab-abc"
"ab-a-abc-a-abc"

These are not errors:

"ab-x-abc-x-abc" // anything goes after x
"ab-x-abc-a-a"   // ditto
"i-default"      // grandfathered

Hope that helps,

Addison

Addison Phillips
Globalization Architect − Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 1 Aug 2006 14:30:57 -0700

What may be useful is that ICU has a test string generator (BNF) that
generates strings that match a specified BNF syntax. It augments the regular
syntax with percent values that indicate relative weights. That is, if you
have

x = (a | b | c)

in the BNF, you can make it

x = (a 25% | b 45% | c 30%)

so that it generates those alternatives with those frequencies.

It is an internal testing class, and doesn't have much documentation, but I
thought I'd mention it in case you'd find it useful.

Mark

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 2 Aug 2006 09:28:48 +0200

On Tue, Aug 01, 2006 at 02:30:57PM -0700,
 Mark Davis <[email-masked]> wrote 
 a message of 117 lines which said:

> What may be useful is that ICU has a test string generator (BNF)
> that generates strings that match a specified BNF syntax.

Interesting. An ABNF2tests that takes RFC 4234 as input and produces
Junit / PyUnit / Hunit / whatever would certainly be an useful tool
for IETF. I do not use Java but I'll have a look.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 2 Aug 2006 09:27:09 +0200

On Tue, Aug 01, 2006 at 02:06:02PM -0700,
 Addison Phillips <[email-masked]> wrote 
 a message of 65 lines which said:

> Some particular problem cases to check would be:

Many thanks for the test cases, I discovered a bug in the parser with
them. Now, it works:

Cases: 58  Tried: 58  Errors: 0  Failures: 0

:-)

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 2 Aug 2006 06:37:23 -0700

I'm sorry I gave the wrong impression. It doesn't use RFC 4234 syntax; it
uses Perl-style syntax, eg

x = ( a | b | c ) d* ( e | f )+ ....
instead of
x=("a"/"b"/"c") *d *1("e"/"f")

It would not take a lot of work to have it also take the older syntax as
well, but it doesn't use it out of the box.

Mark

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 2 Aug 2006 06:47:40 -0700

P.S. It was also ported to C++ (although with some limitations) in wbnf.*

Mark

---

From: Addison Phillips <[email-masked]>
Date: Thu, 03 Aug 2006 08:48:29 -0700

Glad it helped.

I could probably have saved time writing that message if I'd just sent 
my JUnits ;-). Glad to see that others are getting their parsers going.

Just curious: will you also be implementing a validating parser?

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 4 Aug 2006 18:57:20 +0200

On Thu, Aug 03, 2006 at 08:48:29AM -0700,
 Addison Phillips <[email-masked]> wrote 
 a message of 30 lines which said:

> I could probably have saved time writing that message if I'd just
> sent my JUnits ;-).

I do not program in Java so I'm not sure it would have been helpful
:-) The best thing would probably to create four text files containing
tags, one per line (may be with comments allowed), with:

* two files for well-formedness (OK/broken),
* two files for validaty (OK/invalid).

So anyone could use it as a test suite, by converting the text files
to Junit, PyUnit, HUnit, whatever.

> Just curious: will you also be implementing a validating parser?

Yes, it is on the tracks.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 05 Aug 2006 16:33:38 +0200

Stephane Bortzmeyer wrote:

> The best thing would probably to create four text files
> containing tags, one per line (may be with comments allowed),
> with:

> So anyone could use it as a test suite, by converting the
> text files to Junit, PyUnit, HUnit, whatever.

Maybe it's possible to publish this in a 3066ter appendix,
or better a separate document (informational RFC).  I often
miss examples and test cases in other documents.

Frank

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 30 Aug 2006 11:30:00 +0200

On Sat, Aug 05, 2006 at 04:33:38PM +0200,
 Frank Ellermann <[email-masked]> wrote 

> miss examples and test cases in other documents.

I agree and here are the two files I currently use (my parser succeeds
for all these tags). I suggest that they should be displayed in a
public place (is there a Web site for LTRU?) and added in a future
RFC.

Corrections and additions are of course very welcome, as well as test
reports from other implementations, so we can be sure we all use the
same language :-)
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 30 Aug 2006 23:53:52 +0200

Stephane Bortzmeyer wrote:

> is there a Web site for LTRU?

The IETF tools WG page is http://tools.ietf.org/wg/ltru - at
the moment this tool offers links to the charter, drafts, list,
jabber, etc., but no other documents or links.

Addison's page is at http://www.inter-locale.com/ - that had
a complete history of all drafts back to the times before LTRU,
but I don't find it at the moment.

Doug's page is at http://users.adelphia.net/~dewell/ - oops,
with a HTML version of UTN 16, replacing PDF by HTML on my box.

Maybe publish the test suite as dummy I-D, the tools server
should show it as "related" if the third token is "-ltru-",
for a draft-bortzmeyer-ltru-tag-test-suite-00.txt or similar.

IIRC there was a sudden interest in xml2rfc one second after
MARID was closed, and you were one of the interested folks :-)

If it's only about an Archived-At link, that exists already:
http://permalink.gmane.org/gmane.ietf.ltru/5279

GMaNe apparently mangles the attachments, the original is here:
http://article.gmane.org/gmane.ietf.ltru/5279/raw

> additions are of course very welcome

  For well-formed: 

abcd-Latn
AaBbCcDd-x-y-any-x

  For broken:

abcd-efg
aabbccddE

Maybe use abcde instead of abcd if you have alpha4 as "not yet
possible", alpha5 is possible today (individually registered).

Frank

---

From: Addison Phillips <[email-masked]>
Date: Wed, 30 Aug 2006 17:45:49 -0700

> public place (is there a Web site for LTRU?) and added in a future

An informational RFC would be quite easy to produce and I think would be 
worthwhile. I'd even be willing to do some work on it in a couple of weeks.

 > I suggest that they should be displayed in a
 > public place (is there a Web site for LTRU?)

As Frank points out, you could probably do worse than 
http://www.inter-locale.com. It has the HTML versions of the various 
drafts (not all are linked any more), plus other information. I'm happy 
to host other's materials, within reason.

Addison

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 31 Aug 2006 09:18:08 +0200

On Wed, Aug 30, 2006 at 11:53:52PM +0200,
 Frank Ellermann <[email-masked]> wrote 
 a message of 45 lines which said:

> Maybe publish the test suite as dummy I-D, the tools server should
> show it as "related" if the third token is "-ltru-", for a
> draft-bortzmeyer-ltru-tag-test-suite-00.txt or similar.

OK, what's the opinion of the WG about that?
 
> aabbccddE

Thanks, abcd-Latn discovered yet another bug in my parser :-)

---

From: Martin Duerst <[email-masked]>
Date: Thu, 31 Aug 2006 20:15:19 +0900

[chair hat on]

We haven't put development of test suites into our new charter,
but if you think this should be added, you should say so in
response to the call for comments on the IETF list.

In my impression, test suites are often done in coordination
with an IETF WG, but not usually as an official WG task. But
my implession might be wrong. I don't think there is anything
in the IETF process that would forbid a WG to develop a test
suite, or to accept a test suite proposal as a WG draft.
[Ted, can you tell us what/where the IETF rules re. test
suites are?]

So I would in principle not be opposed to the WG (assuming we
get rechartered) taking on a test suite internet draft, but
we should be sure that this won't slow us down on our main
deliverables.

Regards,    Martin.

At 09:45 06/08/31, Addison Phillips wrote:
>An informational RFC would be quite easy to produce and I think would be worthwhile. I'd even be willing to do some work on it in a couple of weeks.
>
>As Frank points out, you could probably do worse than http://www.inter-locale.com. It has the HTML versions of the various drafts (not all are linked any more), plus other information. I'm happy to host other's materials, within reason.
>
>Addison
>
>-- 
>Addison Phillips

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Addison Phillips <[email-masked]>
Date: Thu, 31 Aug 2006 07:15:42 -0700

I was thinking of an individual submission.

Addison

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 31 Aug 2006 16:51:07 +0200

> I was thinking of an individual submission.

Yes, not necessarily an RFC, an I-D is fine, or an "ION" (an
experiment run by Harald), or just a text file on your site.   
 
> Martin Duerst wrote:
>> not usually as an official WG task. But my implession might
>> be wrong.

I guess you're right, "they" don't like conformance testing.
As "official" WG task we'd get into discussions about the
format etc., the ABNF isn't complex enough for such efforts.

Stephane's two lists amended by whatever Addison, Doug, and
others see is actually all...
 
>> we should be sure that this won't slow us down on our main

...it's more or less ready.  Of course it would need an update
if we change the prefix-SHOULD into a prefix-MUST.  We're
talking about 26+39 (or 28+41) lines of text, less than 2 KB.

Frank

---

From: Martin Duerst <[email-masked]>
Date: Fri, 01 Sep 2006 02:54:24 +0900

>I was thinking of an individual submission.

Yes, I know. I mainly brought up the possibility
of a WG draft so that it has been mentioned.

Regards,    Martin.