Subject: Syntax

From: Julian Reschke <[email-masked]>
Date: Sun, 07 Jan 2007 18:12:09 +0100

Hi,

here are a few thoughts regarding the syntax:

1) Although the language is designed to be used in IDs and RFCs, 
restricting it to ASCII here is IMHO a very bad idea. After all, you may 
want to use it for other specifications, and the IETF may lift the 
current restrictions at some point of time. I would suggest to require a 
specific text encoding such as UTF-8, and try to make the definitions 
for the character set of identifiers much something well understood 
(such as XML 1.0).

2) At a later point, it would be great if the format would define and 
register a mime type (for instance, we would want that for easy 
inclusion into RFC2629 artwork).

3) Although a "compact" syntax is great for readability, an alternate 
syntax in XML would be good for usage within XML tool chains. I've 
written several XSLTs targeting GraphViz in the past, and getting 
escaping right can be a nightmare. Having a separate XML based input 
format would make things much easier (if there's interest in that, I'll 
happily contribute).

Best regards, Julian

_______________________________________________
Cosmogol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/cosmogol

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 7 Jan 2007 21:52:55 +0100

On Sun, Jan 07, 2007 at 06:12:09PM +0100,
 Julian Reschke <[email-masked]> wrote 
 a message of 31 lines which said:

> restricting it to ASCII here is IMHO a very bad idea. After all, you
> may want to use it for other specifications, and the IETF may lift
> the current restrictions at some point of time. I would suggest to
> require a specific text encoding such as UTF-8,

Any idea about the support of UTF-8 in typical languages *and* parsing
tools? For instance, with C and Yacc, I assume it is quite
difficult. With Haskell, I'm not sure :-)

The problem of UTF-8 is that many assumptions no longer hold:

* case insensitivity becomes a problem,
* enumeration of "reasonable" characters become a complex task.

I'm a big fan of internationalization and Unicode but, since Cosmogol
is intended for a technical and limited use, is it reasonable? 

Anyway, I recorded the point as a TODO (we do not have a formal issue
tracker) in the draft source. Other advices?

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 7 Jan 2007 21:58:41 +0100

On Sun, Jan 07, 2007 at 06:12:09PM +0100,
 Julian Reschke <[email-masked]> wrote 
 a message of 31 lines which said:

> 3) Although a "compact" syntax is great for readability, 

And authorship, IMHO.

> an alternate syntax in XML would be good for usage within XML tool
> chains.

It is certainly a good idea. What do you think of having an alternate
*non-normative* XML format, with the same terminology and data model
as the compact syntax? A tool like Shadok could certainly translate
from one syntax to another.

> (if there's interest in that, I'll happily contribute).

Please proceed, as long as it is in Relax NG :-)

---

From: Julian Reschke <[email-masked]>
Date: Mon, 08 Jan 2007 10:31:14 +0100

> difficult. With Haskell, I'm not sure :-)

Well, I've been living in Java world for a long time, so I really can't 
say anything useful about other languages anymore.

> * enumeration of "reasonable" characters become a complex task.

That's a problem with Unicode (the character set), not UTF-8 (the encoding).

Yes, case insensitivity is harder, but is this relevant for cosmogol, if 
  everything stays case-sensitive (which I think is the right thing to do)?

Choosing characters for identifiers: again, just borrow from somewhere 
else, such as <http://www.w3.org/TR/REC-xml/#NT-Name>).

> is intended for a technical and limited use, is it reasonable? 

I think inventing a new format, but not taking I18N is very hard to 
defend. As far as I can tell, there's no real chance to get it published.

> tracker) in the draft source. Other advices?

Let's leave it at this for now.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 08 Jan 2007 10:32:42 +0100

>> 3) Although a "compact" syntax is great for readability, 

Yep.

>>from one syntax to another.

Exactly what I was thinking about.

> Please proceed, as long as it is in Relax NG :-)

Yes, I also like XML Schema languages, as long as they don't come from 
the W3C :-).

Best regards, Julian

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 8 Jan 2007 10:29:18 -0800

On a somewhat related note, we often need to do ladder diagrams (or  
call flows, or interactions diagrams depending on your terminology)   
in IETF drafts - I know that this is separate from FSM but they are  
equally needed.

On Jan 7, 2007, at 9:12 AM, Julian Reschke wrote:

> restricting it to ASCII here is IMHO a very bad idea. After all,  
> you may want to use it for other specifications, and the IETF may  
> lift the current restrictions at some point of time. I would  
> suggest to require a specific text encoding such as UTF-8, and try  
> to make the definitions for the character set of identifiers much  
> something well understood (such as XML 1.0).
>
> 2) At a later point, it would be great if the format would define  
> and register a mime type (for instance, we would want that for easy  
> 3) Although a "compact" syntax is great for readability, an  
> alternate syntax in XML would be good for usage within XML tool  
> chains. I've written several XSLTs targeting GraphViz in the past,  
> and getting escaping right can be a nightmare. Having a separate  
> XML based input format would make things much easier (if there's  
> interest in that, I'll happily contribute).
>
> Best regards, Julian
>
>
>

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 8 Jan 2007 21:46:18 +0100

On Mon, Jan 08, 2007 at 10:31:14AM +0100,
 Julian Reschke <[email-masked]> wrote 
 a message of 48 lines which said:

> Choosing characters for identifiers: again, just borrow from
> somewhere else, such as <http://www.w3.org/TR/REC-xml/#NT-Name>).

Did anyone already tried to convert it in ABNF? The production
BaseChar in the XML standard is a bit frightening and may exercice the
limits of some programs. Implementation reports are welcome.

Otherwise, what do you think of the solution used in RFC 4646?

   ASCCHAR    = %x21-25 / %x27-7E / UNICHAR ; Note: AMPERSAND is %x26
   UNICHAR    = "&#x" 2*6HEXDIG ";"
   
   Characters from outside the US-ASCII [ISO646] repertoire, as well as
   the AMPERSAND character ("&", %x26) when it occurs in a field-body,
   are represented by a "Numeric Character Reference" using hexadecimal
   notation in the style used by [XML10] (see
   <http://www.w3.org/TR/REC-xml/#dt-charref>).  This consists of the
   sequence "&#x" (%x26.23.78) followed by a hexadecimal representation
   of the character's code point in [ISO10646] followed by a closing
   semicolon (%x3B).  For example, the EURO SIGN, U+20AC, would be
   represented by the sequence "&#x20AC;".  Note that the hexadecimal
   notation MAY have between two and six digits.

> defend. As far as I can tell, there's no real chance to get it

Hmmm, how many IETF formats are in Unicode? (Apart from those based
only on XML, like Atom in RFC 4287.) ABNF is not, for instance (right,
it is not a few format, the RFC is recent but it derives from an older
format.)

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 9 Jan 2007 00:07:04 +0000

>> somewhere else, such as <http://www.w3.org/TR/REC-xml/#NT-Name>).
[...]
>    UNICHAR    = "&#x" 2*6HEXDIG ";"

I seem to have missed a message or two. What is the problem you're trying
to solve.

If it's identifiers using characters outside the ASCII regime, then you
want people to be able to write them in their own language, not encode
them. Equally, you want *an* encoding for portable use. The way we solved
this in the C Standard is that you can use such characters directly, or you
can encode them as \u#### or \U######## (where # represents a hexadecimal
digit). So I can write "xán" or "x\u00E1n", and the two are
interchangeable.

>    notation

Here they're trying to encode characters for portable storage and exchange.
Is that what we need in this document?

> it is not a few format, the RFC is recent but it derives from an older

For RFC 3977 we only had to use UTF-8 for "content" type things, not for
"control" type things. So, for example, a NNTP command is still ASCII.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
THUS plc            |                            |

---

From: Julian Reschke <[email-masked]>
Date: Tue, 09 Jan 2007 09:01:10 +0100

>    UNICHAR    = "&#x" 2*6HEXDIG ";"

The reason why XML's production is complex is that it excludes 
characters that do not belong into identifiers.

The escaping rule quoted above doesn't solve that problem at all; it's 
just an escaping rule.

> it is not a few format, the RFC is recent but it derives from an older

I just tried to understand how RFC4234 works with non-ASCII characters, 
and it's not obvious at all. Section 2.4 seems to deal with it but 
really sounds a bit like hand-waving.

Best regards, Julian

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 9 Jan 2007 09:17:53 +0100

On Tue, Jan 09, 2007 at 12:07:04AM +0000,
 Clive D.W. Feather <[email-masked]> wrote 

> I seem to have missed a message or two. What is the problem you're trying

Julian Reschke expressed it here:

http://www1.ietf.org/mail-archive/web/cosmogol/current/msg00007.html

Do note that we could be "partly Unicode". We could decide to allow
Unicode only in comments or only in comments and quoted identifiers.

> If it's identifiers using characters outside the ASCII regime, then
> you want people to be able to write them in their own language,

Hmmm, RFC authors are a small minority. RFC readers are more
numerous. An encoding solution, like in RFC 4646 ("Proven&#xE7;al"),
painful for the writer but allowing translators like Shadok to display
nice Unicode characters for the readers (assuming that Graphviz or
other back-ends are Unicode-aware) would be already a big step, it
seems.

> The way we solved this in the C Standard is that you can use such
> characters directly, or you can encode them as \u#### or \U########
> (where # represents a hexadecimal digit). So I can write "xán" or
> "x\u00E1n", and the two are interchangeable.

In C, only in strings, no, not in identifiers? (Cosmogol has only
identifiers.)

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 9 Jan 2007 09:24:02 +0100

> defend. As far as I can tell, there's no real chance to get it

Any pointer on such regulations? Presently, I know:

RFC 2277 / BCP 18 IETF Policy on Character Sets and Languages

RFC 2223 Instructions to RFC Authors

RFC 3536 Terminology Used in Internationalization in the IETF

But none seems to back up your claim.

(In the same way that no document seems to mandate that the RFC be
written in English: I'll translate the draft to French tomorrow.)

---

From: Julian Reschke <[email-masked]>
Date: Tue, 09 Jan 2007 10:02:32 +0100

> RFC 2277 / BCP 18 IETF Policy on Character Sets and Languages

It's not clear to me whether RFC2277 applies, because it speaks about 
protocols, not formats.

If it does apply, we need to consider Section 3.1 (requirement to allow 
UTF-8) and 4 (language tagging).

> ...

Best regards, Julian

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 09 Jan 2007 17:53:28 +0100

> It's not clear to me whether RFC2277 applies, because it speaks about

After you did the "interoperability tests" for RFC 4234 (thanks!) that
will be a full STD at the end of this week, and there are no UTF-8
identifiers in ABNF.

Adding something in the direction of XML 1.1 later would be trivial.

Let's enjoy the last years where that's not yet necessary (RFC 2277
mumbles something about at least 50 years, still 41 years to go. :-)

Frank

---

From: Julian Reschke <[email-masked]>
Date: Tue, 09 Jan 2007 19:32:45 +0100

> will be a full STD at the end of this week, and there are no UTF-8

"Interoperability tests"?

Best regards, Julian

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 9 Jan 2007 23:08:56 +0100

On Sun, Jan 07, 2007 at 09:52:55PM +0100,
 Stephane Bortzmeyer <[email-masked]> wrote 
 a message of 30 lines which said:

> Any idea about the support of UTF-8 in typical languages *and*
> parsing tools? For instance, with C and Yacc, I assume it is quite

Searching and asking C gurus, it seems that C programmers using Lex
and Yacc (or ANTLR) are out of luck here (they would have to do
everything themselves, not relying on these useful tools).

These tools rely heavily on tables, so you can imagine what it would
be to port them to the world of 100 000 characters, after they were
written for a world of 256 characters :-)

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 10 Jan 2007 05:59:50 +0000

> http://www1.ietf.org/mail-archive/web/cosmogol/current/msg00007.html

Okay.

My opinions:

(1) Allow UTF-8 in both quoted and unquoted identifiers. In quoted ones,
allow all characters. In unquoted ones, limit it to an "alphanumeric"
set. Don't attempt to do this limitation in the grammar.

(2) The requirements on an XML notation are very different from those on a
human-readable one (e.g. you don't need the ability to "group" states)
and I'm not convinced they fit in the same document.

> other back-ends are Unicode-aware) would be already a big step, it

But it's very little effort to allow UTF-8 in plain identifiers, so why not
do so? I would limit any encoding solution to quoted names, using them to
provide an ASCII representation.

By the way, I assume that

    begin
and
    "begin"

are interchangeable? Or are they intended to be different? Assuming they're
the same, then:

    xán
    "xán"
    "x\u00E1;n"

would also all be the same.

[I prefer \u because users are more likely to want to write & in a string.]

> In C, only in strings, no, not in identifiers?

In identifiers.

    int xán;

    xán = getchar ();

is legal in the current C Standard.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 10 Jan 2007 09:31:34 +0100

On Wed, Jan 10, 2007 at 05:59:50AM +0000,
 Clive D.W. Feather <[email-masked]> wrote 

> is legal in the current C Standard.

Not yet implemented, then:

unicode.c: In function 'main':
unicode.c:2: error: stray '\303' in program
unicode.c:2: error: stray '\241' in program
unicode.c:2: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'n'
unicode.c:2: error: 'n' undeclared (first use in this function)
unicode.c:2: error: (Each undeclared identifier is reported only once
unicode.c:2: error: for each function it appears in.)
unicode.c:4: error: stray '\303' in program
unicode.c:4: error: stray '\241' in program
unicode.c:4: error: 'x' undeclared (first use in this function)
unicode.c:4: error: expected ';' before 'n'

And this is gcc 4.1.2, a quite recent version.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 10 Jan 2007 09:34:34 +0100

On Wed, Jan 10, 2007 at 05:59:50AM +0000,
 Clive D.W. Feather <[email-masked]> wrote 
 a message of 70 lines which said:

> But it's very little effort to allow UTF-8 in plain identifiers,

As a parser author, I would not say so. Implementation reports are
welcome.

> are interchangeable? 

Correct. That's my understanding.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 10 Jan 2007 10:15:10 +0000

> Not yet implemented, then:

Sorry, I wasn't clear (and my memory was a bit loose).

The Standard says:

       An implementation may allow  multibyte
       characters  that  are not part of the basic source character
       set to appear in identifiers;  which  characters  and  their
       correspondence    to    universal    character    names   is
       implementation-defined.

An implementation *must* accept:

    int x\u00E1n;

It *may* accept:

    int xán;

and, if it does, the two names must [*] be interchangeable.

[*] Absent a level of language-lawyering that you really don't want to know
about.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 10 Jan 2007 10:22:47 +0000

Stephane Bortzmeyer said:
> Any pointer on such regulations?

I'm not sure about regulations, but you should perhaps get advice from the
Area Director. Certainly the advice we had in nntpext was that we had to
take account of UTF-8.

> written in English: I'll translate the draft to French tomorrow.)

Zut alors!

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 10 Jan 2007 10:46:37 +0000

> The escaping rule quoted above doesn't solve that problem at all; it's

Right.

This sort of exclusion is not easy to do in syntax. It's easier to do in a
separate restriction (in the C Standard, the syntax simply says you have a
"universal character name" and then there's a separate requirement:

       [#3] Each universal character name in  an  identifier  shall
       designate  a character whose encoding in ISO/IEC 10646 falls
       into one of the ranges specified in annex D.
).

> really sounds a bit like hand-waving.

I take it as saying that you have to write your grammar to show the
encoding that you want to use, and may want to have alternative grammars
for different contexts. It doesn't let you use non-ASCII characters in
grammars.

In RFC3977 we wrote the following, which seems to be acceptable to IETF:

    UTF8-non-ascii = UTF8-2 / UTF8-3 / UTF8-4
    UTF8-2    = %xC2-DF UTF8-tail
    UTF8-3    = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2UTF8-tail /
                %xED %x80-9F UTF8-tail / %xEE-EF 2UTF8-tail
    UTF8-4    = %xF0 %x90-BF 2UTF8-tail / %xF1-F3 3UTF8-tail /
                %xF4 %x80-8F 2UTF8-tail
    UTF8-tail = %x80-BF

So, putting it together, one approach would be something like this:

    name       = identifier / quoted-name

    identifier = id-initial *(["-"] 1*id-char)
    id-char    = id-initial / DIGIT
    id-initial = ALPHA / UTF8-non-ascii
        ; Each UTF8-non-ascii in an identifier shall designate a character
        ; whose encoding in ISO/IEC 10646 falls into one of the ranges
        ; specified in appendix X.

    quoted-name = DQUOTE 1*q-char DQUOTE
    q-char      = %x21 / %x23-5B / %x5D-7E / UTF-8-non-ascii / q-escape
        ; excludes DQUOTE and BACKSLASH from ASCII
    q-escape    = %x5C.75 4HEXDIG / %x5C.55 8HEXDIG

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 10 Jan 2007 10:48:10 +0000

> As a parser author, I would not say so. Implementation reports are

I'm sure I've done it in lex before now. Last time I looked, at least flex
was 8-bit clean. The "what's valid UTF-8" syntax isn't complicated.

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 10 Jan 2007 17:15:07 +0100

>     q-escape    = %x5C.75 4HEXDIG / %x5C.55 8HEXDIG

<shudder />  I strongly recommend to defer such decisions
for a future "Cosmogol"-bis.  Backslash-escapes are just
too horrible.  If absolutely necessary &#x103456; is bad
enough, and at least it's compatible with CHARMOD and XML.

And maybe Cosmogol-bis could later adopt UTF-8 directly.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 10 Jan 2007 17:23:04 +0100

Clive D.W. Feather wrote:
 
> Allow UTF-8 in both quoted and unquoted identifiers.

This is horrible.  If folks want XML 1.1 they know where to find it.

It's a specification language for state machines.  It should work
with any parser on any platform, including MS DOS before 3.2 or
the BASIC interpreter written by Bill Gates 198? under CP/M. :-(

---

From: Julian Reschke <[email-masked]>
Date: Wed, 10 Jan 2007 17:31:18 +0100

> This is horrible.  If folks want XML 1.1 they know where to find it.

What does this have to do with XML 1.1? Or with XML at all?

> the BASIC interpreter written by Bill Gates 198? under CP/M. :-(

Where did you get that requirement from?

Anyway, if the language allows textual content (such as comments), it 
should better handle non-ASCII characters. And the simplest way to do 
that is to base it on Unicode, and to require UTF-8 as encoding.

Best regards, Julian

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 10 Jan 2007 16:28:27 +0000

Frank Ellermann said:
> It's a specification language for state machines.

Only American state machines? Both English and French use non-ASCII
characters.

I suggest, once again, that we need Area Director advice on this. Either we
can get away with ASCII-only (as 4234 does) or we can't. In the latter
case, we might as well do it properly.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 10 Jan 2007 16:24:59 +0000

> enough, and at least it's compatible with CHARMOD and XML.

I'm not going to die in the ditch over \u versus &#x. I'm simply concerned
that people will want to use & too often in "simple" state machines:

    Ready + REPLY -> WAITING : "Ring bell & sound horn";

If you can tell me your objection to backslash-escapes, perhaps we can find
an agreeable compromise.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 10 Jan 2007 17:36:37 +0100

On Wed, Jan 10, 2007 at 05:31:18PM +0100,
 Julian Reschke <[email-masked]> wrote 

> Where did you get that requirement from?

Until now, and at least until a formal WG is formed, we have no real
requirments, just personal opinions. Discussing them is welcome.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 10 Jan 2007 17:53:21 +0100

> case, we might as well do it properly.

A few thoughts:

(1) With my request for I18N I definitively didn't want to distract from 
the more important discussions. Apologies for that.

(2) It seems to me that some kind of escaping will be needed anyway. If 
it's there, it'll be trivial to define it so that any Unicode character 
can be represented.

(3) I definitely think that working UTF-8 support should be considered 
standard nowadays.

(4) The reason why the situation for RFC4234 is different IMHO is 
historic. If it would be defined from scratch today, I18N would have to 
be considered as well.

Best regards, Julian

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 10 Jan 2007 22:09:05 +0100

> What does this have to do with XML 1.1? Or with XML at all?

XML 1.1 and 1.0 differ wrt to attribute names and other identifiers.
IIRC, I'm too lazy to check this now (but if you want me I'd try)

>> It should work with any parser on any platform, including MS DOS
>> before 3.2 or the BASIC interpreter written by Bill Gates 198?
>> under CP/M. :-(
 
> Where did you get that requirement from?

Looking at my own OS/2 box without UTF-8 I got an old gawk and 
REXX to implement this.  We'd want to transform the state machines
into other languages for presentation etc., for that we need KISS,
and no obscure escape mechanisms requiring native Unicode support.

On GMaNe the descricption for this list is:
"About languages for state machines in RFCs, especially Cosmogol".

RFCs are plain text US ASCII at the moment, and they always were.
At some point in time that will change, but not _here_ and _now_

> if the language allows textual content (such as comments), it 
> should better handle non-ASCII characters

Together with ABNF and similar constructs.  In the same way as
it's done there.  Not in some ad hoc backslash notation borrowed
from C, or the &#x103456 in RFC 4646, we'd want exactly the same
solution as for among others ABNF comments.  Without a crystal
ball we can't tell what this will be - I guess it won't be the
C or RFC 4646 notation, but native UTF-8.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 10 Jan 2007 22:30:48 +0100

Clive D.W. Feather wrote:

> If you can tell me your objection to backslash-escapes,
> perhaps we can find an agreeable compromise.

In CHARMOD the W3C folks recommend hex. in escape sequences,
that's okay for \1234, u+1234, or &#x1234;

They also recommend terminators, because \39fm or u+39fm is
hard to interpret - of course you said 4HEXDIG or 6HEXDIG,
but users would get this wrong no matter what you say.

And they recommend to use some existing mechanism instead
of inventing a new one.  That leaves hex. NCRs.  Which is
also nice for translations into XML 1.0.  But native UTF-8
would be better.  Except for old platforms like my box.

The newsgroup names as in 3977 aren't the same as state
machines or ABNF, besides USEFOR does not yet use this new
feature.

Even EAI (UTF8SMTP) won't allow UTF-8 in header field names,
it's about header field bodies, especially mail addresses,
where MIME offers no way to use use UTF-8.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 10 Jan 2007 22:36:48 +0100

Clive D.W. Feather wrote:

> The "what's valid UTF-8" syntax isn't complicated.

While I agree that the code shown below isn't complicated it's not
what I'd like to copy into any REXX scripts about state machines.

UTF32O:  procedure               /* UTF-8 to UTF-32BE decoder     */
   U.2 = xrange( x2c( '80' ), x2c( 'BF' ))
   SUB = x2c( '0000FFFD' )       ;  DST = ''
   parse arg SRC                 ;  LOS = length( SRC )

   do while LOS > 0
      parse var SRC LB 2 SRC     ;  LOS = LOS - 1
      LB = c2d( LB )             ;  TOP = 0

      if LB < 128 then  do
         DST = DST || x2c( d2x( LB, 8 ))  ;  iterate
      end

      if LOS > 0  then  TOP = c2d( left( SRC, 1 )) % 16
      select                     /* for CESU remove both LB = 237 */
         when  LB < 192             then  LEN = -0 /* trail bytes */
         when  LB < 194             then  LEN = -1 /* bad C0 + C1 */
         when  LB < 224             then  LEN = +1
         when  LB = 224 & TOP =  8  then  LEN = -2 /* E08x is bad */
         when  LB = 224 & TOP =  9  then  LEN = -2 /* E09x is bad */
         when  LB = 237 & TOP = 10  then  LEN = -2 /* EDAx is bad */
         when  LB = 237 & TOP = 11  then  LEN = -2 /* EDBx is bad */
         when  LB < 240             then  LEN = +2
         when  LB = 240 & TOP =  8  then  LEN = -3 /* F08x is bad */
         when  LB < 244             then  LEN = +3
         when  LB = 244 & TOP =  8  then  LEN = +3 /* F48x is ok. */
         when  LB < 248             then  LEN = -3 /* bad F4 - F7 */
         when  LB < 252             then  LEN = -4 /* bad F8 - FB */
         when  LB < 254             then  LEN = -5 /* bad FC + FD */
         otherwise                        LEN = -0 /* bad FE + FF */
      end

      BAD = ( LEN <= 0 )         ;  LEN = abs( LEN )
      if LOS < LEN   then  do
         BAD = 1                 ;  LEN = LOS
      end

      TOP = left( SRC, LEN )     ;  SRC = substr( SRC, LEN + 1 )
      TMP = verify( TOP, U.2 )   ;  LOS = LOS - LEN
      if TMP > 0  then  do       /* eat plausible trailing bytes: */
         BAD = 1                 ;  SRC = substr( TOP, TMP ) || SRC
         LOS = length( SRC )     /* but keep possible valid input */
      end                        /* bytes for the next iteration  */

      if BAD = 0  then  do       /* at this point input is valid: */
         LB  = x2b( d2x( LB ))   ;  LEN = verify( LB, 1 ) - 2
         LB  = copies( 0, LEN ) || right( LB, 6 - LEN )

         do until TOP == ''
            TMP = x2b( c2x( left( TOP, 1 )))
            LB  = LB || right( TMP, 6 )
            TOP = substr( TOP, 2 )
         end

         TOP = b2x( strip( LB, 'L', 0 ))
         DST = DST || x2c( right( TOP, 8, 0 ))
      end
      else  DST = DST || SUB
   end
   return DST

---

From: Julian Reschke <[email-masked]>
Date: Wed, 10 Jan 2007 22:43:58 +0100

> IIRC, I'm too lazy to check this now (but if you want me I'd try)

XML 1.1 has extended the set of name characters, but it included many 
non-ASCII characters in XML 1.0 already. So the situation hasn't changed 
significantly.

> and no obscure escape mechanisms requiring native Unicode support.

Well. Do you have a C compiler? I'm sure there are decent UTF-8 libs, 
such as in expat. It should compile on your OS/2 box, it sure did 
compile on my Atari TT seven years ago.

> At some point in time that will change, but not _here_ and _now_

But that's not for us to decide. I simply think that inventing new 
notations without taking care of non-ASCII characters is a good idea.

> C or RFC 4646 notation, but native UTF-8.

I think that what I proposed *was* native UTF-8.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 10 Jan 2007 22:54:43 +0100

> what I'd like to copy into any REXX scripts about state machines.

We are defining a *new* language here. For now, it's for use in IETF 
documents (which are ASCII only). That requirement may change. Or people 
may want to use it for different target documents, such as the W3C's, or 
OASIS.

Restricting that language because you happen to use a platform which is 
really really old doesn't compute for me. Unicode support has been 
around on all widely deployed platforms for many, many years now.

You would still have the choice to implement your parser without 
supporting non-ASCII characters. That would simple mean that state 
machines authored with non-ASCII characters wouldn't be processable on 
your platform. I don't think that this would be a problem for the 
Internet Community.

I'd really suggest that we defer this problem until later, when more 
implementor feedback is available. Let's focus for now on the key 
features (and I don't think I18N is one of them).

Best regards, Julian

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 10 Jan 2007 22:55:34 +0100

> "Interoperability tests"?

The comparison of Bill's, Harald's, and Julia's validator for
the "4234 to STD" last call, wasn't that you ?   This report:

http://www.ietf.org/IESG/Implementations/RFC4234_implem.txt

The start page for these reports begins with the statement:

"The IETF Standards Process requires at least two independent
 and inter-operable implementations for advancing a protocol
 specification to Draft Standard."

It's hard to say what "inter-operability" means wrt ABNF, but
apparently Lisa decided that Dave's old 2234 report was a tiny
bit too simple for a full standard:

http://www.ietf.org/IESG/Implementations/RFC2234-implementation-report.txt

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 10 Jan 2007 23:01:25 +0100

> features (and I don't think I18N is one of them).

+1 (as proposed in another article)

I wanted to show that "not complicated" isn't the same as "neglible".

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 10 Jan 2007 23:19:27 +0100

Julian Reschke wrote:
 
> Well. Do you have a C compiler?

Sure, MS C 6.0, gcc 2.6.3, and I could try to install an 
OpenWatcom (after buying a new hard disk), or try to revive
the almost dead hard disk with Watcom 10.
   
> I'm sure there are decent UTF-8 libs

Probably.  But I'd prefer to do anything related to strings
in REXX or gawk (3.0.6).  My box is only an example, like
Stephane's examples about older lexx / yacc / flex / bison.

> I simply think that inventing new notations without taking
> care of non-ASCII characters is a good idea.

ASCII is to some degree comptible with a future extension to
UTF-8.  With more or less obscure escape mechanisms we could
pick a bad idea.
 
> I think that what I proposed *was* native UTF-8.

Then we agree that we don't want any escape mechanisms _now_
with a chance to change our mind later if ABNF or RFCs would
go this way (I doubt it, but it's not absolutely clear).

Frank

---

From: Julian Reschke <[email-masked]>
Date: Thu, 11 Jan 2007 09:45:30 +0100

> the "4234 to STD" last call, wasn't that you ?   This report:

No, that wasn't me.

> http://www.ietf.org/IESG/Implementations/RFC4234_implem.txt

Thanks for the link.

Best regards, Julian

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 11 Jan 2007 13:27:41 +0100

Julian Reschke wrote:

>> the "4234 to STD" last call, wasn't that you ?

> No, that wasn't me.

Sorry, I misunderstood "Julian reviewed RFC4234" in
http://permalink.gmane.org/gmane.ietf.apps-discuss/234

Frank

---

From: Julian Reschke <[email-masked]>
Date: Thu, 11 Jan 2007 14:35:26 +0100

> http://permalink.gmane.org/gmane.ietf.apps-discuss/234

OK, just clarifying: I did look at RFC4234, recommended to poll 
tools-discuss for feedback, and also suggest some minor editorial 
tweaks. I wouldn't personally call that a "review".

Best regards, Julian

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 11 Jan 2007 15:03:19 +0000

>> care of non-ASCII characters is a good idea.

I hope you meant "... is *NOT* a good idea"?

> go this way (I doubt it, but it's not absolutely clear).

I would be happy to just allow native UTF-8 at present. However, if we
don't reserve an escape character in quoted strings now, we can never do so
in the future. I had reservations about using & as the escape character.

Actually, on second thoughts, that's not quite true. We could do escaping
via concatenation of constructs, something like:

    "This: " || &#1234; || " is an escaped character."

The main purpose of escape mechanisms is to allow parsers to be represented
in pure ASCII even when they contain non-ASCII characters. If we don't see
that as a requirement, then we don't need escape mechanisms as far as I can
see.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 11 Jan 2007 16:15:16 +0100

Yep.

> in the future. I had reservations about using & as the escape character.

Right. That's why I said we need an escape character anyway.

> that as a requirement, then we don't need escape mechanisms as far as I can

I'm tempted to say let's do what everybody else does, and that IMHO 
means having an escape character...

Best regards, Julian