Subject: RE: [Megaco] The ABNF coding of VALUE doesn't allow to representa generic UTF-8 string/char

From: "Kevin Boyle" <[email-masked]>
Date: Tue, 28 Jun 2005 16:11:58 -0400

I will note that H.248.23 also has avoided this problem by encoding the data
in hex.  This, along with the solution in .9, provides examples of two
possible solutions to both problems.

You continue to point out that ASN.1 doesn't have this problem.  That is
because ASN.1 uses the hex encoding, and can avoid some of the end of string
detection problems that ABNF inherently possesses.  .23 uses this same idea
-- hex encoding in the ABNF allows explicit detection of the end of the
value string without adding a collision with the delimiters.

Steve makes some good points about the fact that the upper half of UTF-8
does not collide with the delimiter, and therefore shouldn't be a problem.
I have no problem with this as a solution.  A contribution should be taken
to SG16 for this proposal.

Kevin 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Contardi Angelo
Sent: Tuesday, June 28, 2005 4:26 AM
To: Steve Cipolli; Megaco IETF - Mail List (E-mail)
Cc: Christian Groves (E-mail)
Subject: R: [Megaco] The ABNF coding of VALUE doesn't allow to
representageneric UTF-8 string/char

Hello Steve,

if i correctely undestend what you mean, you split the problem to carry ALL
UTF-8 chars/strings in TWO steps:

Step 1) Carry ALL pritable UTF-8 chars, except 0x22 (").
Step 2) Carry The rest of other UTF-8 chars.

Your proposal solve just the first step. And the second one, how do you
solve ?
Adding a new form of VALUE? Which ? A generic UTF-8 string shold then be
formed by a concatenation of quoted, for "printable" UTF-8 chars, and
"somethingelse" , for "non printable" UTF-8 chars ? Is it feasible and
reasonably simple to implement ?
Or is better to leave the actual "status quo" ?

I think the problem of UTF-8 chars should be solved "globally" because of
this problem rise from "general packages definition" (paragraphs 12.x.x) in
which it's stated that the VALUEs Type of Paramaters/Properties may be also
(ANY, not a subset) UTF-8 String or Char.
Actually these two types are allowed just by ASN.1 syntax, not by ABNF.
In other points of the syntax where VALEU is used, UTF-8 aren't required
(ASCII it's enought).
I think nobody care to tx UTF-8 strings in "error text" or in other quoted
strings different form packages Parameters or Properties values.
So the extension of ABNF syntax to carry UTF-8 in VALUE should be FULL
support of UTF-8, as actually is the ASN.1 syntax. Otherwise may be adopted
the "trick" used in H.248.9 with the actual ABNF syntax but should be
specified in the "general packages definition" that the
UTF-8 string/char type is valid only for ASN.1 syntax and the implementor of
packages that need UTF-8 string/char type must specify the coding/decoding
UTF-8/xxx scheme (xxx = ASCII or Octect or something else), as already done
in H.248.9.

-----Messaggio originale-----
Da: Steve Cipolli [mailto:[email-masked]]
Inviato: lunedì 27 giugno 2005 20.32
A: Contardi Angelo; Megaco IETF - Mail List (E-mail)
Cc: Christian Groves (E-mail)
Oggetto: RE: [Megaco] The ABNF coding of VALUE doesn't allow to represent
ageneric UTF-8 string/char

See below.

--Stephen

> -----Original Message-----
> From: Contardi Angelo [mailto:[email-masked]]
> Sent: Monday, June 27, 2005 5:59 AM
> To: Steve Cipolli; Megaco IETF - Mail List (E-mail)
> Cc: Christian Groves (E-mail)
> Subject: R: [Megaco] The ABNF coding of VALUE doesn't allow to 
> represent ageneric UTF-8 string/char
> 
> 
> He Steve,
> 
> i don't understand these points:
> 
> 1) Where are included non printing chars from 0x01-0x31 and
> 0x127 in VALUE? Note that the "general
>    definition of packages" (pargraphs 12.x.x of H.248.1) allow those 
> chars too. Otherwise it should
>    be stated somewhere the opposite.
This issue exists for ASCII today.  It is not something the UTF-8
introduces.  I do not disagree that this should be solved, but it is a
separate problem from adding UTF-8.  The 2 problems can be solved
independently.
> 
> 2) Why  %x80-F7 should be included in both "quoted" and "non quoted" 
> form of value. Isn't it enought
>    to include them just in quoted form of VALUE ?  
VALUE takes both quoted and non-quoted forms.  The quoted form is used when
a string contains unsafe characters (ie. those that conflict with the ABNF
syntaxes delimiters).  Typically strings containing only "safe characters"
are not quoted.  Strings containing unsafe characters must be quoted.  I
won't argue the merits of this logic, but this is the current syntax and its
usage.  The UTF-8 chars above 0x7F are "safe characters" (ie. they do not
interfere with the ABNF sysntax for H.24) and therefore do not require
quoting.  Allowing both forms is simply keeping consistency with how VALUE
was originally defined.   Adding UTF-8 does not introduce any new issues in
this regard (ie. it doesn't add unsafe characters) and so should not
introduce any new syntax rules (what characters are safe and what are
unsafe).  
> 
> 3) How to TX 0x22 (")? It's a NEED, not an optional to tx
> 0x22 too, because of, for instance,
>    the package "Text telephone package", described in the 
> specification H.248.2, need to tx ANY
>    UTF-8 string, 0x22 included. I tink this package is used to tx a 
> text writed by users and so
>    it may include 0x22 too. And actually 0x22 can't be tx in ANY VALUE 
> form. So how do you
>    solve this problem? 
This problem exists today without UTF-8.  It is a completely separate issue.
We can solve the UTF-8 problem with or without solving the 0x22 problem.
Likewise we can solve the 0x22 problem with or without solving the UTF-8
problem.  I'm not against solving the 0x22 problem, but it need not be
solved at the same time or in conjunction with the UTF-8 problem.
> 
> -----Messaggio originale-----
> Da: Steve Cipolli [mailto:[email-masked]]
> Inviato: venerdì 24 giugno 2005 21.11
> A: Contardi Angelo
> Cc: Sasha Ruditsky; Christian Groves; [email-masked]; 
> [email-masked]
> Oggetto: RE: [Megaco] The ABNF coding of VALUE doesn't allow to 
> represent ageneric UTF-8 string/char
> 
> 
> Angelo,
> 
> Today all ASCII characters except double quotes can be specified by 
> the VALUE production (some of the characters need to be in the 
> quotedstring form).  Adding UTF-8 does not change this.  The part of 
> UTF-8 that is < 0x80 is exactly ASCII and does not change or need to 
> change the way in which VALUE is encoded.
> 
> UTF-8 characters above 0x7F are encoded as multiple bytes in which 
> each byte of the character is guaranteed to be > 0x7F.
> So UTF-8 characters are either < 0x80 and are simply the equivalent of 
> ASCII or they are encoded into multiple bytes in which every byte of 
> the character is > 0x7F.  There will never be a circumstance where a 
> byte of a UTF-8 multibyte character collides with _any_ ASCII 
> character.
> 
> Perhaps the table below will help clarify
> 
> Scalar Value 			1st Byte 2nd Byte 3rd Byte 4th Byte
> 00000000 0xxxxxxx 		0xxxxxxx
> 00000yyy yyxxxxxx 		110yyyyy 10xxxxxx
> zzzzyyyy yyxxxxxx 		1110zzzz 10yyyyyy 10xxxxxx
> 000uuuuu zzzzyyyy yyxxxxxx 	11110uuu 10uuzzzz 10yyyyyy 10xxxxxx
> 
> Notice that "one byte" UTF-8 characters have their high order bit 
> cleared (0) and that all bytes of all multi-byte UTF-8 encodings have 
> the high-order bit set (1).  In other words all multi-byte character's 
> bytes are > 0x7F.
> 
> So changing the productions to:
>   VALUE                = quotedString / 1*(SafeChar / %x80-F7)
>   quotedString         = DQUOTE *(SafeChar / %x80-F7 / 
> RestChar / WSP) DQUOTE
> 
> as Sasha Ruditsky suggests will allow UTF-8 characters to be used in 
> VALUEs.  ASCII characters (and UTF-8 equvialents <
> 0x80) that are in the RestChar set will need to be quoted just as they 
> have always been.  UTF-8 characters above 0x7F may or may not be 
> quoted, just as all the ASCII characters in the SafeChar set.
> 
> --Stephen
> 	
> 
> > -----Original Message-----
> > From: Contardi Angelo [mailto:[email-masked]]
> > Sent: Friday, June 24, 2005 12:25 PM
> > To: Steve Cipolli
> > Subject: R: [Megaco] The ABNF coding of VALUE doesn't allow to 
> > represent ageneric UTF-8 string/char
> > 
> > 
> > 
> > 
> > -----Messaggio originale-----
> > Da: Steve Cipolli [mailto:[email-masked]]
> > Inviato: venerdì 24 giugno 2005 18.01
> > A: Contardi Angelo; Megaco IETF - Mail List (E-mail)
> > Cc: Christian Groves (E-mail)
> > Oggetto: RE: [Megaco] The ABNF coding of VALUE doesn't allow to 
> > represent ageneric UTF-8 string/char
> > 
> > 
> > 
> > Two points:
> > 
> > 1. It is not necessary to quote the UTF-8 characters as you define 
> > in your second solution.  VALUE can be extended to accept characters 
> > (quoted and unquoted) in the range 0x80-0xFF.
> > 
> > The problem is that VALUE don't allow to "carry" ALL ASCII
> > (0x00-0x7F) too, NEEDED to code UTF-8 characters < 0x80. So the only 
> > "extension" 0x80-0xFF is NOT enough to allow the TX of ALL UTF-8 
> > with ABNF. The quotation is NEEDED becaue of if you "scan" an input 
> > sequence and allow a VALUE as "any sequence of ono or more 
> > 0x00-0xFF", all "tokens" identified as VALUEs. The extension of 
> > quoted form of VALUE i propose don't increment the number of the 
> > forms of VALUE, just EXEND the actual Quoted form of VALUE
> > 
> > 2. Providing a mechanism to allow double quotes (") in a
> > (double) quotedstring is essentially a separate issue, since 
> > addition of UTF-8 chars does not motivate the need for this 
> > mechanism nor complicate its addition.
> > 
> > See point 1.
> > 
> > --Stephen
> > 
> > > -----Original Message-----
> > > From: [email-masked] [mailto:[email-masked]] On 
> > > Behalf Of Contardi Angelo
> > > Sent: Friday, June 24, 2005 11:34 AM
> > > To: Megaco IETF - Mail List (E-mail)
> > > Cc: Christian Groves (E-mail)
> > > Subject: [Megaco] The ABNF coding of VALUE doesn't allow to 
> > > represent ageneric UTF-8 string/char
> > > 
> > > 
> > > Hello,
> > > 
> > > from my previous  private  discussion with Christian
> Groves, i have
> > > deduce this:
> > > 
> > > The ABNF coding of VALUE doesn't  allow to  represent a generic
> > > UTF-8 string or char (RFC 3629) because of:
> > > 
> > > 1) The FIRST byte of an UTF-8 char may be ANY ASCII char in the 
> > > range  0x00-0x7E
> > >    and, for instance, the  ASCII 0x00-0x07  and 0x22 (") are not  
> > > allowed in ANY
> > >    ABNF form of VALUE.
> > >    
> > > 2) Furthermore, UTF-8 chars "greater than 0x7E", need
> "ASCII chars"  
> > > (to better
> > >    say, Octets) in the range 0x80-0xFF (not ALL), not allowed in  
> > > ANY  ABNF form
> > >    of VALUE.
> > > 
> > > So, to "correct" this problem, i can suggest two possible
> solutions:
> > > 
> > > 1) The simple one, code the UTF-8 string/char  in "Octect
> Mode", as
> > > described in
> > >    ANNEX B.3. This is a BAD solution from the efficiency 
> > > transmission  point of
> > >    view because of it "halve the TX band": to TX one  UTF-8 char  
> > > (max 4 Octet)
> > >    i must TX max 2 x 4 = 8 Octet (ASCII chars).
> > > 
> > > 2) The difficult one, allow the ABNF quoted form of VALUE
> to TX ALL
> > > ASCII chars
> > >    0x01-0xFF (the range 0x80-0xFF are more  properly  named OCTET 
> > > in RFC2234 ),
> > >    except 0x22 ("), that should be  ESCAPED with "\", as already  
> > > done for ABNF
> > >    Local and Remote Descriptor (see SDP).  Note that '\0' 
> > > (0x00) is  NOT ALLOWED
> > >    in this new quoted string form as in the present one, but it's  
> > > not a problem
> > >    because in UTF-8 the char '\0' (0x00) is same as in ASCII 
> > > (string terminator)
> > >    and is NOT used to code "non ASCII" UTF-8 chars,  all those 
> > > chars > 0x7F that
> > >    require more than one ASCII chars to be encoded (from 2 to
> > > 4 ASCII chars). In
> > >    fact the "extra chars" needed to code an UTF-8 char are all 
> > > above 0x7F.
> > > 
> > > While the 1st mode doesn't  require any  modification  of ABNF 
> > > syntax (but is it applicable in any case ?), the 2nd one require 
> > > this ABNF modification:
> > > 
> > >       quotedString         = DQUOTE *(quotedChar) DQUOTE
> > >   	
> > >       ; %x22 (") is allowed just if "escaped" with "\"
> > >       quotedChar           = ( %x01-21 / "\" DQUOTE / %x23-FF )
> > > 
> > > In a parser implementation, i think, this  is not a "terrible 
> > > complication" and can be solved in the same way of
> "octetString" of
> > > Local and RemoteDescriptor. It require  also to  implement an 
> > > ESCAPE "strip(rx)/padding(tx)"  mechanism, as already
> required for
> > > SDP.
> > > 
> > > P.S.: I suppose NO ONE need to  send the  "string
> terminator" '\0' 
> > > (0x00)  in an
> > >       UTF-8 string or char. If my  assertion is false,  in the 
> > > "solution 2)" the
> > >       %x00 should also be "escaped". Viceversa,  the "solution 1)"  
> > > can  already
> > >       "transport" %x00 octet.
> > > 
> > > Best regards
> > > 
> > >         o o o o o o o . . .   ___________________________________
> > >         o      _____           ||        Angelo Contardi         |
> > >       .][__n_n_|DD[  ====_____  |   [email-masked]   |
> > >      >(________|__|_[_________]_|________________________________|
> > >      _/oo OOOOO oo`  ooo   ooo  'o!o!o                      o!o!o`
> > > 
> > > _______________________________________________
> > > Megaco mailing list
> > > [email-masked] https://www1.ietf.org/mailman/listinfo/megaco
> > > 
> > > 
> > > 
> > 
> > 
> 
> 

_______________________________________________
Megaco mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/megaco

_______________________________________________
Megaco mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/megaco