Subject: Atoms and phrases (Was: UTF-8 in headers)

From: Pete Resnick <[email-masked]>
Date: Thu, 4 Feb 1999 16:07:37 -0600

I have Bcc'ed this message to [email-masked] since this discussion 
originated there, but have directed the discussion to DRUMS, since 
this is a DRUMS issue. Please direct future followup on this subtopic 
to the DRUMS list.

On 2/4/99 at 2:23 PM -0500, Keith Moore wrote:

>> I think not. I was working from the syntax in DRUMS (admittedly my 
>> version was dated March 1998, so it may be different now). The 
>> syntax I have gives:
>>
>> atom = [CFWS] 1*atext [CFWS]
>>
>> so "foo" and "bar" are atoms (the CFWS is optional).
>
> This probably points to a need for a clarification in DRUMS -
> an atom is intended to be the entire contiguous sequence of
> atext.
>> so "foobar" is a phrase consisting of two words side-by-side, as 
>> allowed by the syntax. It can also be legitimately parsed in many 
>> other ways, of course.
>
> this may be allowed by the grammar, but was not intended.

It was not intended. Back in draft -05, there was a sentence in the 
syntax discussion of the atom definition that said "Two atoms have to 
be separated by some other token, since putting two atoms next to 
each other would create a single atom." It was pointed out to me that 
this was incorrect, since the atom token itself includes the optional 
CFWS, so I removed the sentence. However, in the semantics 
description of atom, it says: "Semantically, the optional comments 
and FWS surrounding the rest of the characters are not part of the 
token; the token is only the run of atext characters in an atom..."

If further clarification is needed in the atom definition, or in the 
definition of phrase, to disambiguate, I'm open for suggestions.

pr
--
Pete Resnick <mailto:[email-masked]>
Eudora Engineering - QUALCOMM Incorporated
Ph: (217)337-6377 or (619)651-4478, Fax: (619)651-1102

---

From: <[email-masked]>
Date: Thu, 4 Feb 1999 17:55:56 -0500

Pete,

>> atom = [CFWS] 1*atext [CFWS]

The fact that "foobar" can be viewed as a single atom, or as a sequence of
two or more atoms (i.e., "foo" and "bar") is, I think, a red herring. If
the 822bis or ABNF grammars allowed a sequence of two atoms without any
intervening non-atext characters, then it would pose a problem, but I
don't believe that it does.

The whole point of a grammar is that it is possible to assign terminals in
a string to non-terminals in a grammar in a way that satisfies the
production rules of the non-terminals, but unless this assignment results
in a valid sentence, then this is an erroneous assignment. The whole
purpose of parsing is to eliminate erroneous assignments, and select a
single valid assignment. Clearly if a grammar is ambiguous, then the
latter is not possible.

Nick

Nick Shelness
Chief Technology Officer
Lotus Development Corp
Tel: (617) 693-7484

Assistant: Kathy McCullen
Tel: (617) 693-7521

---

From: <[email-masked]>
Date: Thu, 4 Feb 1999 23:27:32 +0000

Pete,

I spoke too quickly.

phrase          =       1*word / obs-phrase

This is a cause of formal ambiguity in the grammar. The fact that this ambiguity
has no semantic value is what may make it acceptable, but it would be nice to
find a set of production rules that remove the ambiguity. I'll give it a go.

Nick

Nick Shelness
Chief Technology Officer
Lotus Development Corp
Tel: (617) 693-7484