Subject: [ietf-822] interpretation of whitespace inside obs-phrase

From: "Jay Freeman (saurik)" <[email-masked]>
Date: Sat, 25 Aug 2012 05:25:16 +0000 (UTC)

Hello. I am working on an implementation of an e-mail parsing library, and thereby am getting intimate with RFC5322.

I am consequently trying to understand how to interpret some of the semantics of whitespace while parsing addresses, and I have come across a specific situation where I have not understood the RFC. I was thereby hoping that someone may be able to offer their expertise.

(I will now apologize profusely if this is a misuse of this mailing list. I found a couple previous questions by someone working on a Ruby e-mail library while going through the archives, and thereby figured that it was at least not entirely frowned upon to ask such questions here.)

In this case, the specific examples I am working with are as follows. The part I am concerned with is the display name (although depending on the answer to this issue I may be forced to reevalulate other things I currently believe I understand).

1: |Jay "Freeman"|
2: |Jay R. Freeman|

For references, here are the higher-level (non-character class) rules from RFC5322 that are important for these two parses.

   display-name    =   phrase
   phrase          =   1*word / obs-phrase
   obs-phrase      =   word *(word / "." / CFWS)
   word            =   atom / quoted-string
   atom            =   [CFWS] 1*atext [CFWS]
   quoted-string   =   [CFWS]
                       DQUOTE *([FWS] qcontent) [FWS] DQUOTE
                       [CFWS]

Now, one problem I run into is that the grammar is somewhat ambiguous with regards to the placement of CFWS, however, I will use a greedy expansion for the purposes of interpreting these examples.

In the first case, I end up with a phrase, and in the second an obs-phrase. (I have included two different versions of the second parse, as technically both are greedy but using a different order for the arguments of the alternation in the obs-phrase rule.)

1: atom:|Jay | quoted-string:|"Freeman"|
2: atom:|Jay | atom:|R| "." atom:| Freeman|
2: atom:|Jay | atom:|R| "." CFWS atom:|Freeman|

At this point, the standard is clear that the whitespace surrounding the atoms and the quotation marks surrounding the quoted string (in addition to any whitespace outside of those, although this example has none) are semantically not part of the values.

   Both atom and dot-atom are interpreted as a single unit, comprising
   the string of characters that make it up.  Semantically, the optional
   comments and FWS surrounding the rest of the characters are not part
   of the atom; the atom is only the run of atext characters in an atom,
   or the atext and "." characters in a dot-atom.

   Semantically, neither the optional CFWS outside of the quote
   characters nor the quote characters themselves are part of the
   quoted-string; the quoted-string is what is contained between the two
   quote characters.  As stated earlier, the "\" in any quoted-pair and
   the CRLF in any FWS/CFWS that appears within the quoted-string are
   semantically "invisible" and therefore not part of the quoted-string
   either.

In these cases, I would then presume, I would end up with the display names |JayFreeman| and |JayR.Freeman|. In the first case I find this perfectly reasonable. In the second case, however, I'm somewhat confused by the lack of resulting whitespace.

      Note: The "period" (or "full stop") character (".") in obs-phrase
      is not a form that was allowed in earlier versions of this or any
      other specification.  Period (nor any other character from
      specials) was not allowed in phrase because it introduced a
      parsing difficulty distinguishing between phrases and portions of
      an addr-spec (see section 4.4).  It appears here because the
      period character is currently used in many messages in the
      display-name portion of addresses, especially for initials in
      names, and therefore must be interpreted properly.

Given this description, I would have assumed that the purpose of this expansion is to support clients that don't feel they need to provide quotation marks around names. I feel somewhat vindicated in this understanding due to RFC5536 2.1.

   o  Articles are conformant if they use the <obs-phrase> construct
      (use of a phrase like "John Q. Public" without the use of quotes,
      see Section 4.1 of [RFC5322]), but agents MUST NOT generate
      productions of such syntax.

However, without the whitespace--which I am required to ignore due to the rules on how atoms are parsed--this seems to not be a useful obs- exception. Am I fundamentally misunderstanding this situation? Is whitespace in these contexts actually preserved?

(If whitespace is preserved, how does one handle the whitespace between the display name and the addr-spec, or whitespace between other random atoms in the specification, or whitespace preceeding the display name after the "To:"?)

For completeness, I can also come up with a second way to interpret the second example, which is |JayR. Freeman|, as neither of the above semantics rules indicate that the CFWS in the obs-phrase is to be semantically ignored, and should thereby become a space.

   Runs of FWS, comment, or CFWS that occur between lexical tokens in a
   structured header field are semantically interpreted as a single
   space character.

That said, I am not certain if these are technically even "lexical tokens in a structured header", as my read of other sections of the specification indicate that the structured header itself only has a single token, an addr-list, and inside of that token there must be explicit rules regarding how whitespace is parsed.

Finally, for comparison, I have attempted to parse this using a few implementations to see what they do. (BTW, if this is interesting, I'm happy to do more work and test actual clients.) I have added an additional test, 3:|"Jay"  R  Freeman|, to stress multiple spaces.

Java JavaMail:
1:|Jay "Freeman"|
2:|Jay R. Freeman|
3:|"Jay"  R  Freeman|

Java MIME4J:
1:|Jay Freeman|
2:|Jay R. Freeman|
3:|Jay  R  Freeman|
3:|Jay R Freeman| (Lenient)

Python email.utils:
1:|Jay Freeman|
2:|Jay R. Freeman|
3:|Jay R Freeman|

Ruby TMail:
1:|Jay Freeman|
2:|Jay R.Freeman|
3:|Jay R Freeman|

PHP mailparse_rfc822_parse_addresses:
1:|Jay Freeman|
2:|Jay R.Freeman|
3:|Jay R Freeman|

PHP imap_rfc822_parse_adrlist:
1:|Jay Freeman|
2:|Jay R. Freeman|
3:|Jay  R  Freeman|

Of these results, there is actually very little similarity :(. MIME4J, when using its "lenient" parser returns the same results as Python's e-mail.utils, and Ruby's TMail returns the same results as PHP's mailparse extension.

(Incidentally, I believe that the reason the PHP imap extension returning different results from the PHP mailparse extension is that the imap extension is calling out to c-client, whereas the mailparse extension was coded in-house.)

So, again, if anyone here is willing to help me understand what the correct behavior here is, I would be most appreciative. ;P

Sincerely,
Jay Freeman (saurik)
[email-masked]

---

From: Timo Sirainen <[email-masked]>
Date: Sat, 25 Aug 2012 11:17:12 +0300

>                       DQUOTE *([FWS] qcontent) [FWS] DQUOTE

There is no dot-atom above.

>   or the atext and "." characters in a dot-atom.

So the "dot-atom" mentioned here doesn't apply.

>   semantically "invisible" and therefore not part of the quoted-string

Both of these basically say that the whitespace outside atoms/quoted-strings doesn't matter for that specific atom/quoted-string.

> In these cases, I would then presume, I would end up with the display names |JayFreeman| and |JayR.Freeman|.

But they don't say that whitespace doesn't matter for the entire phrase, only its individual parts. I'm not sure if anything specifically requires you to show the whitespace in any specific way though, so my parser attempts to use a single space character where there originally was whitespace.

The way I handle obs-phrase is to try to parse either quoted-string or "atom-or-dot" and then append a space. The "atom-or-dot" is my own thing, which parses atom chars or dot chars until it reaches something else. So the end result is that you get: 

1:|Jay Freeman|
2:|Jay R. Freeman|
3:|Jay R Freeman|

And also |Jay.R.Freeman| also produces |Jay.R.Freeman| (I think many parsers will add whitespace after dots there).