Subject: [Sip] ABNF issue

From: Christer Holmberg <[email-masked]>
Date: Thu, 05 Dec 2002 12:17:34 +0200

Hi,

The ABNF syntax in RFC3261 says the following things:

LWS              =  [*WSP CRLF] 1*WSP ; linear whitespace

Request          = Request-Line
                          *( message-header )
                          CRLF
                          [ message-body ]

message-body  =  *OCTET

Also, the syntax says that every message-header ends with CTRL.

Now, the spec says that the SIP part of a message ends with CTRL CTRL.

However, if the message-body part starts with one or more WSP, won't the

CRLF between the message-body be interpreted as LWS instead? Ie, there
will NOT be CRLF CRLF at the end of the SIP part of message, but instead

CRLF LWS.

I think this is a bug in the ABNF, since the message-body from a syntax
point
of view is part of the SIP messages, so they can't be "separated".

Now, to solve this the ABNF needs to be changed, so that the definition
for LWS is not valid at the beginning of a new line (when a
message-header has been successfully parsed), ie LWS is only valid
within headers. Then, if a new line starts with CRLF, is is CRLF no
matter what comes after.

Regards,

Christer Holmberg
Ericsson Finland

_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol
Use [email-masked] for questions on current sip
Use [email-masked] for new developments on the application of sip

---

From: Jonathan Rosenberg <[email-masked]>
Date: Thu, 05 Dec 2002 11:32:13 -0500

inline.

> will NOT be CRLF CRLF at the end of the SIP part of message, but instead

I don't know how you come to this conclusion. If the body starts with 
LWS, you will see a CRLF after the last header field, another CRLF 
before the body, and then an LWS which is the first byte of the body. Thus:

CRLF CRLF LWS

which is fine.

-Jonathan R.

-- 
Jonathan D. Rosenberg, Ph.D.                72 Eagle Rock Ave.
Chief Scientist                             First Floor
dynamicsoft                                 East Hanover, NJ 07936
[email-masked]                     FAX:   (973) 952-5050
http://www.jdrosen.net                      PHONE: (973) 952-5000
http://www.dynamicsoft.com

---

From: Christer Holmberg <[email-masked]>
Date: Fri, 06 Dec 2002 00:11:33 +0200

Hi Jonathan,

> before the body, and then an LWS which is the first byte of the body. Thus:

The body does not start with LWS. The body starts with WSP (RFC3261 doesn't
define what WSP is, but if I remember correct it is SP or TAB). So, in that
case you will have:

CRLF CRLF WSP

...which, according to the ABNF is:

CRLF LWS

Regards,

Christer Holmberg
Ericsson Finland

---

From: Jonathan Rosenberg <[email-masked]>
Date: Fri, 06 Dec 2002 01:01:36 -0500

inline.

> ...which, according to the ABNF is:

You are taking this out of context. It is not the case that you would 
parse through the message replaceing all instances of CRLF WSP with LWS.

The BNF specifies that message-header ends with a CRLF. What follows 
that is one of two things - either another CRLF, marking the end of the 
headers, or another message header, which never begins with LWS. Thus, 
if a CRLF is followed by something OTHER than WSP, the next construction 
cannot be LWS.

-Jonathan R.

---

From: Christer Holmberg <[email-masked]>
Date: Mon, 09 Dec 2002 09:16:11 +0200

Hi,

> parse through the message replaceing all instances of CRLF WSP with LWS.

So, where in the rules is it said when I should do that, and when I should not?
To my understanding the LWS rule is defined as a "general rule".

> The BNF specifies that message-header ends with a CRLF.

Yes.

> What follows that is one of two things - either another CRLF, marking the end
> of the
> headers, or another message header, which never begins with LWS.

From a SIP grammar point of view that is correct. However, I don't see it said
anywhere that the definition of LWS isn't valid at the beginning of a new line.

Also, just for the record, where is it said that a sip-header can't start with
LWS? Where is it said that line folding can't be used at the beginning of a line?
The text only says that the sip-header field values can be folded onto multiple
lines.

> Thus, if a CRLF is followed by something OTHER than WSP, the next construction

> cannot be LWS.

Maybe I missunderstand you, but the case I try to describe is when the CRLF IS
followed by WSP.

Regards,

Christer Holmberg
Ericsson Finland

---

From: Robert Sparks <[email-masked]>
Date: 09 Dec 2002 09:29:38 -0600

inline

> To my understanding the LWS rule is defined as a "general rule".

You need to be very careful here.

Just because there is a production rule that says A -> BC
does not mean all occurrences of BC in the language resulted
from the expansion of A. 

Blindly doing a reverse token substitution will produce an incorrect
parse. You can only apply it when you have proven the left side of
the rule fits in a valid parse of the message.

So from your original example. LWS is not defined inside a body - the
body is just a sequence of octets from the perspective of the SIP
grammar. Your SIP parser _cannot_ apply a match against LWS inside it -
there is no valid expansion of the SIP grammar that has LWS appearing
after CRLF CRLF
> anywhere that the definition of LWS isn't valid at the beginning of a new line.

It _is_ valid after CRLF (which is what I think you mean by a new line)
if (and only if) you are still in the expansion of message-header.

> The text only says that the sip-header field values can be folded onto multiple
I don't know what you mean by beginning of a line here. I assume you are
asking why we cant have a header that looks like (\b for blank):
\b\bNewHeader: foo
which is a "\b\bNewheader"?

The grammar answers that. 
extension-header starts with a token.
> Maybe I missunderstand you, but the case I try to describe is when the CRLF IS
If its in a valid expansion of message-header (which will not happen
across a CRLFCRLF string), then you are looking at LWS. The _only_
place that second CRLF matches in the grammar is the literal in
the Request and Response rules. So for your example, any whitespace
following that second CRLF matches message-body = *OCTET, NOT LWS.

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 11 Dec 2002 10:14:40 +0200

Hi,

> the rule fits in a valid parse of the message.

[CHH] Correct. And, I "shift" a message-header when I have found one CRLF (the end of
the header). After that I'm done with that header. After that I continue parsing, and
if the previous message-header was the last one I expect to parse a second CRLF, to
indentify the end of the SIP header part of the message. So far I agree. What we
disagree on is the fact that you say that now the definition of LWS is not valid
anymore (even if we are still parsing the SIP part of the message), and that in this
case a CRLF is a CRLF - no matter what comes after it.

> there is no valid expansion of the SIP grammar that has LWS appearing

[CHH] I don't think there is such thing as CRLF CRLF in the current SIP grammar (it
used to be before, though, I think). Again, the message-header is shifted once I found
one CRLF, so there will be no CRLF CRLF shift.

> >>of the headers, or another message header, which never begins with LWS.
> >

[CHH] That is what I am asking: where is it said that LWS is only valid within the
context of a message-header?

I do agree that LWS is not valid within the message-body part of the message, once we
get there, but I see it nowhere said that I can't parse a LWS AFTER I've shifted a
message-header, when I'm still in the SIP part of the message. No, the SIP grammar may
not allow it, and we may return a parser error, but that is a separate issue.

> \b\bNewHeader: foo which is a "\b\bNewheader"?
>
> The grammar answers that. extension-header starts with a token.

[CHH] Of course the LWS would not be part of the message-header grammar itself. LWS is
not even part of the message-header grammar within a header (ie the SIP grammar doesn't
require any LWS to be parsed). The grammar only says that LWS can be used
message-header parts for line folding, but it's not part of the grammar.

Let's take an example:

INVITE sip:[email-masked] SIP/2.0 <CRLF_1>
Call-ID: [email-masked] <CRLF_2>
<CRLF_3>
<SP_1>CSeq: 100 INVITE <CRLF_4>

Now, the parser shifts the Call-ID header when it has found CRLF_2.

Then, it finds CRLF_3 and SP_1, and returns LWS. So, there is a line-folding before the
beginning of the CSeq header. Again, the LWS is NOT part of the CSeq header itself, but
my question is: where is it said that this LWS is NOT allowed at the beginning of a
line, before the actual message-header, but only WITHIN a message-header? Have I missed
it in the text?

> across a CRLFCRLF string), then you are looking at LWS.

[CHH] I may be wrong, but again I don't think there is such a thing as CRLFCRLF
(DOUBLE-CRLF) in the current SIP grammar.

Each CRLF is treated separately, and if there is another CRLF after a message-header
has been shifted we have found the end of the SIP part of the message. I don't argue on
that point.

My issue is, that once we've shifted a message-header, why isn't the definition of LWS
valid anymore, since we still are parsing the SIP part of the message?

> The _only_ place that second CRLF matches in the grammar is the literal in
> the Request and Response rules.

[CHH] Again, the point is not where it matches in the grammar. I am not talking about
the SIP grammar, but of the definition of the parts/tokens building the grammar.

> So for your example, any whitespace following that second CRLF matches message-body =
> *OCTET, NOT LWS.

[CHH] Again, I don't think we have any token called the "second CRLF" in the SIP
grammar.

I also think it's important to have a look at the definition of OCTET. RFC2234 says:

OCTET          =  %x00-FF

SP                  =  %x20

Now, if we say that CRLF/SP is LWS, but that CRLF/OCTET is not LWS, don't we have a
conflict here, since %x20 is part of both SP and OCTET?

Anyway, I don't want to go on with this argument forever. I am not an ABNF expert, and
it is possible to fix this on the implementation level, so...

Regards,

Christer Holmberg
Ericsson Finland