Subject: [hybi] ABNF issue in draft-ietf-hybi-thewebsocketprotocol

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 28 Sep 2011 09:36:46 -0600

<hat type='AD'/>

Dear HYBI WG (IESG bcc'd):

Mykyta Yevstifeyev has raised an issue with the IESG regarding the ABNF 
rules in Section 5.2 of draft-ietf-hybi-thewebsocketprotocol, and he has 
said that he plans to formally appeal publication of this specification 
(in accordance with Section 6.5 of RFC 2026) if this issue is not 
addressed, since he is of the opinion that (as RFC 2026 describes it) 
"the Working Group has made an incorrect technical choice which places 
the quality and/or integrity of the Working Group's product(s) in 
significant jeopardy".

The issue is that the ABNF definitions for the frame-* rules (e.g., 
frame-fin and frame-rsv1) use a somewhat unconventional encoding. For 
instance, for frame-rsv1 the terminal values %x0 and %x1 are mapped to 
the digits 0 and 1, not to ASCII codes 0 and 1 (which are control codes 
in ASCII). This kind of mapping is not disallowed by the ABNF spec, 
since Section 2.3 of RFC 5234 states:

    Rules resolve into a string of terminal values, sometimes called
    characters.  In ABNF, a character is merely a non-negative integer.
    In certain contexts, a specific mapping (encoding) of values into a
    character set (such as ASCII) will be specified.

To address this issue, the editors added two parenthetical sentences to 
Section 5.2 in version -16:

    This wire format for the data transfer part is described by the ABNF
    [RFC5234] given in detail in this section.  (Note that unlike in
    other sections of this document the ABNF in this section is operating
    on groups of bits.  When encoded on the wire the most significant bit
    is the leftmost in the ABNF.)  A high level overview of the framing
    is given in the following figure.  In a case of conflict between the
    figure below and the ABNF specified later in this section, the ABNF
    version should be considered to be more authoritative.

Apparently Mykyta Yevstifeyev thinks that those sentences do not address 
his issue and that the WebSocket spec is not consistent with RFC 5234. 
WebSocket editor Alexey Melnikov, AD Pete Resnick, and I happen to 
disagree with Mykyta Yevstifeyev on this matter, and have discussed it 
with him at some length off-list.

However, interpretation of RFC 5234 aside, Mykyta Yevstifeyev also 
contends that the aforementioned ABNF rules are simply unnecessary 
because the figure that appears after the first paragraph of Section 5.2 
of draft-ietf-hybi-thewebsocketprotocol-16 is perfectly explanatory. 
During the off-list discussion of this matter, my co-AD Pete Resnick 
agreed that the fields are fully defined by the figure in Section 5.2 
and that the ABNF rules don't seem to add any further information to the 
document. Thus one approach to resolving this issue is not to wrangle 
over proposed text and the interpretation of RFC 5234, but instead to 
remove the ABNF rules from Section 5.2. Because these ABNF rules have 
been in the spec for a long time, instead of asking the editors to 
remove them I would prefer to find out if doing so would be acceptable 
to the WG.

Therefore I have two questions for the WG:

1. Are any implementers currently using the ABNF rules from Section 5.2?

2. Does anyone object to removing the ABNF rules from Section 5.2?

If possible, please reply by the close of business this Friday 
(September 30) so that I can approve the document and send it to the RFC 
Editor team before I go on vacation next week.

Thank you for your patience.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Joe Mason <[email-masked]>
Date: Wed, 28 Sep 2011 11:40:28 -0400

I have implemented a server using the diagram, without using the ABNF (or even ever reading the ABNF).

I have no objection to removing the ABNF.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Peter Saint-Andre
> Sent: Wednesday, September 28, 2011 11:37 AM
> To: [email-masked]
> Cc: Mykyta Yevstifeyev
> Subject: [hybi] ABNF issue in draft-ietf-hybi-thewebsocketprotocol
> 
> <hat type='AD'/>
> 
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Wed, 28 Sep 2011 08:44:34 -0700

I personally find it adds value and would object to its removal. We provide
ABNF for most everything else, it would seem like a rather odd omission to
take this out for what is essentially a style preference by one person who
has not contributed to the WG in a meaningful manner previously (to my
recollection) after it having been in the document for *well over a year*.

On Wed, Sep 28, 2011 at 8:36 AM, Peter Saint-Andre <[email-masked]>wrote:

> rules in Section 5.2 of draft-ietf-hybi-**thewebsocketprotocol, and he has
> said that he plans to formally appeal publication of this specification (in
> accordance with Section 6.5 of RFC 2026) if this issue is not addressed,
> since he is of the opinion that (as RFC 2026 describes it) "the Working
> Group has made an incorrect technical choice which places the quality and/or
> integrity of the Working Group's product(s) in significant jeopardy".
>
> instance, for frame-rsv1 the terminal values %x0 and %x1 are mapped to the
> digits 0 and 1, not to ASCII codes 0 and 1 (which are control codes in
> ASCII). This kind of mapping is not disallowed by the ABNF spec, since
> Section 2.3 of RFC 5234 states:
>
> WebSocket editor Alexey Melnikov, AD Pete Resnick, and I happen to disagree
> with Mykyta Yevstifeyev on this matter, and have discussed it with him at
> some length off-list.
>
> However, interpretation of RFC 5234 aside, Mykyta Yevstifeyev also contends
> that the aforementioned ABNF rules are simply unnecessary because the figure
> that appears after the first paragraph of Section 5.2 of draft-ietf-hybi-*
> *thewebsocketprotocol-16 is perfectly explanatory. During the off-list
> discussion of this matter, my co-AD Pete Resnick agreed that the fields are
> fully defined by the figure in Section 5.2 and that the ABNF rules don't
> seem to add any further information to the document. Thus one approach to
> resolving this issue is not to wrangle over proposed text and the
> interpretation of RFC 5234, but instead to remove the ABNF rules from
> Section 5.2. Because these ABNF rules have been in the spec for a long time,
> instead of asking the editors to remove them I would prefer to find out if
> doing so would be acceptable to the WG.
>
> If possible, please reply by the close of business this Friday (September
> 30) so that I can approve the document and send it to the RFC Editor team
> before I go on vacation next week.
>
> ______________________________**_________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/hybi<https://www.ietf.org/mailman/listinfo/hybi>
>

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 28 Sep 2011 18:20:30 +0200

>his issue and that the WebSocket spec is not consistent with RFC 5234. 

Well, consider

  frame-payload-length-63 = %x0000000000000000-7FFFFFFFFFFFFFFF

As you point out, ABNF's terminals are integers, so that is the same as

  frame-payload-length-63 = %x00-7FFFFFFFFFFFFFFF

Meaning there is no indication in the ABNF what these "groups of bits"
are and without that the ABNF does not define the wire format contrary
to the claim above. This could be fixed by keeping the terminal's width
consistent, so if you have bits, this would be, for instance,

  frame-payload-length-63 = 64( %x0 / %x1 )

I note that a formal grammar may well have a finite subset of the set
of integers as alphabet, and in that sense such a usage of ABNF is not
entirely wrong, but TCP does not transport integers, so as it is used
currently, it cannot serve as a definition for the wire format.

>1. Are any implementers currently using the ABNF rules from Section 5.2?

I have found the current ABNF usage very confusing.

>2. Does anyone object to removing the ABNF rules from Section 5.2?

If it is removed, there needs to be careful review to ensure all the
existing references to symbols in the grammar still make sense.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Wed, 28 Sep 2011 09:24:25 -0700

> consistent, so if you have bits, this would be, for instance,

We did add "This wire format for the data transfer part is described by the
ABNF
  [RFC5234] given in detail in this section.  (Note that unlike in
  other sections of this document the ABNF in this section is operating
  on groups of bits.  When encoded on the wire the most significant bit
  is the leftmost in the ABNF.)  A high level overview of the framing
  is given in the following figure.  In a case of conflict between the
  figure below and the ABNF specified later in this section, the ABNF
  version should be considered to be more authoritative."

---

From: Tobias Oberstein <[email-masked]>
Date: Wed, 28 Sep 2011 09:43:53 -0700

> 1. Are any implementers currently using the ABNF rules from Section 5.2?

I have implemented one server (Python), two clients (Python, Java/Android) and a
WS testsuite (Python) using the diagrams, without looking at the ABNF.

> 2. Does anyone object to removing the ABNF rules from Section 5.2?

No objections ..

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 28 Sep 2011 18:43:57 +0200

>  version should be considered to be more authoritative."

I was responding to exactly this text. The current ABNF grammar does not
"operate on groups of bits" in any recognizable way and does not define
the groups to begin with. `frame-opcode-non-control` for instance refers
to the same bits as `frame-opcode-control`, but that you actually use 4
bits for `frame-opcode-non-control` even though you only need 3 is left
for the reader to guess, much as you have to guess that you use 64 bits
for `frame-payload-length-63`. If you already understand the wire format
you can read through the grammar saying "ah, yes", but if you don't then
the grammar is probably worse than useless in its current form.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Wed, 28 Sep 2011 10:46:00 -0700

> WS testsuite (Python) using the diagrams, without looking at the ABNF.

And we did look at the ABNF in our implementation. I'm not sure where this
is getting us. We're bikeshedding yet again, this working group has come so
far and managed to overcome months of bikeshedding on many other topics, why
are we bikeshedding yet again at the very end?

-Ian

---

From: Tobias Oberstein <[email-masked]>
Date: Wed, 28 Sep 2011 11:50:53 -0700

I just tried to respond to Peter's questions to the WG as short as possible and without questioning them ..

_If_ there is a significant/confusing issue with the ABNF, I think it should be either fixed or the ABNF removed. Fixing prefered ..

Whether the _if_ is true, I have no opinion .. and thus refrain from "bikeshedding" - funny word I had to look btw.

Von: Ian Fette (イアンフェッティ) [mailto:[email-masked]]
Gesendet: Mittwoch, 28. September 2011 19:46
An: Tobias Oberstein
Cc: Peter Saint-Andre; [email-masked]; Mykyta Yevstifeyev
Betreff: Re: [hybi] ABNF issue in draft-ietf-hybi-thewebsocketprotocol

And we did look at the ABNF in our implementation. I'm not sure where this is getting us. We're bikeshedding yet again, this working group has come so far and managed to overcome months of bikeshedding on many other topics, why are we bikeshedding yet again at the very end?

-Ian

No objections ..

_______________________________________________
hybi mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/hybi

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 28 Sep 2011 13:31:06 -0600

On 9/28/11 12:50 PM, Tobias Oberstein wrote:
> _If_ there is a significant/confusing issue with the ABNF, I think it
> should be either fixed or the ABNF removed. Fixing prefered ..
>
> Whether the _if_ is true, I have no opinion .. and thus refrain from
> "bikeshedding" - funny word I had to look btw.

I do not think that this is an issue of great significance.

I do not think that the frame-related ABNF rules have caused great 
confusion among implementers.

I do not think that Mykyta Yevstifeyev is correct in his interpretation 
of RFC 5234.

I do not think that, by including the frame-related ABNF rules, "the 
Working Group has made an incorrect technical choice which places the 
quality and/or integrity of the Working Group's product(s) in 
significant jeopardy" (RFC 2026).

I think that this issue *could* be addressed by (a) clarifying the text 
in Section 5.2.

However, I also think that if no one is depending on these ABNF rules, 
this issue *could* be addressed by (b) removing them from the spec.

That said, if we choose (b), then I note what Bjoern observed:

"If it is removed, there needs to be careful review to ensure all the
existing references to symbols in the grammar still make sense."

In particular, please note that various paragraphs in Sections 5.3, 5.8, 
6.1, and 6.2 make reference to the ABNF rules from Section 5.2. Removing 
all of those rules would thus require somewhat extensive changes *very* 
late in the process.

Thus I would prefer that we find a way to clarify the text in Section 
5.2, if possible. Proposed text would be welcome.

Peter

--
Peter Saint-Andre
https://stpeter.im/

---

From: Gabriel Montenegro <[email-masked]>
Date: Wed, 28 Sep 2011 19:38:43 +0000

I agree that getting rid of this entirely is too risky and uncalled for. How about a clarification along these lines:

Just adding text to the effect that the diagram and not the ABNF is normative in the first paragraph in section 5.2.

OLD:	
   is the leftmost in the ABNF).  A high level overview of the framing
   is given in the following figure.  In a case of conflict between the
   figure below and the ABNF specified later in this section, the ABNF
   version should be considered to be more authoritative.

NEW:
   is the leftmost in the ABNF).  A specification of the framing
   is given in the following figure.  In a case of conflict between the
   figure below and the ABNF specified later in this section, the 
   figure is authoritative.

Gabriel

> Sent: 28 September, 2011 12:31
> To: Tobias Oberstein
> Cc: Pete Resnick; Mykyta Yevstifeyev; [email-masked]
> Subject: Re: [hybi] ABNF issue in draft-ietf-hybi-thewebsocketprotocol
> 
> I do not think that the frame-related ABNF rules have caused great confusion
> among implementers.
> 
> I do not think that Mykyta Yevstifeyev is correct in his interpretation of RFC
> 5234.
> 
> I do not think that, by including the frame-related ABNF rules, "the Working
> integrity of the Working Group's product(s) in significant jeopardy" (RFC 2026).
> 
> I think that this issue *could* be addressed by (a) clarifying the text in Section
> 5.2.
> 
> However, I also think that if no one is depending on these ABNF rules, this issue
> *could* be addressed by (b) removing them from the spec.
> 
> "If it is removed, there needs to be careful review to ensure all the existing
> references to symbols in the grammar still make sense."
> 
> In particular, please note that various paragraphs in Sections 5.3, 5.8, 6.1, and
> 6.2 make reference to the ABNF rules from Section 5.2. Removing all of those
> rules would thus require somewhat extensive changes *very* late in the process.
> 
> Thus I would prefer that we find a way to clarify the text in Section 5.2, if
> possible. Proposed text would be welcome.
> 
> Peter
> 
> --
> Peter Saint-Andre
> https://stpeter.im/

---

From: Joe Mason <[email-masked]>
Date: Wed, 28 Sep 2011 16:36:13 -0400

> the grammar is probably worse than useless in its current form.

Is the diagram enough clarification to make the ABNF useful?

Joe

---

From: Mykyta Yevstifeyev <[email-masked]>
Date: Thu, 29 Sep 2011 17:17:57 +0300

Dear WG,

[Bcc'ed to IESG]

That was me who raised the issue regarding ABNF in the WS draft, and 
taking into account that Peter and most of IESG members actually 
interpret RFC 5234 in the way that is different from mine, and these 
both interpretations were presented in length in Peter's message, I will 
now make a summary, as I see no further comments are going to appear.

1) So we have heard a two people claiming that it is possible to 
implement WS protocol looking only at the figure, and/or ABNF is hardly 
useful here:

> I have found the current ABNF usage very confusing.

2) Ian Fette, one of the document's editors, mentioned:

28.09.2011 18:44, Ian Fette (イアンフェッティ) wrote:
> I personally find it adds value and would object to its removal. We 
> provide ABNF for most everything else, it would seem like a rather odd 
> omission to take this out for what is essentially a style preference 
> by one person who has not contributed to the WG in a meaningful manner 
> previously (to my recollection) after it having been in the document 
> for *well over a year*.

So, to response, I'll ask:  What is the trouble in removing something 
which has been in the document for "well over a year" but is found in 
some way inappropriate to be in the document in the future?

Also, is providing ABNF for everything but not "foo" is a reason for 
providing ABNF for "foo" as well?

3) Bjoern Hoehrmann assumed that ABNF use is inappropriate:

28.09.2011 19:43, Bjoern Hoehrmann wrote:
> The current ABNF grammar does not
> the groups to begin with.

and, as cited above,

> I have found the current ABNF usage very confusing.

4) Again, Bjoern noticed that frame- rules are referenced in the document:

> existing references to symbols in the grammar still make sense.

and ACK'ed by Peter:

28.09.2011 22:31, Peter Saint-Andre wrote:
> In particular, please note that various paragraphs in Sections 5.3, 
> 5.8, 6.1, and 6.2 make reference to the ABNF rules from Section 5.2.

However, I don't think this is a problem, because there are the names of 
the fields such as, eg. "Mask":

>    Mask:  1 bit
>
>       Defines whether the payload data is masked.  If set to 1, a
>       masking key is present in masking-key, and this is used to unmask
>       the payload data as per Section 5.3.  All frames sent from client
>       to server have this bit set to 1.

and you may simply and safely replace all the frame- rules references 
with these names.

5) Gabriel Montenegro proposed making ABNF informative.  However, what's 
the reason as this ABNF is no useful to implementors and everything is 
clear w/o it?

Taking this into account, I would recommend removing ABNF for frame 
entirely, and fixing all the references accordingly.

All the best,
Mykyta Yevstifeyev

---

From: Joel Martin <[email-masked]>
Date: Thu, 29 Sep 2011 10:08:06 -0500

> 1. Are any implementers currently using the ABNF rules from Section 5.2?

No. I implemented websockify based on the diagram and other explanatory
text, not the ABNF.

> 2. Does anyone object to removing the ABNF rules from Section 5.2?

I don't object. Whatever is the shortest path to IESG approval IMO.

Joel Martin

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Thu, 29 Sep 2011 09:10:49 -0700

I am deeply saddened that this working group is constantly held up by random
people coming in and causing problems with things that magically no one else
has been disturbed by in the past. I would like to see us move on to once
again discussing meaningful matters, and I do not believe this current
debate is meaningful. Furthermore, I believe this is well past the proper
time for such discussion. Ideally this would have come in when we were
actually talking about framing, or at the latest during WG LC. Somehow, we
manage to have multiple implementations (servers, multiple browsers
including Chrome, Safari, Firefox, and Internet Explorer) and so I don't
really believe that the text is causing issues, I don't believe it's
incorrect, and i do believe it would further set this WG back to try to
address it more than we already have.

I would really like us to close this issue and wrap up so that we can once
again discuss meaningful issues.

-Ian

On Thu, Sep 29, 2011 at 7:17 AM, Mykyta Yevstifeyev <[email-masked]>wrote:

> Dear WG,
>
> [Bcc'ed to IESG]
>
> That was me who raised the issue regarding ABNF in the WS draft, and taking
> into account that Peter and most of IESG members actually interpret RFC 5234
> in the way that is different from mine, and these both interpretations were
> presented in length in Peter's message, I will now make a summary, as I see
> no further comments are going to appear.
>
> 1) So we have heard a two people claiming that it is possible to implement
> WS protocol looking only at the figure, and/or ABNF is hardly useful here:
>
>> omission to take this out for what is essentially a style preference by one
>> person who has not contributed to the WG in a meaningful manner previously
>> (to my recollection) after it having been in the document for *well over a
>> year*.
>>
>
> So, to response, I'll ask:  What is the trouble in removing something which
> has been in the document for "well over a year" but is found in some way
> inappropriate to be in the document in the future?
>
>> 6.1, and 6.2 make reference to the ABNF rules from Section 5.2.
>>
>
> and you may simply and safely replace all the frame- rules references with
> these names.
>
> the reason as this ABNF is no useful to implementors and everything is clear
> w/o it?
>

---

From: S Moonesamy <[email-masked]>
Date: Thu, 29 Sep 2011 09:58:06 -0700

The Responsible Area Director explained in the first paragraph of a 
message [1] posted on September 28 why this issue was brought to the 
WG mailing list.

It would be good if the issue could be resolved within the WG.  It is 
understandable that some working group participants might find the 
discussion frustrating.  If the proposed resolution is not mutually 
acceptable, the publication of the draft as a RFC might be delayed.

Regards,
S. Moonesamy

1. http://www.ietf.org/mail-archive/web/hybi/current/msg09148.html

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Thu, 29 Sep 2011 10:40:55 -0700

Here is a proposal which I believe makes the ABNF explicit and correct:

 The base framing protocol is formally defined by the following ABNF
   [RFC5234].  It is important to note that the representation of this
   data is binary, not ASCII characters.  As such, a field with length 1
   bit that takes values %x0 / %x1 is represented as a single bit whose
   value is either 0 or 1, not a full byte with value 0 or 1.  A field
   with a length of 4 bits with values between %x0-F again is
   represented by 4 bits, again NOT by an ASCII character or full byte
   with these values.  [RFC5234] does not specify a character encoding -
   " Rules resolve into a string of terminal values, sometimes called
   characters.  In ABNF, a character is merely a non-negative integer.
   In certain contexts, a specific mapping (encoding) of values into a
   character set (such as ASCII) will be specified."  Here, the
   specified encoding is a binary encoding where each terminal value is
   encoded in the specified number of bits, which varies for each field.

   ws-frame                = frame-fin           ; 1 bit
                             frame-rsv1          ; 1 bit
                             frame-rsv2          ; 1 bit
                             frame-rsv3          ; 1 bit
                             frame-opcode        ; 4 bits
                             frame-masked        ; 1 bit
                             frame-payload-length   ;  7 bits
                             [ frame-masking-key ]  ; 16 or 64 bits
                             frame-payload-data     ; n * 8 bits where n >=
0

   frame-fin               = %x0 ; more frames of this message follow
                           / %x1 ; final frame of this message
                                 ; 1 bit

   frame-rsv1              = %x0
                             ; 1 bit, MUST be 0 unless negotiated
                             ; otherwise

   frame-rsv2              = %x0
                             ; 1 bit, MUST be 0 unless negotiated
                             ; otherwise

   frame-rsv3              = %x0
                             ; 1 bit, MUST be 0 unless negotiated
                             ; otherwise

   frame-opcode            = %x0 ; continuation frame
                           / %x1 ; text frame
                           / %x2 ; binary frame
                           / %x3-7
                          ; reserved for further non-control frames
                           / %x8 ; connection close
                           / %x9 ; ping
                           / %xA ; pong
                           / %xB-F ; reserved for further control frames
                                ; 4 bits in length
  frame-masked            = %x0
                          ; frame is not masked, no frame-masking-key
                           / %x1
                          ; frame is masked, frame-masking-key present
                          ; 1 bit in length

Fette & Melnikov        Expires February 2, 2012               [Page 32]
^L
Internet-Draft           The WebSocket protocol              August 2011

   frame-payload-length    = %x00-7D
                           / %x7E frame-payload-length-16
                           / %x7F frame-payload-length-63
                           ; 7 bits in length

   frame-payload-length-16 = %x0000-FFFF ; 16 bits in length

                           ; 64 bits in length

   frame-masking-key       = 4( %0x00-FF )
                             ; present only if frame-masked is 1
                             ; 32 bits in length

   frame-payload-data      = (frame-masked-extension-data
                              frame-masked-application-data)
                           ; frame-masked 1
                             / (frame-unmasked-extension-data
                               frame-unmasked-application-data)
                           ; frame-masked 0

   frame-masked-extension-data     = *( %x00-FF ) ; to be defined later
                                   ; n*8 bits, where n >= 0

   frame-masked-application-data   = *( %x00-FF )
                                   ; n*8 bits, where n >= 0

   frame-unmasked-extension-data   = *( %x00-FF ) ; to be defined later
                                   ; n*8 bits, where n >= 0

   frame-unmasked-application-data = *( %x00-FF )
                                   ; n*8 bits, where n >= 0

On Thu, Sep 29, 2011 at 9:58 AM, S Moonesamy <[email-masked]> wrote:

> The Responsible Area Director explained in the first paragraph of a message
> [1] posted on September 28 why this issue was brought to the WG mailing
> list.
>
> 1. http://www.ietf.org/mail-**archive/web/hybi/current/**msg09148.html<http://www.ietf.org/mail-archive/web/hybi/current/msg09148.html>
>
>

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 29 Sep 2011 11:44:47 -0600

Thanks, Ian. On a quick reading I agree that's much clearer. I'm poring 
over it in more detail now...

On 9/29/11 11:40 AM, Ian Fette (イアンフェッティ) wrote:
>                               frame-payload-data     ; n * 8 bits where
> n >= 0
>
> On Thu, Sep 29, 2011 at 9:58 AM, S Moonesamy <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     It would be good if the issue could be resolved within the WG.  It
>     is understandable that some working group participants might find
>     the discussion frustrating.  If the proposed resolution is not
>     mutually acceptable, the publication of the draft as a RFC might be
>     delayed.
>
>     Regards,
>     S. Moonesamy
>
>     1.
>     http://www.ietf.org/mail-__archive/web/hybi/current/__msg09148.html
>     <http://www.ietf.org/mail-archive/web/hybi/current/msg09148.html>
>
>
>     _________________________________________________
>     hybi mailing list
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/__listinfo/hybi
>     <https://www.ietf.org/mailman/listinfo/hybi>
>
>
>
>

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Joe Mason <[email-masked]>
Date: Thu, 29 Sep 2011 13:46:39 -0400

I agree.  Since it’s been pointed out that removing the ABNF will have add-on effects in other areas of the document that have reference it, I withdraw my “no objection to removing it”.

As far as I can tell, the only point of ambiguity is the exact bit width of each field, which is easily clarified by looking at the diagram.  I think the minimal amendment that removes this ambiguity is to add the bit width of each field in a comment, as is already done with frame-rsv1.  Note that there is already precedent for specifiying bit widths in this way in RFC 5234 itself, which defines OCTET as exactly 8 bits using the notation:

OCTET          =  %x00-FF
                                ; 8 bits of data

So I recommend updating the ABNF as follows:

   ws-frame                = frame-fin
                             frame-rsv1
                             frame-rsv2
                             frame-rsv3
                             frame-opcode
                             frame-masked
                             frame-payload-length
                             [ frame-masking-key ]
                             frame-payload-data

                             ; 1 bit in all cases

   frame-rsv1              = %x0 / %x1

   frame-rsv2              = %x0 / %x1

   frame-rsv3              = %x0 / %x1

   frame-opcode            = frame-opcode-non-control /
                             frame-opcode-control

   frame-opcode-non-control= %x1 ; text frame
                           / %x2 ; binary frame
                           / %x3-7
                          ; reserved for further non-control frames
                             ; 4 bits in all cases

  frame-opcode-control    = %x8 ; connection close
                           / %x9 ; ping
                           / %xA ; pong
                           / %xB-F ; reserved for further control frames
                             ; 4 bits in all cases

   frame-masked            = %x0
                             ; 1 bit in all cases

                             ; 7 bits in all cases

   frame-payload-length-16 = %x0000-FFFF
                             ; 16 bits

                             ; 63 bits

                             ; 32 bits total

   frame-masked-extension-data     = *OCTET
                           ; reserved for future extensibility

   frame-masked-application-data   = *OCTET

   frame-unmasked-extension-data   = *OCTET
                           ; reserved for future extensibility

   frame-unmasked-application-data = *OCTET

   OCTET = %x00-FF
           ; 8 bits of data

I’ve changed frame-*-data to use OCTET, to enforce that there cannot be partial bytes of data, and repeated the definition of OCTET from RFC 5234 since it is defined in Appendix B: Core ABNF of ABNF, which says “Note that these rules are only valid for ABNF encoded in 7-bit ASCII or in characters sets that are a superset of 7-bit ASCII.” Although this caveat doesn’t affect OCTET since it doesn’t specify a specific ASCII character, a pedantic reading (which this WG seems prone to!) would make the entire appendix inapplicable, so best not to depend on it.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ian Fette (????????)
Sent: Thursday, September 29, 2011 12:11 PM
To: Mykyta Yevstifeyev
Cc: [email-masked]
Subject: Re: [hybi] ABNF issue in draft-ietf-hybi-thewebsocketprotocol - summary

I am deeply saddened that this working group is constantly held up by random people coming in and causing problems with things that magically no one else has been disturbed by in the past. I would like to see us move on to once again discussing meaningful matters, and I do not believe this current debate is meaningful. Furthermore, I believe this is well past the proper time for such discussion. Ideally this would have come in when we were actually talking about framing, or at the latest during WG LC. Somehow, we manage to have multiple implementations (servers, multiple browsers including Chrome, Safari, Firefox, and Internet Explorer) and so I don't really believe that the text is causing issues, I don't believe it's incorrect, and i do believe it would further set this WG back to try to address it more than we already have.

I would really like us to close this issue and wrap up so that we can once again discuss meaningful issues.

-Ian

On Thu, Sep 29, 2011 at 7:17 AM, Mykyta Yevstifeyev <[email-masked]<mailto:[email-masked]>> wrote:
Dear WG,

[Bcc'ed to IESG]

That was me who raised the issue regarding ABNF in the WS draft, and taking into account that Peter and most of IESG members actually interpret RFC 5234 in the way that is different from mine, and these both interpretations were presented in length in Peter's message, I will now make a summary, as I see no further comments are going to appear.

1) So we have heard a two people claiming that it is possible to implement WS protocol looking only at the figure, and/or ABNF is hardly useful here:

28.09.2011 18:44, Ian Fette (イアンフェッティ) wrote:
I personally find it adds value and would object to its removal. We provide ABNF for most everything else, it would seem like a rather odd omission to take this out for what is essentially a style preference by one person who has not contributed to the WG in a meaningful manner previously (to my recollection) after it having been in the document for *well over a year*.

So, to response, I'll ask:  What is the trouble in removing something which has been in the document for "well over a year" but is found in some way inappropriate to be in the document in the future?

Also, is providing ABNF for everything but not "foo" is a reason for providing ABNF for "foo" as well?

and, as cited above,

and ACK'ed by Peter:

28.09.2011 22:31, Peter Saint-Andre wrote:
In particular, please note that various paragraphs in Sections 5.3, 5.8, 6.1, and 6.2 make reference to the ABNF rules from Section 5.2.

However, I don't think this is a problem, because there are the names of the fields such as, eg. "Mask":
  Mask:  1 bit

and you may simply and safely replace all the frame- rules references with these names.

5) Gabriel Montenegro proposed making ABNF informative.  However, what's the reason as this ABNF is no useful to implementors and everything is clear w/o it?

Taking this into account, I would recommend removing ABNF for frame entirely, and fixing all the references accordingly.

All the best,
Mykyta Yevstifeyev

---

From: Joe Mason <[email-masked]>
Date: Thu, 29 Sep 2011 13:48:06 -0400

Oops – reading Ian Fette’s similar proposal, I see I made an error!  frame-payload-length-63 should be 64 bits, not 63.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Joe Mason
Sent: Thursday, September 29, 2011 1:47 PM
To: [email-masked]; Mykyta Yevstifeyev
Cc: [email-masked]
Subject: Re: [hybi] ABNF issue in draft-ietf-hybi-thewebsocketprotocol - summary

OCTET          =  %x00-FF
                                ; 8 bits of data

   ws-frame                = frame-fin
                             frame-rsv1
                             frame-rsv2
                             frame-rsv3
                             frame-opcode
                             frame-masked
                             frame-payload-length
                             [ frame-masking-key ]
                             frame-payload-data

   frame-rsv1              = %x0 / %x1

   frame-rsv2              = %x0 / %x1

   frame-rsv3              = %x0 / %x1

   frame-masked            = %x0

   OCTET = %x00-FF
           ; 8 bits of data

-Ian

[Bcc'ed to IESG]

and, as cited above,

and ACK'ed by Peter:

All the best,
Mykyta Yevstifeyev

---

From: Joe Mason <[email-masked]>
Date: Thu, 29 Sep 2011 13:50:20 -0400

I support this proposal – it’s superior to the one I made because of the added explanation.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ian Fette (????????)
Sent: Thursday, September 29, 2011 1:41 PM
To: S Moonesamy
Cc: [email-masked]
Subject: Re: [hybi] ABNF issue in draft-ietf-hybi-thewebsocketprotocol - summary

   ws-frame                = frame-fin           ; 1 bit
                             frame-rsv1          ; 1 bit
                             frame-rsv2          ; 1 bit
                             frame-rsv3          ; 1 bit
                             frame-opcode        ; 4 bits
                             frame-masked        ; 1 bit
                             frame-payload-length   ;  7 bits
                             [ frame-masking-key ]  ; 16 or 64 bits
                             frame-payload-data     ; n * 8 bits where n >= 0

   frame-rsv1              = %x0

   frame-rsv2              = %x0

   frame-rsv3              = %x0

On Thu, Sep 29, 2011 at 9:58 AM, S Moonesamy <[email-masked]<mailto:[email-masked]>> wrote:
The Responsible Area Director explained in the first paragraph of a message [1] posted on September 28 why this issue was brought to the WG mailing list.

It would be good if the issue could be resolved within the WG.  It is understandable that some working group participants might find the discussion frustrating.  If the proposed resolution is not mutually acceptable, the publication of the draft as a RFC might be delayed.

Regards,
S. Moonesamy

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 29 Sep 2011 18:55:13 +0100

On a second thought, I am Ok with that.

Note that there is one minor tweak I would like to do about versioning 
(say that -13 points to the RFC, not to -13). So feel free to commit 
this change if you get Ok from Peter and chairs, I will do the tweak 
tomorrow and submit -17 tomorrow.

Thanks,
Alexey

P.S. Going to get some beer and get drunk with friends now.

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 29 Sep 2011 18:56:41 +0100

> P.S. Going to get some beer and get drunk with friends now.

That was supposed to be a private reply, but I don't think I've said 
anything I would deny publicly ;-).

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 29 Sep 2011 12:04:24 -0600

> tomorrow and submit -17 tomorrow.

I will check it over in more detail here soon.

> P.S. Going to get some beer and get drunk with friends now.

Yes, this work will drive you to drink. :)

/psa

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 29 Sep 2011 14:59:03 -0600

On 9/29/11 11:40 AM, Ian Fette (イアンフェッティ) wrote:
>    value is either 0 or 1, not a full byte with value 0 or 1. 

I've looked at this in more detail. I think it might it be *slightly*
clearer to phrase the third sentence as follows...

                                            As such, a field with a
   length of 1 bit that takes values %x0 / %x1 is represented as a
   single bit whose value is either 0 or 1, not a full byte (octet)
   that stands for the characters "0" or "1" in the ASCII encoding.

Modulo the comment about saying "; 1 bit in length" everywhere instead
of "; 1 bit" (again to forestall any possible confusion), the rest looks
good to me.

/psa

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Thu, 29 Sep 2011 16:25:42 -0700

done

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 30 Sep 2011 13:38:14 +0900

[I wrote all the stuff below yesterday, and though I'd pressed the send 
button, but apparently I didn't. Anyway, I'm very pleased with the 
direction things are moving now. So you may be able to ignore the rest 
of this message.]

Hello everybody,

> in ASCII).

Strictly speaking, I wouldn't say "digits 0 and 1", but "bit values 0 
and 1". A digit is usually a character, and we don't deal with 
characters here.

> This kind of mapping is not disallowed by the ABNF spec,

Yes, indeed ASCII is not the only choice. For an example, please see 
e.g. http://tools.ietf.org/html/rfc3987#section-2.2, where it says:

                                                       The syntax of this
    ABNF is described in [RFC2234].  Character numbers are taken from the
    UCS, without implying any actual binary encoding.  Terminals in the
    ABNF are characters, not bytes.

(UCS is the Universal/Unicode character set, and is defined earlier in 
the RFC. RFC 2234 is the predecessor of RFC 5234.)

> is the leftmost in the ABNF.)

This is all well and good, except that as Björn notes, the length of 
each groups of bits isn't defined in the ABNF (in some instances, there 
are comments such as "; 1 bit", but in other cases, there is no way to 
deduce the length even from the comment in the ABNF).

Ideally, notations such as ABNF are designed to be used for automatic 
checking of data or automatic generation of parsing and/or construction 
logic. Actually, some IETFers have created such tools. While this is the 
ideal, and in our case things such as conditional dependencies (if this 
bit is 0, then this field is present/absent) make this difficult, it's 
not a bad idea to stay close to the ideal. Having to glean the length of 
each group of bits from the diagram (which is less autoritative than the 
ABNF) in order to be able to understand the ABNF is definitely suboptimal.

> A high level overview of the framing

As I haven't seen this off-list communication, I can't say for sure 
about his arguments, but I have to agree that he has a point.

> 1. Are any implementers currently using the ABNF rules from Section 5.2?

I had a student working on implementation a few months ago, but he's 
gone now, so I can't say much to that point.

> 2. Does anyone object to removing the ABNF rules from Section 5.2?

I haven't looked at it in detail. I agree with Gabriel that we shouldn't 
remove the ABNF (too late in the game), but simply reverse the priority 
in case of conflict to make the diagram have priority.

Thinking about this on a higher level, I don't remember having seen 
something like ABNF for bit-fiddling protocols. It seems that we somehow 
ended up with ABNF for the bit-by-bit parts in our document because we 
also have some character-based part in our protocol and because we are 
used to ABNF in the applications area. So over the long term (read WS 
1.1 or whatever), I think getting rid of the ABNF for the bity parts is 
the right thing.

In addition to the change proposed by Gabriel, I also suggest the 
following change:

Just after the following text:

                                                (Note that unlike in
    on groups of bits.

Add either of the following:

Alternative a):
    The length of each group of bits is indicated in a comment.
   [and add the length of each group in a comment
    where not already present]

Alternative b):
    Groups of bits can be of different length; please consult the figure
    for details.

[Just for the record, another way out would be to come up with a %b 
notation, and define that the number of bits in a group is given by the 
length of the 0/1 string after the %b.]

> Thank you for your patience.

Yes indeed. Just for the record, I'd like to note that what we are doing 
here has nothing to do with bikesheding. We are not discussing 
irrelevant details of the protocol (such as whether we'd not better flip 
bit x to mean y when it is 0 and z when it's 1 rather than the other way 
round). We are fixing a potentially harmless, but potentially not so 
harmless, but in any way annoying, mistake (or call it an omission) in 
the spec, which is definitely not just an editorial matter. While the 
spec might have sailed to RFC without anybody noticing the problem, now 
that somebody has pointed it out, it would be highly unappropriate to 
not fix it.

Regards,   Martin.

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 30 Sep 2011 10:17:44 +0100

>    where not already present]

I've used this alternative. Thanks.

> notation, and define that the number of bits in a group is given by 
> the length of the 0/1 string after the %b.]

But this would not be ABNF anymore, so validating it with existing tools 
would not be possible. But maybe something to do in the future.