Subject: [xmpp] Websockets: streams or not streams?

From: Matthew Wild <[email-masked]>
Date: Wed, 23 Oct 2013 01:07:44 +0100

Florian Zeitz, Lance Stout, Paul Aurich, Waqas Hussain and myself have
just been having some discussion about the largest remaining open
issue in the websocket draft.

I'll start explaining from the beginning, assuming no prior knowledge,
because we're looking for input from as many people as possible in
order to reach a consensus (ha).

Unlike standard TCP which offers a stream of bytes, websockets provide
"framing", which means that the underlying stream is split into
individual frames.

We had a general consensus early on that it made sense to use these
frames to separate stanzas, which is especially useful because
generally Javascript clients do not have access to a streaming (e.g.
SAX) XML parser which is required for parsing unframed XMPP streams.
With framing, clients could (theoretically) take each frame and parse
it individually as a small standalone document.

However there is some awkwardness in this approach, because the
concept of a stream is quite central to traditional XMPP.

Here are the two questions we have:

1) If stanzas are going to be parsed as standalone documents, can they
inherit namespaces from the stream header like they can in normal
XMPP? Or should we require entities to always explicitly declare
namespaces on stanzas? (this includes for example adding
xmlns="jabber:client" xml:lang="xyz" to every stanza).

2) Should we keep the existing stream open and close mechanisms? They
are not standalone documents, and so websocket clients will still need
to handle them with special code.

Options for #1:

  Including namespaces will allow clients (libraries) to pass the data
in a frame directly into the XML parser they have.

  If we don't include namespaces (the current situation), clients are
resorting to hacks which include:
   - Not handling namespaces properly, perhaps assuming they are
always jabber:client
   - Capturing the stream header, and preprending it to every incoming
stanza by concatentating them together (example at
https://github.com/legastero/stanza.io/blob/master/lib/websocket.js#L59
)

However the concerns about adding namespaces explicitly to every stanza include:
  - Bandwidth increase
  - General noise and mess (look at it from a JSON fanatic's perspective)

Options for #2:

First option (current):

  OPEN:  <stream:stream xmlns:stream="..." xmlns="jabber:client"
id="abc123" xml:lang="en">
  CLOSE: </stream:stream>
  ERROR: <stream:error>...</stream:error>

  Benefits include:
     - Pretty much what we are used to already with traditional XMPP,
which means we are all familiar with it, and don't need to define
drastically new stuff

  Problems include:
    - Open and close are not standalone documents, which complicates
parsing for clients without streaming parsers. Again, this leads to
client hacks which include:
        * Parsing with regular expressions
        * Replacing the > at the end with />

Second option (goal: make everything a standalone document):

   OPEN: <stream:stream xmlns:stream="..." xmlns="jabber:client"
id="abc123" xml:lang="en" />
   CLOSE: ????
   ERROR: <stream:error xmlns:stream="..." xmlns="jabber:client"
id="abc123" xml:lang="en">...</stream:error>

   Benefits:
       - Re-uses what we already have

   Problems:
       - Not clear what we should use to signal stream closure
       - Although it's re-using what we already have, it is
overloading it somewhat (the 'stream' is no longer a stream...)

Third option (let's try a new way!):

    OPEN: <open xmlns="..." id="abc123" />
    CLOSE: <close xmlns="..." />
    ERROR: <error xmlns="http://etherx.jabber.org/streams">...</error>

   Benefits:
       - Clean, simple

   Problems:
       - Quite a departure from traditional streams now
       - Duplicates some things from the 'stream' element

Questions, comments, feedback welcome! It's worth noting that this
issue won't be unique to websockets, but will likely be re-used by any
transports that support framing (SCTP for example).

Regards,
Matthew

---

From: Lance Stout <[email-masked]>
Date: Tue, 22 Oct 2013 18:11:13 -0700

> (this includes for example adding

A realization while reading this: we technically also would have to require the <?xml ... ?> header in every frame to create complete standalone documents.

--Lance

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 23 Oct 2013 13:06:30 +0200

* Lance Stout wrote:
>A realization while reading this: we technically also would have to 
>require the <?xml ... ?> header in every frame to create complete 
>standalone documents.

The XML declaration is never needed or useful if you don't do anything
crazy ("crazy" includes using the declaration's `standalone` attribute.)
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Waqas Hussain <[email-masked]>
Date: Wed, 23 Oct 2013 11:05:56 -0400

> crazy ("crazy" includes using the declaration's `standalone` attribute.)

Unless our spec explicitly forbids it, someone will add it. After all,
if you are claiming each frame is an XML document, then adding it is
valid by default (adding the prolog is a SHOULD in the XML 1.0 spec).

--
Waqas Hussain

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Wed, 23 Oct 2013 17:10:41 +0000

>  - General noise and mess (look at it from a JSON fanatic's perspective)

The hack of wrapping stanzas in an element with all of the inheritable namespacing seems fairly innocuous.  It could be something like:

<app:wrapper xmlns:app='urn:for:my:app' xmlns:stream='http://etherx.jabber.org/streams' xmlns='jabber:client'>
  {{ insert websocket message here }}
</app:wrapper>

Assuming the client is keeping track of its stream state (which it really needs to).

That said, I personally have a slight preference for them to be proper stand-alone.  The "general noise and mess" is something that one sees with BOSH anyway.

>       - Duplicates some things from the 'stream' element

I have heard off-list from a few developers that dealing with the XMPP streams in WebSockets like raw TCP sockets is enough of a burden for them to have completely ignored the opening <stream:stream>.  I think it's worth doing something different for this reason.  I also think the target audience for the WebSocket-based transport client-side aren't (guaranteed to be) also writing code for dealing with raw TCP sockets.

I had posted a strawman to the list in May; that might be worth considering (or not) < http://www.ietf.org/mail-archive/web/xmpp/current/msg02971.html >.

- m&m

Matt Miller < [email-masked] >
Cisco Systems, Inc.

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 23 Oct 2013 13:18:31 -0600

On 10/23/13 11:10 AM, Matt Miller (mamille2) wrote:
>> Florian Zeitz, Lance Stout, Paul Aurich, Waqas Hussain and
>> myself have just been having some discussion about the largest
>> remaining open issue in the websocket draft.
>> 
>> I'll start explaining from the beginning, assuming no prior 
>> knowledge, because we're looking for input from as many people
>> as possible in order to reach a consensus (ha).
>> 
>> Unlike standard TCP which offers a stream of bytes, websockets 
>> provide "framing", which means that the underlying stream is
>> split into individual frames.
>> 
>> We had a general consensus early on that it made sense to use 
>> these frames to separate stanzas, which is especially useful 
>> because generally Javascript clients do not have access to a 
>> streaming (e.g. SAX) XML parser which is required for parsing 
>> unframed XMPP streams. With framing, clients could
>> (theoretically) take each frame and parse it individually as a
>> small standalone document.
>> 
>> 1) If stanzas are going to be parsed as standalone documents,
>> can they inherit namespaces from the stream header like they can
>> in normal XMPP? Or should we require entities to always
>> explicitly declare namespaces on stanzas? (this includes for
>> example adding xmlns="jabber:client" xml:lang="xyz" to every
>> stanza).
>> 
>> 2) Should we keep the existing stream open and close mechanisms? 
>> They are not standalone documents, and so websocket clients will 
>> still need to handle them with special code.
>> 
>> Options for #1:
>> 
>> Including namespaces will allow clients (libraries) to pass the 
>> data in a frame directly into the XML parser they have.
>> 
>> If we don't include namespaces (the current situation), clients 
>> are resorting to hacks which include: - Not handling namespaces 
>> properly, perhaps assuming they are always jabber:client - 
>> Capturing the stream header, and preprending it to every incoming
)
>> 
>> However the concerns about adding namespaces explicitly to every 
>> stanza include: - Bandwidth increase - General noise and mess
>> (look at it from a JSON fanatic's perspective)
>> 
> 
> The hack of wrapping stanzas in an element with all of the 
> inheritable namespacing seems fairly innocuous.  It could be 
> something like:
> 
> <app:wrapper xmlns:app='urn:for:my:app' 
> xmlns:stream='http://etherx.jabber.org/streams' 
> xmlns='jabber:client'> {{ insert websocket message here }} 
> </app:wrapper>
> 
> That said, I personally have a slight preference for them to be 
> proper stand-alone.  The "general noise and mess" is something
> that one sees with BOSH anyway.
> 
>> Options for #2:
>> 
>> First option (current):
>> 
>> id="abc123" xml:lang="en"> CLOSE: </stream:stream> ERROR: 
>> <stream:error>...</stream:error>
>> 
>> Benefits include: - Pretty much what we are used to already with 
>> traditional XMPP, which means we are all familiar with it, and 
>> don't need to define drastically new stuff
>> 
>> Problems include: - Open and close are not standalone documents, 
>> which complicates parsing for clients without streaming parsers. 
>> Again, this leads to client hacks which include: * Parsing with 
>> regular expressions * Replacing the > at the end with />
>> 
>> id="abc123" xml:lang="en" /> CLOSE: ???? ERROR: <stream:error 
>> xmlns:stream="..." xmlns="jabber:client" id="abc123" 
>> xml:lang="en">...</stream:error>
>> 
>> Benefits: - Re-uses what we already have
>> 
>> Problems: - Not clear what we should use to signal stream closure
>> - Although it's re-using what we already have, it is overloading
>> it somewhat (the 'stream' is no longer a stream...)
>> 
>> OPEN: <open xmlns="..." id="abc123" /> CLOSE: <close xmlns="..." 
>> /> ERROR: <error 
>> xmlns="http://etherx.jabber.org/streams">...</error>
>> 
>> Benefits: - Clean, simple
>> 
>> Problems: - Quite a departure from traditional streams now - 
>> Duplicates some things from the 'stream' element
>> 
> 
> I have heard off-list from a few developers that dealing with the 
> XMPP streams in WebSockets like raw TCP sockets is enough of a
> burden for them to have completely ignored the opening
> <stream:stream>.  I think it's worth doing something different for
> this reason.  I also think the target audience for the
> WebSocket-based transport client-side aren't (guaranteed to be)
> also writing code for dealing with raw TCP sockets.
> 
> I had posted a strawman to the list in May; that might be worth 
> considering (or not) < 
> http://www.ietf.org/mail-archive/web/xmpp/current/msg02971.html >.

You proposed:

###

* begin: <ws:open xmlns:ws='urn:ietf:params:xml:ns:xmpp-websocket'
                  version='1.0'
                  to='example.com'
                  from='juliet at example.com'
                  xml:lang='en'/>

* end: <ws:close xmlns:ws='urn:ietf:params:xml:ns:xmpp-websocket'/>

###

That seems fine to me. Lance made a good point about overloading the
streams namespace to no longer describe a stream. IMHO it would be
better to define something that's specific to the WebSocket binding,
since we already have a precedent of defined something specific to the
BOSH binding. Thus the streams construct is only for the TCP binding
(which is the only place where it really makes sense, I think).

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Gordin <[email-masked]>
Date: Wed, 23 Oct 2013 21:46:29 +0200

Hi, $guy-who-implemented-webwockets-in-StropheJS here.

>>  - General noise and mess (look at it from a JSON fanatic's perspective)

I'm not a fan of this solution, mostly because I don't like to waste
bandwidth with sending the client information it already knows over
and over again.
If namespaces are to be parsed correctly without using a SAX parser
than someone has to capture the header and put it in the messages that
are parsed every time. However in the proposed solution it is phrased
as if capturing the header and merging it with every message is a hack
when the client does it, but ok if the server does it. Personally I
think letting the server add a namespace that the client already knows
is equally "hacky" than letting the client do it. If someone is adding
a namespace to every message anyway it might as well be the client.
I favor the few lines of code in the client code over the constantly
wasted bandwidth here.

> Assuming the client is keeping track of its stream state (which it really needs to).

It doesn't need to be told with every stanza to do that though.

> That said, I personally have a slight preference for them to be proper stand-alone.  The "general noise and mess" is something that one sees with BOSH anyway.

Also, I don't really see how BOSH being messy justifies websocket
getting messier in any way.

>>       - Duplicates some things from the 'stream' element

I'm not 100% sure on the posted solutions here. I'm OK with the
'stream' being selfclosing and no longer being a stream, as every
stanza is parsed on it's own anyway.

What I would vote for is probably kind of a mix of the 3 options above:

OPEN:  <stream:stream xmlns:stream="..." xmlns="jabber:client"
id="abc123" xml:lang="en"/> OR  <stream:open ......../>
CLOSE: <stream:stream/>  OR   <stream:close/>
ERROR: <stream:error>...</stream:error>

So, change the opening and closing tags to be selfclosing and let
everything else as it is.

Pros:
* Only two "/"s moved around, everything else stays as it is and
browsers can now parse everything.
* Still looks like intended for XMPP streams
* Works exactly like current implementations handle things anyway. (Of
course implementations that just plain match the closing tag against
</stream:stream> could actually parse it now)
* stream errors can stay the way they are

Cons:
* Not a real stream any more (like all of the other solutions)
* two stream:stream tags per stream. This could be "fixed" by either
** renaming ":stream" to ":open" in the closing tag and to ":close" in
the closing tag. Doing this would be the clearest solution but would
duplicate the meaning of stream:stream
or
** just renaming the ":stream" to ":close" in the closing tag. This
would avoid duplication of the meaning of stream:stream
or
** just ignoring that there are two "stream:stream"s for one stream
now. First one opens, second one closes.

Of this options I would probably prefer this one
OPEN:  <stream:stream xmlns:stream="..." xmlns="jabber:client"
id="abc123" xml:lang="en"/>
CLOSE: <stream:close/>
ERROR: <stream:error>...</stream:error>

Keep the stream:stream, make it selfclosing and replace the
</stream:stream> by a <stream:close/>

> I had posted a strawman to the list in May; that might be worth considering (or not) < http://www.ietf.org/mail-archive/web/xmpp/current/msg02971.html >.

This is basically like the solution I came up with, just with
"ws:open" instead of "stream:stream" and "ws:close" instead of
"stream:close", so also +1 for this.

> _______________________________________________
> xmpp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xmpp

- gordin

---

From: Florian Zeitz <[email-masked]>
Date: Wed, 23 Oct 2013 23:22:45 +0200

>> On Oct 22, 2013, at 6:07 PM, Matthew Wild <[email-masked]>
[...]
> I favor the few lines of code in the client code over the constantly
There is actually a reason to have this on the server. Which is that
XMPP traditionally follows a simple client, complex server model.
Stamping this on the server side allows you to put whatever you receive
in the DOMParser, no further processing required.
The bandwidth argument is very valid though, and might outweigh the
small inconvenience for clients.
After yesterdays discussion I honest have no strong opinion on this.

> It doesn't need to be told with every stanza to do that though.
Indeed. Both of you should be aware though that neither injecting some
namespaces directly into the stanzas, nor using a fixed wrapper element
is a correct solution here. You want to wrap stanzas in the actual
stream header you received. I'm convinced it's unnecessarily hard if not
impossibly to extract only the "important" attributes from it.

> </stream:stream> could actually parse it now)
This is entirely untrue. While it still looks almost like a normal XMPP
stream to a human, it has VERY different semantics. A self closing
stream tag, does just that. It immediately closes the stream. Prosody's
implementation currently will disconnect you if you send this. It does
not particularly care that you used a single tag to open and close the
stream instead of two tags.
I'm also rather surprised that you're fine with injecting all of the
namespaces, but would prefer if the server injected the single '/'
needed to make a stream start parseable on your behalf.

> "stream:close", so also +1 for this.

So there is something I think we had consensus on yesterday, that did
not make it into Matthew Wild's mail. And that is that some of the
solutions for #1 and #2 are mutually exclusive.

You managed to pick one such combination. If the stream start is
self-closing, namespaces are semantically not defined on the stanzas.
The element within which these namespaces are valid has already ended.
I'd be very uncomfortable if we were mandating to pretend that
everything that was defined in the stream header stays defined.

To me valid choices are really only:
Self-closing start/end + xmlns on every stanzas
or
normal XMPP stream (<stream:stream> stays open, no xmlns required on
stanzas)

Regards,
Florian

---

From: Lance Stout <[email-masked]>
Date: Wed, 23 Oct 2013 18:25:50 -0700

On Oct 23, 2013, at 12:18 PM, Peter Saint-Andre <[email-masked]> wrote:

> IMHO it would be

Whatever pattern we come up with here *will* be re-used for other transports such as
WebRTC data channels for doing P2P sessions, etc. Therefore, I am opposed 
to making any namespace or element names be WebSocket specific.

Here's what I propose as the baseline "do nothing" approach that we need to beat 
significantly if we do go with new protocol.

1. We update the requirements to say that stream open, error, and close must all be
   in their own frames. I'd argue this should be done regardless of if we keep the
   current open/close or mint new ones.

   Right now we allow an edge case of an open+error+close in one frame. Removing that 
   actually removes the 'ugliest' bit of code in my implementation of the current spec.
  
   We had consensus on this at the Summit, although it might require some effort
   to ensure it is implemented properly in servers.

2. We strongly suggest that the 'stream' prefix be used, following what 6120 states
   on the matter. It's actually not terribly much work to figure out the stream prefix,
   but it can involve regex that would only be needed for theoretical compatibility with some
   future server implementation that did this. IIRC, we had consensus on this at the Summit.

We can then suggest an implementation guide for environments lacking streaming parsers:

    StreamEnd = '</stream:stream>'
    StreamStart = ''

    on WebSocketMessage as Data do
      if (Data == StreamEnd) then
         return emit('StreamEnded');
      end

      WrappedData = StreamStart + Data + StreamEnd;
      ParsedData = XMLParse(WrappedData);
 
      if (!StreamStart) then
          StreamStart = Data;
          emit('StreamStarted', ParsedData);
      else
          emit('StreamData', ParsedData);
      end
    end

We already know when we have to send a new stream start signal, so we can reset StreamStart
to '' when needed at predictable, known times.

For completeness, if we make every frame standalone and with new start/end signals and
include xmlns and lang on every stanza:

    on WebSocketMessage as Data do
        ParsedData = XMLParse(Data);

        if (ParsedData.matches('urn:....', 'open') then
           emit('StreamStarted', ParsedData);
        else if (ParsedData.matches('urn:....', 'close') then
           emit('StreamEnded');
        else then
           emit('StreamData', ParsedData);
        end
    end

This is simpler, and an approach I could support. But, discounting blank lines and ends, there's only a 
difference in 4 significant lines if I count correctly. Personally, I would rather require an extra 10 
relatively simple lines in the client over changing the protocol and increasing the bandwidth costs.

I'm not dedicated to any particular approach here, and I've changed my mind *many* times in the last few 
days. But right now I'm not convinced that the costs of new protocol and extra bandwidth are worth the 
benefits since this issue is a very minor piece of the flow of processing and managing a WebSocket XMPP 
stream properly (consider error handling, dispatching, etc).

-- Lance

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 23 Oct 2013 21:29:50 -0600

On 10/23/2013 07:25 PM, Lance Stout wrote:
>> IMHO it would be better to define something that's specific to
>> the WebSocket binding, since we already have a precedent of
>> defined something specific to the BOSH binding. Thus the streams
>> construct is only for the TCP binding (which is the only place
>> where it really makes sense, I think).
> 
> Whatever pattern we come up with here *will* be re-used for other
> transports such as WebRTC data channels for doing P2P sessions,
> etc. Therefore, I am opposed to making any namespace or element
> names be WebSocket specific.
> 
> 
> 
> Here's what I propose as the baseline "do nothing" approach that we
> need to beat significantly if we do go with new protocol.
> 
> 1. We update the requirements to say that stream open, error, and
> close must all be in their own frames. I'd argue this should be
> done regardless of if we keep the current open/close or mint new
> ones.

That strikes me as quite reasonable.

> Right now we allow an edge case of an open+error+close in one
> frame. Removing that actually removes the 'ugliest' bit of code in
> my implementation of the current spec.
> 
> We had consensus on this at the Summit, although it might require
> some effort to ensure it is implemented properly in servers.
> 
> 2. We strongly suggest that the 'stream' prefix be used, following
> what 6120 states on the matter. It's actually not terribly much
> work to figure out the stream prefix, but it can involve regex that
> would only be needed for theoretical compatibility with some future
> server implementation that did this. IIRC, we had consensus on this
> at the Summit.

Do you mean <stream:stream xmlns='http://etherx.jabber.org/streams'/>
or something like <stream:open/> with a different namespace?

> StreamEnd = '</stream:stream>' StreamStart = ''
> 
> on WebSocketMessage as Data do if (Data == StreamEnd) then return
> emit('StreamEnded'); end
> 
> WrappedData = StreamStart + Data + StreamEnd; ParsedData =
> XMLParse(WrappedData);
> 
> if (!StreamStart) then StreamStart = Data; emit('StreamStarted',
> ParsedData); else emit('StreamData', ParsedData); end end
> 
> We already know when we have to send a new stream start signal, so
> we can reset StreamStart to '' when needed at predictable, known
> times.

Agreed.

> For completeness, if we make every frame standalone and with new
> start/end signals and include xmlns and lang on every stanza:
> 
> on WebSocketMessage as Data do ParsedData = XMLParse(Data);
> 
> emit('StreamStarted', ParsedData); else if
> (ParsedData.matches('urn:....', 'close') then emit('StreamEnded'); 
> else then emit('StreamData', ParsedData); end end
> 
> This is simpler, and an approach I could support. But, discounting
> blank lines and ends, there's only a difference in 4 significant
> lines if I count correctly. Personally, I would rather require an
> extra 10 relatively simple lines in the client over changing the
> protocol and increasing the bandwidth costs.

I do think that bandwidth usage is not insignificant here.

> I'm not dedicated to any particular approach here, and I've changed
> my mind *many* times in the last few days. But right now I'm not
> convinced that the costs of new protocol and extra bandwidth are
> worth the benefits since this issue is a very minor piece of the
> flow of processing and managing a WebSocket XMPP stream properly
> (consider error handling, dispatching, etc).

I promise to give it some more thought, too...

/psa

---

From: Lance Stout <[email-masked]>
Date: Wed, 23 Oct 2013 20:49:28 -0700

> or something like <stream:open/> with a different namespace?

I meant <stream:stream xmlns:stream='http://etherx.jabber.org/streams' ...>.

With RFC 6120 4.8.5, implementations are allowed to only accept 'stream' as
the namespace prefix for the stream element. If we allow WebSocket clients
to make that same assumption, we simplify the work for clients using the 
existing protocol as is.

It's not a huge deal if we allow the prefix to be arbitrary or non-existant,
however. IIRC it only added 2 lines to the stream start case to pick it out 
and create what the closing stream tag token should be. But consensus at the
summit was that it wasn't needed in practice.

-- Lance