Subject: Re: [Uri-review] [hybi] ws: and wss: schemes

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 10 Sep 2009 15:55:10 +0900

Sorry to be very late with this.

On 2009/08/12 19:15, Kristof Zelechovski wrote:
>   1.  Encouraging the user to enter a Web Sockets URL does not make sense if
> cross-domain connections are not allowed, which I hope will be the case (the
> draft specification [1] does not contain security considerations).

Could it be that the URI/IRI entered is sent to a server, where there 
are no cross-domain restrictions?

>   2.  While we are at it, a Web Sockets connection is useless without knowing
> the protocol, and the protocol to be used is not contained within the URL.
> That means a ws URL is not self-contained and thus useless as a stand-alone
> locator.

My understanding is that there is a Web Sockets protocol.

Regards,   Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: Křištof Želechovski <[email-masked]>
Date: Thu, 10 Sep 2009 10:03:12 +0200

Web Sockets is a transmission protocol, intended as a substitute for TCP/IP
where it is unavailable by policy.  (Of course, it is a TCP/IP overlay, so
TCP/IP must be technically available.)  TCP/IP is a transmission protocol
and it is useless without a communication protocol.  And so is Web Sockets.
I think the idea to use Web Sockets on the server is void; the server can
use TCP/IP at will.
Cheers,
Chris

-----Original Message-----
From: "Martin J. Dürst" [mailto:[email-masked]] 
Sent: Thursday, September 10, 2009 8:55 AM
To: Kristof Zelechovski
Cc: 'Jamie Lokier'; [email-masked]; [email-masked]; [email-masked]; 'David
Booth'
Subject: Re: [hybi] [Uri-review] ws: and wss: schemes

Regards,   Martin.

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 10 Sep 2009 13:46:18 +0100

> I think the idea to use Web Sockets on the server is void; the server can

Nice theory.  I believe you have correctly described the intentions of
the WebSockets protocol proposers (as I understand them), and that the
theory is denying reality.

It's wrong.  A server cannot use TCP/IP at will in two scenarios:

1. A server must use WebSockets if it's accessing services which a
   different provider has only made available via WebSockets, due to
   the provider only intending the service for web browsers.

   For example, at one time something like Google Maps' image back-end
   would fit this pattern: only intended for one web application, but
   actually used by various third parties because it's useful.

   Judging by the modern trend to 'mashups', expect this practice to
   become widespread

2. When a service is provided by WebSockets to support a web browser,
   and a requirement emerges to provide the same service to other programs.

   Many implementors will use the path of least resistance, which is
   to continue offering using the service over WebSockets in the new
   context, and require the clients to use generic non-browser
   WebSockets code.  That is simpler than specifying and implementing
   a second protocol for the same service.

   For examples of where this has happened before, see SOAP.  It runs
   over HTTP simply to reuse deployed and well understood code and
   infrastructure.  In principle it could run over raw TCP/IP or a
   simple framing protocol, but that's not done in practice.

   Expect the same to occur with WebSockets if it is widely used by web
   applications.  If only because of familiarity and duplication avoidance.

-- Jamie

---

From: Křištof Želechovski <[email-masked]>
Date: Thu, 10 Sep 2009 15:08:07 +0200

AIUI, in order to provide a service over WebSockets, you have to implement
it in raw stream first.  This stream, once implemented, can run quite
happily under tcpd, and the WebSockets shim is only a wrapper.  So there is
one protocol and one implementation thereof, only exposed over two different
transport channels.
The situation with SOAP is rather different, as it relies on various
HTTPisms to work.
IMHO,
Chris

-----Original Message-----
From: Jamie Lokier [mailto:[email-masked]] 
Sent: Thursday, September 10, 2009 2:46 PM
To: Křištof Želechovski
Cc: 'Martin J. Dürst'; [email-masked]; [email-masked]; [email-masked];
'David Booth'

[snip, perhaps]

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 10 Sep 2009 23:41:51 +0100

Křištof Želechovski wrote:
> AIUI, in order to provide a service over WebSockets, you have to
> implement it in raw stream first.  This stream, once implemented,
> can run quite happily under tcpd, and the WebSockets shim is only a
> wrapper.

All those points are false, so the rest of your argument falls apart.

You seem to think WebSockets is a wrapper around a two-way raw byte
stream, but it isn't.  In fact it's quite tricky to transport a raw
byte stream over WebSockets, and you'd have to define your own
protocol on top of WebSockets to do it.

WebSockets provides application-visible framing with NUL-terminated
UTF-8 strings.  (I haven't checked what it does if you have a real NUL
character to transmit - does the spec say?).  You cannot transmit
binary data in those UTF-8 string frames, or even arbitrary non-zero
bytes; they must be valid UTF-8.

Although the protocol has binary frames and room for other frame
types, Javascript browser implementations will not support them for a
long time if ever, therefore binary is not practical to use.  You can
of course encode binary data to UTF-8 strings, but your encoding would
be application-specific.

The framing is explicit; it cannot be altered in transit (unlike TCP),
is visible to applications, and was intentionally designed to map to
Javascript DOM events - one event per UTF-8 frame.  Applications are
very likely to be coded around that assumption.  (Even if binary
frames ever get supported, this differentiates it from a raw stream.)

So, no, you would not code a WebSockets service by writing a raw
stream protocol first and then using WebSockets as a shim.  It would
be a very complicated way to do it.

You would only do that, if you _already_ have a raw stream protocol,
but you'd still need an additional layer for binary-to-UTF-8 encoding
and packetising, of your own design, on top of WebSockets.

> So there is one protocol and one implementation thereof,
> only exposed over two different transport channels.

If that were true (it's not), you'd still need to write non-browser
things which talk to other people's services over WebSockets, in order
to access services that are only offered over one channel because they
don't care to provide the other one.

By the way, I have argued in favour of WebSockets being more like a
raw stream to applications (mostly for efficiency reasons but the shim
argument you've given is also a good one), so I'm only describing what
it is and what consequences to expect, not saying I like it.

-- Jamie

---

From: Ian Hickson <[email-masked]>
Date: Thu, 17 Sep 2009 09:24:34 +0000 (UTC)

On Fri, 4 Sep 2009, Joseph A Holsten wrote:
> On Sep 4, 2009, at 12:33 AM, Ian Hickson wrote:
> > On Fri, 14 Aug 2009, Julian Reschke wrote:
> > > > 
> > > > [...] it now says:
> > > >
> > > >   URI scheme syntax.
> > > >      In ABNF terms using the terminals from the IRI specifications:
> > > >      [RFC5238] [RFC3987]
> > > > 
> > > >           "ws" ":" ihier-part [ "?" iquery ]
> > > 
> > > That is even worse than before, because it now uses productions from 
> > > the IRI spec defining *URI* syntax.
> > 
> > ws: and wss: URLs are i18n-aware; why would we want to limit them to 
> > ASCII?
> 
> URIs are not i18n-aware, you're thinking of IRIs. But since there is a 
> standard mapping for IRIs, it's pretty clear what you actually want. The 
> *URI* syntax should be:
> 
>   "ws" ":" heir-part [ "?" query ]

Ok, done.

> Then the encoding considerations should be something like:
> 
>   Because many characters are not permitted with this syntax, the
>   "heir-part" and "query" elements may contain characters from the
>   Unicode Character Set [UCS] as suggested by URI [RFC3986] using the
>   reg-name and percent-encoding translations of IRI to URI
>   mapping [RFC3937]. Translation is performed by first encoding those
>   Unicode characters as octets to the UTF-8 character
>   encoding [RFC3629]. Replace the reg-name part of the heir-part by
>   the part converted using the ToASCII operation specified in section
>   4.1 of [RFC3490] on each dot-separated label, and by using U+002E
>   (FULL STOP) as a label separator, with the flag UseSTD3ASCIIRules
>   set to TRUE, and with the flag AllowUnassigned set to TRUE. Then
>   only those octets that do not correspond to characters in the
>   unreserved set should be percent-encoded.
>
>   By using UTF-8 encoding, there are no known compatibility issues
>   with mapping Internationlized Resource Identifiers to websocket
>   URIs according to [RFC3987].

I've used the above as a guide for what to put in the spec. I didn't use 
it literally because it seemed to misuse RFC2119 terminology, and it 
wasn't clear to me where the descriptive ended and the normative started. 
I hope the text now in the spec makes sense. Let me know if it needs more 
work.

> > > Furthermore, it still doesn't answer what the semantics of these parts
> > > are. What do "ihier-part" and "iquery" represent in a ws URI?
> > 
> > This is defined by the RFC 3987, no? Surely we wouldn't want IRI
> > components to have different meanings in different schemes?
> > 
> > > What's the effect? How are they used?
> > 
> > This is defined earlier in the Web Socket specification.
> 
> Section 3.1 Parsing Web Socket URLs seems to make the semantics pretty 
> clear to me. How about adding "See Section 3.1" to URI scheme semantics 
> portions of the IANA Considerations sections? Would that be sufficient?

I don't think section 3.1 really adds anything more than what the 
registration at this point says (that the path and query form the resource 
name, and the other components are as defined in the URI spec).

On Fri, 4 Sep 2009, Joseph A Holsten wrote:
>
> Traditionally, every other scheme defined since RFC3987 has defined 
> itself as a URI and defined the exact encoding considerations to handle 
> reserved characters that may occur given the semantics of a particular 
> part. You have very standard semantics: userinfo, host, port, path 
> segments, query. Those that might meaningfully contain reserved 
> characters are userinfo, reg-name segments, and query. reg-name parts 
> get ToASCII, everything else gets mapped with percent-encoding. You 
> actually have to say this because it's not obvious. There's more than 
> one way to do it.

We really should fix these specs so that there isn't more than one way to 
do it for future schemes.

On Sat, 5 Sep 2009, Toby Inkster wrote:
> 
> As I understand it, if you, Ian Hickson, own websockets.net, then you 
> are the authority for deciding what resources are represented by URIs 
> starting with:
> 
> 	http://websockets.net/

I think it would be pretty ridiculous for anyone to claim that the URL 
above represents anything but a host ("websockets.net"), port (80), and 
path (/) that can be used over HTTP (with whatever method makes sense 
given the context in which the URL is found -- e.g. GET if the URL was on 
the side of a bus, POST if it is was in <form method=post action="...">). 
For example, claiming that it represents a person or a book or something 
like that. (Of course, that hasn't stopped the Semantic Web community from 
doing exactly that.)

I certainly wouldn't feel comfortable saying that that URI represented 
anything to do with the Web Sockets protocol.

> In particular, you in your role as authority are free to decree that 
> this:
> 
>    http://websockets.net/example.com/foo
> 
> Represents a Web Sockets path of "/foo" running on port 81 of the host 
> example.com.

That seems like a terrible idea. Surely it represents a Web page/resource/ 
service/whatever you want to call it on "websockets.net" port 80 path 
"/example.com/foo". Treating it as other things seems like a terrible 
layering and orthogonality violation.

On Mon, 7 Sep 2009, Julian Reschke wrote:
> 
> What I still miss is a reference from the URI registration template to 
> the section which defines the syntax (*)

The registration template _is_ the section that defines the syntax.

> and in that section, a statement about what the resource name exactly is 
> good for. (It's definitively not obvious by just reading the parsing 
> algorithm).

What the resource name is good for is entirely up to the server. The spec 
doesn't assign it any particular semantic meaning or purpose beyond what 
the registration text says. So I don't know what more I could say.

> (*) I think that section would be much more readable when it used ABNF 
> as everybody else does.

Assuming you mean the section that says how to parse the URLs, then the 
only part of it that could conceivably use ABNF is the part defined in 
[WebAddresses], so I don't know what it would mean to use ABNF here.

> I hear that by specifying an algorithm you want to exclude certain 
> standard things like fragments, and include error handling; but I think 
> ABNF + prose would be much easier to understand.

Please send such feedback to Larry; I am no longer editing those 
algorithms.

> Furthermore, fragment identifiers are orthogonal to the URI scheme, see 
> <http://greenbytes.de/tech/webdav/rfc3986.html#rfc.section.3.5.p.2>:
> 
> "Fragment identifier semantics are independent of the URI scheme and 
> thus cannot be redefined by scheme specifications."

I've no idea to what you are referring here. Where are fragment 
identifiers even mentioned in the Web Socket protocol spec?

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Sep 2009 11:47:36 +0200

Ian Hickson wrote:
> ...
> ...

I meant Section 3.1, which essentially is useless, as it replicates 
what's said in the ABNF in the registration template.

> Please send such feedback to Larry; I am no longer editing those 

I'm still talking about WebSockets, Part 3.1.

> identifiers even mentioned in the Web Socket protocol spec?

You did mention them on IRC 
(<http://krijnhoetmer.nl/irc-logs/whatwg/20090904#l-1007>):

 > # [23:26] <Hixie> annevk3: and i want the frag-id case to be invalid
 > before conversion

What I'm trying to explain is you can't make frag-ids "invalid", even by 
the way you specify the parsing.

BR, Julian

---

From: Ian Hickson <[email-masked]>
Date: Thu, 17 Sep 2009 10:14:15 +0000 (UTC)

On Thu, 17 Sep 2009, Julian Reschke wrote:
> Ian Hickson wrote:
> > >
> > > (*) I think that section would be much more readable when it used 
> > > ABNF as everybody else does.
> > 
> > Assuming you mean the section that says how to parse the URLs, then 
> > the only part of it that could conceivably use ABNF is the part 
> > defined in [WebAddresses], so I don't know what it would mean to use 
> > ABNF here.
> 

The ABNF doesn't say how you parse the URI, it says how you check if it's 
valid. Section 3.1 doesn't say how you parse the URI, it says how you 
apply the algorithm from [WebAddresses] in a way that extracts the fields 
you need to use Web Sockets.

> > Please send such feedback to Larry; I am no longer editing those algorithms.
> 

I have no idea how ABNF could possibly be used in section 3.1. Could you 
show an example of what you mean?

> the way you specify the parsing.

Ok.

I encourage you to review the actual spec, and not my comments on IRC.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Sep 2009 12:34:06 +0200

> valid. Section 3.1 doesn't say how you parse the URI, it says how you 

Actually, unless it's ambiguous, an ABNF *does* define how to parse.

> you need to use Web Sockets.

As far as I can tell, it's pretty stand-alone, except for a reference to 
the "resolve the URL" algorithm, which doesn't seem to make sense here.

> show an example of what you mean?

Just use the same ABNF, state which parts define the resource name (and 
what it means), and be done with it.

> I encourage you to review the actual spec, and not my comments on IRC.

That conversation seemed to explain why you are including all that 
specification text that IMHO is useless. I was just pointing out that, 
if the reason is to exclude fragment IDs, this is not going to have the 
effect you seem to intend.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Sep 2009 13:39:02 +0200

It now says:

>    Encoding considerations.
>       Characters in the host component that are excluded by the syntax
>       defined above must be converted from Unicode to ASCII by applying
>       the IDNA ToASCII algorithm to the Unicode host name, with both the
>       AllowUnassigned and UseSTD3ASCIIRules flags set, and using the
>       result of this algorithm as the host in the URI.
> 
>       Characters in other components that are excluded by the syntax
>       defined above must be converted from Unicode to ASCII by first
>       encoding the characters as UTF-8 and then replacing the
>       corresponding bytes using their percent-encoded form as defined in
>       the URI and IRI specification.  [RFC3986] [RFC3987]

I think that's good, except that the mention of IRI in the last sentence 
seems to be superfluous. RFC3986 already defines everything that is 
needed here. Or is there something specific from the IRI spec you think 
is relevant? (In which case it should state that more clearly).

BR, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 17 Sep 2009 10:51:27 -0700

On Sep 17, 2009, at 3:34 AM, Julian Reschke wrote:

> Ian Hickson wrote:
>> ...
>>> I meant Section 3.1, which essentially is useless, as it  
>>> replicates what's said in the ABNF in the registration template.
>> The ABNF doesn't say how you parse the URI, it says how you check  
>> if it's valid. Section 3.1 doesn't say how you parse the URI, it  
>> says how you
>

Actually, no, the purpose of an ABNF is to define the grammar for
generating valid strings and testing strings for validity.  It might
be used as a guide by something like lex to create a parser that
enforces validity while parsing, but that generally is not done in
Internet-facing software because of Postel's Law.

For example, RFC 3986 has a very specific grammar for generation
and validity of URIs, but also describes one parsing algorithm
(not the only one, but certainly one in common use) in an
Appendix that will accept any string and parse it into the major
components.

And I'll reiterate, again, that the algorithm for reference parsing
in HTML5 is not definitive of URLs -- it is just a variation on the
appendix in RFC3986 that includes a non-ASCII character encoding
step.  The entire notion that this has anything to do with IRI or URI
definition, or that we need to fix any of the IETF specs to
incorporate browser-specific reference error-handling, is simply
absurd.  They are not the same thing.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Sep 2009 21:58:20 +0200

> Internet-facing software because of Postel's Law.

OK, let me rephrase that: the ABNF does not define parsing, but provides 
sufficient information for a parser that will accept and process valid 
input.

> Appendix that will accept any string and parse it into the major

Right, and I have mentioned that one more than once to those who 
complained about missing error handling in RFC 3986.

> absurd.  They are not the same thing.

Indeed.

BR, Julian

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 18 Sep 2009 18:18:50 +0900

>> result of this algorithm as the host in the URI.

I think this has various problems.

First, it is fixed to IDNA 2003 (I think I may have said this already). 
IDNA 2008 is around the door. It doesn't use terms such as "ToASCII" or 
"AllowUnassigned".

Second, if this is about resolution (rather than about generic 
conversion), and because this is a new scheme, it should not exclude the 
case that some part of a domain name (reg-name) is percent-encoded, 
because both RFC 3986 and 3987 allow this.

Third, wording this as "characters" seems to say that this is a 
character-by-character operation, or that it might be applied to 
subsequent non-ASCII characters in groups, but ToASCII, when used, has 
to be applied to whole labels, not characters.

Fourth, as http://tools.ietf.org/html/draft-iab-idn-encoding-00 shows in 
more detail, assuming that DNS is always used for resolution of 
reg-names, and the technology will never be used e.g. on intranets with 
other resolution services seems to be unnecessarily restrictive.

Ideally, all the above points should be addressed by some work on the 
IRI front ([email-masked] cc'ed), but that work isn't done yet.

> is relevant? (In which case it should state that more clearly).

RFC 3986 indeed defines how to use %-encoding, but except for domain 
names (which are not involved in this case), it does not specify UTF-8, 
which is only done in RFC 3987.

Regards,    Martin.