Subject: [mmox] Keys as numbers in the binary format ; two different protocols

From: Catherine Pfeffer <[email-masked]>
Date: Fri, 27 Feb 2009 13:13:42 +0100

> >  - it would be less simple in a program because you would need to look
> > up in a table to solve the indirections
>
> I don't think that's intellectually honest.

Don't call me dishonest.

I have probably said something stupid here, but please let me the privilege
to recognize it.

> >  - if you have been unable to transmit the packet containing the
> > table, you can't do anything anymore
>
> If I can't send data, I can't send data, that's right! If I can't send
> data, why am I trying to do a live, real-time streaming application
anyway?

I disagree on this one. It is one thing to lose a given packed that would
prevent your avatar from receiving an IM message or from moving 10
centimeters to the right, and it is another thing to fail receiving a list
of numbers that would allow you to decode the entire session. That's
probably the showstopper point.

For the rest, you are right, my arguments don't resist a serious
examination.

> Note that I am not proposing using schema-driven binary interchange for
> transactional or durable applications. I believe we need at least two
> separate systems: a very efficient transmission format for streaming
> live, high-bandwidth data (such as a stream of entity property updates
> during simulation), and a very robust exchange format for transactional,
> batch or offline processing (such as "properly structured" XML).

It makes sense.

Now is it really a good idea to make both systems derive of the binary
serialization? Probably not. As I see it, LLSD is really fit to real time
data, but its XML serialization, at least in its current state, is not what
I would call "properly structured" for all the reasons that I already
explained.

Having two different systems would also remove the problem of negotiation.
Real time data would be binary encoded, transactional data would be XML
encoded (SOAP, XML-RPC, whatever). Different class of problems, different
encodings, so there's nothing to negociate.

Each packet would naturally fall in one of each classes, real time or
transactional. We could even decide that the first class should be
transported by UDP and the second one by HTTPS (Infinity will kill me for
this remark).

Perharps it is really worth rewriting the XML serialisation so that it fits
into good XML practice, after all.

-- 
Cathy

---

From: Jon Watte <[email-masked]>
Date: Fri, 27 Feb 2009 09:57:51 -0800

> > I don't think that's intellectually honest.

To nit pick, I didn't call you dishonest ("I don't think you are ..."), 
I called the argument dishonest ("I don't think that's ..."). I have not 
seen any reason to believe you are anything but an upstanding, smart, 
helpful intellectual. Please accept my apologies if you inferred 
something else from my statement.

> I disagree on this one. It is one thing to lose a given packed that 
> would prevent your avatar from receiving an IM message or from moving 
> 10 centimeters to the right, and it is another thing to fail receiving 
> a list of numbers that would allow you to decode the entire session. 
> That's probably the showstopper point.
>

If some data has to be transmitted before you can continue with the 
session, then the session protocol has to make sure that that data is 
transmitted. Easiest is of course to run the session through TCP. If, 
for some reason, you don't want the data queuing/delay semantic of TCP, 
there are various "reliable over UDP" options you could implement. An 
alternative is also to send the required set-up data using TCP (say, 
when establishing a session), and using a separate UDP channel for lossy 
real-time data.

In other words: your argument says that we have to be careful in 
implementation, but it doesn't say that implementation has to be 
impossible, or even hard.

> serialization? Probably not. As I see it, LLSD is really fit to real 
> time data, but its XML serialization, at least in its current state, 
> is not what I would call "properly structured" for all the reasons 
> that I already explained.
>

And I think LLSD doesn't fit real-time data, because it is too verbose 
to send the tag and type for each and every piece of data, when you can 
omit it with no loss of generality.

I believe, like you, that deriving everything from the binary 
serialization is not the best approach. I believe that you should start 
with a description of the data you need in the abstract, and then 
separately derive efficient binary representations, and durable XML 
representations, from that description. That allows you to keep the data 
description in one place, which is important to avoid versioning 
de-sync, and it also allows you to design each serialization to fit the 
particular format.

> Having two different systems would also remove the problem of 
> negotiation. Real time data would be binary encoded, transactional 
> data would be XML encoded (SOAP, XML-RPC, whatever). Different class 
> of problems, different encodings, so there's nothing to negociate.
>

I fully agree with this. Each service would be defined in terms of which 
serialization it uses. Meanwhile, having a common "root" format for 
describing the data you want to exchange means you don't have to worry 
about double development, version slew, etc. The format would 
necessarily need to have enough information in it such that a good XSD 
and a good binary serialization could both be generated.

Sincerely,

jw