Subject: [mmox] Even more compact binary serialization?

From: Catherine Pfeffer <[email-masked]>
Date: Wed, 25 Feb 2009 03:55:16 +0100

> Can you view a "xxSD" specification file as an alternative DTD? I e, the
> output of a specific xxSD file could be one or more of the following:
> 1) C++ source code that can marshal/demarshal structs to various
> serializations
> 2) An XML DTD
> 3) Python code that uses module "struct" and "xml.dom.minidom" to
> translate between XML and struct formats
> ...

I am not sure I understand this question. I apologize, but English is not my
language and I sometimes miss a few things.

If the question is "could we build a DTD or a schema to describe the tags
(<ObjectData>, <Path>, etc.)", of course we could do it. That would even be
the whole point of all subsequent works in this mailing list: describe the
tags and their semantics to allow for virtual worlds interoperability.

If your question is "Catherine, could you please write the DTD or the schema
for this packet (the existing ObjectAdd packet)?" yes I could do this boring
work, but I don't really see the point.

> However, I think that you still haven't addressed one big concern, which
> is the schema-dependent transmission format. Given your XML-ish version
> of a packet, in binary, it could look something like:

Why would we need two serializations?

My point is, the current binary serialization is very good and could be used
unchanged (with a few data type additions). I would even call it "binary
XML" since it achieves the same goals of interoperability and flexibility as
XML ;-). And it does it with the same mechanism by the way: by transporting
both the data and the markup in one single file.

But if we were to go for XML, then it should be real XML, and not a "dummy"
transposition of the very good binary serialization, because the result does
not match the paradigms behind XML, and is also very inefficient from a
bandwidth point of view.

> 1) It uses the knowledge that the destination will have the schema to
> encode property references using small, context-dependent integers.

Your binary serialization is very compact because it replaces a key (like
"Taper_X" or "NextOwnerMask") with a small number (like 384 or 17). But the
price to pay in terms of flexibility is also very high.

Such "tags are numbers" approach reminds me of ISO 2709 (
http://en.wikipedia.org/wiki/ISO_2709). It's remarkable that such a standard
has been written in the sixties, when room on a magnetic tape was very
limited.

As always, engineering is a matter of compromises. Given the high number of
data types implied, the speed of evolution of virtual worlds, the huge
dimensions of some grids, and the number and heterogeneity of worlds to
interconnect, I think we should pay a lot of attention to flexibility, even
though bandwidth concerns should not be forgotten.

-- 
Cathy

---

From: Jon Watte <[email-masked]>
Date: Wed, 25 Feb 2009 11:48:22 -0800

> If the question is "could we build a DTD or a schema to describe the 
> tags (<ObjectData>, <Path>, etc.)", of course we could do it. That 
> would even be the whole point of all subsequent works in this mailing 
> list: describe the tags and their semantics to allow for virtual 
> worlds interoperability.
>

My question is not just to you, but to the list:
We view the xxSD description language as a source language, from which 
we could compile various outputs.
Could one of those outputs be a DTD and an XSD? Thus, we don't create 
the DTD or XSD directly; we create the mapping from xxSD input to DTD 
and XSD output.

Btw, I forgot one other important case for using the xxSD source file: 
Systems that work on the data as abstract properties would likely parse 
the source file directly, and use it to understand the data, rather than 
trying to go through code gen.

> Why would we need two serializations?

There are at least three serialization scenarios:

1) There is a persistent "session" connection. The connection/session is 
stateful and durable. This connection is likely to talk about the same 
object types many times, and is likely to send lots of updates to object 
property values. This session is real-time, interactive, and saving 
bytes is important. You can assume that the schema will not change while 
the session is connected. A binary format that wrings every last byte 
out of "knowing" the schema (which is established during up-front 
connection negotiation) is what's needed here.

2) There are non-persistent connections, REST-ful, transaction-based, 
request/response type interactions. Here, saving every byte is not as 
important. The schema may, theoretically, change between each request. I 
think the XML serialization is the right choice for this, although some 
people say they prefer JSON (mostly because that's easier to work with 
within HTML client-side code).

3) Persistent storage into database records, logs, etc. Here, the 
ability to reconstruct data that was created on earlier versions of 
schema is paramount. XML is one possible encoding (and certain 
databases, such as ORACLE, actually have specific XML support), whereas 
a more traditional approach would be a flexible, tagged binary format, 
such as IFF or the currently proposed LLSD binary serialization.

> My point is, the current binary serialization is very good and could 
> be used unchanged (with a few data type additions). I would even call 
> it "binary XML" since it achieves the same goals of interoperability 
> and flexibility as XML ;-). And it does it with the same mechanism by 
> the way: by transporting both the data and the markup in one single file.
>

I disagree that the current binary representation is "very good" -- it 
is designed towards case 3, but because XML can solve case 3 just as 
well (using existing tool sets), I'm more interested in serving case 1.

> But if we were to go for XML, then it should be real XML, and not a 
> "dummy" transposition of the very good binary serialization, because 
> the result does not match the paradigms behind XML, and is also very 
> inefficient from a bandwidth point of view.
>

I agree. I think that current practice within Linden Lab code (or, for 
that matter, OLIVE code) is not as important as getting this right for 
the general interoperability case.

> Your binary serialization is very compact because it replaces a key 
> (like "Taper_X" or "NextOwnerMask") with a small number (like 384 or 
> 17). But the price to pay in terms of flexibility is also very high.
>

There is no flexibility price to pay, because those numbers are 
negotiated on a per-session basis. As long as the schema cannot change 
per session, and the mapping from number to semantic (property, tag) is 
re-built at the start of each session, there is no loss of generality or 
flexibility. I took some python code and moved it from approach 3 to 
approach 1, and it literally took less than a day. Compared to the 
bandwidth savings for all future interoperable virtual worlds network 
traffic, I think that's a good trade-off!

>
> As always, engineering is a matter of compromises. Given the high 
> number of data types implied, the speed of evolution of virtual 
> worlds, the huge dimensions of some grids, and the number and 
> heterogeneity of worlds to interconnect, I think we should pay a lot 
> of attention to flexibility, even though bandwidth concerns should not 
> be forgotten.
>

One additional point: In the protocol I have suggested, the requesting 
side subscribes to properties by container type, and the sending side 
will make sure to not send properties that are not interesting to the 
other side. This allows one side to introduce new data types, without 
the binary protocol breaking, because the receiving side will simply not 
subscribe to property identifiers for properties of types that are 
unknown to it.

Sincerely,

jw