Subject: [mmox] XML serialization

From: Catherine Pfeffer <[email-masked]>
Date: Tue, 24 Feb 2009 14:51:05 +0100

(Infinity: there is a question for you at the end of this message)

Jon said:
> A better way is
>
> <map>
>    <value type="boolean" key="success">true</value>
>    <value type="string"
>         key="something_i_like_to_eat_on_sundays">bananas</value>
> </map>
>
> natural way to express it.This has less of the angle bracket tax, and it
> has a VERY straight-forward parsing style. In fact, in my opinion it's
> the most In XPath, it's now trivial: "value[@key=success]"

Yes. I had thought at this solution too, but I had hesitated to propose it
straight away to the mailing list...

This solution has fewer angle bracket tax, but it also has hidden
implications. I assumed (perharps wrongly) that the authors of the XML
serialization had seen these implications, and rejected this solution
because of these implications.

Moving tags (<key>, <boolean>, <string> and so on) to attribute names or
values is not an innocent operation. Here are the consequences I can see:

1) To allow validation of the allowed types, the new "type" attribute would
need to be declared as an enumeration in the DTD or schema.

No big deal so far.

2) More of a problem: if using XML schema for validation, it is more
straightforward to check that
      <float>3.14</float>
and
      <integer>42</integer>
are correct, than to check that
      <value type="float">3.14</value>
      <value type="integer">42</value>
are correct.

(XML schema allows for checking that the data inside the tags conform to a
certain syntax, for example that floats are made of digts and possibly of a
decimal dot and a minus sign).

3) The key ("success" or "something_i_like_to_eat_on_sundays") is not part
of the data flow anymore, but part of the markup.

There are half-religious wars inside the XML world according to what should
be an attribute and what should be text ;-). Personally, in this case, I
would say that an attribute is more suited to this kind of data, because
then we could enumerate all the allowed values for the virtual world context
("name", "taper_x", "socks", "notecard_encoding", whatever is normalized on
top of VWSD), so I see this as an advantage of your solution.

On the other hand, if I understand well, LLSD has been defined to allow keys
to evolve rapidely and to be supported differently across various
implementations, so validating the key values is perharps a bad idea.

4) Other advantages of your solution, Jon: default values for attributes
could be omitted, and we could address size and signed-ness of the scalars
in separate attributes.

Examples:
         <value>42</value>
would be synonymous of
         <value type="integer" size="32" signed="yes">42</value>

         <value type="float" size="64">3.14159265353238</value>
would be synonymous of
         <value type="float" size="64" signed="yes">3.14159265353238</value>

Of course, there is a lot to spare on the angle bracket tax here, since most
data is probably integer, 32 bits,, and signed).

It is also more future-proof, since 'size="128"' could be added afterward
easily, even if not planned initially.

> I'm working on a list of the things I think are wrong with the current
> proposal, of which this is one thing (there are about six). If we're
> open to change on those things (including this), then I could get behind
> the LLSD. Stay tuned :-)

I have a similar list too ;-). Someone here has proposed a VWSD working
group to separate high level concerns (like meshes versus prims) from low
level concerns (like data types and serializations). If we are sure that
xxSD is the way to go (and not ASN or whatever), I would volunteer to be
part of such a group, and so would you, Jon, I guess ;-).

Infinity, how was the XML serialization thought? Have several alternatives
been compared and considered carefully? Or is it just the result of the
inspiration of the person who made it at a given time?

-- 
Cathy

---

From: "Meadhbh Hamrick (Infinity)" <[email-masked]>
Date: Tue, 24 Feb 2009 10:04:18 -0800

yes. several alternatives were considered.

it was noted, however, that...

* the difference in space between the existing standard and something  
like what's proposed here, after applying gzip is minor. (this is  
germane because we expect people to use compressed content encoding if  
they care about bandwidth utilization)
* the difference in parse time is negligible as it turns out you  
either have to parse angle brackets or parse <attribute>=<value>  
inside an element.
* the DTD as it's specified now was considered "easier to code" by the  
person tasked with implementing it
* many of us did not like the idea of making the content of an  
attribute and the name of an element co-dependent, as Zero explained  
in a different email

i'm also a bit confused... what's the difference between

<value type="integer" size="128">99992123123123</value>

and

<integer size="128"> 99992123123123</integer>

???

i might be mis-reading you, but it sounds like you're saying the  
former allows you to add a size attribute, but the latter doesn't.  
which can't possibly be what you're suggesting because i just put  
myself in the situation of having to extend the XML serialization in  
the future and added a size attribute to the integer element and the  
world didn't come to an end.

On Feb 24, 2009, at 5:51 AM, Catherine Pfeffer wrote:

> Yes. I had thought at this solution too, but I had hesitated to  
> propose it straight away to the mailing list...
>
> implications. I assumed (perharps wrongly) that the authors of the  
> XML serialization had seen these implications, and rejected this  
> solution because of these implications.
>
> Moving tags (<key>, <boolean>, <string> and so on) to attribute  
> names or values is not an innocent operation. Here are the  
> consequences I can see:
>
>
> 1) To allow validation of the allowed types, the new "type"  
> attribute would need to be declared as an enumeration in the DTD or  
> schema.
>
> No big deal so far.
>
>
> (XML schema allows for checking that the data inside the tags  
> conform to a certain syntax, for example that floats are made of  
> digts and possibly of a decimal dot and a minus sign).
>
>
> 3) The key ("success" or "something_i_like_to_eat_on_sundays") is  
> not part of the data flow anymore, but part of the markup.
>
> There are half-religious wars inside the XML world according to what  
> should be an attribute and what should be text ;-). Personally, in  
> this case, I would say that an attribute is more suited to this kind  
> of data, because then we could enumerate all the allowed values for  
> the virtual world context ("name", "taper_x", "socks",  
> "notecard_encoding", whatever is normalized on top of VWSD), so I  
> see this as an advantage of your solution.
>
> On the other hand, if I understand well, LLSD has been defined to  
> allow keys to evolve rapidely and to be supported differently across  
> various implementations, so validating the key values is perharps a  
> bad idea.
>
>
> 4) Other advantages of your solution, Jon: default values for  
> attributes could be omitted, and we could address size and signed- 
> ness of the scalars in separate attributes.
>
> Examples:
>          <value>42</value>
> would be synonymous of
> Of course, there is a lot to spare on the angle bracket tax here,  
> since most data is probably integer, 32 bits,, and signed).
>
> It is also more future-proof, since 'size="128"' could be added  
> afterward easily, even if not planned initially.
>
>
> I have a similar list too ;-). Someone here has proposed a VWSD  
> working group to separate high level concerns (like meshes versus  
> prims) from low level concerns (like data types and serializations).  
> If we are sure that xxSD is the way to go (and not ASN or whatever),  
> I would volunteer to be part of such a group, and so would you, Jon,  
> I guess ;-).
>
> Infinity, how was the XML serialization thought? Have several  
> alternatives been compared and considered carefully? Or is it just  
> the result of the inspiration of the person who made it at a given  
> time?
>
> -- 
> Cathy
> _______________________________________________
> mmox mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mmox

---

From: Jon Watte <[email-masked]>
Date: Tue, 24 Feb 2009 11:23:20 -0800

Catherine Pfeffer wrote:
> (XML schema allows for checking that the data inside the tags conform 
> to a certain syntax, for example that floats are made of digts and 
> possibly of a decimal dot and a minus sign).
>

But you don't actually know whether those digits are actual, valid data 
until and unless that data hits some real executable code. Additionally, 
we will likely deal with data types that are naturally atomic ("vector3" 
or "quaternion") but that are not part of the XML data type set. Sending 
a vector3 as "<float name="x">x</float><float name="y">y</float><float 
name="z">z</float>" seems like a bad idea, compared to 
"<vector3>x,y,z</vector3>".

You wouldn't know whether "123000,-5000000,600000" is a proper teleport 
destination, or under the surface of the earth, or outside the allowed 
simulation coordinate space, until you treated that coordinate in the 
target world space. Thus, the validation capabilities of a possible XSD 
are limited at best anyway.

I must admit that the main reason I want things in attributes is 
brevity, and uniformity of handling. If I want to get a node set of all 
property values of a given entity record, I can simply do:

  values = entityNode.getElementsByTagName("value");

Similarly, if I want to get a specific element, I can do:

  thePosition = entityNode.getElementById("pos");

This assumes that the "key" attribute is defined as an ID such that the 
DOM implementation can recognize it. On the other hand, depending on 
structure, it may mean that they key "pos" can't be used on multiple 
objects in the same overall XML document, so it might be better to get 
"matching attribute value" anyway:

  thePosition = entityNode.getElementPath("[@key='pos']")

> If we are sure that xxSD is the way to go (and not ASN or whatever), I 
> would volunteer to be part of such a group, and so would you, Jon, I 
> guess ;-).

Personally, I think it's too early to really worry about it too much. I 
first want rough consensus on the model we're using -- I think most, but 
not all, participants on the list now understand that interoperating 
virtutal world objects practically require simulation hosts to talk to 
other simulation hosts. I also think that most participants understand 
that a specified client/server protocol would not actually solve 
interoperability between disparate simulation systems.

But how do we actually define and determine "rough consensus"? Perhaps 
coming up with some metric for that would be a good first step.

Sincerely,

jw

---

From: Lawson English <[email-masked]>
Date: Tue, 24 Feb 2009 13:58:15 -0700

Jon Watte wrote:
>
>
> Personally, I think it's too early to really worry about it too much. 
> I first want rough consensus on the model we're using -- I think most, 
> but not all, participants on the list now understand that 
> interoperating virtutal world objects practically require simulation 
> hosts to talk to other simulation hosts. 

which simulation hosts in Croquet/Qwaq are we talking about?

> I also think that most participants understand that a specified 
> client/server protocol would not actually solve interoperability 
> between disparate simulation systems.

Depends on how disparate....

Seems to me that the forterra strategy, as implemented in SL, would 
require sufficient extra server sapce to accommodate all 85000+ 
concurrent users visiting 85000+ different simulators at the same time....

> coming up with some metric for that would be a good first step.
How loud the humming is, I understand.

Lawson

---

From: Jon Watte <[email-masked]>
Date: Tue, 24 Feb 2009 13:19:04 -0800

Lawson English wrote:
>> I first want rough consensus on the model we're using -- I think 
>> most, but not all, participants on the list now understand that 

In Croquet, each participating peer would be a simulation host, 
authoritative for the objects that they introduce into the shared world.

> concurrent users visiting 85000+ different simulators at the same 

We already discussed this, if you look in the archives. That is not 
true. If any Second Life customer interoperates with some other system, 
then it is true that the SL servers will have to funnel object 
information through their system to their clients and their objects, so 
that the clients can see the other systems, and the objects can 
interoperate with the objects from the other system. This is not a large 
additional load, because the simulation of the non-Second-Life objects 
is done by other hosts.
As long as one Second Life customer can only be in one place at a time, 
there will only need to be one avatar simulated by the Second Life 
servers, which is pretty much what they're already providing.

Sincerely,

jw

---

From: Lawson English <[email-masked]>
Date: Tue, 24 Feb 2009 14:26:41 -0700

Jon Watte wrote:
> Lawson English wrote:
>> Jon Watte wrote:
>>>
>>>
>>> Personally, I think it's too early to really worry about it too 
>>> much. I first want rough consensus on the model we're using -- I 
>>> think most, but not all, participants on the list now understand 
>>> that interoperating virtutal world objects practically require 
>>> simulation hosts to talk to other simulation hosts. 
>>
>>
> true. If any Second Life customer interoperates with some other 
> system, then it is true that the SL servers will have to funnel object 
> information through their system to their clients and their objects, 
> so that the clients can see the other systems, and the objects can 
> interoperate with the objects from the other system. This is not a 
> large additional load, because the simulation of the non-Second-Life 
> objects is done by other hosts.
> As long as one Second Life customer can only be in one place at a 
> time, there will only need to be one avatar simulated by the Second 
> Life servers, which is pretty much what they're already providing.
>

Which non-SL objects are we talking about? scripted behavior can be 
added to any SL object or objects. Are you suggesting that the SL 
simulator handle the scripted i/o from 15,000 objects living in another 
sim each with potentially hundreds of active scripts?

Lawson

---

From: Jon Watte <[email-masked]>
Date: Tue, 24 Feb 2009 14:25:42 -0800

Lawson English wrote:
> simulator handle the scripted i/o from 15,000 objects living in 
> another sim each with potentially hundreds of active scripts?
>

No, that's exactly what I'm not suggesting. I suggest you actually read 
the proposal, which does lay this out in some detail.

In brief, each system runs its own objects, with its own scripts. Only 
"proxy" information about the object is funneled through the peer 
systems, to allow them to make whatever determination they want (or not) 
about those objects, and present those objects to their users. This 
keeps simulation of each object on the system where the object 
originated, which is a requirement, given that some systems script in 
Java, others in JavaScript, and yet others in C++ and XML.

Sincerely,

jw

---

From: "Meadhbh Hamrick (Infinity)" <[email-masked]>
Date: Tue, 24 Feb 2009 17:22:19 -0800

ah! i think i see a bit of a disconnect here.

jon, you're saying you want to be able to use common DOM apis to  
access data in an XML serialization.

fwiw... that's the atypical use at the lab (<insert comments here  
acknowledging that there's more the the world than linden lab>) and i  
mention it not to imply that this is the type of thing that should  
never be done, but simply to point out that the way LLSD XML  
serialization layer was intended to be used was as a container for  
something that got de-serialized before the application layer gets  
it's mitts on the data.

so.. ya got your stack... (um.. i apologize for the graph... if you  
change the typeface to something vaguely monospace, it'll look much  
nicer)

Application                                                 Application
      | "a"                                                       | "d"
Presentation                                                Presentation
      | "b"                                                       | "c"
  Transport <------ the great mysteries of the network -----> Transport

it sounds like you want to do some processing at points "b" and "c".  
after it has been received by the transport and before it gets un- 
marshalled / de-serialized.

is this just 'cause you're used to writing code that directly uses the  
DOM, or are you routing to specific application endpoints based on the  
contents of the stuff in the XML?

and also... is this a requirement of your system? or just the way that  
seems right to you?

(not trying to pick a fight... just curious)

-cheers
-meadhbh

On Feb 24, 2009, at 11:23 AM, Jon Watte wrote:

> But you don't actually know whether those digits are actual, valid  
> data until and unless that data hits some real executable code.  
> Additionally, we will likely deal with data types that are naturally  
> atomic ("vector3" or "quaternion") but that are not part of the XML  
> data type set. Sending a vector3 as "<float name="x">x</float><float  
> name="y">y</float><float name="z">z</float>" seems like a bad idea,  
> compared to "<vector3>x,y,z</vector3>".
>
> You wouldn't know whether "123000,-5000000,600000" is a proper  
> teleport destination, or under the surface of the earth, or outside  
> the allowed simulation coordinate space, until you treated that  
> coordinate in the target world space. Thus, the validation  
> capabilities of a possible XSD are limited at best anyway.
>
> brevity, and uniformity of handling. If I want to get a node set of  
> all property values of a given entity record, I can simply do:
>
> This assumes that the "key" attribute is defined as an ID such that  
> the DOM implementation can recognize it. On the other hand,  
> depending on structure, it may mean that they key "pos" can't be  
> used on multiple objects in the same overall XML document, so it  
> might be better to get "matching attribute value" anyway:
>
>> prims) from low level concerns (like data types and  
>> serializations). If we are sure that xxSD is the way to go (and not  
>> ASN or whatever), I would volunteer to be part of such a group, and  
>> so would you, Jon, I guess ;-).
>
>
> simulation hosts to talk to other simulation hosts. I also think  
> that most participants understand that a specified client/server  
> protocol would not actually solve interoperability between disparate  
> simulation systems.
>
> But how do we actually define and determine "rough consensus"?  
> Perhaps coming up with some metric for that would be a good first  
> step.
>
> Sincerely,
>
> jw
>
>

---

From: Jon Watte <[email-masked]>
Date: Tue, 24 Feb 2009 17:37:08 -0800

Meadhbh Hamrick (Infinity) wrote:
> after it has been received by the transport and before it gets 
> un-marshalled / de-serialized.
>

Yes. In fact, my system may actually use the XML directly, without 
de-marshaling. The whole point of a data representation is that it is 
useful as-is, IMO :-)

> and also... is this a requirement of your system? or just the way that 

We currently have customer who do something similar based on data they 
get from us. Specifically, they get an entity data stream, and use 
semantically marked-up world geometry to do automated analysis of 
performance. One specific application is where their system will check 
whether soldiers traversing a street is exposing themselves too much to 
possible attack -- the output is qualitative information like "you put 
teammates 2 and 3 in the cone of fire from this doorway for too long at 
time 13:03 into the exercise" or "your team stacked against this 
structurally unsound wall at time 8:30 and stayed there for four minutes."

That external application (not built or controlled by us) receives a 
text representation of entity data, and process that directly. It is not 
currently LLSD XML serialized data (for mostly obvious reasons :-), but 
if "MMSD" was capable of serving this use case in a useful (and not too 
chatty) way, then I would be all for it -- suitable standards are a good 
thing, and our customer would then also be able to tie their application 
into any virtual world that provides the same kind of data stream.

Sincerely,

jw