Subject: [mmox] More LLIDL questions

From: "Hurliman, John" <[email-masked]>
Date: Wed, 18 Feb 2009 13:39:43 -0700

; An LLSD value containing { member1 : true, member2 : true } comes in. Is member3 a string or an int?
&a = { member1 : true, member2 : false, member3 : string }
&a = { member1 : false, member2 : true, member3 : int }

; How is a parser supposed to handle this?
&b = { $ : &b }
; Or this?
&c = { $ : &d }
&d = { $ : &c }

; The first variant has the type uri. The second variant has the type string?
&e = uri
&e = "http://www.google.com/"

; Forward-slash seems like an odd addition to 0-9+A-Z. Why this character?
&f/ = "hello slash"

John

---

From: Mark Lentczner <[email-masked]>
Date: Thu, 19 Feb 2009 17:15:31 -0800

> &a = { member1 : false, member2 : true, member3 : int }

In our LLIDL implementation, if you matched that LLSD against the  
LLIDL value "&a", with those variants in scope, the answer would come  
back INCOMPATIBLE.  This is because no variant matched the value.  In  
the case of the selectors in LLIDL, they all need to match.

Had the variants been:
&a = { member1: true, member3: string }
&a = { member2: true, member3: int }

The answer would be MIXED.  This is because the actual value matched  
each variant with the same result: In each case, the the data had  
parts that were ADDITIONAL (member2 or member1), and parts that were  
DEFAULTED (member3).  The state of having both DEFAULTED parts and  
ADDITIONAL parts is MIXED. Matching variants returns the best match,  
but in this case they were both MIXED, so the result is MIXED.

Hope that was clear.

Please note: The notion of returning a confidence level from a LLIDL  
to actual LLSD match is a function of our library, not part of the  
specification.  This was intentional: LLSD is defined in such a way  
that given *any* LLSD, and code that expects any particular structure  
- the code can safely extract values, receiving defaults if things  
don't match as expected. Hence, LLIDL isn't a way of expressing a hard  
"go/no-go" conformance, it is a way of expressing what is expected.

> ; How is a parser supposed to handle this?

We could declare recursive LLIDL is illegal.  It isn't too hard to  
detect. On the other hand, since LLSD data values can't be recursive,  
even if you have a recursive LLIDL description, any actual match  
operation is bounded.  Our implementation handles recursive LLIDL and  
it took no extra work to do so.

> &e = "http://www.google.com/"

The second doesn't have the type string, it is a selector of a  
specific string.   Here's how our LLIDL would match these LLSDs:
	string "http://www.google.com/" --> MATCHED (matches the selector  
exactly)
	string "http://www.example.com/" --> CONVERTED (convertable to a uri)
	uri "http://www.google.com/" --> MATCHED (matches the type uri)
	uri "http://www.example.com/" --> MATCHED (matches the type uri)
	string "" --> INCOMPATIBLE (neither matches the specific selector  
string, nor is a reasonable conversion to uri)

The notion of "reasonable conversion" is going to be implementation  
specific, since LLSD defines a result no matter what is converted to  
what.  In our case, we took the notion that if the transmitted value  
only converts to the desired value via a defaulting, well, then, that  
doesn't really increase the confidence in the match.  (There is some  
subtlety in our choices here... I can go into it more if you like.)

> ; Forward-slash seems like an odd addition to 0-9+A-Z. Why this  

As a trailer, sure, but we like using slash as a name space delimiter  
in our protocols for resource names, member names, fixed selector  
strings, and, as you indicate, selector names:

%% thing/operation_x
-> { id: uuid }
<- { name: string, system_a/count: int, system_b/count: int, system_c:  
&system_c/info }

&system_c/info = { name: string, count: int }
&system_c/info = { name: string, count: real, extended: true }

	- Mark

Mark Lentczner
Sr. Systems Architect
Technology Integration
Linden Lab

[email-masked]

Zero Linden
[email-masked]

---

From: "Hurliman, John" <[email-masked]>
Date: Thu, 19 Feb 2009 18:56:59 -0700

Great, thank you for the quick responses on all of these questions. Keeping the implementation private is a good way to test the accuracy of the draft and determine if it explains the concepts well enough. For the most part it seems reasonable, but there are parts that I've scratched my head at. Generating strongly typed classes in C# from LLIDL leaves a lot of unnamed ("anonymous", in C#-speak) variables.

Last question. I found this snippet of LLIDL from the agent login draft proposal on the Second Life wiki:

agents: [ first_name: string, last_name: string ... ]

Is that valid LLIDL, or should it be:

agents: [ { first_name: string, last_name: string } ... ]

Allowing key:value pairs in arrays seems to defeat the purpose of the map type.

John

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Mark Lentczner
>Sent: Thursday, February 19, 2009 5:16 PM
>To: [email-masked]
>Subject: Re: [mmox] More LLIDL questions
>
>_______________________________________________
>mmox mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/mmox
>

---

From: Charles Krinke <[email-masked]>
Date: Thu, 19 Feb 2009 19:52:32 -0800 (PST)

In looking at the secondlife wiki at http://wiki.secondlife.com/wiki/MMOX it looks like one of the first proposals for this working group is the LLSD draft specification.

Am I correct in that the gist of the LLSD specification is essentially to propose three mime formats and thats about it for now?

application/llsd+xml
application/llsd+json
application/llsd+binary

Charles

---

From: "Meadhbh Hamrick (Infinity)" <[email-masked]>
Date: Thu, 19 Feb 2009 20:10:05 -0800

http://wiki.secondlife.com/wiki/MMOX is non-authoritative, but yes  
LLSD is one of the work items we have on the list.

the objective of LLSD is to provide an abstract type system with  
multiple serializations. it also defines the mime types.

however, to say that the LLSD draft exists only to define the  
serializations is akin to saying that ASN.1 exists only to provide  
input into DER or BER encodings. the two are linked, but each has a  
particular use.

LLSD as an abstract type system allows us to define and reason about  
the semantics of structured data used in PDUs independent of an  
existing implementation language. the serialization rules allow us to  
format structured data prior to transport and later de-serialize it  
after receipt. the MIME type registrations allow us to identify the  
serialization scheme used on transports that support the use of MIME  
types. LLIDL (pronounced "little") defines the set of expected  
parameters to and responses from a resource access.

LLSD was selected as the first draft to work on as it is used as a  
building block for other protocol interactions, some of which are  
published in draft form on the second life wiki. the further  
development of these protocols is also considered a task of the  
proposed working group.

-cheers
-meadhbh

On Feb 19, 2009, at 7:52 PM, Charles Krinke wrote:

> In looking at the secondlife wiki at http://wiki.secondlife.com/wiki/MMOX 
>  it looks like one of the first proposals for this working group is  
> the LLSD draft specification.
>
> Am I correct in that the gist of the LLSD specification is  
> essentially to propose three mime formats and thats about it for now?
>
> application/llsd+xml
> application/llsd+json
> application/llsd+binary
>
> Charles

---

From: "Meadhbh Hamrick (Infinity)" <[email-masked]>
Date: Thu, 19 Feb 2009 20:22:56 -0800

good eye john.

yes, the purpose of the list is to provide an array of first_name,  
last_name pairs. however... yes... it is an improperly formed array.  
it was supposed to be

[ string ... ]

with the understanding that the array SHOULD contain an even number of  
entries, odd entries representing an avatar's first_name, even entries  
representing an avatars last_name.

-cheers
-meadhbh

On Feb 19, 2009, at 5:56 PM, Hurliman, John wrote:

> Great, thank you for the quick responses on all of these questions.  
> Keeping the implementation private is a good way to test the  
> accuracy of the draft and determine if it explains the concepts well  
> enough. For the most part it seems reasonable, but there are parts  
> that I've scratched my head at. Generating strongly typed classes in  
> C# from LLIDL leaves a lot of unnamed ("anonymous", in C#-speak)  
> variables.
>

---

From: Charles Krinke <[email-masked]>
Date: Thu, 19 Feb 2009 20:34:48 -0800 (PST)

Ok, Meadhbh:

In looking at this draft *and* hoping I dont make my OpenSim peers too unhappy with me, let me ask a question.

"Is there any reason why trying to approve the draft LLSD spec would preclude or pertubate either OGP, HyperGrid or other similar notions from being supported?"

Charles

________________________________
From: Meadhbh Hamrick (Infinity) <[email-masked]>
To: Charles Krinke <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: Thursday, February 19, 2009 8:10:05 PM
Subject: Re: [mmox] LLSD

http://wiki.secondlife.com/wiki/MMOX is non-authoritative, but yes LLSD is one of the work items we have on the list.

the objective of LLSD is to provide an abstract type system with multiple serializations. it also defines the mime types.

however, to say that the LLSD draft exists only to define the serializations is akin to saying that ASN.1 exists only to provide input into DER or BER encodings. the two are linked, but each has a particular use.

LLSD as an abstract type system allows us to define and reason about the semantics of structured data used in PDUs independent of an existing implementation language. the serialization rules allow us to format structured data prior to transport and later de-serialize it after receipt. the MIME type registrations allow us to identify the serialization scheme used on transports that support the use of MIME types. LLIDL (pronounced "little") defines the set of expected parameters to and responses from a resource access.

LLSD was selected as the first draft to work on as it is used as a building block for other protocol interactions, some of which are published in draft form on the second life wiki. the further development of these protocols is also considered a task of the proposed working group.

-cheers
-meadhbh

application/llsd+xml
application/llsd+json
application/llsd+binary

Charles

---

From: "Meadhbh Hamrick (Infinity)" <[email-masked]>
Date: Thu, 19 Feb 2009 22:38:46 -0800

anything that uses LLSD and wants to be considered a "standard" must  
only provide normative references that are also "standards".

essentially the deal here is we think the RFCs that will eventually  
provide the authoritative references for OGP should be put on the IETF  
"standards track." in fact, this is the primary raison d'être for the  
proposed working group. if OGP is to be a "real" IETF standard, all  
the documents it references as "normative" need to be standards as well.

the alternative was to simply continue publishing our interface  
documents on our wiki and perhaps publishing LLSD as an informational  
RFC. however... we thought it better to use our existing OGP work as a  
starting point, put it out in a forum where the expertise of the  
internet engineering community could be utilized to review, comment  
and improve it. our objective is to create a robust, flexible family  
of protocols usable by the widest practical community.

and for what it's worth, in the context of the proposed working group  
OGP/Teleport and HyperGrid are two proposals [1] for doing similar  
things. The two approaches _do_ seem assume different trust models,  
and maybe the future is that both will find wide adoption. the "define  
mechanism, not policy" mantra you sometimes hear some people repeat  
implies that neither approach should preclude implementations from  
implementing both. this is rather like most modern email user agents  
implementing both,POP3 _and_ IMAP. the two protocols do essentially  
the same thing, but there's nothing in either protocol that prevents  
you from using either or both in the same mail client.

but i ramble.

the question as i read it was "would trying to approve the draft of  
LLSD preclude or perturb further protocols?" the answer i would give  
would be "no. it shouldn't."

-cheers
-meadhbh

[1] - yes... actually neither are currently internet drafts, but i  
believe John is working on something HyperGridish and OGP/Teleport was  
listed on the draft charter of the proposed working group.

On Feb 19, 2009, at 8:34 PM, Charles Krinke wrote:

> Ok, Meadhbh:
>
> In looking at this draft *and* hoping I dont make my OpenSim peers  
> too unhappy with me, let me ask a question.
>
> "Is there any reason why trying to approve the draft LLSD spec would  
> preclude or pertubate either OGP, HyperGrid or other similar notions  
> from being supported?"
>
> Charles
>
> existing implementation language. the serialization rules allow us  
> to format structured data prior to transport and later de-serialize  
> it after receipt. the MIME type registrations allow us to identify  
> the serialization scheme used on transports that support the use of  
> MIME types. LLIDL (pronounced "little") defines the set of expected

---

From: "Hurliman, John" <[email-masked]>
Date: Thu, 19 Feb 2009 23:50:31 -0700

Ah, that would work too. In fact, my reading of the draft says that

[ string, string ... ]

Would accomplish exactly that.

>From: Meadhbh Hamrick (Infinity) [mailto:[email-masked]]
>Sent: Thursday, February 19, 2009 8:23 PM
>To: Hurliman, John
>Cc: [email-masked]

---

From: Charles Krinke <[email-masked]>
Date: Fri, 20 Feb 2009 07:28:55 -0800 (PST)

Perhaps  I am being naive here, but I dont see anything in the draft proposal other then the definition of a few variable types and three mime definitions which seems completely innocuous to me.

Admittedly, follow on draft proposals will undoubtedly favor one teams bias at the expense of another teams bias, but I cannot find a reason not to support this variable definition and mime definition draft.

There is lots of time for us all to argue that our own favorite <whatever> is the best for the world, but it seems that any of the thoughts I have heard can all be built with this variable name and type standard.

Charles

Sent: Thursday, February 19, 2009 10:38:46 PM
Subject: Re: [mmox] LLSD

anything that uses LLSD and wants to be considered a "standard" must only provide normative references that are also "standards".

essentially the deal here is we think the RFCs that will eventually provide the authoritative references for OGP should be put on the IETF "standards track." in fact, this is the primary raison d'être for the proposed working group. if OGP is to be a "real" IETF standard, all the documents it references as "normative" need to be standards as well.

the alternative was to simply continue publishing our interface documents on our wiki and perhaps publishing LLSD as an informational RFC. however... we thought it better to use our existing OGP work as a starting point, put it out in a forum where the expertise of the internet engineering community could be utilized to review, comment and improve it. our objective is to create a robust, flexible family of protocols usable by the widest practical community.

and for what it's worth, in the context of the proposed working group OGP/Teleport and HyperGrid are two proposals [1] for doing similar things. The two approaches _do_ seem assume different trust models, and maybe the future is that both will find wide adoption. the "define mechanism, not policy" mantra you sometimes hear some people repeat implies that neither approach should preclude implementations from implementing both. this is rather like most modern email user agents implementing both,POP3 _and_ IMAP. the two protocols do essentially the same thing, but there's nothing in either protocol that prevents you from using either or both in the same mail client.

but i ramble.

the question as i read it was "would trying to approve the draft of LLSD preclude or perturb further protocols?" the answer i would give would be "no. it shouldn't."

-cheers
-meadhbh

[1] - yes... actually neither are currently internet drafts, but i believe John is working on something HyperGridish and OGP/Teleport was listed on the draft charter of the proposed working group.

---

From: Jesrad <[email-masked]>
Date: Fri, 20 Feb 2009 16:41:47 +0100

Just to be sure: LLSD in the context of MMOX is meant to fulfill
roughly the same goal as ASN.1
(http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One) does in
general communications ?

On Fri, Feb 20, 2009 at 4:28 PM, Charles Krinke <[email-masked]> wrote:
> Perhaps  I am being naive here, but I dont see anything in the draft
> proposal other then the definition of a few variable types and three mime
> definitions which seems completely innocuous to me.
>
> Admittedly, follow on draft proposals will undoubtedly favor one teams bias
> at the expense of another teams bias, but I cannot find a reason not to
> support this variable definition and mime definition draft.
>
> There is lots of time for us all to argue that our own favorite <whatever>
> is the best for the world, but it seems that any of the thoughts I have
> heard can all be built with this variable name and type standard.
>
> Charles
>
> anything that uses LLSD and wants to be considered a "standard" must only
> provide normative references that are also "standards".
>
> essentially the deal here is we think the RFCs that will eventually provide
> the authoritative references for OGP should be put on the IETF "standards
> track." in fact, this is the primary raison d'être for the proposed working
> group. if OGP is to be a "real" IETF standard, all the documents it
> references as "normative" need to be standards as well.
>
> the alternative was to simply continue publishing our interface documents on
> our wiki and perhaps publishing LLSD as an informational RFC. however... we
> thought it better to use our existing OGP work as a starting point, put it
> out in a forum where the expertise of the internet engineering community
> could be utilized to review, comment and improve it. our objective is to
> create a robust, flexible family of protocols usable by the widest practical
> community.
>
> OGP/Teleport and HyperGrid are two proposals [1] for doing similar things.
> The two approaches _do_ seem assume different trust models, and maybe the
> future is that both will find wide adoption. the "define mechanism, not
> policy" mantra you sometimes hear some people repeat implies that neither
> approach should preclude implementations from implementing both. this is
> rather like most modern email user agents implementing both,POP3 _and_ IMAP.
> the two protocols do essentially the same thing, but there's nothing in
> either protocol that prevents you from using either or both in the same mail
> client.
>
> but i ramble.
>
> the question as i read it was "would trying to approve the draft of LLSD
> preclude or perturb further protocols?" the answer i would give would be
> "no. it shouldn't."
>
> -cheers
> -meadhbh
>
> [1] - yes... actually neither are currently internet drafts, but i believe
> John is working on something HyperGridish and OGP/Teleport was listed on the
> draft charter of the proposed working group.
>
>> preclude or pertubate either OGP, HyperGrid or other similar notions from
>> being supported?"
>>
>> Charles
>>
>> http://wiki.secondlife.com/wiki/MMOX is non-authoritative, but yes LLSD is
>> one of the work items we have on the list.
>>
>> the objective of LLSD is to provide an abstract type system with multiple
>> serializations. it also defines the mime types.
>>
>> serializations is akin to saying that ASN.1 exists only to provide input
>> into DER or BER encodings. the two are linked, but each has a particular
>> use.
>>
>> LLSD as an abstract type system allows us to define and reason about the
>> semantics of structured data used in PDUs independent of an existing
>> implementation language. the serialization rules allow us to format
>> structured data prior to transport and later de-serialize it after receipt.
>> the MIME type registrations allow us to identify the serialization scheme
>> used on transports that support the use of MIME types. LLIDL (pronounced
>> "little") defines the set of expected parameters to and responses from a
>> resource access.
>>
>> building block for other protocol interactions, some of which are published
>> in draft form on the second life wiki. the further development of these
>> protocols is also considered a task of the proposed working group.
>>
>> -cheers
>> -meadhbh
>>
>>> it looks like one of the first proposals for this working group is the LLSD
>>> draft specification.
>>>
>>> Am I correct in that the gist of the LLSD specification is essentially to
>>> propose three mime formats and thats about it for now?
>>>
>>> application/llsd+xml
>>> application/llsd+json
>>> application/llsd+binary
>>>
>>> Charles

---

From: Charles Krinke <[email-masked]>
Date: Fri, 20 Feb 2009 07:52:26 -0800 (PST)

Certainly, and that is an interesting issue in itself. 

My assumption is that the LL team picked the simplist place to start from in the spirit of gaining momentum with an IETF working group.

I can see lots of interesting and spirited discussions on the horizon over all the things that have been mentioned on this list in the last few days including :

1. Identification & Authorization ala OpenID and others.
2. The balance between server side and client side functionality.
3. Avatar definition consistent to render an avatar with and without scripted attachments in various virtual worlds.
4. Objects moving from one virtual world to another with various associated permissions.

But, none of this is described in the draft at all. As I look at it, it seems this is a reasonable way for this group to gain credibility with the ietf with a very basic draft and then this is to be followed up with more structure later. 

Whether or not the variable and mime definitions help innovation for virtual worlds and if they are, how to lay the foundation for the greatest flexibility certainly seems like a more interesting and relevant discussion then whether or not joe can drive is BatMobile through the FarGate between Croquet and Fortrerra and expect all the scripted weapons to be blazing.

Charles

________________________________
From: Jesrad <[email-masked]>
To: Charles Krinke <[email-masked]>; "[email-masked]" <[email-masked]>
Sent: Friday, February 20, 2009 7:41:47 AM
Subject: Re: [mmox] LLSD

---

From: Mark Lentczner <[email-masked]>
Date: Fri, 20 Feb 2009 09:07:38 -0800

> draft proposal on the Second Life wiki:
Gosh, I hope this won't be the *last* question! I'm really glad you  
are diving into this draft to this level of detail!

> agents: [ first_name: string, last_name: string ... ]

This was valid in an earlier, internal form of LLIDL. The intent was,  
as Meadhbh said, was to describe an array of values, grouped in twos.  
As you pointed out, the current valid way to do this is:

	agents: [ string, string, ... ]

The intent of the member-like names was so that the specification  
could name the members of the grouping, not that those members-like  
names appeared in the LLSD in any way. The confusion that ensued from  
that design seemed more than the utility it provided, so we removed it  
at a later stage. I think this discussion proves the point!

In any event, this kind of construction probably shouldn't be  
encouraged in APIs. Your alternative structure is far preferable from  

	- Mark

Mark Lentczner
Sr. Systems Architect
Technology Integration
Linden Lab

[email-masked]

Zero Linden

---

From: "Meadhbh Hamrick (Infinity)" <[email-masked]>
Date: Fri, 20 Feb 2009 11:11:29 -0800

hey jesrad...

conceptually ASN.1 and LLSD are similar, but LLSD is probably more  
appropriate for accessing RESTful accesses over a HTTP(S)-like  
transport.

specifically...

* LLSD was designed to be "less declarative" than ASN.1 . which is to  
say, all LLSD components are optional, so it is never a transport- 
layer error to not include a "required" parameter.
* LLSD (including LLIDL) is considerably smaller.
* the focus of LLSD is actually on the application layer, not on the  
transport syntax; but yes... we spend an awful lot of space in the  
draft talking about serializations, so this focus is sorta obscured  
there.
* LLSD is implementable by mortals in a tractable amount of time and  
without the use of complicated third-party tools (ASN.1 compilers, etc.)
* LLSD uses more consistent serializations (when compared with BER or  
DER)

so to recap ...

LLSD and ASN.1 are similar, but LLSD was designed to transport  
structured data that accepted extra parameters (or missing parameters)  
in a PDU without the need to recompile and deploy new PDU parsers.

-cheers
-meadhbh

On Feb 20, 2009, at 7:41 AM, Jesrad wrote:

>> provide
>> documents on
>> things.
>> LLSD
>> believe
>>> LLSD is
>>> multiple
>>> input
>>> published
>>>> essentially to

---

From: Jon Watte <[email-masked]>
Date: Fri, 20 Feb 2009 12:21:43 -0800

> without the use of complicated third-party tools (ASN.1 compilers, etc.)

I don't see the LLSD interface compiler as any different in conceptual 
complexity than an ASN.1 compiler. I did bring up the question of ASN.1 
before, and was told by you that you felt it was "too buggy," which I 
think is not fair or accurate.

However, I believe ASN.1 and also LLSD have a problem that I'd like to 
see addressed in a standard: they use implicit schema.

If I want to send a piece of data from point A to point B, and the 
points don't both exactly know what the data is, that's fine -- but, in 
that case, you typically want to use something a little more open-ended 
and more generally standardized, like XML. The benefit of XML is that 
there are tons of tools that already support it (as there are with 
ASN.1, btw); XML also allows well-formed documents to be verified using 
XSD schema. If you need large binary blob support (textures, etc), those 
can be included by reference uri. I think XML is appropriate for those 
cases, and LLSD doesn't quite fit the bill in that case.

If I want to send a piece of data from point A to point B, and point A 
and point B know exactly what this data will be, then data type 
information does not need to be sent as part of the serialization. In 
fact, including data type information is redundant and consumes network 
bandwidth unnecessarily.

Thus, I would assume that any communication where size matters (such as 
for entity property update streams) will be done using negotiated 
(bilaterally understood) schema, with the following properties:
1) Only data elements, not names or types, are sent.
2) Specfic properties are addressed using short Ids that are translated 
in schema.
3) If certain properties are always sent together (say, position x y z 
and orientation x y z w) then schema allows for an "aggregate property" 
to be declared, such that only a single name (Id) is needed to identify 
that entire tuple.

I have written an illustrative proposal up for this that may help 
clarify the issue, at the following URLs:
http://www.interopworld.com/node/29
http://www.interopworld.com/node/28

Sincerely,

jw

---

From: Morgaine <[email-masked]>
Date: Sat, 21 Feb 2009 08:39:14 +0000

On Fri, Feb 20, 2009 at 8:21 PM, Jon Watte <[email-masked]> wrote:

>
> If I want to send a piece of data from point A to point B, and point A and
> point B know exactly what this data will be, then data type information does
> not need to be sent as part of the serialization. In fact, including data
> type information is redundant and consumes network bandwidth unnecessarily.
>
> Thus, I would assume that any communication where size matters (such as for
> entity property update streams) will be done using negotiated (bilaterally
> understood) schema, with the following properties:
> 2) Specfic properties are addressed using short Ids that are translated in
> schema.
> 3) If certain properties are always sent together (say, position x y z and
> orientation x y z w) then schema allows for an "aggregate property" to be
> declared, such that only a single name (Id) is needed to identify that
> entire tuple.
>
>
That is quite close to being a description of Google Protocol Buffers.

One of the reasons why I find LLSD to be "tentatively satisfactory" as a *
base* from which to create an ADT layer suitable for virtual worlds interop
is that LLSD embraces multiple alternative serializations, and to its 3
current serializations I would be adding Protocol Buffers for the very good
properties that you described.

Several other changes would be needed too, in particular the addition of new
data types to remove the conversion ambiguities that others have mentioned,
as well as to support more efficient encoding of common VW elements beyond
LL's native requirements.  Despite the need for all these changes, I do find
LLSD to be a useful *base* from which to start, largely because nobody has
suggested an alternative *that ticks all the same boxes*.

Given an alternative *that ticks all the same boxes* (this is important)
then I would of course re-evaluate the matter.

Morgaine.

---

From: "Meadhbh Hamrick (Infinity)" <[email-masked]>
Date: Sat, 21 Feb 2009 01:18:06 -0800

On Feb 20, 2009, at 12:21 PM, Jon Watte wrote:

>> * LLSD is implementable by mortals in a tractable amount of time  
>> and without the use of complicated third-party tools (ASN.1  
>> compilers, etc.)
>
> I don't see the LLSD interface compiler as any different in  
> conceptual complexity than an ASN.1 compiler. I did bring up the  
> question of ASN.1 before, and was told by you that you felt it was  
> "too buggy," which I think is not fair or accurate.

which LLSD compiler are you referring to? i wasn't aware that one was  
required or available.

> However, I believe ASN.1 and also LLSD have a problem that I'd like  
> to see addressed in a standard: they use implicit schema.
>
> points don't both exactly know what the data is, that's fine -- but,  
> in that case, you typically want to use something a little more open- 
> ended and more generally standardized, like XML. The benefit of XML  
> is that there are tons of tools that already support it (as there  
> are with ASN.1, btw); XML also allows well-formed documents to be  
> verified using XSD schema. If you need large binary blob support  
> (textures, etc), those can be included by reference uri. I think XML  
> is appropriate for those cases, and LLSD doesn't quite fit the bill  
> in that case.

Jon. i think you mis-understand LLSD. there is nothing in the LLSD  
specification that REQUIREs a schema be defined. In the current  
implementations of LLSD, it is not considered a transport-layer error  
to  add an additional field, or even to drop one that's defined by an  
application protocol. In this sense XML is more restrictive because it  
REQUIRES you to create a new DTD to extend an existing message flow  
(though presumably you would configure your DTDs or Schemas to be  
easily extensible.)

for instance.. if a parameter was defined as:

    &response = {
      success     : false ,
      description : string ,
      err_num     : integer
    }

    &response = {
      success     : true,
      description : string
    }
    %% foo <- undef -> & response
and you received an XML serialization that looked like:

<llsd>
   <map>
     <key>success</key>
     <boolean>true</boolean>
     <key>something_i_like_to_eat_on_sundays</key>
     <string>bananas</string>
   </map>
</llsd>

this i not an error. an implementation that attempted to get the value  
of "description" field from the map would retrieve the default for  
type string (i.e. - a zero length string.) that the  
'something_i_like_to_eat_on_sundays' information is passed may be a  
waste of bandwidth, but it might also be a new message format unknown  
by the consumer. a system that ignores unexpected input is a system  
that has a fighting chance of not having to upgraded in tandem with  
the client.

> If I want to send a piece of data from point A to point B, and point  
> A and point B know exactly what this data will be, then data type  
> information does not need to be sent as part of the serialization.  
> In fact, including data type information is redundant and consumes  
> network bandwidth unnecessarily.

however, sending type information in the serialization allows  
intermediary systems to reason about the content of the data they are  
receiving and potentially make routing and caching decisions based on  
the content.

LLSD is intended to be used in a "loosely coupled" environment. one  
"feature" of such an environment is that systems be resilient in the  
face of heterogeneity in terms of component version. for example,  
consider a logging system that received logging messages from a number  
of clients and stores them for later analysis. when the system is  
initially deployed, messages may be defined as the tuple (source,  
severity, description). it is not difficult to imagine the situation  
where one client wishes to add a value to that tuple to make it  
(source, facility, severity, description). in tightly bound systems,  
you might create a new message PDU format, but both the client and the  
server would need to know about this change at exactly the same time  
(otherwise the server would signal an error when it received an  
unknown message format. now imagine the mess if half of the systems  
were in one administrative domain and the rest in another.

---

From: Jon Watte <[email-masked]>
Date: Sat, 21 Feb 2009 10:08:14 -0800

>     <key>something_i_like_to_eat_on_sundays</key>

IMO, that's a terrible serialization, because it depends on ordering. 
While XML is ordered, this particular serialization makes it hard to 
write XPath that selects "the string that comes after key success but 
before key something_i_like_to_eat_on_sundays".

>> fact, including data type information is redundant and consumes 

I believe that, for the high-volume real-time binary stream, the 
bandwidth requirements outweigh the "caching" requirements -- nobody's 
going to be caching the real-time data anyway.

> where one

It turns out we have a logging system in OLIVE, and it stores the actual 
framed data without needing to know the internals at all. The clients 
need to know and understand the internals, but they can do that because 
they have the schema. Now, in OLIVE, the schema is described largely in 
code, not dynamically, as proposed in my proposal, but I think a dynamic 
schema is actually the right thing to do for interoperability -- it, 
too, makes it possible to update the protocol without recompiling 
anything. And it doesn't require sending "here comes a float" for each 
component of each position vector of each object in each packet sent...

I guess I don't see the applicability of LLSD. It's too little to be 
useful for a "verbose" format (like XML documents), and it's too much 
overhead for the real-time stuff. By contrast, XDR uses external schema, 
which is good, but the existing schema tools are not dynamic, which is 
less good. If LLSD can be adapted in one of two directions (or both!) 
then I'd be much less resistive. Either make it support external schema 
encodings in a binary efficient way (see 
http://www.interopworld.com/node/29) and use it for real-time stuff, or 
make it support proper, Xpath compatible XML and use it for the XML-RPC 
stuff.

Sincerely,

jw

---

From: Lisa Dusseault <[email-masked]>
Date: Mon, 23 Feb 2009 16:49:53 -0800

Finally, a topic on which I have more than passing technical familiarity:

On Sat, Feb 21, 2009 at 1:18 AM, Meadhbh Hamrick (Infinity) <
[email-masked]> wrote:

>  In this sense XML is more restrictive because it REQUIRES you to create a
> new DTD to extend an existing message flow (though presumably you would
> configure your DTDs or Schemas to be easily extensible.)

The way XML is used in IETF protocols certainly does not require anybody to
create a new DTD to extend an existing message flow.  E.g. in WebDAV XML
request bodies, a client may insert additional elements at certain places
(other places have defined meanings) and if the server doesn't recognize
them it just ignores them.  Same with Jabber.

> "feature" of such an environment is that systems be resilient in the face of
> heterogeneity in terms of component version. for example, consider a logging
> system that received logging messages from a number of clients and stores
> them for later analysis. when the system is initially deployed, messages may
> be defined as the tuple (source, severity, description). it is not difficult
> to imagine the situation where one client wishes to add a value to that
> tuple to make it (source, facility, severity, description). in tightly bound
> systems, you might create a new message PDU format, but both the client and
> the server would need to know about this change at exactly the same time
> (otherwise the server would signal an error when it received an unknown
> message format. now imagine the mess if half of the systems were in one
> administrative domain and the rest in another.
>
> Again, this is very much the way IETF uses XML, so I don't see this as a
distinguishing point between XML and LLSD.

Also, related to this thread, the IETF rarely does binary protocols in the
layers I've been involved in.  Clear text, perhaps with compression at a
lower layer, makes a protocol much easier to debug, maintain, test and make
interoperate.

Lisa

---

From: Jon Watte <[email-masked]>
Date: Mon, 23 Feb 2009 17:22:57 -0800

Lisa Dusseault wrote:
> Also, related to this thread, the IETF rarely does binary protocols in 
> the layers I've been involved in.  Clear text, perhaps with 
> compression at a lower layer, makes a protocol much easier to debug, 
> maintain, test and make interoperate.

I agree, for services and transactional stuff, text protocols are much 
better.

Real-time entity telemetry data probably needs to be binary, for the 
same reason that streaming video, or VoIP data is binary.

Sincerely,

jw