Subject: [vwrap] thinking about LLIDL and the future

From: Meadhbh Hamrick <[email-masked]>
Date: Thu, 15 Jul 2010 14:01:53 -0700

hey peeps,

i've been pondering LLIDL for a bit now, and there are some things
about it that really stick in my craw. i put together a blog post
about it, but wanted to replicate it here on the list.

from http://blog.meadhbh.org/2010/07/abstract-resource-definitions-vs-llidl.html

so i've been noodling on LLIDL and LLSD for a couple years now, and i
think i've figured something out. i like the concept of an "abstract
resource definition" a lot more than i like an "interface description
language." (if you have no idea what i'm talking about, you may want
to read my previous blog post "VWRAP Essentials : an abstract type
system? hunh?." advanced students can take a look at the internet
draft: draft-ieft-vwrap-type-system-00.)

the background

the objective of the abstract type system is simple and well defined:
it provides system independent type semantics for structured data
developed by protocol designers. that is, it defines types that are
common to popular implementation languages. and it does it in a way
that developers know the details of the types. like whether integers
are 32 or 64 bits and whether radix 10 is supported in floats and so
forth. the LLSD abstract type system describes how a small set of
fundamental types work so implementers know how their "concretized"
implementations should behave. and this is a good thing.

the problem

but after having mulled LLIDL around, i'm starting to see some issues
with it. the main problem i have with it is that the interface is
hopelessly entangled with the resource it describes. that is, you
can't define an abstract resource without defining a method of access.
for example, the current spec describes a way to say something like
this:

&info = {
  name : string,
  status_uri : uri
  enroll_uri : uri
  login_uri : uri
}

%% site_info << &info

in this example, the &info named type describes a structure that
includes a string and three URIs. but the only context you can use the
&info named type in is when you use a HTTP GET to access it. i think
this is wrong.

it's wrong because it links the declaration of the structure of a
resource with the transport used to access it. i have no problem with
accessing resources over HTTP(S), but i do have a problem with ONLY
accessing them over HTTP(S).

i think there's sufficient interest in the VWRAP working group to
support the concept of making resource definition and access
orthogonal. i would personally want to be able to have the option of
using XMPP and (S)RTP when characteristics of those protocols are
advantageous. but i don't want to have to create a new data definition
language every time i want to go access a resource over a new
transport.

furthermore, there are still open questions regarding content
negotiation over HTTP(S). suzy deffeyes, john hurliman and i have
reached a rough consensus on how to handle content negotiation, but we
need to put it in an internet draft so other people can comment on it
and implement it without having to refer to random email archives and
IRC logs.

a proposed solution

i propose we layer an abstract resource definition on top of the
abstract type system. the abstract resource definition can then be
used to describe a transfer syntax (aka serialization scheme) for
resources. we can then describe how each transport (HTTP(S), XMPP,
etc.) carries serialized resource data. we can then describe
interaction semantics independent of resource definition.

the benefit of this is it makes it easier to create systems that are
transport-neutral. some of the lessons we learned from Second Life™
and the Linden Lab Legacy Protocol are that there's a LOT of stuff
that's not "real time" data. I suspect that moving forward, we'll
discover there are a lot of interactions that are modeled with the
request-response pattern. i am proposing a solution that would offer
deployers and protocol designers the ability to describe a resource
without describing the method in which it is accessed.

more details

this proposal describes five components: the abstract type system, the
abstract resource definition, the transfer syntax, the abstract
interaction definition and the transfer mapping.

abstract type system

as mentioned above, the abstract type system defines a core set of
types and type behavior. types are intended to be "concretized" in
programming languages like C, Java, Python, whatever. we create an
"abstract" type system so we're free to define the type behavior we
want while not being tied to type behavior of a particular
implementation language. individual implementers are responsible for
ensuring that type semantics are consistent with the spec.

for example, imagine you were developing an application using an
embedded 8 bit microprocessor and there was a protocol flow that
involved incrementing an integer. a client might send you a 32 bit
integer, assuming you would increment it. you would be responsible for
modeling the behavior of a 32 bit integer despite the fact your
micro-controller probably only natively knows about 16 bit ints.

abstract resource definition

the abstract resource definition is sort of like the abstract type
system for collections. we define a resource as being either an
individual item (a boolean, integer, etc.) or we define it as being a
map or an array. the abstract resource definition system allows
protocol developers to define the "shape" of resources. but because
they are dynamic, implementations must allow for the type of a
resource to differ from it's definition and for elements in map to be
added and removed.

for example, you may want to define the following resource to describe
an user logged into a system:

  RESOURCE agent {
   name string,
   identifier uri,
   current_location uri
 }

the abstract resource definition is used to group named data elements
together into a dynamic structure. these structures will later be used
by the abstract interaction definition.

transfer syntax (aka serialization scheme)

these are basically the three serialization schemes defined in the
type-system draft: XML, JSON and Binary. they define how structured
data is serialized into an octet string in preparation for
transmission across a network. for example, the resource defined above
might be serialized like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE map PUBLIC "+//IDN meadhbh.org//xxdsd 0.1//EN"
"http://home.meadhbh.org/xml/xxdsd/0.1/xxdsd.dtd">
<map>
 <key>name</key>
 <string>Meadhbh Oh</string>
 <key>identifier</key>
 <uri>http://world.secondlife.com/resident/6e7477a7-2de5-4660-bc83-4a47096a18f0</uri>
 <key>current_location</key>
 <uri>http://world.secondlife.com/place/65f385f0-691c-5755-9ef0-15f3bb05c8b7</uri>
</map>

abstract interaction definition

LLIDL currently defines access verb sets that are tied to HTTP verbs.
if we want to be able to access resources using multiple transports,
this is wrong. i propose we define an abstract interaction description
"language" that is later mapped to a particular transport.

i propose we have five "abstract" verbs: CREATE, READ, UPDATE, DELETE
and EVENT. the first four should be pretty straight forward. They're
used to create resources, read and write them and then maybe delete
them. the last verb, EVENT, is used to define a resource an entity may
receive without requesting it. things like object updates, incoming
message session requests, etc.

transport mapping

the transport mapping maps abstract verbs from the abstract
interaction description language to concretized protocol flows in the
transport. so, for instance, you may use this to say that CREATE
abstract access methods map to a POST verb in HTTP or that an EVENT
message is delivered via the HTTP event queue.

conclusion

i'm going to wrap as much of this as i can into an internet draft and
create some example code. but i figured i would get this idea out
there so people can comment on it. just lemme know what you think...

--
meadhbh hamrick * it's pronounced "maeve"
@OhMeadhbh * http://meadhbh.org/ * [email-masked]

---

From: Morgaine <[email-masked]>
Date: Fri, 16 Jul 2010 11:07:40 +0100

I agree with the main thrust of Meadhbh's proposal.

Defining the "Method Access Delimeters" of LLIDL resources in terms of HTTP
operations directly was not generic nor forward looking.  The type system
draft uses highly HTTP-centric language in the LLIDL section, for example: "
*A LLIDL "Resource" represents information or state maintained by a remote
system, accessed via HTTP(S).*"  Clearly this is wrong, if our intention is
to be transport-agnostic.

Likewise, defining the "Method Access Delimeters" of LLIDL in terms of HTTP
operations was quite wrong, not to mention that it gave us rather ugly
syntax too ("<>" for GET/PUT, "<x>" for GET/PUT/DELETE, and "->" and "<-"
for POST).  If we're putting LLIDL resources on a sounder,
transport-agnostic footing, this is a good opportunity for cleaning up the
syntax while we're at it, and making it more flexible.

I support the suggestion that we define the 4 CRUD "abstract verbs" CREATE,
READ, UPDATE and DELETE as terminals in the LLIDL grammar.  This is highly
readable, and it is also semantically clean because CRUD operations are
disjoint.  Note that in client-server protocols, each CRUD operation is
usually implemented as two messages, a request message and a response
message --- this is relevant below.

The proposed 5th verb "EVENT" is more problematic, and I think it's actually
incorrect, as it seems to have confused the concepts of "verb" and
"message".  The reality is that messages received outside of a 2-part
synchronous request-response pair usually carry some kind of type
information that activates an operation handler at the receiving end.  Such
operation is enabled by a prior CRUD verb that sets up the session to allow
asynchronous messages to be sent and received subsequently.  In other words,
EVENT isn't a verb or operation at all, it's an asynchronous response
message to a prior verb.

This is the situation as I see it:  VW protocols are almost always
message-based, and they are almost always request-response in nature but the
"response" can comprise an unlimited sequence of response messages, zero or
more.  The reason I see it this way is that a truly unsolicited message is
actually spam, or worse, an attack packet.  All other messages (including
asynchronous ones) *were* solicited by some previous request which enabled
the sending of asynchronous messages in the first place.

This is why I don't see "EVENT" as a resource operation.  It's a deferred
response message that was emitted after being enabled by some previous CRUD
request, and it carries data in response to one of the 4 CRUD operations
within it, either implicitly or explicitly.

As an example, a client should not receive scenegraph update messages unless
it has first requested them using a CRUD verb on the appropriate region
resource(s).  Once it has requested them, they become valid and are
processed by the client scenegraph handlers, but if it has not requested
them then they are invalid and should never be sent nor received.  Indeed,
if they *are* received without being requested then they should be processed
by the error handler.  In a text-only client for example, there probably
wouldn't even be any scenegraph handlers present in the code.

"EVENT" then is not itself a verb that acts on resources.  It's an
asynchronous response message, part of a sequence of zero or more such
messages, and this potentially unlimited sequence of messages is started by
one CRUD operation and stopped by another CRUD operation.

Notice that the EVENT message can also be a container for CRUD request
messages in the reverse direction, but even such container messages obey the
above rule that they are started by one CRUD verb and stopped by another.

In summary, EVENT is not a resource verb  but an *asynchronous transport
property* of a resource, and if that property is enabled by a CRUD
operation, then EVENT messages are emitted under control of that resource.

Apart from EVENT, I think that this is a great proposal.  Regarding EVENT, I
suggest that we analyze this area further and find a clean way of
integrating EVENTS into our resources-based scheme.  I'm very interested in
this topic.

Morgaine.

=================================

On Thu, Jul 15, 2010 at 10:01 PM, Meadhbh Hamrick <[email-masked]>wrote:

> from
> http://blog.meadhbh.org/2010/07/abstract-resource-definitions-vs-llidl.html
>
>  <uri>
> http://world.secondlife.com/resident/6e7477a7-2de5-4660-bc83-4a47096a18f0
> </uri>
>  <uri>
> http://world.secondlife.com/place/65f385f0-691c-5755-9ef0-15f3bb05c8b7
> </uri>
> </map>
>
> _______________________________________________
> vwrap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/vwrap
>

---

From: Meadhbh Hamrick <[email-masked]>
Date: Fri, 16 Jul 2010 09:09:24 -0700

> EVENT is not a resource verb  but an asynchronous transport
> property of a resource, and if that property is enabled by a CRUD operation,
> then EVENT messages are emitted under control of that resource.
>

hmmm.... okay... point taken. true enough, EVENT describes not a verb,
but a message type.

my motivation here is to identify unsolicited messages. i think it's
unrealistic to assume that EVERY message will be properly modeled as a
request / response. consider the situation when someone wants to open
a chat session with you? is it better to model that interaction as an
unsolicited message (like you receive an event saying: "hey. someone
wants to open a chat session with you") or as a response to the
request (like "this is my request that you alert me on my event queue
when someone wants to chat with me.")

so yeah, i agree that EVENT is not a verb in the way CRUD verbs are,
but i don't agree that EVERY interaction in a virtual world is a
synchronous request / response, which is why the EVENT was added.

we could, however, say that the model implies a reverse client-server
relationship over an event queue and in that case, things that were
events would wind up being resources the server accesses using normal
CRUD message types.

it still doesn't help us if we want to model an object update with the
abstract resource definition or interaction definition scheme.

i think we're dealing not only with whether a message is synchronous,
but also dealing with whether the message has been solicited. i think
REQUIRING interactions to be both synchronous and solicited is a big
mistake, cause it means we can't use the system to model object
updates.

-cheers
-meadhbh

---

From: Morgaine <[email-masked]>
Date: Fri, 16 Jul 2010 19:04:44 +0100

Considering asynchronous messages to be emitted in response to a CRUD
operation on an appropriate resource fits in perfectly with the notion of
Capabilities though.  They ARE solicited, by a request to a resource.

As in my earlier example of the text-only client, the CRUD operation
UPDATE(scenegraph_updates_capability, ENABLE) was not invoked by the client
(or it was invoked with the argument DISABLE), and hence that resource saves
time and bandwidth by not sending the unwanted scenegraph updates.  If
instead the resource were set to ENABLE, those update messages would
certainly be appearing in response to the ENABLE request, so it's a natural
interpretation.  And of course it's very helpful to have such events
controlled by a capability.

The alternative model in which events are considered normal CRUD operations
on an event queue in the reverse direction is actually incorrect, because
then those operations would require responses themselves, and that's not
what we want.  Events don't fit the synchronous request-response model,
because they're not synchronous, and are ill-suited to a 2-way
request-response paradigm for every event message.

That's why I prefer the unidirectional model in which events occur as a
sequence of zero or more messages in response to a normal CRUD operation.
It's a clean model because the responses travel in the reverse direction
perfectly naturally (since they're responses) and do not themselves require
further ACKs.  It's also justified because, after all, we will require a
means of enabling / disabling updates and other asynchronous event
notifications.  This model kills two birds with one stone, and it's a
natural fit for capabilities.

Morgaine.

==================================

---

From: Dzonatas Sol <[email-masked]>
Date: Fri, 16 Jul 2010 11:21:03 -0700

Meadhbh Hamrick wrote:
> my motivation here is to identify unsolicited messages.

Good idea since that is the basic message until they are identified as 
solicited.

One step away from the transport protocol, there are three main flows 
noticed in ReST design: simple queries, complex queries, and task(lets).

Of those three, I said flow because of the direction of data expected: 
unidirectional, bidirection, and only internal.

The GET method is comparable to a simple query. The POST method is 
comparable to a complex query, especially if it expect further 
processing after HTTP_OK|HTTP_ACCEPTED status.

Tasks are pretty much the core type that are abstracted into these other 
types described above and in related thread.

Yes, being able to describe or abstract from these basics I think is 
more desirable than than worry about PUT, DELETE, etc. Those extra 
details that are complex and don't let us easily bridge the digital gap, 
one needed to cross in order to explain to users how to basically extend 
VWRAP.

---

From: Meadhbh Hamrick <[email-masked]>
Date: Fri, 16 Jul 2010 11:45:18 -0700

Interesting. I had been thinking that the message semantics would require a
response on the same channel as the request. What you're talking about seems
to be a little higher level than what I'm pitching. (Thx for pointing out I
wasn't clear about this)

So.. the way I'm modeling it is that you have message interactions that are
tied to the same transport session. So if you ask for something with
http(s), you get it back via http(s) on the same connection.

But it's definitely true that an application layer entity can coordinate
messages, only I think we should make a "virtual" connection that spans
actual connections.

But I still think it's advantageous to think of interaction patterns in
terms of message flows over specific connections, otherwise we have to start
adding application layer transaction id's to correlate responses to
requests. Also, since trust seems to be associated with connections (by way
of (D)TLS), it's probably a good idea to not cross trust boundaries in
interactions across connections, otherwise we would have to model that in
the application layer as well.

At the end of the day, we have message patterns that look like:

* send, no reply expected
* request, response expected
and
* unsolicited message.

The first two can be modeled with request/response, but the last one has
unique characteristics that must be modeled.

-cheers
-meadhbh

On Jul 16, 2010 11:05 AM, "Morgaine" <[email-masked]> wrote:

Morgaine.

On Fri, Jul 16, 2010 at 5:09 PM, Meadhbh Hamrick <[email-masked]>
wrote:
> > EVENT is not a...

---

From: Morgaine <[email-masked]>
Date: Sat, 17 Jul 2010 06:41:09 +0100

That's correct, I was describing flows at the capability level, not at the
transport level.  The transport is exactly as you deduced, namely with
separate transports for capability request-response and for asynchronous
event messaging.

When you send an ENABLE to the capability that controls a particular
asynchronous event stream, you receive back the normal capability OK
response on the same channel as the request (as expected), but the event
messaging that this enables occurs on a distinct channel at the transport
level.

This makes a lot of sense for practical implementations, because event
messaging requires high bandwidth and low latency and so it is best handled
on a separate transport and using the most efficient form of serialization.

As an implementation example, the OK response from the request to the
capability could return the IP address and port of the event messaging
source host, the client would open a TCP stream to it, the source host would
send events asynchronously down the pipe, and the client's event listener
would dispatch the event frames to appropriate event handlers as the
messages arrive.

Different event-generating capabilities could choose to return different
IP:port pairs to harness dynamic load-balanced server pools and to provide
parallel pathing, or they could choose to share a common event transport ---
these would be simple site configuration choices.

Note that this is highly scalable, since it allows any number of servers to
participate in event generation and distribution.  No longer do updates have
to come from a single, horribly overloaded and totally non-scalable
simulator host.

Consistent with David's regular exhortations, nothing in the above precludes
this model from working in the legacy deployment architectures of SL and
Opensim:  single simulator operation is still supported by the design, but
it offers additional capability for more scalable deployments.

Morgaine.

=====================================

---

From: Meadhbh Hamrick <[email-masked]>
Date: Fri, 16 Jul 2010 23:27:12 -0700

okay. i think i understand you and dzonatas well enough to go out and
write another, completely unofficial, possibly experimental draft for
the group. then maybe a little bit of PHP and JavaScript to
demonstrate a few of the features.

again, thanks for the comments (and this is by no means the end of the
commentary on my extended type system.) i think it's just easier for
me to have a conversation about it after having real documents and a
sample implementation to look at.

i'm going to probably drop out of the conversation for a week as i go
off and work on this stuff. i'll keep a lookout for additional
comments, and try to address them in my next draft. but i probably
won't have bandwidth to discuss them for a week or so.

-cheers
-meadhbh

On Fri, Jul 16, 2010 at 10:41 PM, Morgaine
<[email-masked]> wrote:
>> Interesting. I had been thinking that the message semantics would require
>> a response on the same channel as the request. What you're talking about
>> seems to be a little higher level than what I'm pitching. (Thx for pointing
>> out I wasn't clear about this)
>>
>> So.. the way I'm modeling it is that you have message interactions that
>> are tied to the same transport session. So if you ask for something with
>> The alternative model in which events are considered normal CRUD
>> operations on an event queue in the reverse direction is actually incorrect,
>> because then those operations would require responses themselves, and that's
>> not what we want.  Events don't fit the synchronous request-response model,

---

From: Morgaine <[email-masked]>
Date: Sat, 17 Jul 2010 08:34:51 +0100

Righto Meadhbh, I'm looking forward to that.

It's worth noting that this design pattern works for many services and
subsystems in our kind of VW architecture.  Wherever a capability enables a
service that operates over a separate transport, the capability grant can
supply dynamic IP:port pairs for the transport and hence quite trivially
harness load balancing for the service in question.

We can certainly apply this not only to object updates but also to group IM,
world notifications, and to search.  Any region-specific assets such as
terrain also fit the model well for load-balanced streaming, and even baked
avatars served from load-balanced pools fit the model.

It's a generic design that is much more flexible and controllable than
access dispersal to dynamic pools through DNS or through load balancers, and
it provides good security as well, and no single point of failure.

Morgaine.

================================

---

From: Dzonatas Sol <[email-masked]>
Date: Sun, 18 Jul 2010 07:01:02 -0700

> demonstrate a few of the features.

A little more for us to think about... from actual usage...

The simple queries that Icesphere puts together are something like: GET 
http://host:port/abc/def/uuid. If it finds many of these simple queries 
found in the queue, then it combines them all into a single complex 
query: POST http://host:port/abd/def/s. Technical detail on transport, 
the single letter "s" is a false UUID to denote a list of UUIDs in the 
request.

Due to such automatic query transition above, you can probably guess at 
reasons why Icesphere avoids the event queue resource, as described in 
VWRAP. I look at VWRAPs event queue as another possible transition and 
technical detail on transport.

When I mentioned bidirectional earlier, I actually meant fully 
bidirectional and not just simply request/response. For example, in 
Icesphere, the client sends a query to load an asset for an inventory 
item, which returns HTTP_OK|HTTP_ACCEPTED status. The resource is 
released after that status. When the server has the asset, it send a 
query that the asset is loaded to the client, as a separate resource. 
That is a capability that is obviously not active for those that don't 
support full bidirectional queries due to possible firewall that only 
allow unidirectional queries.

I think I rather go with QUERY rather then request/response mainly to 
make direction more significant and to make the request/response a 
technical detail of the transport.

I started to type lots of detail here, yet my coffee pot is empty.

To expand you usage of EVENT into a QUERY we just need to know the 
pivotal data for transitions, like UUIDs.