Subject: [mmox] Unifying the LESS and SL object execution models

From: Morgaine <[email-masked]>
Date: Thu, 5 Mar 2009 08:57:38 +0000

*(Note that I use "LESS" generically in this post to refer to worlds of the
"shared space" model.)*

There has been no shortage of discussion about the *differences* between the
architectural models of the LESS proposal and of Second Life, and of the
impact of these differences on protocol requirements.  This has been useful,
but by itself it is not sufficient to help us arrive at any kind of
consensus about this part of the problem space, let alone the solution
space.  For that, we need to identify common ground or a common approach.

I've been looking at one of the most-often highlighted differences, namely
the *object execution model*.   This appears to be one of only 2 or 3
significant roadblocks that hold the two world models apart, so finding
either commonality or some useful symmetry could conceivably help with
interop.

*Two object execution models: inverse problems*

Jon has accurately described the merits of object execution on an object's
"home" system in LESS-type systems.  This approach benefits from proximity
of the computation to the object's local environment, and the benefits are
huge.  Not only can local objects employ all of the facilities of the local
environment, but local processing can be orders of magnitude faster than
remote execution, less starved of resources, graphically intensive and
behaviourally complex, and highly interactive with the user.  The package of
benefits is simply incontestable, and hence this is a very important
requirement.

On the other hand, object execution on an object's "home" system implies
great distance from the environmental simulation performed in a remote
shared space, and this has severe consequences.  Event response latency can
be orders of magnitude higher through remote telemetry than when the event
source and object execution are colocated.  Physics simulations are
compromised in resolution by response scatter, and compromised in speed,
complexity and event concurrency by communication bandwidth.  All of these
combine to severely limit remote environmental complexity and accuracy of
simulation, compared to remote object execution.  This is a very high price
to pay.

I know that Jon is at this point shouting "Local simulations!" with all his
might (:-), but that is no panacea.  Local simulation of a remote
environment suffers from exactly the same problems as no local simulation at
all, for a variety of reasons.  Most importantly, running simulations in
lockstep generally only works for deterministic simulations, and those are
usually not the kind we encounter in human-populated virtual worlds.  Since
the simulations are not running in lockstep, there can only be one source
for each event, either local or remote, and from this observation it is
clear that all the problems of the preceding paragraph apply here as well,
since a local simulation cannot predict events occurring in a remote
simulation.  In other words, the local simulation provides no advantage for
events arriving from the remote simulation versus a system without local
simulation, and that's why all the problems mentioned above apply.  What's
more, local simulation begins to collapse in utility as world simulations
become diverse anyway, so it is not a general solution.

I'm sure that some of the above will be disputed (correctly), but my general
point applies:  the LESS model has both advantages and disadvantages, as do
all systems.

The object execution model of Second Life is the reverse of the above,
namely object execution in the remote region, and it has the almost exactly
the same advantages and disadvantages but in reverse.  The region simulator
has a high-bandwidth and low-latency relationship with the objects it hosts
and hence provides excellent simulation performance, but objects have almost
nil relationship with the user's local environment so user integration is
almost non-existent and all the great benefits of locality are missing.  In
the OGP scheme which potentially allows Region Domains to be local, one
could obtain the high-bandwidth benefits of locality for oneself but only by
making all other participants in the region lose their locality, so it is no
solution either.

*Solution: split the behaviours and unify the models*

The key thing to observe if one wants a solution is that almost all of the
benefits of local execution are related to the *internal* behaviour of the
object (its implementation), whereas almost all of the benefits of remote
execution are related to the *external* behaviour of the object within and
reacting to its remote simulation environment.  This makes a solution easy
to identify:

   - Define objects to have separate internal and external behaviours, both
   optional.
   - External behaviour works exactly as in SL in response to environmental
   events.
   - Internal behaviour works exactly as in LESS in respect of local object
   execution.
   - If an object has both internal and external behaviours, then the two
   are connected by a telemetry event stream for remote/local behaviour
   integration.
   - Both LESS and SL systems can then gain the advantages of the other, if
   they wish.
   - Neither LESS nor SL need suffer their own respective disadvantages,
   it's their choice.
   - If LESS chooses not to implement remote object behaviour, it requires
   no system change.
   - If SL chooses not to implement local object behaviour, it requires no
   system change.
   - The object interop protocol would however need to support both, whether
   used or not.
   - Remote scripting in SL would be unchanged, except for the addition of
   local peer calls.
   - Local scripting in LESS would be unchanged, except for the addition of
   remote peer calls.
   - Since the object execution model of both is supported, one barrier to
   interop has fallen.

*Extremely simple example:*
Instead of writing very slowly and with difficulty in-world on the side of a
3D whiteboard in SL, use a local 2D canvas in the client as the local side
of the remote whiteboard object.  This could provide all the amenities of a
local drawing application, automatically perform the perspective mapping
onto the side of the whiteboard, and still do this accurately even while the
whiteboard object is being thrown around by physical events in the remote
simulation space.

I wrote this primarily to help analyse the problem space in the area of LESS
and SL object execution models, since this is currently a hard problem for
us.  The suggested solution is a byproduct of that.

To round off, I'm aware that such a solution could be problematic for
parties who may be constrained from extending their existing
implementations, since it would involve substantial work.  However, I felt
that it was worth describing this solution because of the immense gains it
can offer to both models, and because it is a solution to one specific
interop problem that we have identified and which has become an impasse.  In
addition, other MMOX parties might be able to run with it.

It is worth noting that many MMOX participants are involved in open source
VW projects, and typically in these some extra design and implementation
work is not a barrier.  Opensim or others might well find it feasible to
implement such a 2-part execution scheme, reap the benefits of this enhanced
object capability, and maybe take a lead towards interop with LESS-like
systems.

It might also spark other ideas to help us bridge the conceptual gap between
these models.

Morgaine.

---

From: Jesrad <[email-masked]>
Date: Thu, 5 Mar 2009 16:58:41 +0100

That's great, you're getting to the "meat" of the work to do, though I
cannot shed the slight feeling it's still a bit early.

It is definitely preferable to allow both remote and local execution
so that the implementation can take advantage of either situation as
needed, and even better to allow any mixture of both. Incidentally,
this is a problem that games like Unreal Tournament already solved.

In the UnrealScript environment, the same code can be assigned a Role
and RemoteRole so that it will run as local (on the client), as remote
(on the game server) or as both (with either the client or the server
having authority), with duplicate instances of that code sitting
simultaneously on the server and the client. Each side executes its
Role-relevant parts (it's an event driven language, IIRC you can
specify the role at the event level), and the states are synchronized
at the VM level transparently, by replication of variables, full
object state or by RPC depending on an internal logic decision.

There is a good explanation of it here, though it is rather
implementation-oriented: http://unreal.epicgames.com/Network.htm

If you think it's interesting I'll dive back into it and make a
proposal for extending xxSD to support replication semantics that
allow this kind of flexibility.

On Thu, Mar 5, 2009 at 9:57 AM, Morgaine <[email-masked]> wrote:
> (Note that I use "LESS" generically in this post to refer to worlds of the
> "shared space" model.)
>
> There has been no shortage of discussion about the differences between the
> the object execution model.   This appears to be one of only 2 or 3
> Two object execution models: inverse problems
>
> Solution: split the behaviours and unify the models
>
> benefits of local execution are related to the internal behaviour of the
> execution are related to the external behaviour of the object within and
> Define objects to have separate internal and external behaviours, both
> optional.
> External behaviour works exactly as in SL in response to environmental
> events.
> Internal behaviour works exactly as in LESS in respect of local object
> execution.
> If an object has both internal and external behaviours, then the two are
> connected by a telemetry event stream for remote/local behaviour
> integration.
> Both LESS and SL systems can then gain the advantages of the other, if they
> wish.
> Neither LESS nor SL need suffer their own respective disadvantages, it's
> their choice.
> If LESS chooses not to implement remote object behaviour, it requires no
> system change.
> If SL chooses not to implement local object behaviour, it requires no system
> change.
> The object interop protocol would however need to support both, whether used
> or not.
> Remote scripting in SL would be unchanged, except for the addition of local
> peer calls.
> Local scripting in LESS would be unchanged, except for the addition of
> remote peer calls.
> Since the object execution model of both is supported, one barrier to
> interop has fallen.
>
> Extremely simple example:
> _______________________________________________
> mmox mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mmox
>
>

---

From: Charles Krinke <[email-masked]>
Date: Thu, 5 Mar 2009 09:07:20 -0800

How do the Croquet and Wonderland designs fit into this? It seems to me that
we should strive to accomodate both of these also.

Charles Krinke
OpenSim Core Developer
OSGrid Director

-- 
Charles Krinke
OpenSim Core Developer
OSGrid Director

---

From: Jon Watte <[email-masked]>
Date: Thu, 05 Mar 2009 11:58:39 -0800

Morgaine wrote:
>  running simulations in lockstep generally only works for 
> deterministic simulations, and those are usually not the kind we 
> encounter in human-populated virtual worlds.  Since the simulations 
> are not running in lockstep, there can only be one source for each 
> event, either local or remote, and from this observation it is clear 
> that all the problems of the preceding 

Thanks for the analysis! It's clear you've thought about this. 
Unfortunately, it seems as if you have misunderstood the actual model 
behind the LESS protocol, and the meaning of the word "local." I had 
previously tried to improve the description of the execution model to 
make it clearer -- if you haven't read the new version, please take a look:

http://www.ietf.org/internet-drafts/draft-jwatte-less-protocol-01.txt

In one sense, the LESS model (which is a server<->server, or at least 
world<->world protocol, and leaves clients entirely unspecified) is 
actually already a very nice match to the Second Life execution model of 
objects running on sims.

I'll attempt to clarify more below:

I have not been advocating lockstep simulation as a global model. 
Similarly, when I say "local execution environment," it means "local" to 
the authoritative host for that object -- not local to the machine of 
the user who happens to last have interacted with the object. In a 
typical space, shared by a small number of servers from different 
technologies (say, an OLIVE server, a Multiverse server, and a Second 
Life server all contribute objects to the shared hallucination), you can 
assume that interaction between objects hosted on the same system will 
be "better" than interactions between objects hosted on different 
servers. However, in the general case, the transmission latency across 
the Internet backbone is pretty good these days -- it's the "last mile" 
that has the most problems, and in the typical client/server 
implementation, the different servers won't have to contend with the 
"last mile" for interoperability.

> paragraph apply here as well, since a local simulation cannot predict 
> events occurring in a remote simulation.  In other words, the local 
> simulation provides no advantage for events arriving from the remote 
> simulation versus a system without local simulation, and that's why 
> all the problems mentioned above apply.  What's more, local simulation 
> begins to collapse in utility as world simulations become diverse 
> anyway, so it is not a general solution.
>
> I'm sure that some of the above will be disputed (correctly), but my 
> general point applies:  the LESS model has both advantages and 
> disadvantages, as do all systems.
>

Note that LESS model, as proposed, is actually more like the Second Life 
"remote simulation" model, between server peers. The objects hosted on 
"remote" servers (other technology servers from the point of view of a 
particular observer) will be provided to the "local" server (from the 
point of view of that particular observer) as properties that you can 
use for dead reckoning. I think "dead reckoning" and "full simulation" 
are two very different things. In the DIS model (which this is loosely 
based on), the source of the data actually runs a parallel dead 
reckoning from its last sent update, and only sends updates if the 
outcome of the dead reckoning varies from the real outcome by some 
tolerance, plus there is a "baseline" retransmission time. That's not a 
bad place to start implementation.

The bigger point you make is important, though: The Internet gives you 
latency. You can't help that, so you have to accept it, and as far as 
possible embrace it. This means that local objects will see a 
time-delayed copy of remote objects, and vice versa. That's why, in 
general, local/remote interactions (like collision) would usually be 
detected and optionally messaged to the remote object, but only resolved 
locally.

> namely object execution in the remote region, and it has the almost 
> exactly the same advantages and disadvantages but in reverse.  The 
> region simulator has a high-bandwidth and low-latency relationship 
> with the objects it hosts and hence provides excellent simulation 
> performance, but objects have almost nil relationship with the user's 
> local environment so user integration is almost non-existent and all 
> the great 

I think the Second Life server model is no different from other server 
models. I think you've mis-understood the meaning of the world "local" 
in the proposal. It's "local to the execution environment," such as the 
particular technology server, which implements a particular collision 
test API, a particular physics integration API, a particular object 
persistence API, a particular user interface model, etc. It's still, 
quite likely, "remote" from the client observer, except in the case of 
single-user peer-to-peer nodes.

Note that, whether locally controlled objects are further optimistically 
simulated ahead on the controlling client machine is a platform 
implementation detail, and not under the purview of a LESS-like approach.

>     * External behaviour works exactly as in SL in response to
>       environmental events.
>     * Internal behaviour works exactly as in LESS in respect of local
>       object execution.
>

I don't think there's a difference between the SL server environment and 
the kind of execution environment that LESS exposes. Given the time to 
code it up, existing second life objects can expose the LESS properties 
to any peer, today, with no change. Similarly, Second Life servers can 
inject shadow copies of objects received over LESS into their own 
simulations, and "drive" them based on the updates received from the 
remote service, with dead reckoning for the steps for which there is no 
input data.

I know this model works, because I've implemented it, twice (with 
proprietary protocols) in two different code bases. You basically 
implement a "object that is a ghost of some remote entity" generic 
object kind, which you can expose to your local world and local (in the 
technology platform sense) clients, and you build the gateway that sends 
your own data in the right format, and receives the formatted data and 
translates it to input for the ghost objects, and you're done!

In this sense, I think the difference between LESS and the Second Life 
model are minimal, and there's little "unification" needed.

However, the differences between LESS and OGP are huge, because they 
address totally different problems, and make totally different 
assumptions. I've posted what those are already, in detail. In brief, I 
believe that OGP necessarily forces interoperability towards a 
"universal client" model, and towards a "all objects written to a common 
API" model, which I think are counter to the interests of vendor neutral 
virtual world interoperability.

If you read the updated version of the LESS proposal that I posted to 
the list a while back, perhaps it will clarify the model, and the 
meaning of the word "local"?

http://www.ietf.org/internet-drafts/draft-jwatte-less-protocol-01.txt

Sincerely,

jw

---

From: Morgaine <[email-masked]>
Date: Fri, 6 Mar 2009 11:44:08 +0000

On Thu, Mar 5, 2009 at 3:58 PM, Jesrad <[email-masked]> wrote:

>
> In the UnrealScript environment, the same code can be assigned a Role and
> RemoteRole so that it will run as local (on the client), as remote (on the
> game server) or as both ...
> ...
> If you think it's interesting I'll dive back into it and make a proposal
> for extending xxSD to support replication semantics that allow this kind of
> flexibility.
>
>
Excellent, Jesrad!

It just reinforces the common observation that everything in software has
been done before (mostly from 1936 to 1969 :-) and underlines the
ridiculousness of the concept of software patents.  We work and live in an
international sea of ideas, and everything that we do builds upon, and often
just reinvents, the work of those who came before.  :-)

I bow my head to the Unreal designers. :-)

PS. That's a *great* article, very nicely summarizing local/remote variable
replication, thanks for the link!  By all means work on anything that might
help this come about.  However, let's leave the discussion until the group
starts dealing with specific requirements and solutions, since first we have
to tackle the BoF, workgroup formation, and charter (which of course must
not preclude this).  After that though, I will be strongly interested in
ensuring that our protocol has the flexibility to handle a great solution
that was already in use in 1999. :-)

Morgaine.

On Thu, Mar 5, 2009 at 3:58 PM, Jesrad <[email-masked]> wrote:

> general
> exactly

---

From: Jesrad <[email-masked]>
Date: Fri, 6 Mar 2009 13:05:19 +0100

Thanks for the praise, but doesn't it mandate pushing xxSD well beyond
the representation of data and into code execution ? We might find it
better, then, to skip xxSD entirely and adopt Javascript/JSON, leaving
only a slight choice of VM for the coders to implement such MMOX
compliance.

I'm not excluding that possibility (many languages can nowadays
compile into JS, and there's a number of readily available VMs to
choose from), but at the moment I'd rather we come up with a good
uniform API for cross-world script/object behavior and not bother
about full-state object updates, instead limiting state sync to
variable content and RPC. That restricts the number of things we can
get the "other side" to execute locally, but then they don't have to
rebuild their world around a different language and VM. To further
your whiteboard example, it's OK if I need a (local) pen specific to
my VW in order to write on your (remote) board, if that saves me from
rewriting the hand that holds it, and/or the body it is attached to.

Sincerely,
--
Jesrad Seraph

On Fri, Mar 6, 2009 at 12:44 PM, Morgaine
<[email-masked]> wrote:
> PS. That's a great article, very nicely summarizing local/remote variable

---

From: Morgaine <[email-masked]>
Date: Fri, 6 Mar 2009 13:29:14 +0000

On Fri, Mar 6, 2009 at 12:05 PM, Jesrad <[email-masked]> wrote:

We might find it better, then, to skip xxSD entirely and adopt
> Javascript/JSON

I specifically don't want to address the remainder of your post because I
wish to very firmly stick to IETF guidelines on the order in which things
should be done. :-)

(So now is charter and problem space time, not solution space time.)

That said ...

Yes, given the superb precedent of the success of Javascript in web
technology and the reasonable security of its VM when done right, I would
imagine that JS could play a wonderful role in client-side VW object
scripting as well.

*Bear in mind however* that I am wholly language-agnostic by conscious
choice:  engineers use components made of a multiplicity of materials, and
I'm an engineer.  What matters are the interfaces.  So Javascript is great
... but I would strongly oppose any attempt at making it the only
client-side language supported.

As usual, let's provide (extensible) mechanisms, and not enforce policy. ;-)

Morgaine.

On Fri, Mar 6, 2009 at 12:05 PM, Jesrad <[email-masked]> wrote:

> and
> >> On Thu, Mar 5, 2009 at 9:57 AM, Morgaine <
> [email-masked]>
> >> wrote:
> >> > shared space, and this has severe consequences.  Event response
> latency
> >> > can
> >> > I know that Jon is at this point shouting "Local simulations!" with
> all
> >> > his
> >> > simulation.  In other words, the local simulation provides no
> advantage
> >> > for
> >> > point applies:  the LESS model has both advantages and disadvantages,
> as
> >> > do
> >> > all systems.
> >> >
> >> > almost non-existent and all the great benefits of locality are
> missing.
> >> > In
> >> > The object interop protocol would however need to support both,
> whether
> >> > used
> >> > or not.
> >> > Instead of writing very slowly and with difficulty in-world on the
> side
> >> > of a
> >> > onto the side of the whiteboard, and still do this accurately even
> while
> >> > the
> >> > I wrote this primarily to help analyse the problem space in the area
> of
> >> > LESS
> >> > that it was worth describing this solution because of the immense
> gains
> >> > it

---

From: Morgaine <[email-masked]>
Date: Fri, 6 Mar 2009 13:41:02 +0000

On Thu, Mar 5, 2009 at 7:58 PM, Jon Watte <[email-masked]> wrote:

> Unfortunately, it seems as if you have misunderstood the actual model behind
> the LESS protocol, *and the meaning of the word "local."* I had previously
> tried to improve the description of the execution model to make it clearer
> -- if you haven't read the new version, please take a look:
>

The word "local" doesn't appear even once in the new draft.

I am currently trying to figure out its meaning in LESS from context.

Morgaine.

On Thu, Mar 5, 2009 at 7:58 PM, Jon Watte <[email-masked]> wrote:

>
>
> Morgaine wrote:
>
>>  running simulations in lockstep generally only works for deterministic
>> simulations, and those are usually not the kind we encounter in
>> human-populated virtual worlds.  Since the simulations are not running in
>> lockstep, there can only be one source for each event, either local or
>> remote, and from this observation it is clear that all the problems of the
>> preceding
>>
>
> Unfortunately, it seems as if you have misunderstood the actual model behind
> the LESS protocol, and the meaning of the word "local." I had previously
> tried to improve the description of the execution model to make it clearer
> -- if you haven't read the new version, please take a look:
>
> world<->world protocol, and leaves clients entirely unspecified) is actually
> already a very nice match to the Second Life execution model of objects
> running on sims.
>
> Similarly, when I say "local execution environment," it means "local" to the
> authoritative host for that object -- not local to the machine of the user
> who happens to last have interacted with the object. In a typical space,
> shared by a small number of servers from different technologies (say, an
> OLIVE server, a Multiverse server, and a Second Life server all contribute
> objects to the shared hallucination), you can assume that interaction
> between objects hosted on the same system will be "better" than interactions
> between objects hosted on different servers. However, in the general case,
> the transmission latency across the Internet backbone is pretty good these
> days -- it's the "last mile" that has the most problems, and in the typical
> client/server implementation, the different servers won't have to contend
> with the "last mile" for interoperability.
>
>> simulation versus a system without local simulation, and that's why all the
>> problems mentioned above apply.  What's more, local simulation begins to
>> collapse in utility as world simulations become diverse anyway, so it is not
>> a general solution.
>>
> particular observer) will be provided to the "local" server (from the point
> of view of that particular observer) as properties that you can use for dead
> reckoning. I think "dead reckoning" and "full simulation" are two very
> different things. In the DIS model (which this is loosely based on), the
> source of the data actually runs a parallel dead reckoning from its last
> sent update, and only sends updates if the outcome of the dead reckoning
> varies from the real outcome by some tolerance, plus there is a "baseline"
> retransmission time. That's not a bad place to start implementation.
>
> possible embrace it. This means that local objects will see a time-delayed
> copy of remote objects, and vice versa. That's why, in general, local/remote
> interactions (like collision) would usually be detected and optionally
> messaged to the remote object, but only resolved locally.
>
>> almost non-existent and all the great
>>
>
> models. I think you've mis-understood the meaning of the world "local" in
> the proposal. It's "local to the execution environment," such as the
> particular technology server, which implements a particular collision test
> API, a particular physics integration API, a particular object persistence
> API, a particular user interface model, etc. It's still, quite likely,
> "remote" from the client observer, except in the case of single-user
> peer-to-peer nodes.
>
> the kind of execution environment that LESS exposes. Given the time to code
> it up, existing second life objects can expose the LESS properties to any
> peer, today, with no change. Similarly, Second Life servers can inject
> shadow copies of objects received over LESS into their own simulations, and
> "drive" them based on the updates received from the remote service, with
> dead reckoning for the steps for which there is no input data.
>
> proprietary protocols) in two different code bases. You basically implement
> a "object that is a ghost of some remote entity" generic object kind, which
> you can expose to your local world and local (in the technology platform
> sense) clients, and you build the gateway that sends your own data in the
> right format, and receives the formatted data and translates it to input for
> the ghost objects, and you're done!
>
> address totally different problems, and make totally different assumptions.
> I've posted what those are already, in detail. In brief, I believe that OGP
> necessarily forces interoperability towards a "universal client" model, and
> towards a "all objects written to a common API" model, which I think are
> counter to the interests of vendor neutral virtual world interoperability.
>
> If you read the updated version of the LESS proposal that I posted to the
> list a while back, perhaps it will clarify the model, and the meaning of the
> word "local"?
>