Subject: [mmox] Yet another proposal for LLSD

From: Catherine Pfeffer <[email-masked]>
Date: Wed, 11 Feb 2009 13:05:04 +0100

Hi again,

Someone asked my to forward this suggestion for enhancement;

"32 bits data are not future-proof enough, they should be replaced with 64
bits"

If my memories are correct, Infinity already gave some thoughts abour it
during some informal conversation, but not on this list. I apologize in
advance if that has already been proposed.

-- 
Cathy

---

From: [email-masked]
Date: Wed, 11 Feb 2009 11:08:23 -0600

Abstract the serializable type so that other vendors may implement their own.

Note that serialization may not even be required when moving an object  
from one local CPU to local another.

64 vs. 32 bit integers... this isn't an issue for XML but will be for  
binary streams.  Again abstract integers can help here (also with  
big-endian probs too)

---

From: Meadhbh Hamrick <[email-masked]>
Date: Wed, 11 Feb 2009 09:26:23 -0800

right. the answer is basically there are several systems / languages 
that currently don't support 64 bit efficiently. Python, for instance, 
does not support a 64 bit int. if you need an int with a width larger 
than 32 bits, you get an arbitrary lengthed bignum. by defining an int 
as 64 bits, you REQUIRE the use of bignums in Python (and I think PHP 
and ECMAScript) but not C/C++, C# or Java where you simply use longs.

and PLEASE remember... WE DO NOT WANT TO ENSHRINE THE EXISTING SECOND 
LIFE PROTOCOL AS A STANDARD. our purpose is not to take the existing UDP 
message system, that does encode 64 bit values as a binary and create a 
protocol out of it. rather, we are hoping we can all learn from such 
mistakes and "do the right thing" in the future.

so the question is... where do we need 64 bit ints, if we're not taking 
the existing SL protocol and standardizing it? and the sad reply is "we 
don't know cause we haven't worked out the protocol for all aspects of a 
virtual world yet." so we have a situation where a long int may or may 
not be useful in the future, but we know it will be detrimental to 
people using Python (and we think PHP and ECMAScript) and certainly 
people interested in working on mobile clients.

this is the thought process the authors went through to decide the width 
of ints in the abstract type system.

Catherine Pfeffer wrote:
> "32 bits data are not future-proof enough, they should be replaced 
> with 64 bits"
>
> If my memories are correct, Infinity already gave some thoughts abour 
> it during some informal conversation, but not on this list. I 
> apologize in advance if that has already been proposed.
>
> -- 
> Cathy
> ------------------------------------------------------------------------
>
> _______________________________________________
> mmox mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mmox
>

---

From: "Hurliman, John" <[email-masked]>
Date: Wed, 11 Feb 2009 11:03:57 -0700

It actually is a problem in XML. In the current spec, the omission of a 64-bit integer type means 64-bit integers are stored as binary blobs. These end up as Base64 encoded data in the XML. Internally, you have slower access to the larger integers due to typecasting (and possibly endian conversions, depending on when you do the host to network order conversion). Over the wire, you have to go through Base64 encoding/decoding on top of the conversions.

John

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>[email-masked]
>Sent: Wednesday, February 11, 2009 9:08 AM
>To: [email-masked]
>Subject: Re: [mmox] Serial
>

---

From: Mark Lentczner <[email-masked]>
Date: Wed, 11 Feb 2009 09:08:35 -0800

The original design of LLSD had 64 bit integers.  I argued for and  
switch the definition to 32 bit several years ago. My rationale was  
thus:

It is the goal that LLSD data, when in memory and being operated on by  
a program, use native data types as much as possible. In particular,  
when LLSD data is in a scripting language such as Perl, Python or PHP,  
then the types should be the corresponding native data types.

For Integers, choosing 32 bits means that integers are just plain,  
native integers in the vast majority of languages in use today. While  
many of these languages have facilities for dealing with larger  
integral numbers, they are often not uniform to work with, convenient  
or efficient by orders of magnitude than the native, 32 bit integers.

There is also the fact that there has been little call for 64 bit  
integers in the interfaces and APIs that we have built for virtual  
worlds. The only place I know of in our entire protocol is an obvious  
design mistake: Two 32 bit integers are concatenated to form a 64 bit  
identifier. It would have been better to just treat the array of the  
two integers as the identifier.

	- Mark

Mark Lentczner
Sr. Systems Architect
Technology Integration
Linden Lab

[email-masked]

Zero Linden
[email-masked]

---

From: Dahlia Trimble <[email-masked]>
Date: Wed, 11 Feb 2009 10:47:00 -0800

There are other uses of base 64 encoding of integers in LLSD in the LL
protocol in the event queue. the "RegionHandle" and "GroupPowers" fields are
base 64 encoded 64 bit unsigned, and  "ParcelFlags" and "AuctionID" are 32
bit unsigned. Either of these may require endian conversion depending on
architecture. In the case of "ParcelFlags" the original meaning of the data
are further obscured as the meaning of each individual flag is not specified
in the message. It would seem that rather specifying each flag as a boolean
would serve to improve human readability, if human readability were indeed a
goal.
There may be other cases where similar encodings are used but I haven't seen
them yet.

On Wed, Feb 11, 2009 at 9:08 AM, Mark Lentczner <[email-masked]> wrote:

> The original design of LLSD had 64 bit integers.  I argued for and switch
> the definition to 32 bit several years ago. My rationale was thus:
>
> It is the goal that LLSD data, when in memory and being operated on by a
> program, use native data types as much as possible. In particular, when LLSD
> data is in a scripting language such as Perl, Python or PHP, then the types
> should be the corresponding native data types.
>
> For Integers, choosing 32 bits means that integers are just plain, native
> integers in the vast majority of languages in use today. While many of these
> languages have facilities for dealing with larger integral numbers, they are
> often not uniform to work with, convenient or efficient by orders of
> magnitude than the native, 32 bit integers.
>
> There is also the fact that there has been little call for 64 bit integers
> in the interfaces and APIs that we have built for virtual worlds. The only
> place I know of in our entire protocol is an obvious design mistake: Two 32
> bit integers are concatenated to form a 64 bit identifier. It would have
> been better to just treat the array of the two integers as the identifier.
>
>        - Mark
>
> Mark Lentczner
> Sr. Systems Architect
> Technology Integration
> Linden Lab
>
> [email-masked]
>
> Zero Linden

---

From: Jon Watte <[email-masked]>
Date: Wed, 11 Feb 2009 11:07:28 -0800

> two integers as the identifier.

Our system uses 64-bit integers extensively for object identifiers, time 
step counters, virtual currency, and other such potentially "large" 
quantities. I would say that any forward-looking standard needs to 
support 64-bit integer domains. One way to solve this is to leave the 
max size of integers as something that's specified on a per-field basis, 
and/or support continuation encoding of integers (7 bits per byte; if 
high bit is set, shift and concatenate the next 7 bits. Repeat).

Sincerely,

jw

---

From: "Hurliman, John" <[email-masked]>
Date: Wed, 11 Feb 2009 13:08:52 -0700

The integer->binary->base64->binary->integer dance also happens for unsigned 32-bit integers in the current LLSD spec. While 64-bit integers might be hand waved away as unnecessary, it seems odd to make 32-bit unsigned integers so difficult to deal with.

John

>Dahlia Trimble
>Sent: Wednesday, February 11, 2009 10:47 AM
>To: Mark Lentczner
>Cc: [email-masked]
>Subject: Re: [mmox] LLSD -- 64 bit Integers
>
>protocol in the event queue. the "RegionHandle" and "GroupPowers" fields
>are base 64 encoded 64 bit unsigned, and  "ParcelFlags" and "AuctionID"
>are 32 bit unsigned. Either of these may require endian conversion
>depending on architecture. In the case of "ParcelFlags" the original
>meaning of the data are further obscured as the meaning of each
>individual flag is not specified in the message. It would seem that
>rather specifying each flag as a boolean would serve to improve human
>readability, if human readability were indeed a goal.
>
>There may be other cases where similar encodings are used but I haven't
>seen them yet.
>
>
>
>	It is the goal that LLSD data, when in memory and being operated
>on by a program, use native data types as much as possible. In
>particular, when LLSD data is in a scripting language such as Perl,
>Python or PHP, then the types should be the corresponding native data
>types.
>
>many of these languages have facilities for dealing with larger integral
>numbers, they are often not uniform to work with, convenient or
>efficient by orders of magnitude than the native, 32 bit integers.
>
>identifier. It would have been better to just treat the array of the two
>integers as the identifier.
>
>	       - Mark
>
>	Mark Lentczner
>	Sr. Systems Architect
>	Technology Integration
>	Linden Lab
>
>	[email-masked]
>
>	Zero Linden

---

From: "Meadhbh Hamrick (Infinity)" <[email-masked]>
Date: Wed, 11 Feb 2009 12:14:01 -0800

John, i'm not sure what you mean by this.

assuming you're on a system that does 2's complement notation, the  
conversion between unsigned and signed is trivial.

and there's nothing that REQUIREs you to represent a 32 bit integer as  
a binary, but if you want to, and then you want to serialize it,  
maintaining the ability to deserialize it seems like a "good thing."

-cheers
-meadhbh

On Feb 11, 2009, at 12:08 PM, Hurliman, John wrote:

> The integer->binary->base64->binary->integer dance also happens for  
> unsigned 32-bit integers in the current LLSD spec. While 64-bit  
> integers might be hand waved away as unnecessary, it seems odd to  
> make 32-bit unsigned integers so difficult to deal with.
>
> John
>
>> integral

---

From: Ewen Cheslack-Postava <[email-masked]>
Date: Wed, 11 Feb 2009 12:36:27 -0800

On Wed, Feb 11, 2009 at 11:07 AM, Jon Watte <[email-masked]> wrote:
> quantities. I would say that any forward-looking standard needs to support
> 64-bit integer domains. One way to solve this is to leave the max size of
> integers as something that's specified on a per-field basis, and/or support
> continuation encoding of integers (7 bits per byte; if high bit is set,
> shift and concatenate the next 7 bits. Repeat).

There are a number of places I'd expect 64 bit integers to be used as
well, absolute time points being the most obvious.

But beyond that, the reasoning given leads to a *very* slippery slope,
the bottom of which is a "least common denominator" approach.  In this
case the issue is efficiency, but it seems to me that's a bad
argument.  Because python (or some other language) doesn't have native
support for a feature, no language gets to use it.  Now if I want to
represent a time as a 64bit int, I have to use 2x32bit ints and do the
munging manually.  Suddenly my C++ code, which could handle this
efficiently, also becomes inefficient.  Either that or we end up with
layers on top of the standard interface to provide these
features/conversions for "free".  But then we might as well include
them in the standard since they'll be implemented for every language
anyway.

Computational efficiency isn't the only concern either.  Currently
there is only support for double precision in xxSD.  Is this a
foregone conclusion since Ruby doesn't support native single precision
floats?  This is a wire protocol, so there should be efficient
encodings when necessary.  Forcing double precision is wasteful in a
lot of case, e.g. when specifying location information in a local
frame (mesh vertices) or specifying normals, which shouldn't require
so much precision.

The barrier to entry should be "is this feature useful and does it
have a reasonable implementation" not "can this feature be implemented
efficiently everywhere."  If you really need some service to be
implemented very efficiently and it has to handle a lot of data which
potentially uses 64bit ints, then maybe Python isn't the right
language for that implementation.  The intent isn't to start a
language flame war, but to point out that if the desire is to have a
fairly language agnostic type system, then specific considerations for
specific languages generally shouldn't enter the equation.

-Ewen

---

From: "Hurliman, John" <[email-masked]>
Date: Wed, 11 Feb 2009 13:48:41 -0700

If support for unsigned 32-bit integers is implemented by typecasting to a signed integer, sending over the wire, and assuming the other side knows to typecast back to an unsigned, that should be explicitly stated in the spec. How do you state this in LLIDL?

John

>From: Meadhbh Hamrick (Infinity) [mailto:[email-masked]]
>Sent: Wednesday, February 11, 2009 12:14 PM
>To: Hurliman, John
>Cc: [email-masked]

---

From: Dahlia Trimble <[email-masked]>
Date: Wed, 11 Feb 2009 12:54:36 -0800

Wouldn't the capacity to store an order of magnitude be lost by typecasting?
I thought that was one of the reasons to consider using an unsigned int
rather than a signed one.

On Wed, Feb 11, 2009 at 12:48 PM, Hurliman, John <[email-masked]>wrote:

> If support for unsigned 32-bit integers is implemented by typecasting to a
> signed integer, sending over the wire, and assuming the other side knows to
> typecast back to an unsigned, that should be explicitly stated in the spec.
> How do you state this in LLIDL?
>
> John
>

---

From: David W Levine <[email-masked]>
Date: Wed, 11 Feb 2009 16:02:00 -0500

I think Ewen's posited ground rule feels right. The presence or absence of 
a native 64 bit type in a specific language is a very weak reason to 
support it in a wire format intended to be language independent.
Further, per comments from several others, if we're doing this with native 
types and possible type casting in various ways, we need the wire format 
to be quite unambiguous about signed, unsigned, and so on. Ambiguity has 
no place at all in this sort of specification, the fact that questions 
about sign bits showed up today, is a hint to be careful on that. 

- David W. Levine
~ Zha Ewry (ISL)

Ewen Cheslack-Postava <[email-masked]> 
Sent by: [email-masked]
02/11/2009 03:36 PM

To
Jon Watte <[email-masked]>
cc
[email-masked]
Subject
Re: [mmox] LLSD -- 64 bit Integers

>> There is also the fact that there has been little call for 64 bit 
integers
> quantities. I would say that any forward-looking standard needs to 
support

-Ewen

---

From: "Meadhbh Hamrick (Infinity)" <[email-masked]>
Date: Wed, 11 Feb 2009 13:46:20 -0800

no, it isn't.

On Feb 11, 2009, at 12:54 PM, Dahlia Trimble wrote:

> Wouldn't the capacity to store an order of magnitude be lost by  
> typecasting? I thought that was one of the reasons to consider using  
> an unsigned int rather than a signed one.
>
> On Wed, Feb 11, 2009 at 12:48 PM, Hurliman, John <[email-masked] 
> > wrote:
> If support for unsigned 32-bit integers is implemented by  
> typecasting to a signed integer, sending over the wire, and assuming  
> the other side knows to typecast back to an unsigned, that should be  
> explicitly stated in the spec. How do you state this in LLIDL?
>
> John
>
> >>> There are other uses of base 64 encoding of integers in LLSD in  
> the
> >>> LL
> >>> On Wed, Feb 11, 2009 at 9:08 AM, Mark Lentczner <[email-masked] 
> >
> >>> wrote:
> >>>
> >>>

---

From: Morgaine Dinova <[email-masked]>
Date: Thu, 12 Feb 2009 15:17:03 +0000

On Wed, Feb 11, 2009 at 9:02 PM, David W Levine <[email-masked]> wrote:

> a native 64 bit type in a specific language is a very weak reason to support
> it in a wire format intended to be language independent.
> types and possible type casting in various ways, we need the wire format to
> be quite unambiguous about signed, unsigned, and so on. Ambiguity has no
> place at all in this sort of specification, the fact that questions about
> sign bits showed up today, is a hint to be careful on that.
>
> - David W. Levine
> ~ Zha Ewry (ISL)
>
>
I second both parts of that argument.  I also worry that we're not
addressing its second part at all.

By restricting ourselves to a single form of integer, we seem to be heading
directly towards defeat in the battle against ambiguity.  Defining multiple
distinct integer types is almost completely free of cost, yet I've heard
many people reject the idea through an incomprehensible "proliferation"
argument.  Defining numerous types to support myriad very diverse components
most effectively is the basis of engineering, and we should embrace it, not
shy away from it.

Morgaine.

>
>  *Ewen Cheslack-Postava <[email-masked]>*
> [email-masked]  Subject