Subject: [manet] RFC5444 schema-based compression V2

From: Henning Rogge <[email-masked]>
Date: Thu, 24 Jan 2013 15:53:31 +0100

RFC5444 Schema Compression V2
=============================

Changes since V1:
- restructured text to make it easier to read
- added compression for address-TLVs
- simplified Schema description
   (no more differentiation between mandatory and optional TLVs)
- added explanation for TLVs schema fields

Compression idea
----------------

Describe the typical content of a RFC5444 message in your deployment 
with a Schema.

Remove everything from the RFC5444 message that can be constructed from 
Schema. Use bitfields to describe how many TLVs of each kind are present 
in a TLV-block and their data.

Schema content
--------------

Schema field values can a list of possible values (the list can contain 
a single value) or an interval of numbers. Schema fields can be empty 
when the field is not restricted by the Schema in any ways and has 
always to be fully transmitted over the network.

Fields for Message Header:
- each of the fields of the Message Headers schema describes one of the 
possible fields described in RFC5444 chapter 5.2

Fields for TLVs:
- "quantity" limits the number of times a TLV can be present in a 
TLV-block. The VALIDITY_TIME TLV is mandatory for all NHDP messages and 
can only be present once, so its quantity is "1". The INTERVAL_TIME TLV 
is optional for NHDP messages, but must not be there more than once, so 
its quantity is "0-1".
- "extension type" sets the extension type of the TLV if known.
- "value length" restricts the length of the TLVs value (to a range or a 
fixed length).
- "value" restricts the value itself of the TLV.

Example schema
--------------

Schema Number        42

Message Header:
* Type               0
* MHasOrig:          0
* MHasHopLimit:      1
* MHasHopCount:      0
* MHasSeqnum:        1
* Address Length:    4
* Originator Address
* Hop Limit          1
* Hop Count
* Sequence Number

Message TLVs:
* TLV type 0: (INTERVAL_TIME)
+-- quantity         0-1
+-- extension type   0
+-- value length     1
+-- value
|
* TLV type 1 (VALIDITY_TIME)
+-- quantity         1
+-- extension type   0
+-- value length     1
+-- value

Addresses Block TLVs:
* TLV type 2 (LOCAL_IF)
+-- quantity
+-- extension type   0
+-- value length     1
+-- value            0-1
|
* TLV type 3 (LINK_STATUS)
+-- quantity
+-- extension type   0
+-- value length     1
+-- value            0-2
|
+ TLV type 4 (OTHER_NEIGHBOR)
+-- quantity
+-- extension type   0
+-- value length     1
+-- value            0-1

Example message for compression
-------------------------------

The message to be compressed in this example is a NHDP message:
- 4-byte addresses,
- hoplimit 1,
- sequence number 0x1234

- an INTERVAL_TIME Message-TLV with value 0x01
- a VALDIITY_TIME Message-TLV with value 0x10

- an address block with the IP's 10.0.0.1, 10.0.0.2 and 10.0.0.3
- 3-byte head, no prefixes
- 10.0.0.1 has a LOCAL_IF Address-TLV with value 0x00
- 10.0.0.2/3 have a LINK_STATUS Address-TLV with value 0x01 and 0x02
- 10.0.0.2/3 have an OTHER_NEIGHBOR Address-TLV with value 0x01 and 0x00

Compression description
-----------------------

The message always starts with the schema number.

# 42

Add compressed msg-size to message.

# xx xx

If the msg-flags AND msg-addr-length field is specified in the schema, 
skip them. Otherwise add them to the message

# (none in this example)

Add all optional message header fields (Originator, Hoplimit, Hopcount, 
Sequence Number) that are marked in the flags and are not already 
defined in the schema.

# 12 34 (sequence number value)

Add a bitfield with as many bits for each TLV as needed to encode the 
quantity of it (no bit necessary for one fixed quantity, one bit for 
0-1, ...). Append another bit that encodes if 'other TLVs' will follow 
the compressed ones.

binary:
# 1 0 ( 1*INTERVAL_TIME, 1*VALDIITY_TIME, no "other TLVs")

in bytes:
# 80

Add all present TLVs to the compressed message. Leave out all fields 
that can be reconstructed from the Schema. Shorten each TLVs remaining 
fields (except tlv-flags) into bitfield as long as the possible 
combinations need. Only compress value if can be expressed in less than 
8 bits (padd with zero-bits before the value if not).

# 01 (value of INTERVAL-TIME TLV)
# 10 (value of VALIDITY-TIME TLV)

If there are other TLVs (not described in the schema), add the number of 
bytes needed to describe them (2 byte length field similar to a TLV 
block) and the TLVs itself (uncompressed).

# (none in this example)

Add the address-block to the compressed message.

The number of addresses can give an additional restriction to the 
minimum/maximum index the TLVs can have and the maximum quantity of the 
TLVs. Maximum quantity in this example is 3.

# 03 80 03 10 00 00 01 02 03

Add a bitfield with as many bits for each TLV as needed to encode the 
quantity of it (no bit necessary for one fixed quantity, one bit for 
0-1, ...). Append another bit that encodes if 'other TLVs' will follow 
the compressed ones.

binary:
# 01 01 01 0 (1*LOCAL_IF, 1*LINK_STATUS, 1*OTHER_NEIGHBOR, no other TLVs)

or in bytes:
# 54

Add all present TLVs to the compressed message. Leave out all fields 
that can be reconstructed from the Schema. Shorten each TLVs remaining 
fields (except tlv-flags) into bitfield as long as the possible 
combinations need. Only compress value if can be expressed in less than 
8 bits (padd with zero-bits before the value if not).

binary:
# 01010000 01 0 (LOCAL_IF flags, index-start and value)
# 00110100 10 11 01 10 (LINK_STATUS flags, index-start, index-end and 
value 1/2)
# 00110100 10 11 1 0 (OTHER_NEIGHBOR flags, index-start, index-end and 
value 1/2)

in bytes:
# 50 40
# 34 b6
# 34 b8

If there are other TLVs (not described in the schema), add the number of 
bytes needed to describe them (2 byte length field similar to a TLV 
block) and the TLVs itself (uncompressed).

# (none in this example)

Result
------

The complete message has been reduced from

# 00 00 2f 53 01 12 34 (message header)
# 00 08 (message-tlv block length)
# 00 10 01 01 (INTERVAL-TIME 0x01)
# 01 10 01 10 (VALIDITY-TIME 0x10)
# 03 80 03 10 00 00 01 02 03 (3 Addresses, 10.0.0.1/2/3, no prefixes)
# 00 13 (address-tlv block length)
# 02 50 01 01 00 (LOCAL_IF index=1 0x00)
# 03 34 02 03 01 01 02 (LINK_STATUS index=2-3 0x01/0x02)
# 04 34 02 03 01 01 00 (OTHER_NEIGHBOR index=2-3 0x01/0x00)

(47 bytes)

to

# 42 00 18 12 34  (message header)
# 80 01 10 (message tlvs)
# 03 80 03 10 00 00 01 02 03 (address block)
# 54 50 40 34 b6 34 b8 (address tlvs)

(24 bytes)

Comments
--------

A packet-level schema entry should be added to this concept.

Maybe the schema number should be a two-byte identifier?

I think there could be some more compression if the Schema includes a 
"mandatory prefix" for all addresses (or maybe a list of them?). This 
would help to compress both the originator address and the prefix fields 
in address blocks.

In theory there are a few wasted bits here and there, but I decided NOT 
to do any bit-shifting on data which is 1 byte or larger.

One problem I ignored in this version is the presence of multiple TLVs 
of the same type attached to the same address. Maybe the Schema 
"quantity" field for address TLVs should be interpreted as "quantity per 
address?"

A great addition to this idea would be a protocol for requesting unknown 
schema from your neighbor. And a program that scans a RFC5444 stream and 
suggests a schema based on the watched messages.

-- 
Diplom-Informatiker Henning Rogge , Fraunhofer-Institut für
Kommunikation, Informationsverarbeitung und Ergonomie FKIE
Kommunikationssysteme (KOM)
Fraunhofer Straße 20, 53343 Wachtberg, Germany
Telefon +49 228 9435-961,   Fax +49 228 9435 685
mailto:[email-masked] http://www.fkie.fraunhofer.de

---

From: "Dearlove, Christopher (UK)" <[email-masked]>
Date: Thu, 24 Jan 2013 16:48:24 +0000

How is it proposed to put this compressed format on the wire (or over the air more likely)?

You can't just put it on the manet protocol/port, as that would be parsed as 5444, which it isn't.

Approaches I could see include (but this is not an exhaustive list; new, better, ideas welcome).
- This is for specialised uses (e.g. as in something like 6LOWPAN).
- On some other port (I doubt you'd get another protocol).
- Describing a new 5444 version 1. It's a bit of an abuse, as it's not
  exactly a version, but maybe.

I'd also want to be convinced that these schemas allow extensions
without changing the schema. For example suppose I decide to add
to either NHDP's HELLO message or OLSRv2's TC message that these
could advertise blacklisted addresses (whatever that means) using
a BLACKLIST TLV. That needs to be possible without updating the
schema to know about that TLV. (Why? TLVs in the private/experimental
space for a start.)

[This may be obvious, I haven't had the time to study this yet.]

-- 
Christopher Dearlove
Senior Principal Engineer, Communications Group
Communications, Networks and Image Analysis Capability
BAE Systems Advanced Technology Centre
West Hanningfield Road, Great Baddow, Chelmsford, CM2 8HN, UK
Tel: +44 1245 242194 |  Fax: +44 1245 242124
[email-masked] | http://www.baesystems.com

BAE Systems (Operations) Limited
Registered Office: Warwick House, PO Box 87, Farnborough Aerospace Centre, Farnborough, Hants, GU14 6YU, UK
Registered in England & Wales No: 1996687

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Henning Rogge
Sent: 24 January 2013 14:54
To: [email-masked]
Subject: [manet] RFC5444 schema-based compression V2

Changes since V1:

Compression idea
----------------

Schema content
--------------

Fields for TLVs:

Example schema
--------------

Schema Number        42

Message Header:
* Type               0
* MHasOrig:          0
* MHasHopLimit:      1
* MHasHopCount:      0
* MHasSeqnum:        1
* Address Length:    4
* Originator Address
* Hop Limit          1
* Hop Count
* Sequence Number

Message TLVs:

Addresses Block TLVs:
* TLV type 2 (LOCAL_IF)
+-- quantity
+-- extension type   0
+-- value length     1
+-- value            0-1
|

Compression description
-----------------------

# 42

# xx xx

# (none in this example)

binary:

in bytes:
# 80

# (none in this example)

binary:

or in bytes:
# 54

binary:

in bytes:
# 50 40
# 34 b6
# 34 b8

# (none in this example)

Result
------

(47 bytes)

to

(24 bytes)

Comments
--------

********************************************************************
This email and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or
distribute its contents to any other person.
********************************************************************

---

From: Henning Rogge <[email-masked]>
Date: Thu, 24 Jan 2013 18:09:18 +0100

> - This is for specialised uses (e.g. as in something like 6LOWPAN).

We could consider it as some kind of "linklayer
compressor/decompressor" in some cases, which means the UDP packets of
RFC5444 get compressed and decompressed by some device driver.

ROHC (Robust Header Compression) is similar I think.

>   exactly a version, but maybe.
In a certain sense it is... it contains the same "internal structure",
it has just been transformed into a different representation.

If there would be a protocol to query for an unknown schema, we could
even have nodes learning from each other what to do with this packets
as long as they are marked correctly.

> schema to know about that TLV. (Why? TLVs in the private/experimental

Yes, it does.

In each compressed TLV-Block, after all the bits in the bitfield that
mark the presence of certain compressed TLVs there is a final bit that
says "uncompressed TLVs follow".

If this bit is a "1", you will process the compressed TLVs first, then
read a normal 2-byte length field and start processing uncompressed
TLVs.

This way we have another "upgrade path" from one schema to the other one.

Of course when you do so, you might also start working on a new schema
for your deployment too. But I think its mandatory that there is a way
to add things that are not described in the schema.

Henning Rogge

-- 
We began as wanderers, and we are wanderers still. We have lingured
long enough on the shores of the cosmic ocean. We are ready at last to
set sail for the stars - Carl Sagan

---

From: Abdussalam Baryun <[email-masked]>
Date: Thu, 24 Jan 2013 19:32:35 +0100

> # 54 50 40 34 b6 34 b8 (address tlvs)

If compressed then, Why not this way as in your previous example,
# 42 00 18 12 34 80 01 10 03 80 03 10 00 00 01 02 03 54 50 40 34 b6 34 b8
The Schema takes them together or in parts as you done, not sure :(

Please advise,

AB

---

From: Abdussalam Baryun <[email-masked]>
Date: Thu, 24 Jan 2013 19:43:36 +0100

Hi Henning,

comments below;

> RFC5444 Schema Compression V2

Not sure what you mean, I thought the Schema is only for messages,
addresses and tlvs, why you including the packet?

> Maybe the schema number should be a two-byte identifier?

Yes I agree, because to cover all future extensions and updates.

> "quantity" field for address TLVs should be interpreted as "quantity per

That is why I mentioned before there was limitation of TLV and asked
if there is possibility to adjust. I think it is important to consider
this issue,

AB

---

From: Abdussalam Baryun <[email-masked]>
Date: Thu, 24 Jan 2013 21:36:17 +0100

> You can't just put it on the manet protocol/port, as that would be parsed as

A different parse compression, I think this schema is not for 5444
packet, but 5444 messages
>   exactly a version, but maybe.

It is a second version of the parse and compression done before,

> schema to know about that TLV. (Why? TLVs in the private/experimental

I agree that it is better as you mentioned

AB
> _______________________________________________
> manet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/manet
>

---

From: Teco Boot <[email-masked]>
Date: Fri, 25 Jan 2013 08:28:13 +0100

> ROHC (Robust Header Compression) is similar I think.

We could use this schema based compression along stateful compression (send only delta's; ask for full messages if out-of-sync detected and/or send full messages every now and then). Both on link layer, either discovered or configured.

Teco

---

From: Teco Boot <[email-masked]>
Date: Fri, 25 Jan 2013 08:50:20 +0100

> Maybe the schema number should be a two-byte identifier?
It does make sense to use short fields as much as possible, that's the goal of compression, right? 
Having many schema's could bring high compression rates. Limitation on schema numbers brings unneeded restrictions. So why not put integers in variable length fields?

Teco

---

From: Henning Rogge <[email-masked]>
Date: Fri, 25 Jan 2013 09:00:02 +0100

> The Schema takes them together or in parts as you done, not sure :(

The example is easier to read this way.

Henning Rogge

---

From: Henning Rogge <[email-masked]>
Date: Fri, 25 Jan 2013 09:11:48 +0100

> addresses and tlvs, why you including the packet?

Because when we create a mechanism to compress TLV-blocks, we should 
think about allowing nodes to have a schema for their packet-TLVs too, 
so we can also compress them?

> Yes I agree, because to cover all future extensions and updates.

You can reuse schema numbers as soon as you do not use an old number 
anymore in your deployment. If you deploy the same schemas on all nodes, 
1 byte should be enough I think.

But if we want to think about using different schema's for each node, 1 
byte will definitely not enough.

> if there is possibility to adjust. I think it is important to consider

No, you asked about a limitation of optional TLVs based on the mistake 
that the bitfield involved could only extend for 1 byte.

Henning Rogge

---

From: Henning Rogge <[email-masked]>
Date: Fri, 25 Jan 2013 09:14:12 +0100

On 01/25/2013 08:28 AM, Teco Boot wrote:
>>> Approaches I could see include (but this is not an exhaustive
>>> list; new, better, ideas welcome). - This is for specialised uses
>>> (e.g. as in something like 6LOWPAN).
>>
>> compressor/decompressor" in some cases, which means the UDP packets
>> of RFC5444 get compressed and decompressed by some device driver.
>>
> We could use this schema based compression along stateful compression
> (send only delta's; ask for full messages if out-of-sync detected
> and/or send full messages every now and then). Both on link layer,
> either discovered or configured.

We discussed this idea here at the institute too.

ROHC most deals with information flows between two points and with 
(mostly) fixed length information blocks.

I think this "only transmit delta" will be quite difficult to do, so I 
skipped the idea and worked on something I was sure I could do it.

Henning Rogge

---

From: Henning Rogge <[email-masked]>
Date: Fri, 25 Jan 2013 09:17:02 +0100

On 01/25/2013 08:50 AM, Teco Boot wrote:
> It does make sense to use short fields as much as possible, that's
> the goal of compression, right? Having many schema's could bring high
> compression rates. Limitation on schema numbers brings unneeded
> restrictions. So why not put integers in variable length fields?

Not sure we would normally safe that much.

Less than one byte doesn't make much sense, unless you want to start 
bitshift everything in the following data.

What we could do would be to allocate the first bit as a "7 bit/15 bit" 
switch for the schema id.

If the first bit is zero, just take the first byte as the schema number.
If the first bit is one, set it to zero and take the first TWO bytes as 
the schema number.

What do you think?

Henning Rogge

---

From: Henning Rogge <[email-masked]>
Date: Fri, 25 Jan 2013 09:37:53 +0100

> If the first bit is one, set it to zero and take the first TWO bytes as

Second thought, we could use this for the message length field too.

Henning Rogge

---

From: Abdussalam Baryun <[email-masked]>
Date: Fri, 25 Jan 2013 09:47:26 +0100

> If the first bit is one, set it to zero and take the first TWO bytes as

I think that is a good choice,

AB

---

From: "Dearlove, Christopher (UK)" <[email-masked]>
Date: Fri, 25 Jan 2013 10:13:52 +0000

The linklayer concept obviously means one would have to consider 5444 packets, not just messages. It could be argued to be a layer violation, but then several things done in such circumstances could be so considered.

We are in agreement on your last point.

-----Original Message-----
From: Henning Rogge [mailto:[email-masked]] 
Sent: 24 January 2013 17:09
To: Dearlove, Christopher (UK)
Cc: Henning Rogge; [email-masked]
Subject: Re: [manet] RFC5444 schema-based compression V2

----------------------! WARNING ! ----------------------
This message originates from outside our organisation,
either from an external partner or from the internet.
Keep this in mind if you answer this message.
Follow the 'Report Suspicious Emails' link on IT matters
for instructions on reporting suspicious email messages.
--------------------------------------------------------

Yes, it does.

Henning Rogge

---

From: Henning Rogge <[email-masked]>
Date: Fri, 25 Jan 2013 11:42:01 +0100

On 01/25/2013 11:13 AM, Dearlove, Christopher (UK) wrote:
> The linklayer concept obviously means one would have to consider 5444
> packets, not just messages.

Yes, I agree. The packet-compression level is not there because I had to 
start at one point to write the idea down. I WANT a compression scheme 
for all parts of RFC5444 (including packet level and addresses), not 
just the TLVs of messages.

 > It could be argued to be a layer
> violation, but then several things done in such circumstances could
> be so considered.

Yes, thats right.

A "version 1" of packet-bb might be a better approach.

Henning Rogge

---

From: Abdussalam Baryun <[email-masked]>
Date: Fri, 25 Jan 2013 12:07:25 +0100

> A "version 1" of packet-bb might be a better approach.

V1 is better approach because more general than V2 for MANETs. If you
make a way in Schema to have the options both include or not-include
the 5444 packet, then will be better for more general MANETs,

my thoughts, not sure yet :(

AB

---

From: Teco Boot <[email-masked]>
Date: Fri, 25 Jan 2013 12:52:06 +0100

Op 25 jan. 2013, om 09:17 heeft Henning Rogge het volgende geschreven:

> Less than one byte doesn't make much sense, unless you want to start bitshift everything in the following data.
> 
> What we could do would be to allocate the first bit as a "7 bit/15 bit" switch for the schema id.
> 
> If the first bit is one, set it to zero and take the first TWO bytes as the schema number.
> 
> What do you think?

My thoughts are: this is done before: Using Self-Delimiting Numeric Values in Protocols.
http://tools.ietf.org/html/rfc6256

Further optimization, with values near to to 0xFF / 0xFFFF: flip the bits. The flip indicator could be in the schema or an additional bit, e.g. first octet, second bit leaving 6 bits for payload.

Teco

---

From: "Dearlove, Christopher (UK)" <[email-masked]>
Date: Fri, 25 Jan 2013 11:55:00 +0000

I have mixed views over this being version 1, but let's see where we go technically.

-----Original Message-----
From: Henning Rogge [mailto:[email-masked]] 
Sent: 25 January 2013 10:42
To: Dearlove, Christopher (UK)
Cc: [email-masked]
Subject: Re: [manet] RFC5444 schema-based compression V2

Yes, thats right.

Henning Rogge

---

From: "Dearlove, Christopher (UK)" <[email-masked]>
Date: Fri, 25 Jan 2013 12:00:26 +0000

If you know that this will be two bytes maximum, then RFC 6256 is wasteful, it has a maximum of 14 information bits rather than 15 information bits. Do we need to consider cases where message or TLV block length is more than 32K? (Though 5444 supports these in theory.)

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Teco Boot
Sent: 25 January 2013 11:52
To: Henning Rogge
Cc: [email-masked]
Subject: Re: [manet] RFC5444 schema-based compression V2

Teco

---

From: Teco Boot <[email-masked]>
Date: Fri, 25 Jan 2013 13:22:50 +0100

There is the SDNV-8/-16 method. Maybe leave this open and use the schema to select.

Teco

---

From: Henning Rogge <[email-masked]>
Date: Fri, 25 Jan 2013 16:00:39 +0100

If we are only need to encode 16 bits, we could do it this way:

1) if you encode 0-127, just use a single byte value.
2) if you encode 128 - 32511 (0x7eff), add 0x8000 and and use a two byte 
value.
3) if you encode 32512 - 65535, first write 0xff and then write the full 
two bytes.

Henning Rogge

---

From: Henning Rogge <[email-masked]>
Date: Fri, 25 Jan 2013 16:20:51 +0100

> There is the SDNV-8/-16 method. Maybe leave this open and use the schema to select.

After looking through RFC6256, I think we could easily use this one. The 
one-byte more overhead for TLV-blocks longer than 16K Byte is not really 
that relevant.

Henning Rogge

---

From: Teco Boot <[email-masked]>
Date: Fri, 25 Jan 2013 16:27:23 +0100

I'm fine as long as it is unlikely to face higher overhead encoding for values that are frequently used. My worries are for the 0xFFxx range.

Teco

Op 25 jan. 2013, om 16:20 heeft Henning Rogge het volgende geschreven:

> After looking through RFC6256, I think we could easily use this one. The one-byte more overhead for TLV-blocks longer than 16K Byte is not really that relevant.
> 
> Henning Rogge
> 
> --

---

From: Henning Rogge <[email-masked]>
Date: Fri, 25 Jan 2013 16:29:14 +0100

On 01/25/2013 04:27 PM, Teco Boot wrote:
> I'm fine as long as it is unlikely to face higher overhead encoding
> for values that are frequently used. My worries are for the 0xFFxx
> range.

I do not think compressed message length or Schema numbers larger than
16000 will be common for RFC5444 packets.

Henning Rogge

---

From: Abdussalam Baryun <[email-masked]>
Date: Fri, 25 Jan 2013 15:49:55 +0000

> 3) if you encode 32512 - 65535, first write 0xff and then write the full
For the third one do you mean it will be three bytes: 0xff  xx  xx    = 24
bits  not as the condition,
or do you mean encode 32512 - 32767, first write 0xff and complete the two
byte = 16 bits

Not sure how do you distinguish between values 32767 and 65535 and more
others?

Please help because I want to understand this,

AB

---

From: Abdussalam Baryun <[email-masked]>
Date: Fri, 25 Jan 2013 15:53:46 +0000

16000 is large for the cases, I agree too,

AB

On Fri, Jan 25, 2013 at 3:29 PM, Henning Rogge <
[email-masked]> wrote:

> mailto:henning.rogge@fkie.**fraunhofer.de<[email-masked]>
> http://www.fkie.fraunhofer.de
>
>

---

From: Teco Boot <[email-masked]>
Date: Fri, 25 Jan 2013 17:58:47 +0100

> 16000 will be common for RFC5444 packets.

I was thinking on tlv-fulltype, with experimental TLV types.
OK, maybe not optimize for experiments. And stick to RFC6256.

Teco

---

From: Abdussalam Baryun <[email-masked]>
Date: Fri, 25 Jan 2013 17:07:27 +0000

I agree to stick with RFC6256,

AB

---

From: "Dearlove, Christopher (UK)" <[email-masked]>
Date: Fri, 25 Jan 2013 18:07:44 +0000

If we are using this for lengths - of messages, TLV blocks etc. I sincerely hope we never see a value as high as 0xFF00.

(I don't think we are planning to use it for two byte values that aren't weighted low in that manner (i.e. not for sequence numbers etc.)

-----Original Message-----
From: Teco Boot [mailto:[email-masked]] 
Sent: 25 January 2013 15:27
To: Henning Rogge
Cc: Dearlove, Christopher (UK); [email-masked]
Subject: Re: [manet] RFC5444 schema-based compression V2

Teco

---

From: Henning Rogge <[email-masked]>
Date: Fri, 25 Jan 2013 20:58:53 +0100

> If we are using this for lengths - of messages, TLV blocks etc. I sincerely hope we never see a value as high as 0xFF00.

For length and maybe for the Schema-ID itself.

> (I don't think we are planning to use it for two byte values that aren't weighted low in that manner (i.e. not for sequence numbers etc.)

Yes, that would be stupid to do.

But I am currently thinking about a few additions to help with address
compression.

Henning Rogge

---

From: "Taylor, Rick" <[email-masked]>
Date: Mon, 28 Jan 2013 10:56:42 +0000

+1 for RFC6256 - and thanks for drawing my attention to it Teco, I had used it elsewhere without realising there was an RFC.

I do have a concern with schema compression at the packet level.  In order to maintain backwards compatibility, any packet received by a compliant RFC5444 parser must be able to determine the <message-type> and total-length of each message in the packet *even if schema-compressed*, just for the purposes of discarding the message if it is not a handled type.

If you ignore such back-compatibility then you are forced into using a separate port for schema-compression and I would much rather see schema-compression developed as a backwards-compatible extension to the RFC5444 message format, than a separate meta-protocol.

That being said, I am in favour of schema-compression, but only if it can be made to play nicely with RFC5444.

I also have some concerns over formal schema definition 'languages'. Although I hate them for their pedantry, I do see the need for them.  May I suggest using an XML format for formal specification of schemas, as I think there might be a danger trying to invent our own schema definition syntax that just adds complexity.

Rick Taylor

> Sent: 25 January 2013 19:59
> > If we are using this for lengths - of messages, TLV blocks etc. I
> sincerely hope we never see a value as high as 0xFF00.
>
> > (I don't think we are planning to use it for two byte values that aren't
> weighted low in that manner (i.e. not for sequence numbers etc.)
>
> > I'm fine as long as it is unlikely to face higher overhead encoding for
> values that are frequently used. My worries are for the 0xFFxx range.
> >
> > Teco
> >
> >> After looking through RFC6256, I think we could easily use this one.
> The one-byte more overhead for TLV-blocks longer than 16K Byte is not
> really that relevant.
> >>
> >> Henning Rogge
> >>
> >> --
The information contained within this e-mail and any files attached to this e-mail is private and in addition may include commercially sensitive information. The contents of this e-mail are for the intended recipient only and therefore if you wish to disclose the information contained within this e-mail or attached files, please contact the sender prior to any such disclosure. If you are not the intended recipient, any disclosure, copying or distribution is prohibited. Please also contact the sender and inform them of the error and delete the e-mail, including any attached files from your system. Cassidian Limited, Registered Office : Quadrant House, Celtic Springs, Coedkernew, Newport, NP10 8FZ Company No: 04191036 http://www.cassidian.com

---

From: Henning Rogge <[email-masked]>
Date: Mon, 28 Jan 2013 12:30:56 +0100

On 01/28/2013 11:56 AM, Taylor, Rick wrote:
> I do have a concern with schema compression at the packet level.  In
> order to maintain backwards compatibility, any packet received by a
> compliant RFC5444 parser must be able to determine the <message-type>
> and total-length of each message in the packet *even if
> schema-compressed*, just for the purposes of discarding the message
> if it is not a handled type.

I don't think the current idea supports mixing compressed and 
uncompressed messages in the same packet. There is also no bit left in 
the message-header flags we could use to mark "this is compressed".

But one way to determine "this is compressed" might be the packetbb 
version number.

> If you ignore such back-compatibility then you are forced into using
> a separate port for schema-compression and I would much rather see
> schema-compression developed as a backwards-compatible extension to
> the RFC5444 message format, than a separate meta-protocol.

I don't think IANA will give us another port/ip-protocol number, just 
for a compressed format.

But we still have to think about how to transmit it over the network.

> can be made to play nicely with RFC5444.

+1

> I also have some concerns over formal schema definition 'languages'.
> Although I hate them for their pedantry, I do see the need for them.
> May I suggest using an XML format for formal specification of
> schemas, as I think there might be a danger trying to invent our own
> schema definition syntax that just adds complexity.

I have already played with the idea of using ABNF (RFC5234) to specify 
the schema format. XML would also work, but we might also need a binary 
representation of the schema if we want to transmit it over the network.

Henning Rogge

---

From: "Taylor, Rick" <[email-masked]>
Date: Mon, 28 Jan 2013 11:57:33 +0000

My 'off-the-cuff' thoughts on schema discovery:

Specify an optional RFC5444 TLV to be allowed in schema-compressed message blocks that contains the URL that hosts the schema definition 'document' (another advantage for XML).  The URL could just refer to an FTP or HTTP service on the sender for private networks.

This would allow retrieval and/or extended validation of schemas by full-featured parsers (useful for a development environment) but not require support in all parsers (good for embedded devices) and act a little like namespaces in XML.

Embedding entire schemas within RFC5444 messages just seems too 'heavy', some level of indirection seems better.  It avoids "give me the latest schema text" transactions over datagram transport.

Rick Taylor

> Sent: 28 January 2013 11:31
> To: [email-masked]

---

From: Henning Rogge <[email-masked]>
Date: Mon, 28 Jan 2013 13:16:05 +0100

On 01/28/2013 12:57 PM, Taylor, Rick wrote:
> Specify an optional RFC5444 TLV to be allowed in schema-compressed
> message blocks that contains the URL that hosts the schema definition
> 'document' (another advantage for XML).  The URL could just refer to
> an FTP or HTTP service on the sender for private networks.

I don't think FTP or HTTP are a good idea in this context.

RFC5444 is made for transmitting packets/messages of ROUTING protocols.

If a node cannot understand this because of a missing schema, it will 
not have access to any services except the one the neighbors provide.

> This would allow retrieval and/or extended validation of schemas by
> full-featured parsers (useful for a development environment) but not
> require support in all parsers (good for embedded devices) and act a
> little like namespaces in XML.

> Embedding entire schemas within RFC5444 messages just seems too
> 'heavy', some level of indirection seems better.  It avoids "give me
> the latest schema text" transactions over datagram transport.

I think a binary format for the schemas would be a good idea (in 
addition to a normal "text form" ?).

This would also solve the transmission problem. Most schemas will be 
pretty short, I don't see a problem transmitting them in a single UDP 
packet, maybe even encapsulated in a RFC5444 packet-TLV or a special 
(and uncompressed) RFC5444 message..

Henning Rogge

---

From: "Taylor, Rick" <[email-masked]>
Date: Mon, 28 Jan 2013 13:17:47 +0000

> RFC5444 is made for transmitting packets/messages of ROUTING protocols.

I agree: RFC5444 is for ROUTING protocol messages, not meta-information messages.  I would always want to move meta-information out-of-band.

As I see it, either the schema is fixed (as in a final protocol) in which case the schema is documented and widely available, and all participants use the correct final version, or the schema is in flux (as in a prototype or development scenario) in which case participants need a method to check if they have a valid schema for compression/decompression.  In this later case I see it as perfectly sensible that the retrieval of the schema is out of band - either the parser does something automated to get and parse the schema, or it just errors with "Mismatched or unknown schema".

Adding the extra specification to cover the transmission of schema meta-information within RFC5444 is just making extra work for yourself.  I would much rather see a simple/flexible/lightweight solution such as an optional message-tlv that held a reference to the schema, rather than specifying an RFC5444 compliant schema format.  HTTP and FTP a perfectly sensible transport mechanisms, and in a production environment one would not want dynamically updating schemas anyway, so they would be disabled in the parser.

> not have access to any services except the one the neighbors provide.

In an embedded device this might be exactly what is desired. In a fuller-featured device being able to use HTTP or FTP (or whatever out-of-band mechanism is required) may be practical, allowing much more flexibility.

> addition to a normal "text form" ?).

I think it would make any schema compression RFC longer, harder to ratify and harder to implement.  I just don't see the need.  Let's reuse ABNF or XML or something (almost) human readable and keep the data out-of-band.  Binary formats shouldn't be transmitted and are implementation details for the parser developers.

> (and uncompressed) RFC5444 message..

But then someone asks about reliability, and we end up building TCP/RFC5444/IP and that takes 4 years to reach an uneasy consensus... ;)

Rick Taylor

---

From: Henning Rogge <[email-masked]>
Date: Tue, 29 Jan 2013 09:14:02 +0100

On 01/28/2013 01:16 PM, Taylor, Rick wrote:>> On 01/28/2013 12:45 PM,
Taylor, Rick wrote:
> One potential solution would be to add a NUL message-block (length =
>  0, 4 bytes) to mark the end of the version-0 message-blocks, and
> the start of the schema-compressed blocks.  Not nice, and adds 4
> bytes, and it still breaks version-0 parsers, but does allow a
> mix-and-match approach and general-purpose parsers.

If we are willing to break compatibility, we can easily add a "number of
uncompressed/compressed messages" byte into the message. No need for
some hack with a message with zero length (which would not be valid
RFC5444 anyways).

> Agreed, it's not the nicest solution, but it does allow
> schema-compressed messages to be handled safely by a pre version-0
> parser, at a cost of 3 or 4 bytes per schema-compressed
> message-block. This is very much a work-around but it does allow best
> compatibility.

I don't think its worth the effort.

> As I said above, I think this is out of scope for RFC5444, some
> higher level protocol can determine this.

I don't like the idea bootstrapping a lowlevel protocol with a highlevel 
one.

 > Does the packetbb security RFC go into this at all?

No.

On 01/28/2013 02:17 PM, Taylor, Rick wrote:
> I agree: RFC5444 is for ROUTING protocol messages, not
> meta-information messages.  I would always want to move
> meta-information out-of-band.

Its still routing information, so I think RFC5444 could be a place for 
this. It would also solve the bootstrap problem easily.

> As I see it, either the schema is fixed (as in a final protocol) in
> which case the schema is documented and widely available, and all
> participants use the correct final version, or the schema is in flux
> (as in a prototype or development scenario) in which case
> participants need a method to check if they have a valid schema for
> compression/decompression.  In this later case I see it as perfectly
> sensible that the retrieval of the schema is out of band - either the
> parser does something automated to get and parse the schema, or it
> just errors with "Mismatched or unknown schema".

Errors like this in a routing protocol are a good way to break your 
manet apart.

> Adding the extra specification to cover the transmission of schema
> meta-information within RFC5444 is just making extra work for
> yourself.  I would much rather see a simple/flexible/lightweight
> solution such as an optional message-tlv that held a reference to
> the schema, rather than specifying an RFC5444 compliant schema
> format. HTTP and FTP a perfectly sensible transport mechanisms, and
> in a production environment one would not want dynamically updating
> schemas anyway, so they would be disabled in the parser.

I am not sure "lightweight" and HTTP/FTP mix well in a Manet.

In addition to this including a full URL "where to download the schema" 
into a RFC5444 message/packet would be a huge waste of bytes.

> I think it would make any schema compression RFC longer, harder to
> ratify and harder to implement.  I just don't see the need.  Let's
> reuse ABNF or XML or something (almost) human readable and keep the
> data out-of-band.

I don't agree here, but we have to see the final structure of a schema 
before we can decide on this.

Henning Rogge

---

From: Christopher Dearlove <[email-masked]>
Date: Tue, 29 Jan 2013 08:42:02 +0000

I've yet to form a definite opinion on this work, but I have one definite opinion. No breaking compatibility. A minimum requirement is if version is zero, no changes to format. And creating a version one has definite issues that's part of my yet to firm a definite opinion. (As do other solutions - encapsulating in TLV, link layer changes.) 

--  
Christopher Dearlove
[email-masked] (iPhone)
[email-masked] (home)

On 29 Jan 2013, at 08:14, Henning Rogge <[email-masked]> wrote:

> I don't like the idea bootstrapping a lowlevel protocol with a highlevel one.
> 
> Its still routing information, so I think RFC5444 could be a place for this. It would also solve the bootstrap problem easily.
> 
> Errors like this in a routing protocol are a good way to break your manet apart.
> 
> In addition to this including a full URL "where to download the schema" into a RFC5444 message/packet would be a huge waste of bytes.
> 
> I don't agree here, but we have to see the final structure of a schema before we can decide on this.
> 
> Henning Rogge
> 
> --

---

From: "Taylor, Rick" <[email-masked]>
Date: Tue, 29 Jan 2013 10:37:27 +0000

I think we are all agreed that breaking RFC5444 is not an option.

My questions and comments were more about compatibility between RFC5444 and schema-compression, and the options I see are:

A) No compatibility - schema-compression feels like RFC5444 but is sufficiently different to require a different parser to use.

B) Backwards compatible - A schema-compression parser can parse RFC5444 and schema-compressed packets.

C) Forwards compatible - An RFC5444 parser can parse a packet containing schema-compressed messages, handling unrecognised messages according to the rules of RFC5444.

All my suggestions so far have been directed at solving options B and C.  The suggestions do come at a cost in extra bytes in the RFC5444 packet and some loss of elegance, but they do allow for some level of compatibility.

Really, to make progress, there needs to be some consensus on what level of compatibility people wish to see defined.

My personal recommendation would be that any schema-compression protocol maintains backwards compatibility (B), and an appendix is added to any specification detailing optional functionality for forwards compatibility (C), such as packet level RFC5444 TLVs.

Rick Taylor

P.S> From my reading of RFC5444, a message with no TLVs is perfectly valid - am I wrong?

> From: Christopher Dearlove [mailto:[email-masked]]
> Sent: 29 January 2013 08:42
> To: Henning Rogge
> Cc: Taylor, Rick; [email-masked]
> I've yet to form a definite opinion on this work, but I have one definite
> opinion. No breaking compatibility. A minimum requirement is if version is
> zero, no changes to format. And creating a version one has definite issues
> that's part of my yet to firm a definite opinion. (As do other solutions -
> encapsulating in TLV, link layer changes.)
>
> --
> Christopher Dearlove

---

From: Henning Rogge <[email-masked]>
Date: Tue, 29 Jan 2013 11:40:25 +0100

> P.S> From my reading of RFC5444, a message with no TLVs is perfectly valid - am I wrong?

See https://tools.ietf.org/html/rfc5444#section-5.2

A minimum RFC5444 message is made from a message-header AND a tlv-block 
(without TLVs inside).

The length field of this message would contain the value 6.

Henning Rogge

---

From: Henning Rogge <[email-masked]>
Date: Tue, 29 Jan 2013 15:02:52 +0100

Use the link that is below every MANET list to go to the webinterface of 
the mailing list:

Put your email (the one you are using for this list) into the field 
besides the "unsubscribe or edit options" button and press the button.

Click on the "unsubscribe" button.

You should receive an "unsubscribe" mail with a link you have to click 
on... afterwards you are removed from the list.

Henning Rogge

On 01/29/2013 02:15 PM, Bush, Stephen F (GE Global Research) wrote:
> Can someone please remove me from this list?
>
> The email server said I was removed, but I'm still receiving messages.
>
> Thanks,
> Stephen Bush
> Scientist, GE Global Research
> http://www.research.ge.com/~bushsf
> Chair IEEE P1906.1 Nano/Molecular Communication http://standards.ieee.org/develop/project/1906.1.html
>
>
>
> Sent: Tuesday, January 29, 2013 5:40 AM
> To: Taylor, Rick
> Cc: [email-masked]
> Subject: Re: [manet] RFC5444 schema-based compression - compatibility
>
> A minimum RFC5444 message is made from a message-header AND a tlv-block (without TLVs inside).
>
> Diplom-Informatiker Henning Rogge , Fraunhofer-Institut für Kommunikation, Informationsverarbeitung und Ergonomie FKIE Kommunikationssysteme (KOM) Fraunhofer Straße 20, 53343 Wachtberg, Germany

---

From: Joseph Macker <[email-masked]>
Date: Tue, 29 Jan 2013 13:11:22 -0500

Definitely check the ROHC community documentation for any relevant lessons
learned.

---

From: Henning Rogge <[email-masked]>
Date: Wed, 30 Jan 2013 08:37:06 +0100

On 01/29/2013 07:11 PM, Joseph Macker wrote:
> Definitely check the ROHC community documentation for any relevant
> lessons learned.

Do you have a special document in mind (maybe with an URL) ? I have 
worked with ROHC in the past here at the institute, but I don't remember 
a 'community documentation'.

Henning Rogge