Subject: LZS compression for IPPCP

From: Bob Monsour <[email-masked]>
Date: Fri, 01 Aug 1997 09:25:46 -0700

It has not yet been announced on the list, but there is a new draft that
describes the use of LZS compression with the IP Payload Compression
Protocol (ippcp).

Title: IP Payload Compression Using LZ

Abstract

   This document describes a IP compression method based on the LZS
   compression algorithm. This document defines the application of the
   LZS algorithm to the IP Payload Compression Protocol [Thomas].
   [Thomas] defines a method for applying lossless compression to the
   payloads of Internet Protocol datagrams.

It can be found at:
	
	ftp://ds.internic.net/internet-drafts/draft-ietf-ippcp-lzs-00.txt

I will be presenting it in Munich at the ippcp wg meeting.

Please let me know if you have any questions.

Just to preempt one question that has already been raised, the payload
format described in the draft has a 4-byte field, of which only 1 bit is
specified. This bit, C, is specified to indicate whether or not a datagram
is compressed. Since the release of this draft, an alternate solution has
been proposed which would not require this 4-byte area to be consumed.

Regards,

-Bob

---

From: Avram Shacham <[email-masked]>
Date: Fri, 01 Aug 1997 16:14:21 -0700

At 04:03 PM 8/1/97 -0700, Daniel Harkins wrote:
>> What about IP compression without IPSec?  For example, IP compression can
>> prevent firewalls from "seeing" ULP fields, such as TCP port.
>
>  Do you see this as a benefit? 

No. That is the reason I suggested to detail such scenarios in "Security
Consideration".

>I hear enough grumbling that IPSec masks
>that information and renders typical switching and queueing methods void.
>With IPSec I at least can try to convince customers that this is a "feature" 
>because it prevents traffic analysis. 
>  I'm not saying compression without IPSec is bad (in fact, I'm pretty 
>agnostic about it), 

IP compression without IPSec may be useful, for example in the environment
of congested/slow links where bandwidth is expensive and CPU time is cheap.

>just that preventing firewalls, or just plain ol' routers,
>from seeing ULP fields is _not_ good.
>

Agree. 

avram

---

From: Bob Monsour <[email-masked]>
Date: Fri, 01 Aug 1997 17:01:41 -0700

At 03:36 PM 8/1/97 -0700, Avram Shacham wrote:
[snip]
>>   The LZS algorithm uses a sliding window of 2,048 bytes.
>
>In the typical IP over Ethernet environment, where MTU is 1500 bytes, why
>use 2048 bytes as a sliding window?

The LZS algorithm has been in use for quite a while in PPP, where the
longer the window, the better the compression. It was originally invented
with a 2048-byte window. For most of our customers who already have
licenses, there is no change to their code. They can use their existing
code to support IPComp. Clearly, shorter window versions can easily be
defined, and those would have lower memory requirements (however, since IP
compression is stateless, the mem reqmts are low already).

>>2.2 Anti-expansion of Payload Data
[snip]
>These paragraphs already appear in the protocol draft and afterwards in the
>architecture draft. Why duplicate and not just refer to the
>architecture/protocol I-D?

Agree. They'll come out of the next draft.

>>4. Security Considerations
[snip]
>prevent firewalls from "seeing" ULP fields, such as TCP port.

Other than DanH's comment, this was a hold over from my earlier draft.

>>7. Appendix: Compression Efficiency versus Datagram Size
[snip]
>In the IP environment, where each packet is compressed without history (as
>you defined in sec 2.1 of this draft), how can one compress a 3.2MB file "as
>a single payload"?  Instead, could you refer to the results of IP
>compression using LZS, that I presented in IETF #38?  The results of the
>tests show a ratio of 1.78:1 for the Calgary files when MTU of 1500 bytes is
>used.

The "single payload" statement was to give a data point on the overall
compressibility of the data, not to say that it was a realistic payload
(unless it is an IPv6 jumbo datagram, which I fully expect we'll all see in
our lifetime). Your 1.78 clearly lands between the 1.74 and 1.91 for the
1024 and 2048 byte packets as expected. While your 1500-byte MTU is very
real, I would expect a reader to understand that 1500-bytes would get
somewhere between the 1.74 and 1.91.

>Moreover, the same results show that packets of size less than 128 bytes
>rarely compress.

All I'm trying to show here is that, with the Calgary data set, 64-byte
stateless packets compress, on average at 1.18:1 and 128-byte stateless
packets compress, on average, at 1.28:1. It's just a data point, I should
add encouragement to prospective users to take their own data as
comrpession varies by data type and packet size.

---

From: Bob Monsour <[email-masked]>
Date: Fri, 01 Aug 1997 17:05:13 -0700

At 04:14 PM 8/1/97 -0700, Avram Shacham wrote:
[snip]
>of congested/slow links where bandwidth is expensive and CPU time is

I would expect that in the "environment of congested/slow links where
bandwidth is expensive and CPU time is cheap", that PPP compression is in
use, typically getting a higher compression ratio due to the statefulness
of PPP. That sounds a lot like the end-user dial-up environment. Is that an
incorrect expectation?

-Bob

---

From: Avram Shacham <[email-masked]>
Date: Fri, 01 Aug 1997 17:12:12 -0700

>of PPP. That sounds a lot like the end-user dial-up environment. Is that an

No. The scenario described is Ethernet/Token-ring, where many users are
sharing the congested infrastructure. No PPP involved.

avram

---

From: Avram Shacham <[email-masked]>
Date: Fri, 01 Aug 1997 17:42:22 -0700

>compression is stateless, the mem reqmts are low already).

Historic reasons are common in products, but if you can prevent unused
memory - that's better. Every driver-writer or user should appreciate
reducing memory requirements by (2048-1500)*n, where n is number of bytes
the LZS algorithm is using per 1-byte window.

>somewhere between the 1.74 and 1.91.

The data that I measured, plus feedback from several participants in the
last IETF, point to the ratio of 1.78:1 as realistic for the IP environment
when transferring the benchmark Calgary files.

Your current draft details the usage of LZS in the IP environment. Twice in
your draft you used the compression ratio of 2:1 or better. How do you base
this data using IP datagrams? 3.2MB IP datagrams are yet to be in common use.

>comrpession varies by data type and packet size.

Again, how do you derive these ratios? And could you detail what percentage
of the packets of size less than 128 bytes fail to compress (i.e., expand)?

avram

---

From: Bob Monsour <[email-masked]>
Date: Fri, 01 Aug 1997 17:57:22 -0700

At 05:42 PM 8/1/97 -0700, Avram Shacham wrote:
[snip]
>the LZS algorithm is using per 1-byte window.

As I said, it is possible to define LZS with varying window sizes. All of
the implementations that I know of can decompress packets compressed with
smaller window sizes. We will have to define them as unique compression
algorithms, however, since an a packet compressed using 2048-byte window,
may not be decompressible by one that uses a smaller window.

>this data using IP datagrams? 3.2MB IP datagrams are yet to be in common

The draft states "Each entry in the table shows the compression ratio that
was achieved when LZS was applied to a test file using datagrams of a
specified size." What this means (and perhaps the description is not clear
enough) is that the file was split into "datagrams" of the size shown in
the table. For example, the column labeled 1024 shows a compression ratio
of 1.74:1. This means that from the start of the file, 1024-byte chunks of
the file were compressed independently, producing an average compression
ratio of 1.74:1. This only treats the 1024 as "payload" and does not
account for any protocol overhead; it is simply the reduction in size of
the datagram payload.

>Again, how do you derive these ratios?

See explanation above.

>And could you detail what
>percentage

I don't have that data right now, but we certainly can provide it. In my
next draft, I will also refer to the data you presented at the Memphis
meeting.

---

From: Roy Pereira <[email-masked]>
Date: Mon, 4 Aug 1997 14:12:17 -0400

I would much rather see a generic IPCOMP protocol header document and
then individual compression algorithm documents that use that protocol
header.  Thus this document should either use Matt's protocol header and
just describe how LZS works with it or be split up into two documents
(header and algorithm)

>----------
>From: 	Bob Monsour[SMTP:[email-masked]]
>Sent: 	Friday, August 01, 1997 12:25 PM
>To: 	[email-masked]
>Subject: 	LZS compression for IPPCP
>

---

From: Bob Monsour <[email-masked]>
Date: Mon, 04 Aug 1997 12:40:43 -0700

>just describe how LZS works with it or be split up into two documents

I agree. When I went to draft this, I found Matt's draft somewhat lacking,
so I added some things that I thought belonged in the "whole solution". I
think we should factor out several things from my draft and add them to
Matt's. I agree with the one header spec and multiple algorithm spec approach.

-Bob
[snip]