Subject: Re: IMAP/POP compression for slow links
From: Ned Freed <[email-masked]>
To: Dan Wing <[email-masked]>
Cc: [email-masked]
Date: Sun, 01 Aug 1999 18:31:51 -0700 (PDT)

> As someone else pointed out, POP and IMAP is where the users feel an
> impact.  I would argue that few people run SMTP servers on slow links, and
> while most users at work are on fast links (typically ethernet), most
> users at home or on the road are on slow links.

> Thus, perhaps an IMAP (and POP) extension to allow on-the-fly compression
> would give the biggest bang for the buck.  It has an additional advantage
> of not causing the interoperability problems with a new Content-Encoding.

Again, you can use TLS with both POP and IMAP (and this is supported by
quite a few servers). And TLS provides compression as an option, which
is good since TLS will severely limit the ability of modems and such to
perform link-level compression.

> Such an extension would be useful even after RESCAP and some kind of
> Content-Encoding compression were being deployed and would provide a
> benefit to users even if the sender didn't support RESCAP or the new
> Content-Encoding compression.

Actually, the extension that's needed in IMAP is one to remove CTEs completely.
Do this and use TLS compression and you should have a reasonably effective
solution for large messages. And its one that even works for messages already
in message stores.

				Ned

Date: Sun, 01 Aug 1999 12:12:33 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: compressed content-transfer-encoding?
To: Keith Moore <[email-masked]>
Cc: Larry Masinter <[email-masked]>, Keith Moore <[email-masked]>, [email-masked]
Sender: [email-masked]

> > Even though application/zip is lame, other mechanisms might be
> > worse. Maybe we should look harder at providing what would be
> > necessary to make 'application/zip' actually work, e.g., some
> > top-level indication of what's actually in the package?

> this has the same deployment barrier as adding a new
> content-transfer-encoding - in either case the mime mail
> reader needs to know what to do with the extra parameter
> or the new c-t-e.

Actually the problems are far worse than a new CTE. We end up having to  add a
bunch of really gross nonorthogonal nonsense to MIME headers and the butchery
we'd need on the IMAP front is too terrible to contemplate. And besides, this
is effectively a non-lead CTE, and as such runs smack into the no-nested-CTE
consensus that allowed MIME to deploy in the first place.

				Ned

From: Keith Moore <[email-masked]>
To: Jacob Palme <[email-masked]>
cc: [email-masked], "Keith Moore" <[email-masked]>
Subject: Re: compressed content-transfer-encoding? 
Date: Sun, 01 Aug 1999 15:11:53 -0400
Sender: [email-masked]

> At 12.02 -0400 99-07-31, Keith Moore wrote:
> > > Even though application/zip is lame, other mechanisms might be
> > > worse. Maybe we should look harder at providing what would be
> > > necessary to make 'application/zip' actually work, e.g., some
> > > top-level indication of what's actually in the package?
> >
> >this has the same deployment barrier as adding a new
> >content-transfer-encoding - in either case the mime mail
> >reader needs to know what to do with the extra parameter
> >or the new c-t-e.
> 
> No, it has one very important advantage: It will co-work,
> with existing software using application/zip, in the way
> people are already accustomed to sending and receiving
> compressed e-mail in that format.

it will co-work with existing software that most people don't
have, in the way that a small minority of people are already
accustomed to working.  the vast majority of people will have
to either install new software or learn how to cope with
zip files, or both.  as long as people have to install new
software, why not have them install a new MIME mail reader?

saying that people are already accustomed to using zip is a
lot like saying that AOL comes with built-in MIME support.

> At 13.36 -0400 99-07-31, Keith Moore wrote:
> >so what you are proposing to do is to clutter up the MIME
> >architecture and degrade the recipient's user interface
> >just so a minority of users who already use zip don't have
> >to upgrade immedately.  in the long run I don't think it's
> >worth it.
> 
> The zip format may be in a minority of all mail attachments,
> simply since most mail attachments are not compressed
> using any compression format. But the zip format is
> certainly in a large majority of the e-mailed attachments
> which are compressed at all, and which are sent to me
> by various people around the world. The only other
> compression formats which are common in e-mail are
> the JPEG and GIF formats. I hardly ever get any e-mail
> with attachments in any other compression format.

you're missing the point.  just because you have zip installed
on your computer doesn't mean that the majority of computer
users have zip installed.

eith

Date: Sun, 01 Aug 1999 12:06:57 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: RE: compressed content-transfer-encoding?
To: Larry Masinter <[email-masked]>
Cc: [email-masked], [email-masked]
Sender: [email-masked]

> > > Even though application/zip is lame, other mechanisms might be
> > > worse. Maybe we should look harder at providing what would be
> > > necessary to make 'application/zip' actually work, e.g., some
> > > top-level indication of what's actually in the package?
> >
> > this has the same deployment barrier as adding a new
> > content-transfer-encoding - in either case the mime mail
> > reader needs to know what to do with the extra parameter
> > or the new c-t-e.

> I'm not sure this is true. Unaware MIME mailers will just
> see they have application/zip, that they either recognize
> or not, and users of existing MIME mail programs have simple
> way of configuring their mail reader to at do something
> sensible with it.

> Adding a new content-transfer-encoding has a more serious
> deployment problem, because most deployed systems don't have
> any kind of extensibility built in for CTE.

Believe me, from a tech support perspective the problems here are far worse
than those associated with a CTE.  Think about the implications in the context
of IMAP, for example, where separate fetch of body parts is an important
feature.

> Mailing around zip files is common practice. Many people are
> used to dealing with getting a zip file. Leveraging this
> just means making the common practice easier to accomplish
> for senders, and less awkward for receivers; it's a product
> enhancement that mail client vendors could add today.

Summary and unconditional rejection of ZIPs is also common because of the
inability in some environments to check the content for viruses.

> In the menu for 'attach file', it could just have a check
> box for 'send zipped', for example, and a setting for making
> 'send zipped' the default for attachments, or for attachments
> that aren't known to already have better media-specific
> compression.

In case it isn't clear, I am absolutely opposed to increased use of
application/zip within MIME. We need to solve the real problem here, which is
to allow end-to-end knowledge of what C-Ts and CTEs are supported. Once this is
done we can deploy new CTEs or whatever else we chose willy-nilly. And until it
is done none of this stuff, with the exception of TLS-based compression, stands
a snowball's chance in hell of being widely deployed.

				Ned

Date: Sun, 1 Aug 1999 12:00:46 +0200
To: [email-masked]
From: Jacob Palme <[email-masked]>
Subject: Re: compressed content-transfer-encoding?
Cc: "Keith Moore" <[email-masked]>
Sender: [email-masked]

At 12.02 -0400 99-07-31, Keith Moore wrote:
> > Even though application/zip is lame, other mechanisms might be
> > worse. Maybe we should look harder at providing what would be
> > necessary to make 'application/zip' actually work, e.g., some
> > top-level indication of what's actually in the package?
>
>this has the same deployment barrier as adding a new
>content-transfer-encoding - in either case the mime mail
>reader needs to know what to do with the extra parameter
>or the new c-t-e.

No, it has one very important advantage: It will co-work,
with existing software using application/zip, in the way
people are already accustomed to sending and receiving
compressed e-mail in that format.

At 10.05 -0700 99-07-31, Paul Hoffman / IMC wrote:
>I see a problem here with making a content type (zipped) act like a
>c-t-e. Zipped seems fine for "attachments", that is, leaves in the
>MIME tree. But some of the requirement for making messages smaller
>would want to compress a whole message, which might be a nested
>multipart. At this point, zipped hides the lower layers. If we want
>a rule that say "you can't use app/zipped for multiparts", how does
>this become different than a c-t-e?

Zip is not used to compress multiparts, just leaves. This is no
serious restriction, since the main part of e-mail messages seldom
needs compression. It is the attachments which are often large and
bulky. Even when the main part is in HTML format, all the images are
in other than the main body part.

At 13.36 -0400 99-07-31, Keith Moore wrote:
>so what you are proposing to do is to clutter up the MIME
>architecture and degrade the recipient's user interface
>just so a minority of users who already use zip don't have
>to upgrade immedately.  in the long run I don't think it's
>worth it.

The zip format may be in a minority of all mail attachments,
simply since most mail attachments are not compressed
using any compression format. But the zip format is
certainly in a large majority of the e-mailed attachments
which are compressed at all, and which are sent to me
by various people around the world. The only other
compression formats which are common in e-mail are
the JPEG and GIF formats. I hardly ever get any e-mail
with attachments in any other compression format.
------------------------------------------------------------------------
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/~jpalme

To: Keith Moore <[email-masked]>
Cc: [email-masked] (Kai Henningsen), [email-masked]
Subject: Re: compressed content-transfer-encoding?
From: Arnt Gulbrandsen <[email-masked]>
Date: 01 Aug 1999 20:36:34 +0200
Lines: 18
Sender: [email-masked]

Keith Moore <[email-masked]>
> depends on which PPP compression algorithm is used. the typical
> (for me at least) "BSD compress" algorithm is almost the same 
> algorithm as the one used in modems.

Yes, but the data is different :) A PPP-based algorithm can look ahead
until the end of the IP packet without cost.  A modem has to wait for
each byte coming in across a comparatively slow connection, so good
compression means waiting for a "long" time before emitting output.

In the extreme case, a 56k modem with a 115,200bps host link and
twelve-bit BSD compression can't read more than about 1.5-2 characters
before it has to emit output or introduce latency.

(Maybe I've just understood why people use these newfangled host
connections - EPP or whatever.  I forget the abbreviation.)

--Arnt

From: Keith Moore <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
cc: [email-masked] (Kai Henningsen), [email-masked]
Subject: Re: compressed content-transfer-encoding? 
Date: Sun, 01 Aug 1999 14:12:09 -0400
Sender: [email-masked]

> > There's a fairly simple reason. This type of compression must keep  
> > latencies from growing too much. This *really* hurts compression.
> 
> I see what you mean.  However, PPP compression (RFC 1962) should not
> have this problem, since the additional time taken to start sending
> each packet should be more than offset by the decrease in transmission
> time.

depends on which PPP compression algorithm is used. the typical
(for me at least) "BSD compress" algorithm is almost the same 
algorithm as the one used in modems.   but then again I use 
NetBSD. I don't know what compression Mac or Windoze stacks use.

my understanding is that with compressing modems, the main additional 
value of PPP compression is in reducing interrupt overhead.

Keith

From: Keith Moore <[email-masked]>
To: [email-masked] (Kai Henningsen)
cc: [email-masked]
Subject: Re: compressed content-transfer-encoding? 
Date: Sun, 01 Aug 1999 14:06:15 -0400
Sender: [email-masked]

> I don't know about you, but in my experience, modem compression (and  
> similar schemes) typically does a fairly poor job.
> 
> There's a fairly simple reason. This type of compression must keep  
> latencies from growing too much. This *really* hurts compression.

actually my experience is just the opposite ... modem compression
(usually based on LZW) does a reasonably good job ... not as good
as gzip or bz2 but close enough.   and yes, one difference between
the effectiveness of these algorithms is the amount of latency/
lookahead that is needed.  LZW is amazingly good for an algorithm
needing only a fixed amount of memory and one octet lookahead.

but for me the disappointing thing about modems is that they add far 
more latency than you would expect given their speed.  I don't think 
this is the fault of the compression so much as the error correction
(I suspect that many modems rely on error correction to compensate 
for marginal analog circuitry)

> > you'll get more connection time savings from pipelining smtp than you
> > will from compressing the data.
> 
> That's only true if the time is dominated by RCPT TO: handling, that is,  
> you're sending many small mails. When you're sending a few large mails,  
> pipelining is completely irrelevant, and compression is very important,  
> because you spend most of the time in the DATA phase anyway.

right.  but even these days most messages are small.  and I was assuming 
that this traffic was already going over compressing modems, which would 
save almost as much bandwidth as compressing smtp data.

Keith

To: [email-masked] (Kai Henningsen)
Cc: [email-masked]
Subject: Re: compressed content-transfer-encoding?
From: Arnt Gulbrandsen <[email-masked]>
Date: 01 Aug 1999 19:24:08 +0200
Lines: 13
Sender: [email-masked]

[email-masked] (Kai Henningsen)
> I don't know about you, but in my experience, modem compression (and  
> similar schemes) typically does a fairly poor job.
> 
> There's a fairly simple reason. This type of compression must keep  
> latencies from growing too much. This *really* hurts compression.

I see what you mean.  However, PPP compression (RFC 1962) should not
have this problem, since the additional time taken to start sending
each packet should be more than offset by the decrease in transmission
time.

--Arnt

Date: 01 Aug 1999 13:39:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: compressed content-transfer-encoding?
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Dave Crocker)  wrote on 31.07.99 in <[email-masked]>:

> At 07:28 AM 7/29/99 , Keith Moore wrote:
> > > We already have a standard for sending compressed data in e-mail.
> > > We have an IANA registered content-type application/zip.
> >
> >application/zip is not a standard.
>
> Just to beat this one into the ground:
>
> It's also not a compression mechanism.
>
> It does "bagging" of independent obkects, but does not compress the bits.

This turns out not to be the case.

The application/zip format typically compresses the contained objects with  
deflate. ("typically" because there are other algorithms of only historic  
significance, and also the option not to use compression; but it's been a  
long time since I last saw anything except deflate.)

Now, I certainly agree that this is a poor match with MIME, but the reason  
is that it uses a different, incompatible mechanism to describe the  
contained objects.

MfG Kai

Date: 01 Aug 1999 13:46:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: compressed content-transfer-encoding?
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Keith Moore)  wrote on 31.07.99 in <[email-masked]>:

> > We really do need to remember just how limited bandwidth (in particular)
> > and "Internet performance" are in much of the Internet.  Added to that is
> > that much of the user community pays a usage fee for connection time.
>
> sure, but compressing modems *are* widely deployed.  if the "connection"
> uses a compressing modem, any additional savings in "connection time"
> resulting from smtp or mime compression is truly marginal.

I don't know about you, but in my experience, modem compression (and  
similar schemes) typically does a fairly poor job.

There's a fairly simple reason. This type of compression must keep  
latencies from growing too much. This *really* hurts compression.

> you'll get more connection time savings from pipelining smtp than you
> will from compressing the data.

That's only true if the time is dominated by RCPT TO: handling, that is,  
you're sending many small mails. When you're sending a few large mails,  
pipelining is completely irrelevant, and compression is very important,  
because you spend most of the time in the DATA phase anyway.

I don't think anyone really wants compression for small mails.

MfG Kai