Subject: [NNTP] Compressed LIST answers

From: Julien ÉLIE <[email-masked]>
Date: Sat, 21 Nov 2009 14:32:52 +0100

Hi,

Another idea suggested by Urs is to have LIST variants which
send compressed answers.
Especially for ACTIVE and NEWSGROUPS.

The data should be compressed and probably also encoded.

Should we try to standardize that?

LIST ZACTIVE, LIST ZNEWSGROUPS?
or a new capability ZLIST (therefore ZLIST ACTIVE, ZLIST NEWSGROUPS
and any other keywords -- which would have exactly the same meaning
as its LIST variant)?

Should we force a format?  (lzma-compressed and base64-encoded?)
Or should we negotiate it?  (saying that lzma+base64, at least, MUST
be implemented)

For instance:
ZLIST LZMA BASE64 ACTIVE
ZLIST NONE NONE ACTIVE         -> which would be like LIST ACTIVE
ZLIST LZMA NONE ACTIVE         -> for binary-clean implementations
ZLIST BZIP2 YENC ACTIVE comp.*

The problem is that IANA does not register, as far as I can see,
compression names or encoding names like these.
I was imagining something like SASL mechanisms could be useful
to have here.
Because if we force lzma+base64, what if new algorithms more
efficient arrive?  There is lzma2 for instance.  And base64 is
not the best encoding...

Any comments on that?

The same mechanism could also be applied to OVER, HDR, NEWNEWS, etc.
Maybe ARTICLE...  and for clients, IHAVE, TAKETHIS if we want.

But if that is the case, maybe we will need a generic command
for compressed commands...  Because it will be a lot if we have
ZOVER, ZHDR, ZNEWNEWS, ZTAKETHIS, etc.

Any idea?
Or any other suggestion to handle LIST compression and possibly
all NNTP commands?

-- 
Julien ÉLIE

« Ce qui m'intéresse chez une femme, c'est ce que
  je n'ose pas lui demander. » (Wolinski)

---

From: Julien ÉLIE <[email-masked]>
Date: Sat, 28 Nov 2009 09:38:53 +0100

Hi all,

> ZLIST BZIP2 YENC ACTIVE comp.*

Is that construction a bad idea?

Is it best to have new commands?

  ZHDR LZMA BASE64 Injection-Info 1-
  ZLIST LZMA BASE64 ACTIVE
  ZOVER LZMA BASE64 1-
  ZTAKETHIS LZMA NONE 123 <valid@mid>

or a generic one (name to be defined) like:

  ZCOMMAND LZMA BASE64 0 HDR Injection-Info 1-
  ZCOMMAND LZMA BASE64 0 LIST ACTIVE
  ZCOMMAND LZMA BASE64 0 OVER 1-
  ZCOMMAND LZMA NONE 123 TAKETHIS <valid@mid>

The advantage of the first form is that it can be defined
for each command (but more exhausting to define).
The generic form would allow anybody to use compressed
commands for anything that supports it.
We could have a new capability entry:
  ZCOMMAND HDR LIST NEWNEWS OVER STREAMING
showing the compressed commands understood (the name of
a capability follows ZCOMMAND in CAPABILITIES).

But how to mention that HEAD can be compressed...  Unless
HEAD and CAPABILITIES are the only commands that cannot
be compressed.  The resulting output is small, though.
So maybe it is OK to do that.

Note that I added a third argument (default to 0 when
unused) because we can send unencoded binaries.  So we need
to know the number of bytes to receive.

  ZCOMMAND LZMA NONE 5 TAKETHIS <valid@mid>
  12345\r\n
  .\r\n

Any comment about that?
Is it a wrong direction I am suggesting here?
How could it be done better?

> efficient arrive?  There is lzma2 for instance.  And base64 is

Could we imagine to have a IANA registry for that? (GZIP, LZMA, LZMA2,
YENC, BASE64, etc.)

-- 
Julien ÉLIE

« Et cette même nuit, c'est-à-dire trois semaines plus tard... » (Astérix)

---

From: Russ Allbery <[email-masked]>
Date: Sat, 28 Nov 2009 01:38:39 -0800

>  ZCOMMAND LZMA NONE 123 TAKETHIS <valid@mid>

I think a generic command is an interesting idea and definitely better
than creating a bunch of new separate commands.

Although I wonder if, at that point, whether we want to just provide some
facility negotiate compression of all subsequent traffic on the NNTP
connection.  In other words, rather than treating this on a
command-by-command basis, what if we model it after STARTTLS?  It has the
advantage of simplicity of description, but it has the serious drawback
that it's another data layer, and I know from SASL and TLS that managing
layers can be a huge pain and it's very easy to do it poorly.

> Could we imagine to have a IANA registry for that? (GZIP, LZMA, LZMA2,

There sort of is one already for the HTTP Content-Coding header, but it
only specifies compress, gzip/deflate, and something called pack200-gzip
which is apparently used with Java.  It seems like a reasonable IANA
registry to me.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Julien ÉLIE <[email-masked]>
Date: Sat, 28 Nov 2009 10:56:47 +0100

Hi Russ,

> than creating a bunch of new separate commands.

All right.

> layers can be a huge pain and it's very easy to do it poorly.

Another drawback would be that one would for instance want compressed
LIST ACTIVE and OVER results but not compressed ARTICLE, POST or IHAVE
results because articles (binaries) are already compressed and it would
mean to compress+encode already compressed+encoded articles...  Whence
a loss of performance and compression.

> which is apparently used with Java.  It seems like a reasonable IANA

OK, I see it here <http://www.iana.org/assignments/http-parameters>.

I also see for PGP:
<http://www.iana.org/assignments/pgp-parameters/pgp-parameters.xhtml#pgp-parameters-15>
(with zip, zlib and bzip2).

-- 
Julien ÉLIE

« Je suis adroit de la main gauche et je suis gauche de la main droite. » (Raymond Devos)

---

From: Ade Lovett <[email-masked]>
Date: Sat, 28 Nov 2009 11:49:44 -0800

> a loss of performance and compression.

Honestly, I think this a solution looking for a problem.  LIST ACTIVE is, at most, a couple of meg in size.  It could _possibly_ be useful in pulling down very large OVER subsets, but I'd submit that the additional complexity and cpu usage on both server and client side to implement it is not worth the reduction in size of the datasets involved.

-aDe

---

From: Sabahattin Gucukoglu <[email-masked]>
Date: Sun, 29 Nov 2009 06:07:28 +0000

A few usenet providers already do compressed list output.  If only we had their documentation, we might have some real-world justification to go on.  As things stand, both GigaNews and AstraWeb support it using different methods: GigaNews uses a capability and AstraWeb overloads the xover command with a compressed xzver alternative.

GigaNews's take (uses capability "XFEATURE COMPRESS GZIP"):
http://www.giganews.com/accelerator.html
and the related blog posting:
http://www.giganews.com/blog/2008/07/why-you-should-be-using-giganews.html

Cheers,
Sabahattin

---

From: Urs Janßen <[email-masked]>
Date: Mon, 30 Nov 2009 16:06:28 +0100

> than creating a bunch of new separate commands.

but what about response code then? (i.e. HDR returns 221, OVER 224 and LIST
ACTIVE 215 on success) - with seperate commands we could keep this.

> layers can be a huge pain and it's very easy to do it poorly.

I dislike another layer as it's usualy muich more complicated to implemnt
(correct) and not adding a compession layer has the advantage that clients
could use compression for only the few things which usualy produce a lot of
data (LIST ACTIVE/NEWSGROUPS, OVER). and it avoids recompression of already
compressed data (i.e. binarie postings).

just adding compression for multiline server -> client responses would be a
big win (and if compression/decompession speed is an issue one could go with
LZO (<http://www.oberhumer.com/opensource/lzo/>) as this is where users
spend time to wait for responses from the servers; client -> server
communication usualy is usualy less critical (exception might be batched
postings or the like).

ZLIST, ZHDR, ZOVER might suffice (ZNEWGROUPS, ZNEWNEWS, ZPAT, ZLISTGOUP
might be handy but usualy do not generate that much output).

I'd like to see BASE64 encoding instead of YENC as it's 7bit clean, has
short and fixed line length (except for the last), is well documented and
a decoder is usualy already implemented in readers. if there will be more
than one encoding to chose from BASE64 should IMHO be the mandatory one.

For compression I don't care, XZ (LZMA2) (<http://tukaani.org/xz/>) gives
good compression but GZIP and BZIP2 are much more common, LZO is fast but
less efficient. if there will be more than one compression to chose from
GZIP should probabely the mandatory one.

urs
-- 
"Only whimps use tape backup: _real_ men just upload their important stuff
 on ftp, and let the rest of the world mirror it ;)" - Linus

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 30 Nov 2009 15:35:03 +0000

> than creating a bunch of new separate commands.

Given that (bad, IMO) choice, I'd go for a generic command.

> command-by-command basis, what if we model it after STARTTLS?

I have a vague memory that we discussed this during the development of
3977, and decided that this was the direction to go. In additon, TLS
already does compression, so we don't even need to invent anything.

> It has the

True.

On the other hand, I'm with Ade on this: I'm not convinced that there is
enough gain to be made from compression to make this worthwhile. If there
was, we'd see it in other protocols as well and compression would be done
at the IP layer or below (didn't PEP modems use to compress data
transparently?).

-- 
Clive D.W. Feather          | If you lie to the compiler,
Email: [email-masked]     | it will get its revenge.
Web: http://www.davros.org  |   - Henry Spencer
Mobile: +44 7973 377646

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 30 Nov 2009 15:50:42 +0000

> ACTIVE 215 on success) - with seperate commands we could keep this.

In order to be compatible with NNTP section 3.2, we'd have to do something
like:

[C] COMPRESSED STAT <[email-masked]>
[S] 202 Underlying command successful with single line response
[S] 223 0 <[email-masked]>
[S] .

[C] COMPRESSED LIST ACTIVE
[S] 203 Underlying command successful with compressed multiline response
[S] 215 list follows
[S] <compressed data>
[S] .

[C] COMPRESSED LIST WIBBLE
[S] 204 Underlying command failed
[S] 501 Unknown LIST keyword
[S] .

with the response always being multiline.

> big win

How big?

If it's only LIST ACTIVE and OVER, how much do these get used?

> (and if compression/decompession speed is an issue one could go with
> spend time to wait for responses from the servers;

Would an NNTP-specific compression be better for these two cases?

> I'd like to see BASE64 encoding instead of YENC as it's 7bit clean,

That's a bad reason. NNTP connections are 8 bit, so what on earth is the
point of wasting a huge amount of encoding space. BASE64 uses 6 bits per
character; NNTP is capable of carrying about 7.9.

If we have to go in this direction, let's have a well-designed compressor
that squeezes out the maximum usage of the channel.

---

From: Urs Janßen <[email-masked]>
Date: Mon, 30 Nov 2009 17:15:35 +0100

> If it's only LIST ACTIVE and OVER, how much do these get used?

tin usualy sends "LIST ACTIVE" (and LIST NEWSGROUPS) at startup and OVER
for each group entered during the session. It may resend LIST ACTIVE during
the session.

on my small server I have 1481 groups in the active file:

akk3% zcat active.gz| wc
   1481    5924   65629
akk3% mimencode -b < active.gz| wc
    373     373   27185

that will be a win of 58.5% or 37kb

on a server with 17277 groups that would be

akk3% zcat big-active | wc 
  17277   69108  809175
akk3% mimencode -b < big-active.gz | wc  
   4698    4698  342946

57.5% / 455KB win.

as for OVER, in a group with about 3100 articles (deep nested thread in
there), plain OVER data is:

akk3% zcat over.gz | wc 
   3099  103293 2257618
akk3% mimencode -b < over.gz | wc 
   3110    3110  226990

which will be a win of 89% or 1.9mb

urs

---

From: Russ Allbery <[email-masked]>
Date: Mon, 30 Nov 2009 08:50:46 -0800

Julien ÉLIE <[email-masked]> writes:

> mean to compress+encode already compressed+encoded articles...  Whence a
> loss of performance and compression.

A real NNTP compression layer would send binary data, so we'd save space
for anything currently on Usenet.  That would change if we added real
binary article transport to NNTP and people started using it, but that's
some distance away.