Subject: Re: [NNTP] Compressed LIST (and other commands) answers

From: "Jeffrey M. Vinocur" <[email-masked]>
Date: Tue, 1 Dec 2009 20:25:42 -0500 (EST)

On Mon, 30 Nov 2009, Clive D.W. Feather wrote:

> Russ Allbery said:
> > I think a generic command is an interesting idea and definitely better
> > than creating a bunch of new separate commands.
> 
> Given that (bad, IMO) choice, I'd go for a generic command.

A third (probably also bad) option would be to have something like "MODE
COMPRESS" that a client could send, after which the semantics of
multi-line responses would change (i.e. instead of just dot stuffing, 
the server/client would apply a more complex transformation) but all of 
the commands and response codes could stay the same.

> > Although I wonder if, at that point, whether we want to just provide some
> > facility negotiate compression of all subsequent traffic on the NNTP
> > connection.  In other words, rather than treating this on a
> > command-by-command basis, what if we model it after STARTTLS?
> 
> I have a vague memory that we discussed this during the development of
> 3977, and decided that this was the direction to go. In additon, TLS
> already does compression, so we don't even need to invent anything.

I agree in princple, but am curious where this stands in terms of SASL 
which should be our long-term framework -- Ken, do you have any thoughts?

> On the other hand, I'm with Ade on this: I'm not convinced that there is
> enough gain to be made from compression to make this worthwhile. If there
> was, we'd see it in other protocols as well and compression would be done
> at the IP layer or below (didn't PEP modems use to compress data
> transparently?).

Hmmm doesn't HTTP have a way to do encryption within the protocol (as a 
content encoding or something?) that is fairly widely used?

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: Ade Lovett <[email-masked]>
Date: Wed, 2 Dec 2009 01:35:27 -0800

> the commands and response codes could stay the same.

As you said, "bad".  This would be an utter nightmare to implement.

> which should be our long-term framework -- Ken, do you have any thoughts?

If we _really_ want compression, then it should be here, in SASL, where things are relatively well-defined, as opposed to trying to retrofit something into a _very_ simple protocol (NNTP).

> content encoding or something?) that is fairly widely used?

Yes, but that's done at a layer above generic-HTTP.  Pass some magic keywords around in the initial request, and assuming the server understands them, everything goes compressed.

The _key_ point here is that HTTP itself says _nothing_ about compression, it's up to a handshake between client and server to then decide to talk in compressed bits as opposed to regular 1's and 0's.

So, if someone wants to come up with an overlay concept whereby a modified client + server starts talking compressed for some or all of the transaction, I'm all for it.

However, it SHOULD NOT (in the RFC terminology) be part of the protocol itself.

-aDe

---

From: Urs Janßen <[email-masked]>
Date: Wed, 2 Dec 2009 11:07:15 +0100

> As you said, "bad".  This would be an utter nightmare to implement.

JFTR giganews has implemented something like this:

| < 200 news.mozilla.org
| > CAPABILITIES
| < 101 capability list
| < VERSION 2
| < READER
| < POST
| < LIST OVERVIEW.FMT ACTIVE ACTIVE.TIMES NEWSGROUPS
| < XFEATURE 1
| < XFEATURE-COMPRESS GZIP TERMINATOR
| < XFEATURE-METADATA BYTES
| < XFEATURE-USERAGENT
| < XFEATURES 1
| < .
| > list active *.test
| < 215 newsgroups follow
| < netscape.public.mozilla.test 611 2 y
| < mozilla.test 4354 2 y
| < .
| > XFEATURES
| < 390 send features
| > COMPRESS GZIP
| > .
| < 290 features updated
| > list active *.test
| < 215 newsgroups follow [COMPRESS=GZIP]
| < xËK-)NN,HÕ+(MÊÉLÖËÍ¯ÊÌÉIÔ+I-.Q034T0R¨äåB5165¢¨§.
| > group netscape.public.mozilla.test
| < 211 610 2 611 netscape.public.mozilla.test
| > xover -  
| < 224 xover information follows [COMPRESS=GZIP]
[binarie data]
| > quit
| < 205 goodbye

urs
-- 
"Only whimps use tape backup: _real_ men just upload their important stuff
 on ftp, and let the rest of the world mirror it ;)" - Linus

---

From: Ade Lovett <[email-masked]>
Date: Wed, 2 Dec 2009 02:16:35 -0800

> JFTR giganews has implemented something like this:

I am fully aware of what Giganews has implemented.

It's a crock.  Plain and simple.  If they had _any_ respect for the community of which they are a part, they'd have offered up some form of communication on this list.

The fact remains that they've chosen to implement some random ad-hoc solution, no doubt driven by just-out-of-school business development folks, surrounded by a "the family can do no wrong" attitude, to implement something that, quite frankly, is complete garbage.

Ask yourselves these two questions:

1.  If I'm attempting to download binaries faster, what does <compression-technique> give me?
2.  If I'm spending a lot of time in high-traffic text newsgroups, what does <compression-technique> give me?

Pay particular attention to the fact that consumers of (1) and (2) above are most likely already on high speed internet connections (heck, I pay $19.95/mo for 10/3) and so are _not_ the target audience.

Again, I ask folks.  Is this really a "problem" that needs to be "solved" ?

-aDe

---

From: Sabahattin Gucukoglu <[email-masked]>
Date: Wed, 2 Dec 2009 16:38:17 +0000

> The fact remains that they've chosen to implement some random ad-hoc solution, no doubt driven by just-out-of-school business development folks, surrounded by a "the family can do no wrong" attitud…
Sounds like a whole lot of very sour grapes you have there.  I don't see any logical connection between quality of implementation and respect for standardisation; the two tend to be beneficial to each other but are basically exclusive.  I would much rather you told us flat out how big bad Giganews made your life miserable by being so successful and yet, like so many large companies, such obvious sharks.

To be clear: I am a customer of theirs because I like their service (read: lots of browsable text news with awesome retention on the lowest plan).  And yes, compression helps; large listings are constructed and compressed once and then consumed many many many times.

I am well aware of the, erm, roughness of the implementation, though I hasten not to call it "A crock" and "Garbage".  It serves their needs.  I would add compression on text-only groups, myself.  My request for documentation, in order to in some way appease the many non-Windows (their proxy is Windows only) users, never made it to the blog comments on their oh-so-fabulous proxy accelerator promotion.  But Newsbin, a popular binary grabber for Windows, managed just fine without it.

> 1.  If I'm attempting to download binaries faster, what does <compression-technique> give me?

Nothing.

> 2.  If I'm spending a lot of time in high-traffic text newsgroups, what does <compression-technique> give me?
Everything.

> Pay particular attention to the fact that consumers of (1) and (2) above are most likely already on high speed internet connections (heck, I pay $19.95/mo for 10/3) and so are _not_ the target audi…
Whether you benefit depends on whether the bottleneck is CPU or network.  On my Pentium II, compression is not welcome; on my dual-core MacBook, it sure wouldn't hurt.

(I was lying when I said "Nothing" or "Everything".)

> Again, I ask folks.  Is this really a "problem" that needs to be "solved" ?

If we can get past all the loathing, I'd say it wasn't a *serious* consideration, but for fear of multiple implementations doing the same thing in many stupid ways, I think we should try our hardest to best the lot of them.  Excellence is, after all, what the IETF is good for.  And when we have those specifications, we can legitimately point to the NSPs who ought to know better, and to the implementers who will appreciate the one right way, and bring them aboard for features people will actually like.  Right now, it's a bit hard to deny that compression is a popular idea that might just have merit.

Cheers,
Sabahattin

---

From: Julien ÉLIE <[email-masked]>
Date: Wed, 2 Dec 2009 20:11:23 +0100

Hi Ade,

> If we _really_ want compression, then it should be here,
> in SASL, where things are relatively well-defined, as opposed
> to trying to retrofit something into a _very_ simple protocol (NNTP).

Why in SASL?
If we do that only with SASL, It would require people to authenticate
(via AUTHINFO SASL) in order to start using compression.  And of course
a mechanism which negotiates a compression layer would be needed.

-- 
Julien ÉLIE

« -- Il est parti comme il est venu...
  -- Il ne faisait que passer... » (Astérix)

---

From: Julien ÉLIE <[email-masked]>
Date: Wed, 2 Dec 2009 20:48:00 +0100

Hi Clive,

> already does compression, so we don't even need to invent anything.

I see RFC 3943 "Transport Layer Security (TLS) Protocol Compression Using
Lempel-Ziv-Stac (LZS)" but OpenSSL 1.0.0 does not implement it.
It seems that the only compression that OpenSSL currently recognizes
is DEFLATE.

I have just tested with INN:

% openssl s_client -connect localhost:563 -ssl3
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 1024 bit
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : SSLv3
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 94688B87D8D12DCE21C33B050F194E949A78875D7098DF05FD391DC89839D7BA
    Session-ID-ctx:
    Master-Key: 89B44D4B8C09B70ACFF109E8C77BFD9BA2ADE7F07D28DD89990E53CCA4DAD0B227A5A6BEEACD5BE02A697302583AF98F
    Key-Arg   : None
   Compression: 1 (zlib compression)
    Start Time: 1259781851
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---

It seems to use "zlib compression".  Shouldn't it return DEFLATE instead?
Well, never mind.

It is strange that I have to force "-ssl3"; otherwise, no compression is used
(the default is "-tls1"):

---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 1D8D6322CD08A4244C911898CA438C7D7170B70358E76943CFAA5F27013061B9
    Session-ID-ctx:
    Master-Key: F3393B2457EC130F8433285B36466BCB7344A719FEC28A2D32C8A24F6CB64EB3A4B42267DEABCF0EBB5EB33B0179D167
    Key-Arg   : None
    Start Time: 1259781874
    Timeout   : 300 (sec)
    Verify return code: 18 (self signed certificate)
---

The problem I currently see in INN is that we do not use buffered answers
for all commands...  For instance:

CAPABILITIES
write to 0x80bef18 [0x80c9b88] (90 bytes => 90 (0x5A))
0000 - 17 03 00 00 20 40 42 78-a4 c1 b1 44 ec 62 0a 93   .... @Bx...D.b..
0010 - ef c0 df 67 70 3e aa 70-cf e0 ed 6f ef 64 3b 00   ...gp>.p...o.d;.
0020 - 9a c5 11 c0 d3 17 03 00-00 30 46 42 a6 85 9f 2d   .........0FB...-
0030 - 3c bb 4a 47 15 b4 ea 29-76 fd 2e 29 1f 17 96 c8   <.JG...)v..)....
0040 - 6f d8 5f ff a6 96 24 79-46 a4 23 6d 5f 02 88 4b   o._...$yF.#m_..K
0050 - 6f c7 9f bc 4d f5 55 1a-3e 6f                     o...M.U.>o
read from 0x80bef18 [0x80c5378] (5 bytes => 5 (0x5))
0000 - 17 03 00 00 40                                    ....@
read from 0x80bef18 [0x80c537d] (64 bytes => 64 (0x40))
0000 - c4 75 e5 9d a9 06 48 39-fe 7f 05 d1 f3 39 4f 25   .u....H9.....9O%
0010 - d6 60 d7 e8 e2 53 0e cd-6c 64 c9 f9 93 e5 ac ca   .`...S..ld......
0020 - 2e 66 62 4b 7e 9d 24 47-da c4 4f c0 d3 35 20 63   .fbK~.$G..O..5 c
0030 - 01 d2 97 f3 b0 60 36 fe-de 3d 2c d0 d0 9f 64 e8   .....`6..=,...d.
101 Capability list:
read from 0x80bef18 [0x80c5378] (5 bytes => 5 (0x5))
0000 - 17 03 00 00 30                                    ....0
read from 0x80bef18 [0x80c537d] (48 bytes => 48 (0x30))
0000 - 12 3c 94 cc a6 98 91 3a-9c f6 4f 01 0e f8 56 21   .<.....:..O...V!
0010 - af be 42 34 6c 41 a1 06-86 29 32 85 39 86 90 12   ..B4lA...)2.9...
0020 - fd 70 41 ef 4c 6f 44 e2-82 79 83 95 f1 6a 27 61   .pA.LoD..y...j'a
VERSION 2

90 bytes for "CAPABILITIES\r\n".
5+48 bytes for "VERSION 2\r\n".

That's pretty bad!

However, an OVER of about 13200 bytes is transmitted in about 1600 bytes.
That's a huge gain!

Is it the best approach we should take?
News clients and news servers bufferize their output and send it compressed,
using STARTTLS?

Note that there will still be a lot of overhead for commands like:
BODY
CHECK <mid>
GROUP group (and its answer)
...

-- 
Julien ÉLIE

« Le sel de l'existence est essentiellement dans le poivre qu'on y met. »

---

From: Ade Lovett <[email-masked]>
Date: Thu, 3 Dec 2009 04:23:33 -0800

I am still waiting to see evidence (anecdotal or otherwise) that there is a sufficient "problem" out in the wild that compression (in whatever form) will "solve".

NNTP's strengths (and weaknesses) are a result of its inherent simplicity.  One can implement a compliant server in a few thousand lines of code, over the course of a couple of days (yes, I have 3977-compliant proof-of-concept servers written in perl, python, ruby, and if you give me a week or so, I can probably knock one out in ksh).

The fact of the matter is that, as yet, no-one has provided a compelling reason as to why compression (or lack thereof) should be embedded in the protocol itself, as opposed to being layered over the top (in a similar manner that stunnel provides for "secure" transmission over <random protocol>)

We live in days of multi-megabit home connections.  Sure, not everyone has them, but I would hazard a guess that the majority do, further compounded by the fact that said majority are downloading non-compressable data.

All I ask, before spending time and effort on standardizing a compression methodology, is that the proponents offer up a real-world, real-case scenario, where such an implementation will provide _significant_ improvement above and beyond the current NNTP experience.

-aDe

---

From: Antti-Juhani Kaijanaho <[email-masked]>
Date: Thu, 3 Dec 2009 18:02:38 +0200

On Thu, Dec 03, 2009 at 04:23:33AM -0800, Ade Lovett wrote:
> All I ask, before spending time and effort on standardizing a compression
> methodology, is that the proponents offer up a real-world, real-case
> scenario, where such an implementation will provide _significant_ improvement
> above and beyond the current NNTP experience.

I believe this is a good point.  However, if someone wants to do the work
regardless, an Experimental RFC might be a good way.

-- 
Antti-Juhani Kaijanaho, Jyväskylä, Finland
http://antti-juhani.kaijanaho.fi/newblog/
http://www.flickr.com/photos/antti-juhani/