Subject: [DNSOP] CBOR-DNS compression is 30% the size of gzip compressed PCAP

From: George Michaelson <[email-masked]>
Date: Tue, 15 Nov 2016 14:59:27 +0900

Terry clarified, the 30% size compression is not from uncompressed,
but is the comparision with compressed gzip pcap.

So, compressed DNS-CBOR is *significantly* smaller than pcap.gz

-G

---

From: Sara Dickinson <[email-masked]>
Date: Tue, 15 Nov 2016 15:18:12 +0900

> On 15 Nov 2016, at 14:59, George Michaelson <[email-masked]> wrote:
> 

Yup - sorry if I wasn’t clear in the presentation. With a general purpose compression tool like xz:

- Compressed C-DNS is around 30% the size of the compressed PCAP.

-  It requires around an order of magnitude less CPU to compress the C-DNS file.

Sara.

---

From: Jerry Lundström <[email-masked]>
Date: Tue, 15 Nov 2016 07:09:13 +0000

Hi,

> - Compressed C-DNS is around 30% the size of the compressed PCAP.

I am seeing similar results for CDS [1]:

Gzipped         PCAP        CDS         Factor

client          108136      45944       0,4248
zonalizer       12468329    2485620     0,1993
large ditl      327227203   117569598   0,3592
small ditl      539323      253402      0,4698

Xzipped         PCAP        CDS         Factor

client          76248       36308       0,4761
zonalizer       7894356     1695920     0,2148
large ditl      267031412   86747604    0,3248
small ditl      442260      206596      0,4671

> -  It requires around an order of magnitude less CPU to compress the C-DNS file.

Can you clarify this, do you mean that it takes longer to compress PCAP
then C-DNS?  If so, do you have numbers?  And how did you get them?

Cheers,
Jerry

[1] https://github.com/DNS-OARC/dnscap/blob/develop/CBOR_DNS_STREAM.md

---

From: Maarten Wullink <[email-masked]>
Date: Tue, 15 Nov 2016 07:51:21 +0000

Did you consider columnar storage formats like Apache Parquet + Snappy compression?
It would be an interesting to compare this to cbor-dns.

https://parquet.apache.org/
https://github.com/google/snappy

> Op 15 nov. 2016, om 07:18 heeft Sara Dickinson <[email-masked]> het volgende geschreven:
> 
> 
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

---

From: Jim Hague <[email-masked]>
Date: Tue, 15 Nov 2016 15:09:01 +0000

> then C-DNS?  If so, do you have numbers?  And how did you get them?

We looked at compressing raw PCAP versus the same data encoded as C-DNS as 
part of the C-DNS design process. We ran

$ /usr/bin/time xz -9 <input>

and logged the reported user and system CPU times and maximum resident set 
size. The idea was to get an idea of the resource consumption of the 
compression, and look at the obvious question of whether it was better than 
just compressing PCAP.

We used 'xz -9' to get some idea of the upper bound of achievable compression.

We looked at 5 minute capture samples for 3 different servers, with loadings I 
might roughly describe as light, medium and heavy.

You'll need fixed font to comfortably view the following table of results:

                 File size                    CPU
Traffic Format  Input   Output VM space    User  System   
Light    PCAP   47.92M   4.14M  490.56M    8.27    7.29
Light	  C-DNS   5.12M   1.78M  113.44M	  3.39	   1.19
Medium	  PCAP	 245.77M	17.96M	 690.05M	102.53	  15.91
Medium	  C-DNS  23.57M   6.88M	 276.47M	  9.38	   0.11
Heavy	  PCAP	 677.61M	45.25M	 690.05M	180.15	   6.61
Heavy	  C-DNS  62.85M	16.83M	 620.37M	 29.36	   0.95

This is really what one would expect. C-DNS presents the compression with much 
smaller input, so less to chew through, and groups data likely to be similar 
together in its header tables. So the compressor is given an easier job, and 
needs fewer resources.
-- 
Jim Hague - [email-masked]          Never trust a computer you can't lift.

---

From: Jim Hague <[email-masked]>
Date: Tue, 15 Nov 2016 15:29:39 +0000

On Tuesday, 15 November 2016 07:51:21 GMT Maarten Wullink wrote:
> Did you consider columnar storage formats like Apache Parquet + Snappy
> compression?  It would be an interesting to compare this to cbor-dns.
> 

I did look briefly at Snappy to compare it to gzip and xz, but our focus at the 
time was on the smallest possible output and so Snappy wasn't really what we 
wanted.

We didn't look directly at Parquet, but Tomáš Gavenčiak from CZ.nic did some 
experiments with us on a simple columnar storage format. After compression, 
the output sizes did get close to the C-DNS sizes, but C-DNS retained a small 
advantage in post-compression file size so we stuck with the current design.
My records indicate an input 370Mb PCAP produced a 12.34Mb output with the 
columnar output after xz compression, versus C-DNS at 10.34Mb.  But you needed 
good compression to get that close - with gzip or lz4, the columnar final size 
was more like 1.6-18x C-DNS. I did not look at compression resources.

I believe that Tomáš is using Parquet in his project, which has similarities 
but slightly different requirements (and so trade-offs) to our brief.
-- 
Jim Hague - [email-masked]          Never trust a computer you can't lift.

---

From: Jerry Lundström <[email-masked]>
Date: Tue, 15 Nov 2016 15:50:31 +0000

Hi Jim,

> $ /usr/bin/time xz -9 <input>

Was -9 also used when comparing compression?

Cheers,
Jerry

---

From: Jim Hague <[email-masked]>
Date: Tue, 15 Nov 2016 16:20:41 +0000

Hi Jerry,

> Was -9 also used when comparing compression?

For the numbers I posted in the table in response to your message, yes.

Generally when comparing compression (and I have data for lz4, gzip and xz) we 
used the default settings, so -6 for xz.
-- 
Jim Hague - [email-masked]          Never trust a computer you can't lift.