Subject: Re: [DNSOP] Fwd: New Version Notification for draft-dickinson-dnsop-dns-capture-format-00.txt

From: Jerry Lundström <[email-masked]>
Date: Tue, 1 Nov 2016 08:12:12 +0000

Hi Sara,

On 10/31/16 18:22, Sara Dickinson wrote:
> https://github.com/dns-stats/draft-dns-capture-format

An overall question, did you consider a CBOR extension tag (which does
not require a RFC)?

The "file type id" content is not specified but in the CDDL it says
"DNS-STAT", if any should it not be "C-DNS" ?

"format version" exists but there is no indication which format this is.

Description of "Block statistics" is missing.

"Timestamp" in "Block preamble map" is limited to microseconds, maybe
add that each element within the array after the first is a /million to
also allow nano/pico?

Same goes for "time-useconds" and "delay-useconds", maybe allow them to
have a mixed type to either specify microseconds or a "Timestamp" offset
(as array with [seconds,micro,pico...]).

About malformed and other data in the stream, would it not be possible
to add optional "unrecognized-data" byte string to the appropriate
objects and let the implementation decide?  This could also include padding.

Cheers,
Jerry

---

From: Philip Homburg <[email-masked]>
Date: Tue, 01 Nov 2016 17:08:57 +0100

>We have just published a new draft on a proposed format for DNS packet 
>capture - please see below for details. We would very much appreciate 
>feedback on the overall problem discussed here in addition to the 
>details of the format proposed.

Did you consider not (partially) decoding the DNS payload and instead just
storing DNS payloads directly as binary blobs?

Experience with RIPE Atlas shows that binary the DNS data has a number of
advantages:
- future proof
- no maintainance required
- can store anything no matter how broken
- lack of processing equals lack of bugs
- parsers can be based on original DNS standards instead of a new scheme
  plus all DNS standards for the details.

Another issue is to consider whether the format would benefit from local
extensions. For example, enrichtment of data according to local specifications.
If so, then BSON would be another format to consider.

---

From: Sara Dickinson <[email-masked]>
Date: Tue, 1 Nov 2016 16:20:13 +0000

> On 1 Nov 2016, at 08:12, Jerry Lundström <[email-masked]> wrote:
> 
> Hi Sara,
> 
>> https://github.com/dns-stats/draft-dns-capture-format <https://github.com/dns-stats/draft-dns-capture-format>

Hi Jerry, 

Thanks for the comments and the detailed review. 

> An overall question, did you consider a CBOR extension tag (which does

Good question - not really at this stage but it something that could be considered (particularly if this draft moves towards adoption). 

My understanding of tags is that they are a useful convenience for labelling data items but not a necessity. 

> "DNS-STAT", if any should it not be "C-DNS” ?

Yes, good spot :-)  Agreed it should be C-DNS if anything. 

> "format version" exists but there is no indication which format this is.

Implicitly version 0, but will make this explicit in the next version. 

> Description of "Block statistics" is missing.

Yup - will also add in next version. Here is a quick summary

total-packets                   - Total packets received in the stream
total-pairs                        - Total number of matched DNS Q/R pairs written to the C-DNS file
unmatched_queries        - Unmatched DNS queries  written to the C-DNS file
unmatched_responses   - Unmatched DNS queries  written to the C-DNS file
malformed-packets         - Malformed DNS packets
non-dns-packets             - Non-DNS packets in the packet stream
out-of-order-packets       - Number of packets delivered out of order from the capture library
missing-pairs                  - Number or matched DNS Q/R pairs collected, but not written to C-DNS file
missing-packets             - Number or received packets in the stream not written as PCAP
missing-non-dns             - Number or malformed/non-DNS packets in the stream not written as PCAP

out-of-order-packets - this is a count of the number of times that a packet is received from the capture library where the timestamp on the packet is less than the timestamp on the previously received packet. There is a world of intrigue here when you go digging into the details of how/why this can happen….

The last 3 are somewhat application specific (all are optional). They are so that a collecting application can signal issues with overload where it was not able to output all the packets received in the the stream during the collection window for some reason. Such a collecting application could, for example, write out the malformed/non-DNS packets in PCAP format for later processing in parallel to writing the C-DNS data. 

> add that each element within the array after the first is a /million to

That seems reasonable, if it can be done in a flexible way where the additional resolution is only written when needed (i.e when not 0) so as to not increase file size unnecessarily. 

> objects and let the implementation decide?  This could also include padding.

We do want to come up with some mechanism to represent the malformed DNS packets where possible. That is one option to look at. 

Sara.

---

From: "Paul Hoffman" <[email-masked]>
Date: Tue, 01 Nov 2016 09:48:13 -0700

This seems like a really good design, and one that CBOR was specifically 
designed for. This design is a bit complicated, but that complication 
directly matches one of the main design goals: "a format that has a 
relatively low overhead to produce and minimizes the requirement for 
further potentially costly compression". It would be good if that design 
goal was all the way up in the Abstract or the beginning of the 
Introduction, not buried at the end of Section 3.

Two small questions on the format itself:

- Why isn't everything in BlockTables optional?

- It feels like combining both class and type into ClassType might be 
over-optimization. Since Class will almost always be IN, why not just 
have this as its own object member?

--Paul Hoffman

---

From: Jim Hague <[email-masked]>
Date: Tue, 1 Nov 2016 16:50:36 +0000

Hi Philip,

> storing DNS payloads directly as binary blobs?

We did, and we entirely appreciate the advantages of having the binary.
But from a processing point of view, this seemed to us to be equivalent to
using PCAP. Our aim was to produce files that are as small as possible, 
but while
consuming minimal machine resources. We expect files in our format to be 
run through
general purpose compression. Our measurements indicated that compressing 
raw PCAP
is much more expensive in terms of CPU and working set size, and 
delivers files that
are still twice the size of our format after compression. We concluded that
any format shipping binary blobs would not meet our goals.

> If so, then BSON would be another format to consider.

We deliberately specified CBOR maps for most of the data structures to allow
other fields to be added, either in later versions or in local 
modifications. We intend
that decoders should just ignored any fields they don't recognise.

We looked at a variety of binary forms, and I think I did at least look 
briefly
at BSON. It didn't seem to have any major advantages over CBOR, though
obviously I may have missed something. We looked closer at Apache Avro 
and Protocol
Buffers; Avro was the closest competition, but in the end did not offer any
significant advantage, so we went with the format with the IETF standard.
-- 
Jim Hague - [email-masked]          Never trust a computer you can't lift.

---

From: Philip Homburg <[email-masked]>
Date: Tue, 01 Nov 2016 18:54:36 +0100

>any format shipping binary blobs would not meet our goals.

If find it hard to believe that after compression, the BSON encoded
version of the DNS data would be a lot smaller than just the
raw DNS data.

There is a not a lot of redundancy in the DNS encoding.

On the other hand, the IP and UDP headers are quite big, certainly compared
to most queries and also compared to an NXDOMAIN answer.

In addition, IP headers may have random IDs, and UDP headers have random
checksums. Which compress rather poorly and are stored twiced in the pcap.
The UDP port of the client also compresses poorly and is stored once in BSON
and twice in a PCAP.

So I don't think it follows from badly compressing pcaps that storing
raw DNS would compress badly as well. Unless I missed some tricks
why the CBOR version compresses a lot better.

>significant advantage, so we went with the format with the IETF standard.

The downside of CBOR, certainly as used here is that uses integers to
identify fields in what JSON calls objects.

So anybody who writes a local extension is likely to just continue numbering
fields, which leeds to mutually incompatible extensions.

In contrast, formats like XML, JSON, but also BSON where fields have names
make it less likely that people will pick the same identifier for 
completely different purposes.

I looked at the BSON specs, and BSON can do a lot of things in this
regard. But also seems to bring a lot of complexity and how to do it right
is easily lost in all the details.

---

From: "Paul Hoffman" <[email-masked]>
Date: Tue, 01 Nov 2016 11:25:57 -0700

On 1 Nov 2016, at 10:54, Philip Homburg wrote:

> I find it hard to believe that after compression, the BSON encoded

Please note the use case in the draft: they don't want to burden the CPU 
of the box with compressing with gzip, bzip, and so on.

A good, tuned compressor will end up making a jump table similar to what 
this design has in order to make the output small.

> fields, which leads to mutually incompatible extensions.

This may be a misunderstanding of CBOR. A single CBOR object can have 
keys of multiple types. This draft uses the smallest size keys (the 
small integers), but that does not mean extensions need to use integers 
at all. In fact, the spec should probably say "pick a key that is 
unlikely to be used by anyone else" or "here's a trivial-to-get-into 
IANA registry" (but not both).

> completely different purposes.

I disagree on that point. I think that if a spec says "the defined keys 
are 'a', 'n', 'x', and 'y'", and you have an extension you want that is 
about security, you're likely to choose "s", which is also likely to be 
chosen by someone else doing an extension on stability. You gotta have a 
registry or much longer names in order to prevent name collision.

--Paul Hoffman

---

From: Jim Hague <[email-masked]>
Date: Tue, 1 Nov 2016 19:31:29 +0000

> There is a not a lot of redundancy in the DNS encoding.

Certainly there is not a lot of redundancy in the DNS encoding of a 
single packet,
and there is a fair amount of poorly compressible data in transport 
headers in
the PCAP.

What we're exploiting, though, is the redundancy in DNS encoding in a 
stream of
packets. We're building tables of data that is often repeated in a 
stream - names,
addresses, etc. - and storing references instead of repeating data. We 
can do this
cheaply during writing of the CBOR output, because we know where the 
redundancy
will be located. A general purpose compression engine will end up doing 
much the
same, but will have to work harder to locate that specific redundancy. 
Also, by
writing all e.g. names in a table, we're both grouping data that is 
likely to have
significant internal redundancy that we're not exploiting and making the 
size of the input
data to the compression much smaller, both of which again makes the general
engine's job much easier.

> why the CBOR version compresses a lot better.

We did experiment with simple CBOR and Avro encodings of individual DNS 
packets with
minimal transport information, which I think would be comparable. Our 
data showed that
the size of input to the compressor was ~10x the size of our format, and 
the final size
after compression was still significantly greater (~25-30%) than our 
format after compression.
We did not take compression resource measurements in that case, but 
given our experience
I would be surprised if the compression resources required were not also 
significantly
greater.

> completely different purposes.

CBOR does not have to use integers as key values. It can use strings in 
exactly
the same way as BSON. The reason for using integers is simply one of 
space and
hence file size and minimising load on the final compressor. Key values 
occur in the
data stream in both CBOR and BSON for every item with that key, so using 
strings as key
values is not consistent with a goal of minimum file size.

We expect it would be possible, given the CDDL specification of the 
format, to use that
specification to turn keys values back into text for, say, a conversion to
JSON, but no such tool currently exists, as far as we are aware.
-- 
Jim Hague - [email-masked]          Never trust a computer you can't lift.

---

From: Jerry Lundström <[email-masked]>
Date: Wed, 2 Nov 2016 07:53:30 +0000

Hi Sara,

On 11/01/16 16:20, Sara Dickinson wrote:
>> On 1 Nov 2016, at 08:12, Jerry Lundström <[email-masked]
> That seems reasonable, if it can be done in a flexible way where the
> additional resolution is only written when needed (i.e when not 0) so as
> to not increase file size unnecessarily. 

That was kinda the point:

  [ seconds [, optional microseconds [, optional picoseconds ] ] ]

Cheers,
Jerry

---

From: Jerry Lundström <[email-masked]>
Date: Wed, 2 Nov 2016 08:05:15 +0000

Hi Philip,

On 11/01/16 17:54, Philip Homburg wrote:
> Unless I missed some tricks why the CBOR version compresses a lot better.

The trick isn't really CBOR (which you can compare to protobuf/message
pack) but to discard "unnecessary" information (according to the C-DNS
specification).

Cheers,
Jerry

---

From: Jerry Lundström <[email-masked]>
Date: Wed, 2 Nov 2016 08:09:18 +0000

Hi Paul,

> have this as its own object member?

I was also looking at this and there are some values which are very
common so you could add to the specification "if not specified then X is
Y" assumptions.

Cheers,
Jerry

---

From: "Paul Hoffman" <[email-masked]>
Date: Wed, 02 Nov 2016 15:16:08 -0700

> common so you could add to the specification "if not specified then X 

Arrrgh. I think I now understand that Class/Type tables (Section 7.10) 
are a way to say "there are a very limited number of combinations of 
Class and Types, so let's combine the two into single entries for 
pointing to". If I have that right, then making ClassType a list would 
save two bytes per entry in the table.

Current:
    ClassType = {
        type  => uint,
        class => uint,
    }
    type  = 0
    class = 1

Proposed:
    ClassType = [ type, class ]

I cannot imagine a case where a query or a response doesn't have both a 
class and a type.

--Paul Hoffman, who totally admits that this totally dives into the 
format and not the picture of whether it is useful and to whom...

---

From: Jim Hague <[email-masked]>
Date: Thu, 3 Nov 2016 10:28:37 +0000

> save two bytes per entry in the table.

Yep. That was exactly the intention. Probably not a big saving overall, 
but a saving.

 > format and not the picture of whether it is useful and to whom...

I'm impressed anyone is reading it that closely at this stage :-)
-- 
Jim Hague - [email-masked]          Never trust a computer you can't lift.

---

From: Robert Edmonds <[email-masked]>
Date: Mon, 28 Nov 2016 15:57:58 -0500

Paul Hoffman wrote:
> Please note the use case in the draft: they don't want to burden the CPU of
> the box with compressing with gzip, bzip, and so on.

This draft compares lz4 vs gzip vs xz. If minimizing CPU usage is a
concern, I'd be curious to see zstd (http://zstd.net/) added to the set
of compressors being compared. I've found that zstd is often capable of
beating gzip's compression ratio while consuming much less CPU.

-- 
Robert Edmonds