Subject: [DNSOP] I-D Action: draft-ietf-dnsop-dns-capture-format-03.txt

From: [email-masked]
Date: Mon, 03 Jul 2017 02:08:34 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Domain Name System Operations of the IETF.

        Title           : C-DNS: A DNS Packet Capture Format
        Authors         : John Dickinson
                          Jim Hague
                          Sara Dickinson
                          Terry Manderson
                          John Bond
	Filename        : draft-ietf-dnsop-dns-capture-format-03.txt
	Pages           : 49
	Date            : 2017-07-03

Abstract:
   This document describes a data representation for collections of DNS
   messages.  The format is designed for efficient storage and
   transmission of large packet captures of DNS traffic; it attempts to
   minimize the size of such packet capture files but retain the full
   DNS message contents along with the most useful transport metadata.
   It is intended to assist with the development of DNS traffic
   monitoring applications.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-dnsop-dns-capture-format/

There are also htmlized versions available at:
https://tools.ietf.org/html/draft-ietf-dnsop-dns-capture-format-03
https://datatracker.ietf.org/doc/html/draft-ietf-dnsop-dns-capture-format-03

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-dnsop-dns-capture-format-03

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: "John Dickinson" <[email-masked]>
Date: Mon, 03 Jul 2017 10:11:24 +0100

> 	Filename        : draft-ietf-dnsop-dns-capture-format-03.txt

Hi,

This new version just adds an Implementation Status as defined in RFC7942.

regards
John

> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

John Dickinson

http://sinodun.com

Sinodun Internet Technologies Ltd.
Magdalen Centre
Oxford Science Park
Robert Robinson Avenue
Oxford OX4 4GA
U.K.

---

From: Marc Groeneweg <[email-masked]>
Date: Mon, 3 Jul 2017 12:34:32 +0000

Hi all,

What’s the status of the IPR filed by Ray Bellis? Is this a viable IPR claims? Ray states himself that the claim may have implications, but is not sure. And in my opinion, no one at Nominet has claimed this IPR. Can we drop this claim, and proceed this draft to standard?

Best regards,
Marc Groeneweg

On 03/07/2017, 11:11, "DNSOP on behalf of John Dickinson" <[email-masked] on behalf of [email-masked]> wrote:

---

From: Ray Bellis <[email-masked]>
Date: Mon, 3 Jul 2017 13:52:27 +0100

On 03/07/2017 13:34, Marc Groeneweg wrote:

> What’s the status of the IPR filed by Ray Bellis? Is this a viable
> IPR claims? Ray states himself that the claim may have implications,
> but is not sure. And in my opinion, no one at Nominet has claimed
> this IPR. Can we drop this claim, and proceed this draft to
> standard?

I didn't make a "claim" - I made a third party disclosure on the belief
and understanding that implementations of this draft *might* be affected
by this patent application should it be granted.

Whilst I am one of the two authors of the patent application in
question, I am not the assignee and therefore cannot make any binding
statements (nor claims) about them.

I did notify Nominet's legal counsel of the IPR declaration but heard
nothing from them.

kind regards,

Ray

---

From: "Paul Hoffman" <[email-masked]>
Date: Mon, 03 Jul 2017 09:55:18 -0700

On 3 Jul 2017, at 5:34, Marc Groeneweg wrote:

> IPR claims?

The status is that he filed it. It doesn't matter whether there are 
viable claims in it.

> Ray states himself that the claim may have implications, but is not 
> sure.

Correct.

> And in my opinion, no one at Nominet has claimed this IPR.

That opinion matches a search of the IPR database for 
draft-ietf-dnsop-dns-capture-format.

> Can we drop this claim, and proceed this draft to standard?

There is no need to "drop this claim" in order to move the draft 
forward. Plenty of documents in the IETF have claims like this 
associated with them.

--Paul Hoffman

---

From: Richard Gibson <[email-masked]>
Date: Mon, 3 Jul 2017 19:22:14 -0400

I looked over this draft in detail, and found a handful of ambiguous points
("Clarifications" and "Potentially Missing Data" below). But more
importantly, it is very close to defining a format that could replace much
of my organization's in-house technology. Would you consider some
generalizations to take it over the finish line ("Extension Fields" and
"Opt-in Lossyness")? Only the suggestions related to representing time and
"classtype" items would change the representation of existing data in such
a way that implementations already supporting the draft specification would
require changes.

*Clarifications*
* Items in the "classtype" table (section 7.11) are missing data type
documentation. Both "type" and "class" should be unsigned numbers.
  * And speaking of 7.11, why are CLASS/TYPE pairs represented as CBOR maps
instead of more efficient two-item arrays? If it was an intentional
decision for clarity, then maybe the section 7.7 block preamble
"earliest-time" field should also be promoted to a map ("time-seconds",
"time-useconds", "time-pseconds", mirroring Q/R items) for the same reason.
* In "query-sig" table items (section 7.13) "transport-flags" field, the
bit corresponding to "trailing bytes" shouldn't be limited to UDP.
* In section 7.18, "and an unsigned key" appears to be meaningless and
should probably be removed.

*Potentially Missing Data*
* In "query-sig" table items (section 7.13), "transport-flags" should
probably be extended to include a TLS bit (cf. RFC 7858).

*Extension Fields*
Of the many potentially open-ended key-value maps (file preamble, file
preamble configuration, block preamble, block statistics, query signatures,
Q/R data), only block statistics allows for "implementation-specific
fields", and no further guidance is provided. I think all maps should allow
such fields, with a recommendation that they use an implementation-specific
prefix to avoid collisions with fields added by other implementations or
later versions of C-DNS. Example use cases:
* Extend file preamble configuration (section 7.5) to document aggregation
of queries answered by wildcard names.
* Extend the block preamble (section 7.7) to override file preamble fields
like "host-id" and "server-addresses", enabling fleet-wide file merges.
* Extend query signatures (section 7.13) to indicate EDNS Client Subnet use
(cf. RFC 7871) or selection of context-dependent handling like views.
* Extend Q/R data (section 7.18) to include actual EDNS Client Subnet
values or even a "count" field (which would come with *tremendous* space
savings, sufficient for my organization to abandon its internal
summarization format in favor of C-DNS).

*Opt-in Lossyness*
The format is generally quite good about allowing for detail without
requiring it. However, there are some areas where more space savings could
be had:
* Communicate aggregation of IP addresses into prefixes (i.e., the
irrelevance of least-significant bits in ip-address values) with new
"client-prefix-length-ipv4" and "client-prefix-length-ipv6" and
"server-prefix-length-ipv4" and "server-prefix-length-ipv6" file preamble
configuration options.
* Communicate case-normalizing aggregation of names (e.g., transforming
"eXaMpLe.com" into "example.com") with a new boolean-valued
"name-normalization" file preamble configuration option.
* In "rr" table items (section 7.15), "ttl" should be optional to
accommodate decrementing in recursive resolver responses.
* In Q/R data items (section 7.18) and malformed packet records (section
7.20), I'd like "time-useconds" broken out into "time-seconds" and optional
"time-useconds", both for parity with block-preamble "earliest-time" and
for space savings in applications that are content with second-level
resolution.
* For truly customizable aggregation, I think all query signature (section
7.13) and Q/R (section 7.18) data item fields should be optional... but
especially Q/R data "client-port" and "transaction-id".

---

From: Tim Wicinski <[email-masked]>
Date: Tue, 4 Jul 2017 07:04:09 -0400

> forward. Plenty of documents in the IETF have claims like this 

Paul is correct about this.  We can move the document forward, and let 
the "process" sort out the issue.

The current only outstanding item in this draft is the storing of 
malformed packets (per the authors).  The chairs feel if this version 
gets some solid review and the group resolves this last issue, it is 
ready for working group last call.

tim

---

From: Jim Hague <[email-masked]>
Date: Wed, 5 Jul 2017 13:05:26 +0100

On 04/07/2017 00:22, Richard Gibson wrote:
> I looked over this draft in detail, and found a handful of ambiguous
> points ("Clarifications" and "Potentially Missing Data" below). But more
> importantly, it is very close to defining a format that could replace
> much of my organization's in-house technology. Would you consider some
> "Opt-in Lossyness")? Only the suggestions related to representing time
> and "classtype" items would change the representation of existing data
> in such a way that implementations already supporting the draft
> specification would require changes.
>
> *Clarifications*

Thanks. Yes, the type needs adding.

>   * And speaking of 7.11, why are CLASS/TYPE pairs represented as CBOR
> maps instead of more efficient two-item arrays? If it was an intentional

All the tables in the BlockTables section that are multi-valued are
maps. I did consider making the Class/Type pairs an array, but decided
to go for consistency with the other block tables containing composite
data and make it a map. Yes, two-item arrays would be more
space-efficient, saving two bytes per item. However, in the data we've
observed, the number of entries in the Class/Type table is, as you'd
expect, small, typically 15-20 entries. So we're looking at a saving of
maybe 40 bytes per block. By the time you've run through compression,
that advantage will be further eroded, so I ended up deciding that the
cost of consistency here was worth paying.

We've also considered specifying an implicit Class entry of IN (i.e. if
the Class items isn't present in the map, assume IN), but as, again, the
space saving is negligible prefer to keep the values explicit.

Timestamps, on the other hand, I always regarded as a basic data type,
so naturally a structure. Plus, of course, there's one per
query/response item, so in a block the size savings are in the 10-15k
bytes region, which is rather more significant.

> bit corresponding to "trailing bytes" shouldn't be limited to UDP.

Interesting point. We haven't to date observed trailing data over TCP,
but that's not to say that somebody won't try it.

> should probably be removed.

In most places where we are discussing a map, we've specified the type
of the map key in the text, though I notice we're not 100% consistent
with that.

> probably be extended to include a TLS bit (cf. RFC 7858).

Agreed. We should also look at indicators for DNS-over-HTTP,
DNS-over-QUIC and any other exotica.

> preamble configuration, block preamble, block statistics, query
> signatures, Q/R data), only block statistics allows for
> "implementation-specific fields", and no further guidance is provided. I
> think all maps should allow such fields, with a recommendation that they
> use an implementation-specific prefix to avoid collisions with fields
> added by other implementations or later versions of C-DNS.

You are right that extensibility of the tables is not something we have
considered deeply up to now, and it's definitely something that should
be done. FWIW, my initial inclination is to designate as
implementation-specific all key values above a threshold that allows
plenty of growth space for standardised fields, as long as we can be
sure that generic readers can safely skip over the fields they don't
understand.

This is a topic we need to discuss and flesh out.

 Example use
> cases:
[...]
> * Extend the block preamble (section 7.7) to override file preamble
> fields like "host-id" and "server-addresses", enabling fleet-wide file
> merges.

I don't quite follow why you'd need to put this informational-only stuff
into the block preamble rather than the file preamble/configuration. Can
you expand on that a bit?

> requiring it. However, there are some areas where more space savings
> could be had:
> "server-prefix-length-ipv4" and "server-prefix-length-ipv6" file
> preamble configuration options.
> "eXaMpLe.com" into "example.com <http://example.com>") with a new
> boolean-valued "name-normalization" file preamble configuration option.

These are items that could be addressed by implementation-specific
fields, though I do see the motivation behind wanting a standardised
representation for interchange.

This raises a question about a tension between the background of C-DNS
to date and the slightly different angle you are coming from. We've been
very much focused on using C-DNS to record traffic in a form where the
packets can be recreated in wire format (i.e. as PCAP). The optional
data items mean that data may be missing from those packets, but the
core query and response will still be present.

and (skipping out of order) your final:

> * For truly customizable aggregation, I think all query signature
> (section 7.13) and Q/R (section 7.18) data item fields should be
> optional... but especially Q/R data "client-port" and "transaction-id".

moves the recording to a point where reconstructing wire format means
that the application doing the reconstruction has to not just omit
information not present in C-DNS, but must start generating values to
fill in for the missing items. This feels a bit like a step that needs
discussion; we need to think over the design from your point of view.
Possibly those fields should be optional, but with recommendations for
how to populate them when/if generating PCAP.

> 7.20), I'd like "time-useconds" broken out into "time-seconds" and
> optional "time-useconds", both for parity with block-preamble
> "earliest-time" and for space savings in applications that are content
> with second-level resolution.

time-useconds is a time offset, rather than absolute time. Splitting
into seconds and optionally useconds means that people wanting usecond
resolution must pay a size overhead in their collection files as there
will will be an additional field, a byte for each query/response. If
there is a general demand for only having time resolution at the second
level, possibly there should instead be a configuration field indicating
whether the offset is in usec or sec. I see that second offsets would
provide for notable size savings.

I'm interested to hear whether others also have similar use cases to the
above.

We'll be doing a 10 minute slot on C-DNS in Prague, and would welcome
discussion there.
-- 
Jim Hague - [email-masked]          Never trust a computer you can't lift.

---

From: Richard Gibson <[email-masked]>
Date: Wed, 5 Jul 2017 13:06:12 -0400

> bytes region, which is rather more significant.

In that case, I'm even more convinced that all of them (block preamble
"earliest-time", Q/R data item "time" and "delay", malformed packet item
"time") should have the same structure—either variable-precision arrays or
{"seconds", "useconds", "pseconds"} maps.

> of the map key in the text, though I notice we're not 100% consistent

All the keys in those maps (and in every map, as far as I can tell) are
strings, for which "unsigned" is a meaningless concept.

> you expand on that a bit?

Some of our processing can benefit from merging telemetry (e.g., all hosts
at a site), which would produce files containing blocks that don't share
values for fields currently in the file preamble. In fact, we might even
decide to treat "block" as the only relevant unit, using block preamble
extension fields for *everything* in the standard file preamble except
version information.

> core query and response will still be present.

Agreed. But it's _so close_ that I felt these suggestions were worth
raising—not to water down the primary use case, but to cover a few more
with some tradeoffs.

> how to populate them when/if generating PCAP.

That sounds great.

> discussion there.

I don't expect to be there personally, but we'll have a well-informed
representative.

---

From: Jim Hague <[email-masked]>
Date: Thu, 06 Jul 2017 10:17:26 +0100

> {"seconds", "useconds", "pseconds"} maps.

Q/R data item and malformed packet items have the same structure, though it's 
not an explicit data type in CDDL as Timeval is at the moment. I'll amend 
that. However, they are time deltas, not timestamps, so I don't think they 
should have the same type as the block earliest time. It it turns out that no-
better-then-second resolution is a common use case, there are potential 
savings to be made in using seconds as the time delta, because the deltas will 
tend to be very small numbers with consequent space savings in CDDL. We'll 
look at this.

> strings, for which "unsigned" is a meaningless concept.

No. All keys are unsigned ints, with values specified in the CDDL. We should 
make this more explicit in the text.

String keys would bloat the output file size enormously.

> extension fields for *everything* in the standard file preamble except

Ah, so you are talking about doing a merge at the block level rather than the 
Q/R item level? We'll think about the implications of that.

The fields you cited, host-id and server-addresses, are purely informational 
anyway.

> raising—not to water down the primary use case, but to cover a few more

Oh, absolutely! You've give an important use case we hadn't considered - we 
need to look into it.

> I don't expect to be there personally, but we'll have a well-informed

We'd definitely like to talk, then.
-- 
Jim Hague - [email-masked]          Never trust a computer you can't lift.

---

From: Richard Gibson <[email-masked]>
Date: Thu, 6 Jul 2017 12:25:33 -0400

> String keys would bloat the output file size enormously.

Oh, wow. Yes, more explicit documentation in the text would be
appreciated. I completely missed that, even in the CDDL where the key
definitions follow maps referencing them. It seemed like you were using
string keys and relying on general compression to eliminate the bloat, but
this greatly clarifies your "designate as implementation-specific all key
values above a threshold" statement regarding extension fields (and in that
case, might I recommend reserving all nonnegative integers and perhaps also
all strings lacking an implementation-extension prefix for standard key
values? That would suggest negative numbers and prefixed strings for
extension keys, a set encompassing one full major type and then some).

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 16 Jul 2017 15:49:08 +0200

On Mon, Jul 03, 2017 at 07:22:14PM -0400,
 Richard Gibson <[email-masked]> wrote 

> preamble configuration options.

Yes, it is very important in privacy-conscious settings, when you want
to blur the data.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 16 Jul 2017 15:44:53 +0200

On Mon, Jul 03, 2017 at 02:08:34AM -0700,
 [email-masked] <[email-masked]> wrote 

> 	Filename        : draft-ietf-dnsop-dns-capture-format-03.txt

Jim Hague and myself worked at the IETF 99 hackathon on C-DNS,
producing a proof-of-concept DNSTAP-to-C-DNS translator
<https://github.com/bortzmeyer/golang-dnstap/tree/c-dns> This raised
some issues.

There is a semantic mismatch between dnstap and C-DNS. C-DNS makes
most information mandatory, dnstap accepts that you omit almost
everything. (The current dnstap-to-C-DNS converter solves the problem
by generating dummy values, such as all-zeroes for IP addresses.)
Should C-DNS be more lax? Some people would prefer the security of
konwing everything is there, some may have the problem that the
information is unknown on the beginning. May be having two profiles of
C-DNS, a "Full" where almost everything is mandatory, a "Partial"
where a lot of things are optional? (Tools could refuse to convert
"Partial" C-DNS files to pcap.)

Many arrays in the spec syntactically allow empty arrays
(Block.queries, BlockTables.ip-address, etc). When the field is
optional (BlockTables.qrr), it means there are two ways to omit the
array: not having the field at all, and having the field going to an
empty array. Is it a good thing to have both?

When the field is mandatory (BlockTables.classtype), what is the use
of an empty array? Note that we cannot refer to an empty array: the
draft says 'An index value of 0 is reserved to mean "not present".'
but it is not accepted by the current tools (Compactor). Is the tool
wrong?

I find no way to indicate in C-DNS the bailiwick of the response, or
the fact that it was a cache hit or miss in the resolvers. These are
two important information in dnstap, that you cannot get by sniffing
the traffic. May be related to the "private extensions" discussion
<https://mailarchive.ietf.org/arch/msg/dnsop/K7ZnFFa3tW0QDZnJXgef0-1WzOA>?

As already mentioned in
<https://mailarchive.ietf.org/arch/msg/dnsop/WBdp12r1NFOPeUW6H60SdohRI6o>,
the fact that keys are always integers and not the strings used in the
description is not obvious. A warning would be welcome.

There is no mention of the CBOR encoding of the maps and arrays. Can a
C-DNS producer use at will indefinite or definite lengths? Should it
be explicit in the draft or do we think that "Read RFC 7049, you CBOR
n00b" is enough?

Should we have an "advice to implementers" section? For instance, if a
C-DNS producer runs for a long time and then crashes, losing the
opportunity to add a CBOR break at the end of an array, the file
becomes corrupted (pcap does not have this problem). Is it worth a
warning?

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 17 Jul 2017 11:34:15 +0200

On Sun, Jul 16, 2017 at 03:44:53PM +0200,
 Stephane Bortzmeyer <[email-masked]> wrote 

> <https://github.com/bortzmeyer/golang-dnstap/tree/c-dns> This raised

And if you want the gory details on the code:

http://www.bortzmeyer.org/c-dns-tests.html