Subject: [DNSOP] Open Aggregated Datasets and stats on DNS (.NL ccTLD)

From: "Giovane C. M. Moura" <[email-masked]>
Date: Thu, 3 Sep 2015 15:32:12 +0200

[We apologize for multiple copies.]

*************************************************************
OPEN AGGREGATED DATASETS AND STATS ON DNS (.NL ccTLD)
https://stats.sidnlabs.nl/
*************************************************************

SIDN Labs[1] is happy to announce our new stats and open aggregated
datasets to the Internet/Research/DNS communities. It provides:

   *  Visualizations of the DNS traffic for .nl, which is the
country-code top-level domain (ccTLD) for the Netherlands. It also
contains stats on domain registrations, DNS queries, DNSSEC usage, plus
layer-3 and layer-4 information.
   * Longitudinal data: the data we provide are continuous and
longitudinal (starting from May 2014), and daily updated.
   * The respective datasets in JSON format.
   * URL: https://stats.sidnlabs.nl/

Currently, we only provide aggregated data and visualizations, which
means that we do not offer PCAP/Netflow datasets, similarly to the
former Internet2 Netflow Weekly report[2].

However, if you have a novel research idea that would improve the
security and stability of the Internet and would require our data, then
let us know so that we can consider sharing some of the underlying data
with you (subject to our privacy framework and EU and Dutch privacy laws
[3]). Mail us at [email-masked].

ABOUT SIDN AND SIDN LABS:

SIDN[1] manages .nl, the Internet's country-code domain for the
Netherlands. As the Dutch national domain name registry, we enable
Internet users all over the world to register.nl domain names and use
them safely.

We operate the .nl zone of the Domain Name System (DNS) and handle over
a billion DNS queries a day for the 5.5 million-plus registered .nl
domain names. More than 2.4 million of those domain names are secured
with DNSSEC, making .nl the largest secured Internet extension in the
world. We operate our own DNS anycast network to maximise the
availability of the .nl zone and we provide backend services for .aw
(Aruba, a Dutch overseas territory) and .amsterdam.

SIDN Labs[4] is SIDN's research and development team, which develops and
evaluates new technologies and systems with a view to further enhancing
the stability and security of .nl, the DNS and the wider Internet.

DATA DESCRIPTION

The data were obtained by analyzing the DNS traffic received on one of
our authoritative name servers for .nl. They represent approximately 10
per cent of the total traffic, or about 130 million queries per day.
Please note that most of the traffic comes from DNS resolvers, which are
typically located at ISPs/CDNs, and not from end users.

Data interpretation:

Please be aware that, due to DNS caching by the resolvers operated by
ISPs and CDNs, the data are actually a sample of all the .nl queries
generated by clients.

Data format:  The data are made available in JSON format.

Data processing: We use our Hadoop-based ENTRADA platform and our
Domain Registration System (DRS). ENTRADA[5] is an acronym for ENhanced
Top-level domain Resilience through Advanced Data Analysis. The goal of
the ENTRADA platform is to develop new applications and services with a
view to further increasing the security and stability of .nl, the DNS
and the wider internet. ENTRADA enables us to store all the DNS queries
received on our authoritative name servers. Automated analysis of the
stored data helps us to quickly detect threats and anomalies.

Privacy: The data and datasets provided here do not include any
personally identifiable information [3].

Usage/License: The data on this website are licensed under a Creative
Commons Attribution 4.0 International Licence.

Contact: Do you have any questions about this site or the statistics? If
so, mail us at [email-masked].

REFERENCES

[1] https://sidnlabs.nl
[2]
https://web.archive.org/web/20130724070129/http://netflow.internet2.edu/weekly/
[3]
https://www.sidnlabs.nl/uploads/tx_sidnpublications/SIDN_Labs_Privacyraamwerk_Position_Paper_V1.4_ENG.pdf
[4] https://www.sidn.nl/
[5]
https://www.sidnlabs.nl/uploads/tx_sidnpublications/NCSC-presentatie-BIG-data-pub.pdf

Giovane C. M. Moura, PhD. |Data Scientist|SIDN Labs
SIDN | Meander 501 | 6825 MD | Postbus 5022 | 6802 EA | ARNHEM
[email-masked] | https://www.sidn.nl/

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 3 Sep 2015 15:27:45 +0000

> https://stats.sidnlabs.nl/

Quick question/observation about the TLSA query portion of the
data-set.  At least for SMTP, the query pattern is:

    ; sent to .nl authoritative servers when cache is cold
    ;
    Q: example.nl. IN MX ?
    A: example.nl IN MX 0 mail.example2.nl

    ; sent to .nl authoritative servers when cache is cold
    ; and the MX host domain is not enclosed by the next-hop
    ; domain (otherwise the query goes to the next-hop domain's
    ; authoritative servers learned above).
    ;
    Q: mail.example2.nl. IN A ?
    A: mail.example2.nl. IN A 192.0.2.1

    ; sent to example2.nl authoritative servers when cache is cold,
    ; bypassing the .nl servers, because of the immediately preceding
    ; address lookup 
    ;
    Q: _25._tcp.mail.example2.nl. IN TLSA ?
    A: _25._tcp.mail.example2.nl. IN TLSA 3 1 1 <pkey-digest>

So it seems that TLSA queries observed at the .nl level will
substantially under-represent the actual query load even for cold
caches.

I'd be curious to know what you're seeing for the dominant "_<port>"
number in the observed TLSA queries, and whether any particular
resolvers are responsible for the bulk of the "_25" queries.  It
may well be that these are mostly originated by testing tools
(rather than real MTAs) that skip or parallelize the required
address queries.

-- 
	Viktor.

---

From: "Giovane C. M. Moura" <[email-masked]>
Date: Mon, 21 Sep 2015 11:28:46 +0200

Hi Viktor,

Sorry for the delay in getting back to you.

I am not a DANE expert, so I asked for help here from Marco and Jelte
and they helped here with an answer.

>     A: example.nl IN MX 0 mail.example2.nl

Strictly speaking, we will only return a referral to the NS-set of
example.nl. But I guess you meant that implicitly.

> substantially under-represent the actual query load even for cold

The figures as shown on http://stats.sidnlabs.nl/ are indeed relative.
The absolute numbers on the left are no good representative figures. The
graph is only useful to look for certain trends. Reason for this is off
course, that we only gather a subset of the total amount of .nl-queries
and also because of the reason as mentioned above. Once a resolver has
cached the NS-set for example2.nl (because it is trying to resolve
mail.example2.nl), it will probably not return to us for
_25._tcp.mail.example2.nl.

> resolvers are responsible for the bulk of the "_25" queries.

Sure, please check the attached zip file. It has two files:

* ip_count.csv: total number of queries per source IP address (omitted
due to privacy reasons)
* port_count:csv: total number of queries per client source port

The evaluated period is Jan 1st, 2015 to Sept 20th, 2015.
We had for this period a total of 136,500 TLSA queries. For both cases,
it shows a long tail distribution.

Does this answer your question?

>  It may well be that these are mostly originated by testing tools

Yes, that could very well be the case. I'll have to get back to you
later on that, since this will take more time.

Best,

/giovane

-- 
SIDN | Meander 501 | 6825 MD | Postbus 5022 | 6802 EA | ARNHEM
[email-masked] | www.sidn.nl <http://www.sidn.nl/>

---

From: "Giovane C. M. Moura" <[email-masked]>
Date: Mon, 21 Sep 2015 14:23:15 +0200

>> resolvers are responsible for the bulk of the "_25" queries.

In the previous e-mail I sent the client's source ports counts, in
addition to the counts per IP.

Now I see you meant  something else, the port number listed in the
qname: E.g: the "25"  part in _25._tcp.mail.example2.nl.

So for that, I parsed the query as follows, in SQL/Impala syntax:

select regexp_extract(qname, "^(_[0-9]*)\.",0) as label, count(1) as
total from  dns.queries where qtype=52 and year='2015' group by label
order by total desc;

The results, for the top 20, are:

Port,TotalQueries
25,93592
443,36536
NA,4011
587,885
465,878
8443,98
2222,70
4040,66
8888,38
2223,35
5222,25
48443,24
5001,23
1443,20
448,20
2087,16
4443,15
14443,15
2443,14

So it's more "popular" on e-mail than web. Just let me know if you need
the entire csv.

/giovane

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 21 Sep 2015 15:51:03 +0000

> qname: E.g: the "25"  part in _25._tcp.mail.example2.nl.

Yes, and I am specifically interested in whether for those qnames,
a particuler set of client resolvers account for the bulk of the
queries seen by the .nl nameservers.  The theory is that such
queries might more likely originate from testing tools than real
DANE MTAs, because the latter would first resolve the A/AAAA records
of the SMTP server, and only then check for TLSA, in which case
the query should typically go the authoritative server rather than
.nl.

> ...

Thanks, most of the queries being SMTP makes sense.  The question
is then whether drilling down into just the port 25 queries, you
see any interesting distribution of client IPs (if available in
your dataset).

Ways in which TLSA lookups from "real" MTAs might hit "cold"
caches and make it to .nl are:

   * MTA's local resolver is configured with multiple forwarders,
     and the A/AAAA queries were forwarded to a different forwarder
     than the TLSA query.

    * Postfix or Exim MTA misconfigured to directly use multiple remote
      iterative resolvers (remote hosts in /etc/resolv.conf).

    * Queries to multiple remote iterators from some MTA other than
      Postfix or Exim that does validation in the application rather
      than relying on AD-bit from resolver.

    * Some MTA other than Postfix or Exim that either does not determine
      the security status of the address records before TLSA lookups, or
      does both "in parallel".

I am not aware of any MTAs other than Postfix and Exim that implement
client-side DANE, but perhaps they exist.

-- 
	Viktor.