Subject: [Ntp] NTS pool support

From: Christer Weinigel <[email-masked]>
Date: Sat, 20 Jul 2019 22:37:22 +0200

Hi all,

there have been some discussions on how to handle NTS pool support but
I don't think there was any definite conclusion to the discussions.

Here's my naive suggestion on how to handle NTS pools.

Considerations
==============

1) A user should not have to do anything to start using NTS if the
    NTP client that comes with the OS adds NTS support.

2) It should be possible for someone to manage a pool of NTS servers
    without having to manage the TLS private keys for each individual
    NTSKE server.

3) Reuse as much existing infrastructure as possible.

Proposal
========

Today an NTP server pool is implemented with A/AAAA records, where a
pool domain name resolves to multiple IP addresses.  For example I've
set up a small pool for my company's domain which looks like this:

$ dig time.weinigel.se a

time.weinigel.se.       86400   IN      A       77.72.227.121
time.weinigel.se.       86400   IN      A       71.19.158.104
time.weinigel.se.       86400   IN      A       37.46.169.123

(these IP-addresses correspond to fpga-lab.sth.netnod.se,
www.weinigel.se and zoo.weinigel.se respectively).

This is not suitable for a pool of NTS servers though, mostly because
the NTSKE server has to have a server certificate which matches the
domain name used to contact the NTSKE server.  This means that each
NTSKE server would have to be issued a certificate with the name of
the pool.  It also makes it trickier to talk to a specific server,
unless the server is issued multi-domain certificates which will
probably be an administrative nightmare when the pool is in part of
one domain and the server is on another domain (such as
time.weinigel.se and the NTSKE server on fpga-lab.sth.netnod.se).

Additionally, there is no way for a client know if a certain IP
address returned by a pool supports NTS or not.  It could try a well
known port (when/if NTS is assigned it's own port, or if NTS will use
tcp/123) but the client would have to attempt a TCP connection to see
if it's supported or not and this would be rather wasteful.  An ad hoc
solution would be to have a naming convention where one looks for the
same domain prefixed with nts, i.e. nts.time.weinigel.se, but that
feels rather ugly IMNSHO.

Luckily, some smart people have already solved most of this problem:
DNS SRV records.  There is a pretty decent explanation of DNS SRV
records on Wikipedia:

https://en.wikipedia.org/wiki/SRV_record

Basically, it's a way of associating services with a domain with a DNS
record which has the form:

_service._proto.name. TTL class SRV priority weight port target.

Just add a "_ntske._tcp" service for each domain that want to add NTS
support.  (I'm not sure if "_ntske" is the best name, maybe it should
be just be "_nts", or "_ntske1" or some other name, and it has to be
registered with IANA anyway.  But I'll use "_ntske" for my example.)

For my domain "time.weinigel.se", two of my NTP servers also support
NTS, and the NTSKE server listens to port 4446 on both machines.  This
means that I have added the following DNS SRV records to my pool:

$ dig _ntske._tcp.time.weinigel.se srv

_ntske._tcp.time.weinigel.se. 86400 IN SRV 1 10 4446 zoo.weinigel.se.
_ntske._tcp.time.weinigel.se. 86400 IN SRV 2 10 4446 fpga-lab.sth.netnod.se.

Since the DNS SRV records are in a completely different namespace than
the A records used for a NTP pool today it won't affect any existing
NTP clients.  A new client which supports NTS can look up the SRV
records to find the target (host) and port for the NTSKE servers.

The best thing about DNS SRV records is that the target (host) is a
fully qualified domain name, which means that when the NTS client
connects to the NTSKE server it will match the name that's present in
the TLS certificate for the server.  This means that the administrator
of the pool does not have to manage the TLS certificates and private
keys, that's up to whoever manages each NTSKE server.

In my case, I manage all of these machines, and have used letsencrypt
to create the TLS certificates for both zoo.weinigel.se and
fpga-lab.sth.netnod.se, but it could just as well have been that case
that it's Netnods operations team that would handle the certificates
for the machine on their network using their preferred CA.

Notes
=====

Microsoft does something similar for plain NTP if you run a skype for
business server.  They use a SRV record for "_ntp._udp" on domains to
point at the NTP server:

https://docs.microsoft.com/en-us/skypeforbusiness/plan-your-deployment/network-requirements/dns

Just for fun, I have really set up time.weinigel.se just the way I
have described above, and created a simple application in Python do do
the SRV record lookup needed in a client:

https://github.com/Netnod/nts-poc-python/blob/master/lookup.py

$ python lookup.py time.weinigel.se

NTP servers
     37.46.169.123
     71.19.158.104
     77.72.227.121

NTS servers
     fpga-lab.sth.netnod.se.weinigel.se:4446
     zoo.weinigel.se.weinigel.se:4446

Policy for load balancing/failover
==================================

Load balancing and failover can be done at multiple levels.  One can
have multiple DNS SRV records in a pool and achieve load balancing
that way.  If someone wants to have multiple NTSKE servers with the
same host name and do load balancing using A/AAAA records for that
name they can do that as long as the TLS certificates match the host
name.  And someone who wants to have multiple NTS/UDP timestamping
servers using the cookies from one or more NTSKE servers which share
the server keys (the keys needed to decrypt the cookie) they can do
that.

It's up to a client how it behaves when it gets multiple responses, or
if there are both DNS SRV and plain old NTP A/AAAA records is up to
the implementation.  I would suggest something like this though:

Treat multiple DNS SRV _ntske._tcp.$DOMAIN records as members of a
pool.  A smart client should try to use as many as it can at the same
time and weigh the responses from the servers.

Treat multiple A/AAAA records for the target pointed out by the DNS
SRV record as alternate addresses for the same server.  The main use
for this is load balancing/failover.  The client should treat all IP
addresses as the responses from the same server and not weigh them
separately.  The client can try all addresses at the same time, or try
one at a time, but should only use the response from one of them.

If the NTPv4 Server Negotiation record contains a host name and the
A/AAAA lookup returns multiple addresses, treat them as alternate
addresses for the same server.  The main use for this is load
balancing / failover.  The client should treat all IP addresses as the
responses from the same server and not weigh them separately.  To be
as robust as possible a client remember the list of IP addresses if
the NTS UDP timestamping server it's currently using goes down and
switch to using one of the other addresses in that case

I believe that this ought to work for just about all cases of pool
management and gives enough flexibility that one can do load balancing
and failover at multiple levels.  But I don't know if I have missed
some use cases here.

TLS Certificate PKI
===================

I don't think we want to build up our own PKI for the TLS certificates
used with NTS, so I'd suggest that we do the same thing everyone does
for HTTPS.  Yes, this means trusting every CA out there, and yes, it
means that the security of the certificates is generally set by the
least secure CA, but that's something I think we'll have to live with.
And it makes life easier, if you know how to get a HTTPS server
certificate you also know how to get a certificate for your NTSKE
server.  We shouldn't requre any special extensions, just plain
"same certs as for HTTPS".

It is possible to raise the security a bit using DNSSEC and TLSA
(RFC6698), just as for HTTPS.  With DNSSEC whoever maintains the DNS
entries for a pool or DNS entries for a a NTSKE server can ensure that
the DNS information is signed all the way from the root servers.  With
TLSA it's possible to put a hash of the NTSKE server certificate (or
CA) in DNS and thus bridge the gap from DNS to TLS session.

A TLSA record for zoo.weinigel.se would look something like this:

_4446._tcp.zoo.weinigel.se. IN TLSA (3 0 0 deadbeef...)

It basically says that deadbeef is a hash of the certificate used for
TLS connections to tcp port 4446 of zoo.weinigel.se

How to use this information is up to a client.  The default would
probably be for a NTP client to use both plain NTP servers and NTS
servers to begin with and only check the certificates against the
normal HTTPS CAs.  A NTS client which understands DNSSEC and TLSA
could add even more weight to NTSKE servers which use those.  A client
could also be configured to require NTSKE, DNSSEC and/or TLSA (maybe
per pool).

Question: Are there any best practices for HTTPS somewhere out there
that we could reuse?  Since NTSKE is a new protocol we could drop any
deprecated features of HTTPS that web browsers still have to support
for compatibility reasons.

Still not solved
================

There's one major problem with all of this.

To be able to validate the certificate of the NTSKE server the client
needs to know time, and to be able to securely get time a client needs
to be able to NTS.  A client which doesn't have any idea at all about
what time it is will have to ignore the time-related checks of the
NTSKE server certificate (or fall back to plain NTP).

This becomes even worse if one uses DNSSEC since DNSSEC requires the
DNS client to know time to within 5 minutes (IIRC) of the DNS server's
time.  Without time it can't do the DNSSEC validation.

A client will need some kind of policy on what to do here.

Thoughts, flames, have I missed anything vital?

   /Christer

---

From: Ask Bjørn Hansen <[email-masked]>
Date: Sat, 20 Jul 2019 20:18:57 -0700

> On Jul 20, 2019, at 1:37 PM, Christer Weinigel <[email-masked]> wrote:
> 

In this scenario, for the client to know that the server behind the SRV record are trusted by the pool system, it will have to either do end-to-end DNSSEC (and the pool DNS has to support it), or the CA used for the server need to be explicitly tied to the pool configuration. If not then your ISP can easily hi-jack the service in a way that’s not possible with “regular NTS”. Or maybe I am missing something. :-)

Ask

---

From: "Patrik Fältström" <[email-masked]>
Date: Sun, 21 Jul 2019 09:28:53 +0200

On 21 Jul 2019, at 5:18, Ask Bjørn Hansen wrote:

> In this scenario, for the client to know that the server behind the SRV record are trusted by the pool system, it will have to either do end-to-end DNSSEC (and the pool DNS has to support it),

DNSSEC do not really work that way. But close :-)

You must trust whoever is doing the validation. You extrapolate that to you having to do the validation, but that builds upon you not trusting (for example) the access provider that runs the recursive resolver for you also do the validation.

> or the CA used for the server need to be explicitly tied to the pool configuration. If not then your ISP can easily hi-jack the service in a way that’s not possible with “regular NTS”. Or maybe I am missing something. :-)

I think Christers idea is good(*), as it collapses to "if you trust your DNS, then you can use SRV". This irrespectively of whether DNSSEC is in use or not.

That trust can be built by either just trusting DNS, trust the validating recursive resolver run by a third party, or you run your own validating resolver on ::1.

Saying SRV can not be used because you do not trust DNS is not working, as we have other issues with NTS if we do not trust DNS. NTP Pools rely on us trusting the DNS as the pool is built in DNS as the database.

   Patrik

---

From: Hal Murray <[email-masked]>
Date: Sun, 21 Jul 2019 15:15:05 -0700

> Here's my naive suggestion on how to handle NTS pools.

I think you have to be very careful about what you mean by "pool".  In most 
NTP discussions, "the pool" usually means pool.ntp.org, a large collection of 
NTP servers run by volunteers.  Trust and secure don't make sense with that 
type of operation.  There is no point in using NTS.

Actually, I think we should NOT use NTS with "the pool".  It would give a 
false sense of security.

If the pool technology used with other pools?

Is there a term for a collection of NTP servers that are all run by the same organization?  I'll use "cluster".

Do we need pool technology if one organization runs the whole thing?  Why not just use separate names for each server?  How many servers will one organization want to run?  ...

One approach for clusters is for all the NTP servers to use the same certificate.  That means they all need the same secret key.  Cloudflare thinks that is reasonable.  They have lots of servers with the same certificate/key but use something other than pool technology to locate them.

We could avoid the key sharing by having the KE servers return the name on the certificate the server will use along with the IP Address.  That seems like a good idea.

For a more pool like approach, we need an option to return several batches of whatever is returned now - maybe with separate K2S and S2K for each batch.

---------

Or we could invent an alternate protocol.  The first stage just returns a list of names - what would have been in the SRV records.  The second stage does NTSKE with each name.  That's just reinventing SRV records in an environment we know is secure.

If we trust DNS we could use real DNS/SRV records.  I'm not a DNS wizard.  I don't know how to setup DNSSEC.  I'd be happy to use DNS/SRV if we find a DNS wizard who can point to a simple HOWTO type document that tells how to secure DNS.  We may need a separate document for each OS/distro.

What's the general status of DNSSEC these days?  Is it ready for prime time?  Should we be using it to help build critical mass?  Or using something else until more OSes/distros have good support?

Is there a simple/portable API to get SRV records?

-- 
These are my opinions.  I hate spam.

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 22 Jul 2019 06:02:19 +0200

On 22 Jul 2019, at 0:15, Hal Murray wrote:

> If we trust DNS we could use real DNS/SRV records.

There are a few different approaches if one look at DNS:

- SRV

_prefix.domain. SRV port host

- NAPTR

domain. NAPTR service_type port host [sort of]

- URI

_prefix.domain. URI uri

> I'm not a DNS wizard.  I don't know how to setup DNSSEC.

DNSSEC consists of two parts:

1. Have a zone signed. Or in reality, each RRSet (all records of the same type, class and domain name) is signed by adding a signature to it, using asymmetric keys.

2. Have a signed response (i.e. RRSet) validated when it is sent as a response to a query.

Many people mix up the two.

> I'd be happy to use DNS/SRV if we find a DNS wizard who can point to a simple HOWTO type document that tells how to secure DNS.  We may need a separate document for each OS/distro.

Yes and no.

I think we should decide what we build the trust on, and where we se risks. We could say "for this to work, zone with records should be signed, and validation should happen or else the following attack vector exists".

For "how to use DNS", we must refer to DNS material. Like we do with other non-NTP stuff. And how non-NTP people should refer to us.

I do not want to write the "NTS for DNS people" guide :-)

> What's the general status of DNSSEC these days?

Regarding the two functions I talked about above:

- Signed zones is pretty good. Almost all TLDs are signed, which means if you have a domain in a signed TLD you should be able to sign your zone. You can do it automatically for example in Bind or other software, and many DNS providers automatically sign your zone even if you do not ask for it. The trouble is that whenever you have signed your zone, the public key you use for signing must be sent to the registry of the TLD where you have your domain, so that it can be signed and a chain of trust can be built. Not a problem unless you roll your keys.

- Validation is easy to do by "just" turning it on in your resolver. Unbound do it automatically for you for example. If you use your ISP resolver, the result varies. Google open DNS 8.8.8.8 and as far as I know all open DNS resolvers do validate DNSSEC signed responses.

> Is it ready for prime time?

Absolutely!

> Should we be using it to help build critical mass?

DNS people like me think one definitely can build other things like NTS using DNSSEC, given you first have something like rough time as the digital signatures in DNSSEC can be validated :-)

That said, we should write "you must have a DNSSEC chain of trust, or else build an equivalent chain of trust in some other way".

> Or using something else until more OSes/distros have good support?

It's all in there. It is not always deployed because people are lazy. Like IPv6. Similar syndrom. Although DNSSEC can be deployed even if your ISP have not deployed it. IPv6 is hard if you can not get IPv6 transit.

> Is there a simple/portable API to get SRV records?

Only the normal DNS API "one level deeper" than gethostbyname. But not more hard that that. You can still do sync calls, and do not have to deal with async socket stuff.

   Patrik

---

From: Ask Bjørn Hansen <[email-masked]>
Date: Sun, 21 Jul 2019 23:25:19 -0700

> On Jul 21, 2019, at 0:28, Patrik Fältström <paf=[email-masked]> wrote:
> 

Hi Patrick,

I know how DNSSEC works, but I failed at explaining my idea well!

> You must trust whoever is doing the validation. You extrapolate that to you having to do the validation, but that builds upon you not trusting (for example) the access provider that runs the recurs…

Right. I don’t believe trusting the access provider (typically) nearly comes close to the goal of NTS in terms of security.

With “regular NTS” you don’t trust NTS, but rather the chosen TLS infrastructure.  Changing the trust model to rely on DNS doesn’t seem right.

What I meant with the above was that if the client does DNSSEC validation and the pool service does DNSSEC signing you (might?) get the same security guarantees back. There are a lot more TLS capable clients than DNSSEC validating ones, so this doesn’t seem like a good trade-off to me.

> That trust can be built by either just trusting DNS, trust the validating recursive resolver run by a third party, or you run your own validating resolver on ::1.

This doesn’t seem realistic for most clients, at least of the general pool service.

As Hal said, if your “pool” is all run by one organization it’s really just a cluster and you have reasonable deployment options already.

> Saying SRV can not be used because you do not trust DNS is not working, as we have other issues with NTS if we do not trust DNS.

For most TLS-based security you don’t need to trust DNS (because a DNS-hijacker won’t have the right TLS certificate); I don’t understand why NTS is different. If DNS is lying/hijacked/bad the NTSKE connection won’t work, right?

> NTP Pools rely on us trusting the DNS as the pool is built in DNS as the database.

NTS is an opportunity to improve on that. :-)

I’ll expand on my suggestion regarding the certificate authority above in a reply to Hal’s message.

Ask

---

From: Ask Bjørn Hansen <[email-masked]>
Date: Sun, 21 Jul 2019 23:25:20 -0700

> On Jul 21, 2019, at 3:15 PM, Hal Murray <[email-masked]> wrote:
> 

I don’t think that’s true. Implemented right a client could know for sure who it’s talking to.

With short lived certificates, the client could know that the pool system had validated the monitoring data within X hours/days. It could be a solution to having a client know if the server had been removed from the pool system.

One suggestion for this that’d work, I think, with Christer’s SRV suggestion would be (as I suggested a few days ago) to configure the client with a certificate authority that’s managed by the pool system and only trust this CA for servers returned by that specific pool DNS name.  (Basically Christer’s TLSA suggestion, but for clients not doing DNSSEC they could configure the CA in their configuration).

The servers participating in the pool would (automatically) request/get an updated certificate every so many days.

This would be an improvement over the current system in that the client will know for sure who it’s talking to and that the server it’s talking to is part of the pool.

(DNSSEC would still be an improvement in that the client would know the particular DNS response was generated by the pool system and not “improvised”, but it wouldn’t be required for NTS to work).

(Another version, as per what Ragner Sundblad suggested last year in https://tools.ietf.org/html/draft-dansarie-nts-00#section-8 <https://tools.ietf.org/html/draft-dansarie-nts-00#section-8> would be to require the pool system to run the NTSKE servers and have another key exchange method with the servers. I think that proposal underestimated the number of distinct clients using the big public pool-like systems (NTP Pool Project and NIST, as two of them).  Christer’s proposal would support “load balancing” the NTPKE work better).

Ask

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 22 Jul 2019 08:58:45 +0200

> I know how DNSSEC works, but I failed at explaining my idea well!

:-)

> Right. I don’t believe trusting the access provider (typically) nearly comes close to the goal of NTS in terms of security.

Ok, if you do not trust DNS, then I think there are many more attack vectors for NTS anyways, as you can do all different kind of tricks as any CA can issue a certificate for any domain name.

> With “regular NTS” you don’t trust NTS, but rather the chosen TLS infrastructure.  Changing the trust model to rely on DNS doesn’t seem right.

Well, you must trust a combination of the cloud of CA's in the world where any CA can issue a certificate for any domain name out there (unless you tie it down by placing special resource records in the DNS), OR you use DANE and place your cert in the DNS and trust it there.

So, we can split hairs what in your sentence is implied by "rely on". I claim we already in NTS as in other things rely quite heavily on DNS to the degree that DNSSEC should always be used. Just like for any use of the Internet.

> What I meant with the above was that if the client does DNSSEC validation and the pool service does DNSSEC signing you (might?) get the same security guarantees back.

It's hard to compare and claim "same security".

> There are a lot more TLS capable clients than DNSSEC validating ones, so this doesn’t seem like a good trade-off to me.

Ehh...I can not do that comparison. It's for me a layer violation.

1. You must have DNSSEC signed data and do validation to get some security on *any* use of DNS, and given the broken CA mess in the world, the only way to tie that down (for example by informing what CA you use) you do that in DNS.

2. You must have TLS (or similar mechanisms) on the application layer to tie that down, with the help of all tools you can get in the lower layers (for example by using DNSSEC and RPKI for the routing).

> This doesn’t seem realistic for most clients, at least of the general pool service.

Ok, there is something I do not understand here.

I have been both signing zones and validating them since around 2003. The .SE TLD have been signed since 2007 I think, and >90% of DNS responses in Sweden are validated.

This is not difficult. At all. And if you do not have your DNS infrastructure up to par with what have been around for >10 years in all mainstream DNS software, then I do not think there is much we in NTS land can do.

> As Hal said, if your “pool” is all run by one organization it’s really just a cluster and you have reasonable deployment options already.

Correct. As you know, we do that at Netnod although we have the cluster even more tightly bundled and use IP anycast for the servers.

> For most TLS-based security you don’t need to trust DNS (because a DNS-hijacker won’t have the right TLS certificate); I don’t understand why NTS is different. If DNS is lying/hijacked/bad the NTSK…

Yes you do have to trust DNS for TLS as DNS include various things that makes TLS robust by including a CAA record in DNS and signing your zone. Else any CA can issue a certificate for your domain. I.e. I disagree with your statement between the lines that is not possible.

What I agree with is that it is as possible to get a certificate for TLS used in NTS as for TLS used for HTTPS. And in both cases you today to get a robust service must sign the zone, include CAA records and validation must happen.

There are too many MITM attacks going on against these kind of things to claim TLS alone is ok, because that (as you write) relies on the antagonist not being able to issue/get a valid cert.

> I’ll expand on my suggestion regarding the certificate authority above in a reply to Hal’s message.

Thanks!

I do not think we disagree much here. What we disagree on is I claim how much you must trust DNS anyways to get things working when using TLS.

   Patrik

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 22 Jul 2019 09:07:46 +0200

> I don’t think that’s true. Implemented right a client could know for sure who it’s talking to.

I must think quite a bit on this proposal, and I completely trust you guys doing NTS right, BUT, my view is:

1. Keep the pool-mechanisms as separate from "NTS-server" mechanisms -- i.e. I am always nervous of feature creep in protocols.

2. We must trust the DNS anyways, and whether one sign and/or validate DNS is a question orthogonal to this discussion. The use of DNSSEC is today *strongly* encouraged, and if you choose to still not do it, you are to some degree toast. But that is a choice.

3. We can not design NTS to be robust if people have chosen other insecure things regarding their environment. There must be a known context within which we design the NTS (and NTS-pool) related issues. We can for example not solve routing problems, and we can not solve DNS security issues, or issues with fake SSIDs (evil hot spots), fake bootp or neighbor discovery messages, or other things.

4. What we *can* do is to make a robust protocol that to some degree for example withstand MITM attacks, but, to get complete protection one must do validation of DNSSEC signed responses, one must have RPKI "strict" for the IP addresses, one must... etc, and that list which is "basic" for todays Internet start to be quite long :-) NTS should fit in that echo system so that other people say "you should use NTS (+roughtime for bootstrap) or else..." i.e. I do not want DNS or HTTP or BGP people start to expand their protocols "just because people might only use NTP and not NTS because NTS is so hard". ;-)

Given that context, on a quick read, I sort of like the below, but think we can do slightly better IF we rely on DNS a bit more. I.e. we can do something that is easy for everyone, and get a good pool structure, and then we say "if the zone is signed, and validation takes place, THEN the whole thing is...".

But I must think a bit more.

   Patrik

> _______________________________________________
> ntp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ntp

---

From: Leif Johansson <[email-masked]>
Date: Mon, 22 Jul 2019 15:37:59 +0200

> Ok, if you do not trust DNS, then I think there are many more attack vectors for NTS anyways, as you can do all different kind of tricks as any CA can issue a certificate for any domain name.

I think this is why Ask proposed a private/"per-pool" CA.

The issue is not (imo) about weather you trust DNS, TLS or carrier
pidgeons. An operator of a pool probably wants to express multiple
properties using different technical representations:

- DNSSEC represents trust in domain ownership and control
- TLS certs issued by a private CA represents trust in certain
properties of the pool (which can be made *explicit* in CA policy)

These are not mutual exclusive nor do they even overlap that much.

	Cheers Leif