Subject: Re: [Ntp] Rate limiting: DDoS, KoD, Pool, NAT

From: Hal Murray <[email-masked]>
Date: Mon, 02 Mar 2020 02:21:49 -0800

[email-masked] said:
> Has anybody else considered this area?  Is there a rate limiting that will
> serve NAT sites without being useful for DoS redirection?

Suppose we put the requesters IP Address into the cookie.  The NTP server can 
check that and drop requests that don't match.

That will degrade the case where a laptop goes from one site to another if it 
is smart enough to save cookies while sleeping or powered off.  Are there 
other interesting cases?

-- 
These are my opinions.  I hate spam.

---

From: "Salz, Rich" <[email-masked]>
Date: Mon, 2 Mar 2020 12:49:31 +0000

That kind of explicit client-tracking will get some of the IESG folks up in arms.  I tend to agree with them.

---

From: Hal Murray <[email-masked]>
Date: Tue, 03 Mar 2020 21:41:39 -0800

> arms.  I tend to agree with them. 

Thanks for the heads up.

In the normal case, the IP Address in the cookie would match the IP Address in 
the packet so there is no information leaked.

The interesting case would be when the client changes its IP Address and then 
uses an old cookie.  Can the client reliably detect when its IP Address 
changes?  Even if that works, it feels like the sort of thing that is likely 
to break when somebody makes an innocent change to some unrelated code.

Even without an IP Address in the cookie, just using the same set of servers 
from a different IP Address is giving a lot of info to trackers.  If you are 
using the poop, an obvious fix for that is to discard all current pool servers 
and do a new DNS lookup.  That also fixes tracking by looking in cookies.

Does it matter if the IP Address in the cookie is encrypted?  That doesn't 
help if you are worried about the NTP server tracking you, but it does make it 
harder for the bad guys who are watching traffic in the middle of the internet.

Where do the anti-tracking people hang out?  Is there a mailing list?  WG?

---

From: Tony Finch <[email-masked]>
Date: Wed, 4 Mar 2020 22:36:00 +0000

> > up in arms.  I tend to agree with them.

[snip]

Dunno how well this matches what you want for NTP but it might be worth
taking a look at DNS cookies, RFC 7873.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Shannon, Southwest Rockall: Variable 4, backing southwesterly 5 to 7, then
veering northwesterly 6 to gale 8, perhaps severe gale 9 later. Rough or very
rough, occasionally high later. Showers. Mainly good.

---

From: Miroslav Lichvar <[email-masked]>
Date: Thu, 5 Mar 2020 08:09:46 +0100

> taking a look at DNS cookies, RFC 7873.

NTP already has a cookie, which servers could use for tracking to
avoid rate limiting a large number of well-behaved clients behind NAT.
It's called origin timestamp. But with the minimization draft we now
recommend clients to not use it.

-- 
Miroslav Lichvar

---

From: Hal Murray <[email-masked]>
Date: Thu, 05 Mar 2020 01:27:06 -0800

[email-masked] said:
> NTP already has a cookie, which servers could use for tracking to avoid rate
> limiting a large number of well-behaved clients behind NAT. It's called
> origin timestamp. But with the minimization draft we now recommend clients to
> not use it.

I don't get it.  How does a server use the origin timestamp to decide whether 
to respond to a request or ignore it?

When I said "cookie", I was thinking of the NTS cookie.  Not stated was the 
assumption that sites using many machines behind NAT (and thus generating 
enough traffic to look like it might be a DoS attack) would have to use NTS.

What do you mean by cookie?  To me, it means something that came from the 
server and the client is giving back to the server.  In the context of web 
servers, it is usually used to identify the client where "client" may be an ID 
or account that is logged in or a handle used to recover some state like the 
contents of your shopping cart.

Do you have things backwards?  The server gives the origin timestamp back to 
the client rather than the client giving something back to the server.

---

From: Miroslav Lichvar <[email-masked]>
Date: Thu, 5 Mar 2020 10:42:05 +0100

> to respond to a request or ignore it?
...
> the client rather than the client giving something back to the server.

A "full" NTP client (not implementing the data minimization) gives the
server back its transmit timestamp as the origin timestamp. This is
due to the client/server mode being a special case of the symmetric
mode. A server from the NTP point of view is just a stateless peer
which doesn't synchronize to its peers.

The server could save its transmit timestamps, or to avoid wasting
memory it could encode a flag in the low-order bits of the timestamp
in order to detect existing client associations and give them a
priority in rate limiting. Of course, this goes against the privacy
issues that the data minimization draft is addressing.

-- 
Miroslav Lichvar

---

From: Hal Murray <[email-masked]>
Date: Thu, 05 Mar 2020 02:12:03 -0800

> ... but it might be worth taking a look at DNS cookies, RFC 7873.

Thanks.

At one point, I suggested that the IP Address could be encrypted in the 
cookie.  That would prevent eavesdroppers from learning anything but still 
allow the server to easily figure out the IP Address that the cookie was given 
to.

That RFC suggests a slightly different approach.  Rather than the IP Address, use a one-way hash of the IP Address - or tangle the IP Address into the authentication.  That way, the server can still test for a match, but if it doesn't match, the server can't (easily) track which IP Address the client was using previously.

IPv4 addresses are only 32 bits so it's reasonable to use brute force.  I'm not sure IPv6 solves the problem, but I don't have a simple example.

---

From: Hal Murray <[email-masked]>
Date: Thu, 05 Mar 2020 02:47:04 -0800

[email-masked] said:
> A "full" NTP client (not implementing the data minimization) gives the server
> back its transmit timestamp as the origin timestamp. This is due to the
> client/server mode being a special case of the symmetric mode. A server from
> the NTP point of view is just a stateless peer which doesn't synchronize to
> its peers. 

Ahhh.  Thanks.

What fraction of the clients are "full"?  Is there an easy way for a server to 
test?  If so, I'll see if I can collect some data.  That seems like a 
generally useful piece of data for discussions like this.

Does that work in the case where there are several/many clients behind a NAT 
box?

---

From: Miroslav Lichvar <[email-masked]>
Date: Thu, 5 Mar 2020 12:11:54 +0100

> generally useful piece of data for discussions like this.

Using tcpdump you can collect transmit timestamps of the server and
origin timestamps of the clients and then find matching timestamps.

Most clients are SNTP and set the origin timestamp to zero or a
different constant. IIRC, in my tests it's not more than 10-20% of
clients being "full". It varies between countries.

> Does that work in the case where there are several/many clients behind a NAT 

Yes, it does.

-- 
Miroslav Lichvar

---

From: [email-masked]
Date: Sun, 22 Mar 2020 08:48:56 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Network Time Protocol WG of the IETF.

        Title           : Network Time Security for the Network Time Protocol
        Authors         : Daniel Fox Franke
                          Dieter Sibold
                          Kristof Teichel
                          Marcus Dansarie
                          Ragnar Sundblad
	Filename        : draft-ietf-ntp-using-nts-for-ntp-26.txt
	Pages           : 44
	Date            : 2020-03-22

Abstract:
   This memo specifies Network Time Security (NTS), a mechanism for
   using Transport Layer Security (TLS) and Authenticated Encryption
   with Associated Data (AEAD) to provide cryptographic security for the
   client-server mode of the Network Time Protocol (NTP).

   NTS is structured as a suite of two loosely coupled sub-protocols.
   The first (NTS-KE) handles initial authentication and key
   establishment over TLS.  The second handles encryption and
   authentication during NTP time synchronization via extension fields
   in the NTP packets, and holds all required state only on the client
   via opaque cookies.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-ntp-using-nts-for-ntp/

There are also htmlized versions available at:
https://tools.ietf.org/html/draft-ietf-ntp-using-nts-for-ntp-26
https://datatracker.ietf.org/doc/html/draft-ietf-ntp-using-nts-for-ntp-26

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-ntp-using-nts-for-ntp-26

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Ragnar Sundblad <[email-masked]>
Date: Sun, 22 Mar 2020 17:13:22 +0100

All,

We would again like to thank all of you who have sent comments and
discuss issues, and for the overwhelming response from the reviewers
and others.

We also got some comments on the comments. With this version, we hope
to have addressed those too.

- No TLS 1.2
During this work, it became apparent that keeping support for TLS 1.2
and correctly covering for the differences between TLS 1.2 and TLS 1.3
would be a lot of work, with the risk of getting something wrong or
missing on important issues, in the specification or in the
implementations.

The last time the issue of 1.2 support was up for debate, there were
some operating systems that would not support TLS 1.3, and there was a
concern that this would impair the protocol adoption rate. Today most
(all?) major operating systems do have TLS 1.3 capable libraries in
their later versions.

Several of the reviewers questioned why TLS 1.2 was supported, and the
same have others done before.

We therefore decided to drop support for TLS 1.2.

For older operating systems, an NTS enabled NTP implementation can
still be linked with a separate TLS library that supports 1.3, a bit
inconvenient, but doable.

- New TLS export disambiguating label string
As we had a techincal change in there already, we also changed the TLS
export disambiguating label string to be more aligned with what is the
standard form today, to "EXPORTER-network-time-security”.

Best regards,
Ragnar Sundblad

> On 22 Mar 2020, at 16:48, [email-masked] wrote:
> 
> 
> _______________________________________________
> ntp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ntp

---

From: Hal Murray <[email-masked]>
Date: Sun, 22 Mar 2020 21:24:09 -0700

[email-masked] said:
> - New TLS export disambiguating label string As we had a techincal change in
> there already, we also changed the TLS export disambiguating label string to
> be more aligned with what is the standard form today, to "EXPORTER-network-tim
> e-security”.

Are we going to coordinate a flag day, or just change things as we get around 
to it, tolerate the down time, and watch the recovery heuristics while we have 
the opportunity?

---

From: Watson Ladd <[email-masked]>
Date: Sun, 22 Mar 2020 21:29:45 -0700

I see no usage so flag day is fine with me.

---

From: Ragnar Sundblad <[email-masked]>
Date: Mon, 23 Mar 2020 08:38:35 +0100

> On 23 Mar 2020, at 05:29, Watson Ladd <[email-masked]> wrote:
> 

What would be useful? Friday this week?

If you actually have users:
Note that it isn’t forbidden to support both, but preferably only for a while, I believe in the server would be easiest, the most natural, and then you could log any usage of the old one.

Best,

Ragnar

---

From: Hal Murray <[email-masked]>
Date: Mon, 23 Mar 2020 01:08:10 -0700

[email-masked] said:
> If you actually have users: Note that it isn’t forbidden to support both, but
> preferably only for a while, I believe in the server would be easiest, the
> most natural, and then you could log any usage of the old one. 

How do you support both?  How does either the client or server know which one 
the other is going to use?  That string doesn't get sent across the wire.

We are talking about the "TLS export disambiguating label string" used to make the client-2-server and server-2-client keys.  Right?

---

From: Miroslav Lichvar <[email-masked]>
Date: Mon, 23 Mar 2020 09:19:13 +0100

On Mon, Mar 23, 2020 at 01:08:10AM -0700, Hal Murray wrote:
> 
> [email-masked] said:
> > If you actually have users: Note that it isnâ€™t forbidden to support both, but

I think you could encode both pairs of keys in the cookies, try
them both for authentication on the server side, and use the same pair
for the response. Not worth the trouble IMHO.

If there is a flag day, I'd prefer if it was the same as for the
NTS-KE port.

-- 
Miroslav Lichvar

---

From: Hal Murray <[email-masked]>
Date: Mon, 23 Mar 2020 01:37:03 -0700

[email-masked] said:
> I think you could encode both pairs of keys in the cookies, try them both for
> authentication on the server side, and use the same pair for the response.
> Not worth the trouble IMHO. 

Good hack.  Thanks.  I agree that it's not worth the trouble.

> If there is a flag day, I'd prefer if it was the same as for the NTS-KE port.

We don't need a flag day for the port.  The server can easily listen to both 
old and new.

---

From: Miroslav Lichvar <[email-masked]>
Date: Mon, 23 Mar 2020 09:50:12 +0100

> We don't need a flag day for the port.  The server can easily listen to both 

It could, but I suspect more people would use the old port in future
than now due to obsolete documentation or configuration examples
coming up in search results, and closing the port later would actually
cause more problems. I'm not planning to support the old port.

-- 
Miroslav Lichvar

---

From: Ragnar Sundblad <[email-masked]>
Date: Mon, 23 Mar 2020 11:02:12 +0100

> On 23 Mar 2020, at 09:08, Hal Murray <[email-masked]> wrote:
> 
> 
> [email-masked] said:

Right, sorry!

The server would have to keep track of that client and try the other one
the next time, or something like that, and you would have to switch
at the next connection before TLS starts (ugly, almost horrible...)

> We are talking about the "TLS export disambiguating label string" used to make the client-2-server and server-2-client keys.  Right?

We are.

/ragge

---

From: Karen O'Donoghue <[email-masked]>
Date: Tue, 24 Mar 2020 04:23:28 +0000

Folks,

Per the guidance provided after the cancellation of the F2F IETF 107 meeting, I have requested a meeting slot for 6 April at 15:00 - 17:00 UTC with the following draft agenda. I will confirm this once the meeting is approved. Also, this is a *draft* agenda so please feel free to add items of discussion. Drafts are being accepted by the datatracker. 

Regards,
Karen

DRAFT NTP WG Agenda
==========================================================

1.  Administrative and Agenda Bashing 
    Note Well, Note Takers, Jabber Scribes, Agenda Bashing

2.  NTP WG Document Status Review
    
3.  Port Randomization in the Network Time Protocol Version 4 
    https://datatracker.ietf.org/doc/html/draft-ietf-ntp-port-randomization/

4.  A Secure Selection and Filtering Mechanism for NTPv4
    https://datatracker.ietf.org/doc/draft-ietf-ntp-chronos/

5.  Interleaved Modes
    https://datatracker.ietf.org/doc/draft-ietf-ntp-interleaved-modes/

6.  NTS Measurement and Testing

7.  NTP v5 requirements 

8.  AOB

---

From: Jiangyuanlong <[email-masked]>
Date: Fri, 27 Mar 2020 09:56:55 +0000

Dear Karen,

We posted an I-D in the TICTOC list , but there will be no TICTOC meeting scheduled (it used to be arranged together with NTP in the past).
Could this agenda also accommodate for this TICTOC topic (please see https://tools.ietf.org/html/draft-jiang-tictoc-syncphy-yang-00)?
If not, is there any other IETF working group you would recommend us to introduce this piece of work?

Best regards,
Yuanlong

-----Original Message-----
From: ntp [mailto:[email-masked]] On Behalf Of Karen O'Donoghue
Sent: Tuesday, March 24, 2020 12:23 PM
To: [email-masked]
Subject: [Ntp] Draft agenda and date for NTP wg virtual interim.

Folks,

Regards,
Karen

DRAFT NTP WG Agenda

    

5.  Interleaved Modes

7.  NTP v5 requirements 

8.  AOB

---

From: Watson Ladd <[email-masked]>
Date: Fri, 27 Mar 2020 09:03:19 -0700

I'd like to add NTS deployability to the agenda: the time is now to
decide if we need a coordinated new UDP port for NTP.

---

From: Ragnar Sundblad <[email-masked]>
Date: Fri, 27 Mar 2020 17:17:26 +0100

> On 27 Mar 2020, at 17:03, Watson Ladd <watson=[email-masked]> wrote:
> 

The NTS-KE server can choose to point to any port, so the user don’t even
have to know.

But is that what the ISP:s really want - they kill port 123, so that they
totally loose control?

There is a discussing going on about this on the nanog mailing list, called
"UDP/123 policers & status" (slowly, there is currently a lot of talk
around the covid-19 outbreak).

Ragnar

---

From: Watson Ladd <[email-masked]>
Date: Fri, 27 Mar 2020 09:41:34 -0700

> The NTS-KE server can choose to point to any port, so the user don’t even

Corporate firewalls might have problems with random ports. The other
thing to discuss is timelines and a coordinated push to evangelize
switching to NTS.

---

From: Ragnar Sundblad <[email-masked]>
Date: Fri, 27 Mar 2020 17:47:38 +0100

> On 27 Mar 2020, at 17:41, Watson Ladd <[email-masked]> wrote:
> 
> Corporate firewalls might have problems with random ports.

Yes, there is no single easy solution as far as I know. :-(

I would prefer if the ISP:s could stop size blocking port 123. But I
don’t know if that is realistic.
Rate limiting I think we must accept, maybe encourage as an alternative.

> The other

Right! Any ideas?

Ragnar

---

From: Miroslav Lichvar <[email-masked]>
Date: Mon, 30 Mar 2020 10:10:07 +0200

> don’t know if that is realistic.

An ISP told me they would have to replace an expensive piece of HW to
block only the amplification-prone modes (6 and 7). I suspect some of
the "anti-DDoS" midleboxes have the packet matching in the silicon and
care only about the NTP port number.

> Rate limiting I think we must accept, maybe encourage as an alternative.

NTS works poorly if there is a large packet loss. Too many NTS-KE
requests.

-- 
Miroslav Lichvar

---

From: Hal Murray <[email-masked]>
Date: Mon, 30 Mar 2020 02:11:37 -0700

[email-masked] said:
> NTS works poorly if there is a large packet loss. Too many NTS-KE requests. 

Are packet losses statistical or reproducible?

I'd expect that routes change infrequently and that a particular route either 
works or doesn't.  So "poorly" doesn't quite fit.

---

From: Ragnar Sundblad <[email-masked]>
Date: Mon, 30 Mar 2020 12:17:22 +0200

> On 30 Mar 2020, at 10:10, Miroslav Lichvar <[email-masked]> wrote:
> 

Yes, if they throw away more than 7/8 of the traffic for a long time,
there will likely be a new KE exchange. Hopefully that does not happen
very often.

Ragnar

---

From: Miroslav Lichvar <[email-masked]>
Date: Mon, 30 Mar 2020 12:23:10 +0200

> works or doesn't.  So "poorly" doesn't quite fit.

Some routes may not work at all for (large) NTP packets, but some
networks seem to have NTP-specific rate limiting. I sometimes see up
to a 90% loss for packets send to or from the port 123. The percentage
is changing over time. I guess it depends on how much NTP (or maybe
non-NTP) traffic is passing through the network.

It's a common problem for public NTP servers. The pool.ntp.org monitor
had to implement some changes to work around this issue. It was
dicussed many times on https://community.ntppool.org/.

-- 
Miroslav Lichvar

---

From: "Dieter Sibold" <[email-masked]>
Date: Mon, 30 Mar 2020 16:11:54 +0200

> there will likely be a new KE exchange. Hopefully that does not happen

I think it is too early to conclude that NTS-KE will fail because of 
rate limitations. Up to now there is not enough data available for a 
serious forecast. My own experience is that I didn’t experience any 
issue at all. But that said, it is clear that this might not be 
representative.

Dieter