Subject: [Ntp] NTPv5 and anycasting

From: Philip Prindeville <[email-masked]>
Date: Tue, 1 Dec 2020 22:41:01 -0700

Sorry if this has been discussed already, but do we want to have a specific prohibition against the use of anycasting with NTPv5?

I can’t see the point in sending packets non-deterministically to one of possibly many servers with different clock values, RTT’s, etc.

---

From: Miroslav Lichvar <[email-masked]>
Date: Wed, 2 Dec 2020 09:08:39 +0100

> I can’t see the point in sending packets non-deterministically to one of possibly many servers with different clock values, RTT’s, etc.

There is a section on anycast in the NTP BCP document. It can be
useful. I don't see a reason why NTPv5 specifically should prohibit
use of anycast. 

-- 
Miroslav Lichvar

---

From: Philip Prindeville <[email-masked]>
Date: Wed, 2 Dec 2020 10:48:43 -0700

> On Dec 2, 2020, at 1:08 AM, Miroslav Lichvar <[email-masked]> wrote:
> 

What’s the scenario where non-determinism is a good thing?

---

From: Paul Gear <[email-masked]>
Date: Thu, 3 Dec 2020 07:37:22 +1000

> What’s the scenario where non-determinism is a good thing?

Hi Philip,

I agree, and I'm curious to hear Miroslav's answer to this as well.

I can think of two scenarios where anycast might be useful, but neither
is really non-deterministic under normal conditions:

 1. Public services where the same well-known IP (or set of IPs) is
    anycast simultaneously from geographically dispersed locations and
    only moves when its prefix is withdrawn from BGP due to maintenance
    or faults.  (Similar to the various DNS services from Cloudflare,
    Google, etc.)  I would argue that the existing DNS-based public pool
    is a better solution than this for most use cases.
 2. A corporate WAN where clients are configured with a single NTP
    server IP, which is anycast from the local branch router.  My
    argument against using anycast here would be that an internal DNS
    pool is simpler and more robust, but I can see some organisations
    wanting to use anycast due to a mix of platform limitations,
    management software, and available skills.

I think RFC8633 section 7 was rather too gentle in its downplaying of
anycast, and would support wording stating that anycast SHOULD not be
used with NTP.  Of course, that's not really within scope of Miroslav's
v5 proposal.

Regards,
Paul

---

From: Steven Sommars <[email-masked]>
Date: Wed, 2 Dec 2020 22:29:31 -0600

Clients should be able to tell if a server uses anycast.
E.g., the client may not want to use *prefer* with an anycast NTP server

On Wed, Dec 2, 2020 at 3:40 PM Paul Gear <ntp=
[email-masked]> wrote:

> On Dec 2, 2020, at 1:08 AM, Miroslav Lichvar <[email-masked]> <[email-masked]> wrote:
>
> I can think of two scenarios where anycast might be useful, but neither is
> really non-deterministic under normal conditions:
>
>    anycast simultaneously from geographically dispersed locations and only
>    moves when its prefix is withdrawn from BGP due to maintenance or faults.
>    (Similar to the various DNS services from Cloudflare, Google, etc.)  I
>    would argue that the existing DNS-based public pool is a better solution
>    than this for most use cases.
>    server IP, which is anycast from the local branch router.  My argument
>    against using anycast here would be that an internal DNS pool is simpler
>    and more robust, but I can see some organisations wanting to use anycast
>    due to a mix of platform limitations, management software, and available
>    skills.
>
> anycast, and would support wording stating that anycast SHOULD not be used
> with NTP.  Of course, that's not really within scope of Miroslav's v5
> proposal.
>
> Regards,
> Paul
> _______________________________________________
> ntp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ntp
>

---

From: Philip Prindeville <[email-masked]>
Date: Wed, 2 Dec 2020 22:17:30 -0700

Or iburst.

> On Dec 2, 2020, at 9:29 PM, Steven Sommars <[email-masked]> wrote:
> 
> E.g., the client may not want to use prefer with an anycast NTP server 
> 
>

---

From: Watson Ladd <[email-masked]>
Date: Thu, 3 Dec 2020 07:23:50 -0800

As the operator of one of the biggest anycast NTP deployments in the
world, how about not?

At least in our case,  we use ECMP on the router, so if an association
uses a persistent port, it will get to the same server modulo
announcement changes due to server failure. I'd be happy to
collaborate in measurement research to quantify the magnitude of any
such effects when this is not the case. Route flaps are pretty rare,
so that's not much of an issue. I know the synchronization inside the
datacenter is quite good.

If we had to use DNS based load balancing that would be equally
nondeterministic, but would be significantly slower to adapt to server
faults, more susceptible to improper DNS or association caching by
clients, and complicated for boring architectural reasons. Anycast
scales much better without IP address consumption (for v4), while DNS
gives more flexibility in load balancing in the working case. DNS
would enable clients to use multiple different points of presence, and
figure out which has the best time, vs. being subject to the whims of
BGP. I don't know how much of an advantage that gives in realistic
scenarios.

In practice I think many single-homed servers are in fact anycasted
over a small number of servers behind a single Internet facing router
to provide redundancy. Once you're in the world of uptimes greater
than that of a single server, these sorts of setups are essential.

We also would not be able to join the pool without lots of operational issues.

I'm happy to collaborate in answering these questions: there are no
real secrets in the time service, and the NTP community has been very
helpful. We'd love to return the favor.

Sincerely,
Watson Ladd

---

From: Denis Reilly <[email-masked]>
Date: Thu, 3 Dec 2020 15:32:53 +0000

Hello Paul and Philip (and Watson):

The intent behind adding NTP over anycast to the BCP was to document a mode of operation that many network operators are already using. In my experience, it has its main value in Paul’s scenario 2. And when a network operator uses it on its corporate WAN, they need to understand that their choice will not lead to optimal time transfer, particularly when the network changes which server a client is talking to without that client’s knowledge. Perhaps that doesn’t happen often enough to cause concern? Ultimately, it is up to the operator to determine what their synchronization requirement is. Their application may be able to tolerate the slop when Anycast association changes, and so making configuration easier is worth the tradeoff.

This can also be accomplished via DNS, as you note, but I do know that there are some organization who prefer their anycast setup.

While I was composing this, Watson provided his reply, with much more relevant detail than I could have provided.

Best Regards,

--
Denis Reilly  |  Technical Lead  |  [email-masked]<mailto:[email-masked]>  (585)321-5837<tel:(585)%20321-5837>

From: ntp <[email-masked]> On Behalf Of Paul Gear
Sent: Wednesday, December 02, 2020 4:37 PM
To: [email-masked]
Subject: Re: [Ntp] NTPv5 and anycasting

CAUTION: This email originated from outside of the organization.
Do not click links or open attachments unless you recognize the sender and know the content is safe.
On 3/12/20 3:48 am, Philip Prindeville wrote:

On Dec 2, 2020, at 1:08 AM, Miroslav Lichvar <[email-masked]><mailto:[email-masked]> wrote:

use of anycast.

Hi Philip,

I can think of two scenarios where anycast might be useful, but neither is really non-deterministic under normal conditions:

  1.  Public services where the same well-known IP (or set of IPs) is anycast simultaneously from geographically dispersed locations and only moves when its prefix is withdrawn from BGP due to maintenance or faults.  (Similar to the various DNS services from Cloudflare, Google, etc.)  I would argue that the existing DNS-based public pool is a better solution than this for most use cases.
  2.  A corporate WAN where clients are configured with a single NTP server IP, which is anycast from the local branch router.  My argument against using anycast here would be that an internal DNS pool is simpler and more robust, but I can see some organisations wanting to use anycast due to a mix of platform limitations, management software, and available skills.

I think RFC8633 section 7 was rather too gentle in its downplaying of anycast, and would support wording stating that anycast SHOULD not be used with NTP.  Of course, that's not really within scope of Miroslav's v5 proposal.

Regards,
Paul

---

From: Philip Prindeville <[email-masked]>
Date: Thu, 3 Dec 2020 12:17:45 -0700

If the clients are (1) using 123 as both the source and destination ports as they should be, (2) behind a single masquerading IP address as most SoHo routers are, then (3) the first client out typically gets the source port reservation (unless the router itself is already using the port, in which case random source port assignments happen for everyone else).

> On Dec 3, 2020, at 8:23 AM, Watson Ladd <[email-masked]> wrote:
>

---

From: Kurt Roeckx <[email-masked]>
Date: Thu, 3 Dec 2020 23:42:16 +0100

> I can’t see the point in sending packets non-deterministically to one of possibly many servers with different clock values, RTT’s, etc.

I don't see how anycast changes anything at all. It's true that
because of routing changes that the RTT might change, but
that's not different then any other route change. The only thing
that's really different is that because of routing changes you
might actually talk to a different server, but without anycast
you can also talk to a different server each time because there is
a load balancer in between. And NTP should deal with all those
without any problem.

Kurt

---

From: Miroslav Lichvar <[email-masked]>
Date: Mon, 7 Dec 2020 11:34:24 +0100

> What’s the scenario where non-determinism is a good thing?

Probably none. I was responding to the first sentence about
prohibiting anycast in general.

-- 
Miroslav Lichvar