Subject: Re: [Ntp] NTP over PTP

From: Miroslav Lichvar <[email-masked]>
Date: Thu, 1 Jul 2021 12:24:54 +0200

On Wed, Jun 30, 2021 at 03:42:53PM +0200, Heiko Gerstung wrote:
> > Well, yes, but the delay measurement is separate from the offset
> > measurement. If you can log and plot the offsets measured by PTP and
> > NTP in a network without PTP support, you will see how the
> > distribution is different due to the different calculation.
> 
> This depends, as pointed out earlier, on the network itself. If you have highly dynamic network paths where traffic patterns change quickly and therefore result in wildly changing queueing and forwarding delays, you will see slightly different delay distribution in NTP and PTP logs. In less dynamic environments, you will probably not notice something like that at all.

I'm referring to the distribution of the offset as measured by PTP
and NTP. Maybe this plot will make it more clear.

https://fedorapeople.org/~mlichvar/tmp/ntp-ptp-offset.png

It shows offset over time as calculated by NTP and PTP in identical
conditions. Hopefully you can see that the offset on the top has a
symmetric distribution and one on the bottom does not. With the PTP
offset calculation (same as in the NTP broadcast mode), the
distribution of the network delay in the server->client direction
transfers to the distribution of the offset. In the NTP client/server
mode the offset is calculated from the sum of delays in both
directions. It's not difficult to guess what is easier to work with
when controlling a clock.

> And in networks with partial or full on-path support, you will see that PTP is performing great with its inferior delay measurement approach. 

With full on-path support, yes, it performs great. That's what it was
designed for. Without full support on a mostly idle network or in a
network where PTP messages are prioritized, it can still work ok. But
as soon as PTP messages start waiting in longer queues, it falls apart
very quickly. NTP can generally perform well in worse conditions as it
measures the offset and delay at the same time, while PTP assumes the
delay didn't change when it measures the offset.

Of course, there are differences between implementations. A more
advanced PTP implementation can certainly perform better than a less
advanced NTP implementation. PTP implementations can also ignore the
PTP specification and use the NTP approach.

> >> Yes, there are implementations which take that into account by applying
> >> static correction values to compensate for link speed asymmetry. I believe this
> >> also affects NTP, but in most cases hundreds of nanoseconds are not a problem
> >> for applications relying on NTP synchronization.
> > 
> > NTP is much less impacted as it timestamps the end of the reception.
> > A software timestamp is captured after the packet is received.
> 
> That is a very confusing argument. 

I'm talking here specifically about the error that is caused by server
and client being on different link speeds, not other sources of error
due to processing delays, etc. NTP using trailing timestamps (which
requires transposition of hardware timestamps) is another reason why
NTP should be preferred over PTP in networks that don't have a full
on-path support.

> When we connect one of our grandmaster clocks (PTP server) to one of our PTP client devices over a direct crossover cable or fiber link, we can measure an offset of less than 20ns between the PPS generated by the GNSS synchronized master clock and the client. 

I see the same with NTP here.

> > Isn't that an issue for both NTP and PTP unicast using high rate sync
> > and delay requests?
> 
> Yes, but unicast PTP servers with hardware time stamping are typically commercial products with dedicated network hardware to support the large number of timestamps that have to be stored and processed.

Would this hardware not work with NTS-over-PTP, but work with
NTS4UPTP?

> The i210 has a time stamping queue length of 4, i.e. it stores a maximum of 4 timestamps in a ring buffer. A unicast PTP server with 10 clients running at 128 sync and delay pkt/s will generate 1,280 time stamps per second. Even on some serious hardware there will be a significant number of timestamps that are lost due to queue overruns.  

I think you are confusing the I210 with something else. It can
timestamp all received packets, at any rate.

Its performance in a server is limited by TX timestamping. It can
timestamp only one packet at a time. But the maximum rate is
definitely higher than 1280 per second. In my tests it's about 35000
TX timestamps per second. That's with the stock Linux driver which
seems to rely on an interrupt. I think it could be modified to poll
the timestamp register for a much higher rate of timestamps. In any
case, I'd say that is pretty good for a $50 NIC.

> > I think the best approach is for the hardware to timestamp all packets
> > as many NICs already do. The problem is with existing hardware that
> > cannot do that. I agree it doesn't look great when you have to run NTP
> > over PTP, but that seems to be the only way to get the timestamping
> > working on this hardware.
> 
> I believe there should be a standard that adds a hardware timestamp to the end of every Ethernet frame. It requires a NIC vendor to implement a hardware clock and a time stamping engine into their silicon. The bandwidth between the NIC and the OS layer (driver) has to accommodate the extra data (but you could reduce the MTU of course). 

That's how it works with the I210 and other NICs, except the timestamp
is before the frame, not after.

-- 
Miroslav Lichvar

---

From: Heiko Gerstung <[email-masked]>
Date: Fri, 2 Jul 2021 12:57:04 +0200

I am trimming the quoted text a bit to improve readabililty...

>>> [...]
>> This depends, as pointed out earlier, on the network itself. If you have
>> highly dynamic network paths where traffic patterns change quickly and
>> therefore result in wildly changing queueing and forwarding delays, you will
>> see slightly different delay distribution in NTP and PTP logs. In less dynamic
>> environments, you will probably not notice something like that at all.
> 
> and NTP. Maybe this plot will make it more clear
Thank you for this. It has no scale, but I think I understand where you are coming from here.

> when controlling a clock.

Agreed, that was why I referred to PTP's delay measurement mechanism as inferior to NTP. In practice, 
the PTP approach is good enough(tm) to provide a similar performance when compared to NTP.

>> And in networks with partial or full on-path support, you will see that PTP
>> is performing great with its inferior delay measurement approach.
> 
Agreed as well. 

> PTP specification and use the NTP approach.
Yes, with timing it is very easy to mess things up considerably as the whole chain 
is important. 

>>>> [...]

> NTP should be preferred over PTP in networks that don't have a full
OK. Thank you for the additional explanation.

>> When we connect one of our grandmaster clocks (PTP server) to one of our PTP
>> client devices over a direct crossover cable or fiber link, we can measure an
>> offset of less than 20ns between the PPS generated by the GNSS synchronized
>> master clock and the client.
> 
With hardware timestamping, yes. If all conditions are the same (high packet rates, 
network conditions and hardware timestamping) both NTP and PTP can achieve similar
performance. 

>> Yes, but unicast PTP servers with hardware time stamping are typically
>> commercial products with dedicated network hardware to support the large number
>> of timestamps that have to be stored and processed.

> Would this hardware not work with NTS-over-PTP, but work with
Some of the hardware would work, for sure. However, there needs to be software 
that is implemented by the commercial vendor. 

>> The i210 has a time stamping queue length of 4, i.e. it stores a maximum of 4
>> timestamps in a ring buffer. A unicast PTP server with 10 clients running at
>> 128 sync and delay pkt/s will generate 1,280 time stamps per second. Even on
>> some serious hardware there will be a significant number of timestamps that are
>> lost due to queue overruns.
> 
I was talking about TX timestamps, sorry if that did not come 
across (server sends sync messages).

> case, I'd say that is pretty good for a $50 NIC.
RX is no problem, obviously. A unicast server needs to send a lot of packets 
(and then has to read out the timestamps in 2-step mode). We ran some tests 
with this chip and ran into problems when trying to use it in server mode. 

However, I wholeheartedly agree that this NIC is a great value for the money. We have two of those
on every CPU board and love them for their reliability and versatility.  We do not use them for PTP 
but that might change at some point. 

If you managed to get 35k TX timestamps out of the chip, it is good for a maximum of ~270 clients, which
is probably good enough for quite a number of setups. As I mentioned earlier, our hardware 
can do 250k TX timestamps and is good for ~2k clients. But of course there are not a lot of applications out there
requiring that firepower, and maybe in those cases the i210 will perfectly do the job.

>>> [...]
>> I believe there should be a standard that adds a hardware timestamp to the
>> end of every Ethernet frame. It requires a NIC vendor to implement a hardware
>> clock and a time stamping engine into their silicon. The bandwidth between the
>> NIC and the OS layer (driver) has to accommodate the extra data (but you could
>> reduce the MTU of course).
> 
Yep, thanks for pointing that out. I am not sure if the i210 can hardware timestamp every packet, but even 
if it does, this is only feasible for the RX path as you need some sort of ring buffer to store TX timestamps
until they are read out by a userspace application. So on the TX path it might be better to not timestamp
everything and instead be able to flag a frame/IP address for timestamping. 

The i210 also seems to support 1step, which might make it possible to send out even more packets as 
you do not actually need to read out the timestamp from userspace. It is really a great piece of hardware 
(the "datasheet" is 870 pages .....). 

If you agree, I propose to close this thread here. We definitely move into more and more off-topic discussions 
and I would like to re-focus on the draft documents that you, I and others submitted. Happy to carry on off list,
of course.
 
Regards,
   Heiko

> --
> Miroslav Lichvar
> 
> _______________________________________________
> ntp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ntp

-- 
Heiko Gerstung 
Managing Director 
 
MEINBERG® Funkuhren GmbH & Co. KG 
Lange Wand 9 
D-31812 Bad Pyrmont, Germany 
Phone: +49 (0)5281 9309-404 
Fax: +49 (0)5281 9309-9404 
 
Amtsgericht Hannover 17HRA 100322 
Geschäftsführer/Management: Günter Meinberg, Werner Meinberg, Andre Hartmann, Heiko Gerstung 
 
Email: 
[email-masked]
Web: 
Deutsch https://www.meinberg.de
English https://www.meinbergglobal.com
 
Do not miss our Time Synchronization Blog: 
https://blog.meinbergglobal.com
 
Connect via LinkedIn: 
https://www.linkedin.com/in/heikogerstung

---

From: "Ulrich Windl" <[email-masked]>
Date: Tue, 06 Jul 2021 10:31:41 +0200

Hi!

Reading RFC 8633, Appendix A, specifically A.2, I was confused a bit:
The last line says "restrict source..." (and the preceding lines had "restrict default...").
The point is that "default" is a literal keyword, but "source" is not (AFAIK).
If that's true there should be a sentence stating that "source" is a placeholder, not a literal keyword.
Unfortunately the manual page for ntp.conf also has "source in bold face in "... you'll  want  a  restrict source ...".
As it's not explained what "source" means, I guess it's a placeholder, and this (now talking about the manual page) it should be typeset in italics, not in bold face.

Regards,
Ulrich

---

From: "Ulrich Windl" <[email-masked]>
Date: Tue, 06 Jul 2021 10:39:10 +0200

>>> "Ulrich Windl" <[email-masked]> schrieb am 06.07.2021 um
10:31 in Nachricht <[email-masked]>:
> Unfortunately the manual page for ntp.conf also has "source in bold face in 

Ignore the double-quote in the line above, or add another one after 'source'.

> "... you'll  want  a  restrict source ...".
> As it's not explained what "source" means, I guess it's a placeholder, and 
> this (now talking about the manual page) it should be typeset in italics, not 
> in bold face.
> 
> Regards,
> Ulrich
> 
> 
>

---

From: Martin Burnicki <[email-masked]>
Date: Tue, 6 Jul 2021 10:55:43 +0200

Hi Ulrich,

> The point is that "default" is a literal keyword, but "source" is not (AFAIK).

Yes, "restrict source ..." it valid syntax, which has been introduced 
int ntpd 4.2.8, if I remember correctly.

The HTML docs of the current ntpd distribution says (in access.html):

"restrict source" configures a template restriction automatically added 
at runtime for each association, whether configured, ephemeral, or 
preemptable, and removed when the association is demobilized.

If I remember correctly, this was introduced to fix a problem e.g. with 
pool server entries. If you had a restrict line like

restrict default ... nopeer

then automatic associations e.g. due to the "pool" directive wasn't 
possible.

restrict source ...

now adds the specified permissions to such associations that are created 
automatically.

Martin
-- 
Martin Burnicki

Senior Software Engineer

MEINBERG Funkuhren GmbH & Co. KG
Email: [email-masked]
Phone: +49 5281 9309-414
Linkedin: https://www.linkedin.com/in/martinburnicki/

Lange Wand 9, 31812 Bad Pyrmont, Germany
Amtsgericht Hannover 17HRA 100322
Geschäftsführer/Managing Directors: Günter Meinberg, Werner Meinberg, 
Andre Hartmann, Heiko Gerstung
Websites: https://www.meinberg.de  https://www.meinbergglobal.com
Training: https://www.meinberg.academy

---

From: "Ulrich Windl" <[email-masked]>
Date: Tue, 06 Jul 2021 14:09:08 +0200

>>> Martin Burnicki <[email-masked]> schrieb am 06.07.2021 um
10:55 in
> int ntpd 4.2.8, if I remember correctly.

Sorry for that. I looked up "man ntpd.conf" for ntp-4.2.8p15-94.2.x86_64 of
SLES 12 and there "source" is not explained.

> now adds the specified permissions to such associations that are created 

OK, but still an explaining sentence wouldn't have been that wrong ;-)

Regards,
Ulrich

---

From: Harlan Stenn <[email-masked]>
Date: Tue, 6 Jul 2021 12:52:22 -0700

In 'restrict source ...' the word 'source' is a literal keyword.

-- 
Harlan Stenn <[email-masked]>
http://networktimefoundation.org - be a member!