Subject: Re: [Ntp] Frequency transfer in NTP

From: Miroslav Lichvar <[email-masked]>
Date: Mon, 1 Feb 2021 10:37:09 +0100

On Sun, Jan 31, 2021 at 11:30:03PM +0100, Magnus Danielson wrote:
> perfectly. The first PLL loop will feed the "worst case" input into the
> next where it will amplify it most, and so it goes. The problem is
> already well analyzed classically, and what you need to do is to
> increase the damping factor, and hence reduce Q, of the PLL and this is
> done by increase the P factor of a PI-loop PLL.

Yes, you can dampen the loops to avoid the overshoot, but that will
have a negative impact on the timekeeping performance. You can
minimize the phase error, or frequency error, but not both at the same
time.

> You also use a statement that you cannot in the protocol convey both
> time and frequency errors. This is just plain incorrect. A phase
> measurement will include both.

The point that I'm trying to make is that transfering time and
frequency separately works better than time transfer alone.

> I have before mentioned that incorrect setting of parameters compared to
> polling rate will create overshoots. This is because P and I parameters
> needs to be scaled correctly with regards to the polling rate, or else
> the balance between them will alter the actual damping factor and thus
> going from over-damped to under-damped.

PI loops are extremely simple, but difficult to tune. I have plenty of
experience with them in PTP. I find it interesting that you would
consider using one in NTP, where the conditions are more variable,
there is more filtering, non-constant update interval, etc. Making one
comparable to the standard NTP loop would be tricky.

> I can provide a range of references, unfortunatly not all in online
> sources. I could do a fairly long derivation of equations for it, and
> you can find posts from me where I do that.

If you wanted to be constructive, it would be best if you showed us an
example of a loop that doesn't overshoot in the step response and
otherwise performs similarly to ntp or chrony.

We can compare it a chain of servers and see if it performs better
than the ntpd and chronyd patched for frequency transfer.

-- 
Miroslav Lichvar

---

From: Magnus Danielson <[email-masked]>
Date: Mon, 1 Feb 2021 12:00:50 +0100

Miroslav,

> minimize the phase error, or frequency error, but not both at the same
Turns out that as you measure the phase stability in MTIE and TDEV, as
well as frequency stability in ADEV, you need to remove that overshot /
resonance for best stability. This becomes especially true as you build
a chain of them, as it will grow worse along the chain.
> frequency separately works better than time transfer alone.

I have yet to see that work and make a distinction which is not
expressable by other means.

It is only when you do frequency transfer by some completely different
means that you can get a benefit, but not within the same protocol. Been
there, done that, did not pan out.

> PI loops are extremely simple, but difficult to tune.

While extremely simple, difficult to tune has not been my experience.
Once one have done once gain scaling right, properly orthogonal setting
of damping and frequency have been straight-forward. Getting optimum
performance given a certain condition is as always a bit tricker, but
not impossible.

As for simple, I have taught my colleagues how the loop works, and they
always got it lock up quickly and with very little training they have
been able to hand-tweak the parameters to about the right neighborhood,
after which we used good measurement tools to fine-tune.

>  I have plenty of

Non-constant update intervals is something you can parameter in. It goes
into the gain of the integrators of the loop, as always. The
polling-rate scale factor to the loop is not too hard to scale up.

The PI loop is simple, but very robust if treated right, so all one
needs to focus on is doing that, and that comes down to setting damping
factor and gain factors appropriatly.

A Kalman filter would adapt dynamically, and that's nice, but as it
converge to a stable steady state setting, it is just as a PI loop and
you have the same tuning issue with regard to the damping factor.
Semi-Kalman heuristics can be done fairly simple with a PI loop if you want.

I've worked with that assortment of tools, and they fit different needs.
Turns out, often you can do most of it just fine with a PI-loop and not
waste too much time and energy on it. Some cases needs the Kalman for
sure. Networks will not only need Kalman, you need non-linear
assistance, and that is what I have done when needed.

> otherwise performs similarly to ntp or chrony.
I could provide an example of scaling with poll-rate, sure. I just don't
have the setup to do measurement of a chain, so that would take time and
effort that I might not be able to pull together, but others may have that.
> than the ntpd and chronyd patched for frequency transfer.

Recommended reading: Trischita & Verma "Jitter in Digital Transmission
Systems" from Artech House 1990. You can then find more context in ITU-T
G.810-813 and G.823-825. It's interesting to note that the reference
network used is a chain of 60 nodes, so they have for sure been looking
at the long chain issue. I can provide you with more references if you
like. I've always found Stefano Bregni "Synchronization of Digital
Telecommunications Network" a very handy book, and you should be able to
locate his home page and lots of reading material.

You might object that this does not match up with how NTP work. Well,
turns out that terminology etc. might be different, some technical
details different, but once one look at things coming down to time/phase
measures and frequency it ends up being more or less the same. You can
do the same for GPS/GNSS tracking as well. While the technical details
may look different, the overal dynamics ends up being quite similar. The
one thing that makes packet timing a bit unique, is the statistics of
pakets. As you do a two-way time-transfer you at least can get a
balanced "noise", but using non-linear methods you can do much better.
Core timing dynamics of the control loop ends up being more or less the
same situation.

I've also spend my time to build chains at work, I recall a 26 node
chain at one time, and it shows for sure as we measure. The theory and
practice add up. It really does. We have one setup running constantly,
but it is not quickly adapted to test this, but we learned this the hard
way, I've already done the exercise, and what you are saying does not
match up with my experience, both failures and ways to make things work
really well. In practice, both me and my customers run chains beyond 15
nodes in length on daily basis, delivering critical signals (at least
that's what our customers say they are), so if we got this wrong, let me
assure you that I would hear about it. In particular, as some of the
signals we carry is really very sensitive to timing disturbtions. So,
when I advice about this, it's not just an opinion, it's based on both
my own and others hard earned experience. I have designed and
implemented a time-transfer system independent of PTP, so I have played
the same game, both over telecom links and packets.

Turns out that jitter tolerance is another aspect, and high deviations
ends up being problematic there, again driving the Q down.

The same issue shows up in high speed digital jitter links, relating to
signal integrity and the link bit error rate. There is an extensive
report on that

So I disagree with you, based on all that science and personal
experience. I spent a lot of time to understand the issue, I've done the
same mistakes as you, and recovered from them.

Cheers,
Magnus

---

From: Miroslav Lichvar <[email-masked]>
Date: Mon, 1 Feb 2021 12:38:56 +0100

> a chain of them, as it will grow worse along the chain.

Right. You cannot minimize the phase error upper in the chain without
disturbing the clocks furher down in the chain. A sacrifice needs to
be made to keep the chain stable unless the frequency is transfered
separately. That's what I tried to explain in the post.

> performance given a certain condition is as always a bit tricker, but

The tricky part is making it adaptive. It's easy to tune a PI loop for
a well-characterised clock and network, but making it work well over a
wide range of conditions, as is common in NTP, is hard.

> effort that I might not be able to pull together, but others may have that.

I'd like to see an example of a loop (it doesn't have to be PI) that
performs similarly or better than an implementation using the proposed
frequency transfer. I don't think that is possible. You say it is, so
an example would be a good way to show that.

Just post the code of the loop. In any language you like. I can
integrate it with NTP and compare it with the existing
implementations.

-- 
Miroslav Lichvar

---

From: Magnus Danielson <[email-masked]>
Date: Mon, 1 Feb 2021 13:18:15 +0100

Miroslav,

> separately. That's what I tried to explain in the post.

Once the phase error is small enough, other performance characteristics
tend to dominate, such as frequency stability (which is ADEV for random
noise and MAFE for systematic noise) and phase stability (which is TDEV
for random noise and MTIE for systematic noise). Turns out it works
really well in practice to get very low offsets doing this, as this is
what I've spent a very long time to perfect. You did not really achieve
the goal in the post as it lacked a number of considerations that I
tried to point out.

Also, the overshots tends to extend the time before equilibrium is
achieved, so it takes longer time to get that stable offset.

> wide range of conditions, as is common in NTP, is hard.
For sure. That's why you end up having to consider adaptive filters.
However, that adaptive filter needs to respect the basic rules
regardless, so one needs to get the resonance part right to start with
and only then one make it auto-tuneable. Those is orthogonal properties.
The PI loop is sufficient to understand the basic. We then only want to
tune the frequency for best cut-over between link noise and local
oscillator noise (and drift).
> an example would be a good way to show that.

Let me see if I can whip up a little simulator for you then. I've done
that before and made it match up with the reality just nicely.

There is one thing you can do to improve things, but it does not involve
frequency but rater phase. That's where the is place for possible
improvements in my mind. I also said that before we can look at
benefits, we need to fix the basic problems of the setup. Only after
that we can measure things, and I did point out that frequency aiding
alters the damping factor and hence the reason we see damping factor
improvements, but the initial damping factor was way of what it should
be for a chain network, so it is not relevant comparison. If I do an
effort, you will have to do one where you fix your extremely unfair
comparisons, as you end up not proving improvement over know good
situation. What you have ends up being meaningless comparison for me, it
needs to start from a well-tuned setup and then show further improvements.

Also, what you show is transient behavior, not long-term errors, not the
usual measure of stability. You need to do that. My experience is that
transient behaviors needs to be taken way down, and first things to fix
is damping. Once damping, and associated scaling with polling rate is
done, only then it becomes meaningful to consider fine-tuning of
frequency which is the adaptive filter trap. However, even a
non-adaptive setup should not be too bad, and the chain problems show up.

I recommend you to look at TimeLab and Stable32 for analysis. Stable32
you can download for free from IEEE UFFC, it used to be a commercial
software.

> integrate it with NTP and compare it with the existing
Notice that no "code" will just work as you drop it in. The gain factors
needs to be corrected for any implementation. Building code to tune that
is more or less painful depending on the implementation. Things needs to
be normalized and parametrized for this to have a chance of working. I
don't have time to make it fool proof.

Cheers,
Magnus

---

From: Miroslav Lichvar <[email-masked]>
Date: Mon, 1 Feb 2021 16:19:29 +0100

> that before and made it match up with the reality just nicely.

That would be great.

> is damping.

I think the response to a large time/frequency step indicates how well
it will perform in a chain. You can scale it down to any numbers you
like. In normal operation there are smaller offsets, but the response
is identical in proportion. It's a series of responses that add up,
together with the frequency errors coming from the clock drift and
measurement noise due to network jitter.

As for long term errors, I have some data for chrony:

st	no-freq		freq
2	11.7		12.0
3	18.1		15.4
4	24.7		17.2
5	30.1		20.7
6	40.1		22.5
7	50.4		25.9
8	64.2		26.9
9	77.1		29.0
10	91.7		31.4

It is the RMS offset (in microseconds) at different strata of the
chain. As you can see, the error grows much faster without frequency
transfer. At stratum 10 it's already 3x better.

If I slowed down the loop as you suggest, the error would not grow as
quickly, but it would start at a larger value.

> don't have time to make it fool proof.

Ok, just specify the conditions. A simulation is fine. To show the
impact of independent frequency transfer, I'm mostly interested in the
case where frequency errors dominate noise in measurements, i.e. when
the Allan intercept is too short for the polling interval. Let's say a
polling interval of 64 seconds, a network jitter of 10 microseconds
and a random walk of 1 ppb/s for the clock?

-- 
Miroslav Lichvar