Subject: [ntpwg] Documents, slides, etc. from WG meeting

From: brian at innovationslab.net (Brian Haberman)
Date: Tue, 23 Oct 2007 12:15:31 -0400

The minutes from the meeting have been posted to the IETF Proceedings 
page.  They are available at:

http://www3.ietf.org/proceedings/07jul/minutes/ntp.txt

Regards,
Brian

Brian Haberman wrote:
> Danny,
>      The -07 version of the spec and the pdf of the last call comments
> and resolutions are available at www.innovationslab.net/ntp.  When the
> minutes are available, they will be posted to this list for review.
> 
> Regards,
> Brian
> 
> 
> Danny Mayer wrote:
>> Could someone make available the pointers to the change documents, etc.
>> plus whatever slides where shown and minutes when available? Also could
>> we get the attendee list? It was hard to make out who was who on the
>> streaming audio.
>>
>> I was able to listen in at least to most of the meeting and I'm happy
>> with the progress you made. Good job everyone!
>>
>> Let's get this round of RFC's finished!
>>
>> Danny
>> _______________________________________________
>> ntpwg mailing list
>> ntpwg at lists.ntp.org
>> https://lists.ntp.org/mailman/listinfo/ntpwg
> _______________________________________________
> ntpwg mailing list
> ntpwg at lists.ntp.org
> https://lists.ntp.org/mailman/listinfo/ntpwg

---

From: mayer at ntp.isc.org (Danny Mayer)
Date: Tue, 23 Oct 2007 23:51:36 -0400

There was this interesting comment at the end of the minutes:

> A long discussion at end of meeting discussing the future of 
> timekeeping here in the IETF.  Is the future 1588 over IP?  
> NTP over IP?  No consensus was reached.

Could some explain exactly what that means? Is this a question of
certain protocols (DNS and Kerberos come immediately to mind) that
require that two interacting servers keep their clocks within X minutes
of each other in order for various security requirements to work
correctly or ro be assumed to be valid? Is it something else?

Thanks,
Danny

---

From: brad at shub-internet.org (Brad Knowles)
Date: Wed, 24 Oct 2007 02:34:40 -0500

>  correctly or ro be assumed to be valid? Is it something else?

Well, NTP is inherently an IP protocol.  From reading the IEEE 1588 
page, it's not clear whether this works over IP or not, although it 
does seem to work over Ethernet (and maybe others?).  But IEEE 1588 
seems to focus on ultra-high resolution, like sub-microsecond or even 
sub-nanosecond.  Meinberg appears to have a clock that does both NTP 
and IEEE 1588.

But I'm not seeing how IEEE 1588 can provide synchronization to an 
outside reference, and I'm not seeing how it can be used outside of 
certain limited circumstances on a pretty much pure Ethernet network.

-- 
Brad Knowles <brad at shub-internet.org>
LinkedIn Profile: <http://tinyurl.com/y8kpxu>

---

From: stbryant at cisco.com (Stewart Bryant)
Date: Wed, 24 Oct 2007 09:45:35 +0100

Brad
>   From reading the IEEE 1588 
> page, it's not clear whether this works over IP or not.
In discussing 1588, you need to make sure you are  talking about the V2 
spec.
V1 is  not likely to see significant deployment, whist V2 is likely to 
get significant
hardware support and hence be widely deployed.

Both V1 and V2 can use multicast IP encapsulation and you can in 
principle build an
IP multicast net, but I have always regarded this mode of operation as a 
little
strange and unlikely to ge deployed outside a highly controlled network. 
I doubt
if we will ever see the 1588 multicast mode on the Internet.

However V2 can run over a unicast IP connection, and this is likely to find
use in wide area networks. Indeed it exists largely to support the needs of
the telecommunications operators for a method of providing clock sync over
packet.
> , although it 
I am not sure what you mean by this. How the grand master acquires and 
locks to
an external clock reference is outside the scope of the protocol, but 
it's also outside the
scope of the NTP. However 1588 does have a grand master election mechanism
that - given honest representation by the clocks - selects the best 
clock as the
grand master to which the slaves synchronise and avoids the formation of
timing looks.

Stewart

---

From: brad at shub-internet.org (Brad Knowles)
Date: Wed, 24 Oct 2007 05:54:07 -0500

On 10/24/07, Stewart Bryant wrote:

>  spec.  V1 is  not likely to see significant deployment, whist V2 is likely
>  to get significant hardware support and hence be widely deployed.

Therein lies the problem.  I've been going through the page at 
<http://ieee1588.nist.gov/>, and I can't find any real discussion of 
the protocol, or specifics of the implementations.  I didn't even 
find any reference to V1 versus V2, and hence did not know that there 
was an updated version.

Sure, they reference specific chipsets as implementing IEEE 1588, and 
other specific devices (this is how I found out about the clock that 
Meinberg makes).

But when the description given for IEEE 1588 is completely circular 
in nature, just exactly what good does that do for anyone?

>  principle build an IP multicast net, but I have always regarded this
>  mode of operation as a little strange and unlikely to ge deployed
>  outside a highly controlled network. I doubt if we will ever see the
>  1588 multicast mode on the Internet.

We've talked to one of the guys at ISC about broad use of multicast 
on the Internet, and they've explained why they believe that would be 
a bad idea.

You may get multicast internal to a given network at a single entity, 
but you're unlikely to get it anywhere else.

>  the telecommunications operators for a method of providing clock sync over

If it's used by telecom operators, why do they need IP or anything 
else that heavy?  I mean, if we're talking about something like SONET 
synchronization, and they're worried about sub-nanosecond resolution, 
then it seems to me that IP is a very hefty hammer to be wielding.

>  locks to an external clock reference is outside the scope of the protocol,
>  but it's also outside the scope of the NTP.

But we have reference clocks, right?  And we have a defined way to 
interface with them?

Honestly, I'm not trying to argue.  I'm trying to understand better 
where IEEE 1588 fits into the picture, and I'm just not finding much 
in the way of what I consider to be useful documentation on the IEEE 
website.

 From what little I can tell, it seems to me that IEEE 1588 is much 
more oriented towards a closed single-entity local area (or 
near-local area) network environment, where you don't care if you're 
sync'ed to any given external reference, or how close or far you may 
be from The One True Time, you just care that all your internal 
slaves are sync'ed to your given master, and that you have some 
pretty tight tolerances on how far out of sync the slaves can get.

In contrast, it seems to me that NTP works well on the broader WAN 
multi-entity internet (global and beyond), and that it can work just 
fine on smaller scale single-entity networks where sub-nanosecond 
accuracy timekeeping is not required.

The two protocols seem to complement each other pretty well, and it 
would appear that the folks at Meinberg agree.

Ahh, okay.  Found the URL for one of the tutorials at 
<http://ieee1588.nist.gov/tutorial-basic.pdf>, which was a little 
more diffcult to find than it needed to be, thanks to the frame-based 
way the web pages are laid out.

Anyway, Page 6 of the PDF is where we want to start, where they 
specifically compare IEEE 1588 and NTP, among others.  I think 
they're wrong to target the peer-to-peer mode exclusively for NTP, 
but that is certainly one potential model that could be used with 
NTP.  There are other master/slave relationships that would be more 
typical for use with NTP, but that doesn't really affect the 
conclusions you can draw from those slides.

Slide 9 is also useful:

	Objectives of IEEE 1588

	+ Sub-microsecond synchronization of real-time clocks in
	  components of a networked distributed measurement and
	  control system [0]

	+ Intended for relatively localized systems typical of
	  industrial automation and test and measurement environments. [0]

	+ Applicable to local area networks supporting multicast
	  communications (including but not limited to Ethernet)

	[0] indicates objectives that may be extended in version 2

Hmm.  Did they fix the assumption of symmetric delays between master 
& slave in V2?

Or the complete lack of any kind of security?  I mean, UDP is 
trivially easy to spoof, and given that everyone on the subnet is 
supposed to be operating with the exact same information and the 
exact same algorithms, it seems like it would be simple to see who 
the current master clock is and spoof them as being bogus and force a 
new master clock to be selected -- which you should be able to 
guarantee is your selected trojan machine....

---

From: heiko.gerstung at meinberg.de (Heiko Gerstung)
Date: Wed, 24 Oct 2007 14:07:27 +0200

> certain limited circumstances on a pretty much pure Ethernet network.

There are several mappings for IEEE1588 in V2, I guess that the UDPv4, 
UDPv6 (layer 3) and Ethernet (layer 2) based mappings will see a larger 
installation base in the future. As far as I remember, V2 is still being 
worked on in the standardization process, although it looks like it is 
almost done. That might be the reason why the IEEE1588 page does not say 
anything about it.

The main purpose of PTP/IEEE1588 is not "absolute" synchronization to 
UTC but "relative" synchronization to the same source. Since one of its 
design goals (as far as I interprete it) is sub-microsecond accuracy, it 
is not working very well over WAN links and other asymmetric network 
infrastructures (imagine the unpredictable delay a regular 
store-and-forward switch introduces when it is queueing packets) . NTP 
does a much better job in such an environment, no doubt about that.

 From our perspective (as a vendor selling products in both worlds) we 
see that PTP is trying to solve a lot of problems that NTP takes care of 
since a couple of years, like redundancy of sync references, unicast 
operation and security. The one thing that really enables PTP to reach 
an impressive accuracy level over an Ethernet/UDP connection is the fact 
that it uses hardware time stamping and therefore requires specialized 
hardware (if you want to benefit from that superior level of accurcacy).

Our GPS synchronized grandmaster is in fact a stratum 1 NTP timeserver 
with an add-on module which adds a PTP/IEEE1588 network port (with 
hardware timestamping) to the package. Of course there are other 
products (GPS synchronized grandmaster clocks) from other vendors. This 
type of product results in a PTP network being synchronized to an 
outside source (GPS), just like a GPS synchronized NTP server does the 
same for its clients.

Cheers,
Heiko

---

From: stbryant at cisco.com (Stewart Bryant)
Date: Wed, 24 Oct 2007 13:25:19 +0100

Brad Knowles wrote:
> the protocol, or specifics of the implementations.  I didn't even find 
> any reference to V1 versus V2, and hence did not know that there was 
> an updated version.
V2 has not been published yet. It is currently mid way though the IEEE 
ballot process at the moment. You may also wish to be aware of the 
802.1AS project which is based on 1588 and is also in IEEE ballot.

You could also look at the proceedings of a number of the timing 
conferences over the past couple of years - WSTS, ITSF and the IEEE1588 
conference for more implementation and application information.
> But when the description given for IEEE 1588 is completely circular in 
> nature, just exactly what good does that do for anyone?
I am not sure what you mean by that.
> We've talked to one of the guys at ISC about broad use of multicast on 
> the Internet, and they've explained why they believe that would be a 
> bad idea.
>
They are migrating off SONET to Ethernet, and hence loose the ability to 
deliver sync to some applications (for example cell sites) that need 
sync. There is some work on Synchronous Ethernet (meaning Ethernet with 
synchronous carrier, NOT synchronous packets, but that requires an end 
to end sync path which may not be available, hence the interest in sync 
over packet. There are many other applications for tiem and frequency 
sync, but the mobile phone network is the poster child.
> But we have reference clocks, right?  And we have a defined way to 
Sure, you have Caesiums and GPS locked Quartz and Rhubidiums clocks, 
some subset of these speak 1588, just as some subset of them currently 
speak NTP, and by analogy some subset speak GPS.
> in the way of what I consider to be useful documentation on the IEEE 

You could read John Eidson's book, though this is mainly about V1. 
However it does give a lot of context and the applications are invariant 
to a V1 or a V2 solution.
> more oriented towards a closed single-entity local area (or near-local 
> area) network environment, where you don't care if you're sync'ed to 
> any given external reference, or how close or far you may be from The 
> One True Time, you just care that all your internal slaves are sync'ed 
> to your given master, and that you have some pretty tight tolerances 
> on how far out of sync the slaves can get.
Absolutely not.

One of the GM selection criteria is external reference and that includes 
GPS, Atomic etc.
> would appear that the folks at Meinberg agree.
The key issue is that we need better accuracy in the wide area.
> <http://ieee1588.nist.gov/tutorial-basic.pdf>, which was a little more 
> diffcult to find than it needed to be, thanks to the frame-based way 
> the web pages are laid out.
>
> specifically compare IEEE 1588 and NTP, among others.  I think they're 
> wrong to target the peer-to-peer mode exclusively for NTP, but that is 
> certainly one potential model that could be used with NTP.  There are 
> other master/slave relationships that would be more typical for use 
> with NTP, but that doesn't really affect the conclusions you can draw 
> from those slides.
>
> Slide 9 is also useful:
>
>     Objectives of IEEE 1588
>
> Hmm.  Did they fix the assumption of symmetric delays between master & 
> slave in V2?
No. The asymmetry is a  fundamental issue in all two way time transfer 
protocols
> supposed to be operating with the exact same information and the exact 
> same algorithms, it seems like it would be simple to see who the 
> current master clock is and spoof them as being bogus and force a new 
> master clock to be selected -- which you should be able to guarantee 
> is your selected trojan machine....
>
There is an informational description of a security mechanism in the V2 
specification, although I think that is something that the IETF should 
take a look at if we write and IETF 1588 profile.

Stewart

---

From: mayer at ntp.isc.org (Danny Mayer)
Date: Wed, 24 Oct 2007 08:35:04 -0400

Brad Knowles wrote:

> diffcult to find than it needed to be, thanks to the frame-based way the
> web pages are laid out.

This is all very nice but doesn't answer the question that I asked: How
does all of this affect IETF that there was a long discussion about it?
It's IETF that I was asking about not what IEEE1588 implements.

Danny

---

From: stbryant at cisco.com (Stewart Bryant)
Date: Wed, 24 Oct 2007 14:28:40 +0100

Heiko
> The one thing that really enables PTP to reach 
There is nothing to stop NTP doing h/w timestamping per se. The critical 
difference is that PTP (1588)  does on path correction through the use 
of the TC mechanism. If TC were added to NTP the two protocols would 
provide the same level of accuracy.

Stewart

---

From: martin.burnicki at gmx.de (Martin Burnicki)
Date: Wed, 24 Oct 2007 15:45:37 +0200

Hi Brad,

> pretty tight tolerances on how far out of sync the slaves can get.

Yep. However, it depends. Basically IEEE 1588 (PTP) is capable of sync'ing all 
clients to a server/grandmaster with high accuracy. If the application does 
not only rely on the _same_ time for all nodes but also on the _right_ time 
the you just have to take care that the grandmaster is sync'ed e.g. to a GPS 
receiver.

BTW, the same is essentially true for NTP.

> would appear that the folks at Meinberg agree.

I absolutely agree. 

The main difference between NTP and PTP is that the statistics implemented by 
the NTP algorithms yield quite good results even over WAN connections, and 
just using standard hardware.

PTP can achieve much higher accuracy than NTP, but this is _only_ if special 
hardware is used which explicitely supports PTP. And this basically applies 
to both PTP v1 and v2.

Imagine you want to send a packet with "the right time" from one PC to
another.

1.) The sending program picks up a time stamp and puts it into
    a network packet.

2.) The network packet is then passed to the IP protocol stack where
    it is passed down from the sending user space application to the
    network drivers which partially run in kernel space, where it
    finally ends up in a send queue. This introduces a delay which
    depends on the CPU power and the system load (interrupt requests).

3.) The network driver waits until the network wire is unused and
    starts to transmit the packet. If there's a collision on the wire
    then transmission is aborted and retried after a random delay.
    This also causes a random delay.

4.) Once the packet is on the wire the propagation delay is pretty
    constant, depending on the length of the wire. If there's a
    network hub between the sender and the receiver then this also
    introduces an additional delay, which is pretty constant, though.
    If there's a router or switch between the 2 nodes then the packet
    may be queued for an undetermined amount of time, which also
    results in an unknown delay.

5.) If the packet arrives at its destination then the network driver
    generates an interrupt request to let the packet be fetched by
    the protocol drivers. It also takes an unknown amount of time
    until this is done, depending on the CPU power, whether there
    are higher-prioritized interrupts just being handled, etc.

6.) Finally the packet is moved up the protocol stack, moved from
    kernel space back to user space, and passed to the application
    which then takes a time stamp of its own system time in order
    to compute the difference to the time stamp from the incoming
    packet.

There's a bunch of unknown delays there, isn't it?

The only delay which is constant and can be measured is the propagation delay 
on the cable.

The other effects introduce a more or less random delay which can only be 
estimated by statistical methods. That's what the NTP algorithms do with 
quite good results. The advantage is that no special hardware is required, 
but the disavantage is that you can only yield a limited accuracy.

In order to compensate the receive delay (i.e. when a packet comes in from the 
wire until it arrives at the application) you have to take a time stamp when 
a packet comes in from the wire. This is done by a time stamp unit which 
includes a pattern matcher which has to identify incoming packets in the bit 
stream from the wire, and take a time stamp if such a packet is detected. 
Both the network card driver and the application have to provide a way (API 
call) which lets the application retrieve that time stamp from the NIC driver 
and assign it to the associated (right) packet. This way the application can 
compute the difference between the time it has received the packet and the 
time the packet has arrived from the wire, and account for that delay.

Obviously the same has to be done for outgoing packets, i.e. determine the 
time interval from when the packet is sent by the application until it really 
goes onto the wire. The calculated delay has to be passed to the receiver 
which has to account for that delay. Unfortunately the time stamp can only be 
taken when the packet goes out onto the wire, so when the time stamp is 
available the packet has already been sent. The PTP protocol accounts for 
this situation by sending a so-called follow-up packet which contains the 
time stamp of the previous packet. The receiver then gets the original packet 
which is time-stamped when coming in, plus the follow-up packet which 
contains the transmission delay and can thus account for both the delays. 

Using a point-to-point connection between the transmitting and the receiving 
node you can yield an accuracy down to a couple of nanoseconds by hardware 
timestamping.

However, there's still a last delay which is not yet known by our server and 
client. This is the propagation delay across intermediate nodes like switches 
and routers. For example, if a switch receives an incoming packet at one 
port, and the outgoing port is just be used by another packet then the 
incoming packet is queued internally in a FIFO. This can take up to several 
tens of milliseconds (!), depending on the network load and the queue depth.

The problem is that neither the transmitting nor the receiving node can 
determine whether a packet has been passed on directly, or has been queued, 
and for how long it has been queued.

So even if both your endpoints provide a way for hardware time stamping, a 
single standard switch between them can screw up the accuracy. The only way 
to avoid this are either to use "dumb" hubs which just duplicate the packets 
without queueing them, or to use special switches which are aware of special 
timing packets and handle them in a special way.

The PTP protocol defines a special "transparent" or "boundary clock" which can 
be implemented in switches or routers in order to handle the PTP packets in a 
special way which compensates the switch's delays. The Hirschmann switch 
which we offer as part of our PTP starter kit
http://www.meinberg.de/english/ptp-starterkit
supports the PTP protocol that way.

Since standard PC NICs don't provide hardware TSU support nowadays, you must 
install a special PTP NIC (normally a PCI card) in each of the PCs anyway.

Here is a short summary on the 3 ways to assign timestamps to specific 
outgoing and incoming network packets:

1.) Inside the application, if a packet is sent or received. This is very 
portable since it runs completely in user space, and this is the way 
currently used by NTP. You can also run this on operating systems where you 
don't have access to the source code of the NIC drivers. However, this is the 
most inaccurate way since the time stamps depend on the latency of the IP 
stack, and on network collisions.

2.) Inside the NIC driver, e.g. in the interrupt service routine. This avoids 
the latencies of the IP stack, but you must modify the source code of the 
driver, or have a driver which takes time stamps if certain packets are being 
sent or received. Also, you don't know the latency due to network collisions 
when sending packets.

3.) By the NIC hardware. You need a time stamp unit (TSU) which listens on the 
data lines between the MAC and the PHY. The TSU contains a pattern matching 
unit which detects the pattern of the desired packet type in the serial bit 
stream on those data lines. If such a pattern is detected then the pattern 
matching unit must capture a time stamp of a high speed counter which is also 
part of the TSU. Since this method takes time stamps when a packet goes on 
the wire or arrives on the wire is also capable to eliminate the latency due 
to network collisions. 
The driver must be able to read those time stamps from the TSU and either 
assign them to the packets or pass them up the protocol stack to the 
application which then assigns the time stamps to the packets. The TSU can be 
implemented in a programmable logic device like an ASIC or FPGA.

The latter method is the most accurate, but it requires a NIC with a TSU, and 
a kernel driver which supports the TSU. The problem here is that most 
commonly used NICs have the MAC and the PHY in a single chip, and the data 
lines on which the TSU must listen is not accessible outside the chip. 
However, there are also some new NIC chips available which have a built-in 
TSU.

As a conclusion you can say that 

1.) NTP can achieve pretty good accuracy in both small and large networks 
where you don't know which routes the packets take, and you can not and don't 
have to rely on special hardware support for the protocol.

2.) PTP can yield very high accuracy provided that the network infrastructure 
fully supports the protocol. Obviously this is easier to implement in a 
closed network where the administrators have full control over the equipment.

3.) Once the special hardware support for PTP is not available, PTP suffers 
from the same limitations as NTP, i.e. the unknown latencies mentioned above. 
Undere these conditions NTP can even yield better results due to the 
statistical methods it uses. 

The other way round, we have made tests with NTP using the same hardware 
timestamping methods, and we could see that NTP can yield the same accuracy 
as PTP if the basic conditions are similar.

The problem here is that the current specification of the NTP protocol does 
not provide a method to send a follow-up message to the client in order to 
let the client know when the original packet really made its way onto the 
wire. If you add such a method then it breaks compatibility with existing 
implementations of NTP.

It's an advantage for the PTP protocol that it could be implemented in a way 
which adds that option missing for NTP.

Best regards,

Martin
-- 
Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany

---

From: heiko.gerstung at meinberg.de (Heiko Gerstung)
Date: Wed, 24 Oct 2007 16:06:02 +0200

Stewart Bryant schrieb:
> Heiko
>> The one thing that really enables PTP to reach an impressive accuracy 
>> level over an Ethernet/UDP connection is the fact that it uses 
>> hardware time stamping and therefore requires specialized hardware (if 
>> you want to benefit from that superior level of accurcacy).
>>   

I agree. The TC concept introduces quite a number of security problems, which 
have been addressed in V2 but obviously lead to an increased complexity. Adding 
a few things like hw-timestamping and on path correction to NTP would result in 
a synchronization protocol which offers a very good performance over both a WAN 
link and a controlled LAN environment (plus compatibility to millions of 
existing clients). Well, I have to go and talk to my engineers now :-)

Best Regards,
  Heiko

-- 
------------------------------------------------------------------------

*MEINBERG Funkuhren GmbH & Co. KG*
Lange Wand 9
D-31812 Bad Pyrmont, Germany
Tel.: ++49 (0)5281 9309-25
Fax: ++49 (0)5281 9309-30
eMail: heiko.gerstung at meinberg.de <mailto:heiko.gerstung at meinberg.de>
Internet: www.meinberg.de <http://www.meinberg.de/>

------------------------------------------------------------------------

Meinberg radio clocks: 25 years of accurate time worldwide

---

From: imp at bsdimp.com (M. Warner Losh)
Date: Wed, 24 Oct 2007 08:21:08 -0600 (MDT)

In message: <p06240801c344a38ce792@[10.240.20.124]>
            Brad Knowles <brad at shub-internet.org> writes:
: On 10/23/07, Danny Mayer wrote:
: 
: >>  A long discussion at end of meeting discussing the future of
: >>  timekeeping here in the IETF.  Is the future 1588 over IP?
: >>  NTP over IP?  No consensus was reached.
: >
: >  Could some explain exactly what that means? Is this a question of
: >  certain protocols (DNS and Kerberos come immediately to mind) that
: >  require that two interacting servers keep their clocks within X minutes
: >  of each other in order for various security requirements to work
: >  correctly or ro be assumed to be valid? Is it something else?
: 
: Well, NTP is inherently an IP protocol.  From reading the IEEE 1588 
: page, it's not clear whether this works over IP or not, although it 
: does seem to work over Ethernet (and maybe others?).

IEEE 1588 uses multicast IP addresses.

: But IEEE 1588 
: seems to focus on ultra-high resolution, like sub-microsecond or even 
: sub-nanosecond.

Well, sub-microsecond is easy and comes from the timestamping of
packets in hardware.  If NTP had extensions to do that, it could get
sub-microsecond time.  Sub-nanosecond time is possible, but it takes
very specialized hardware.  Most commercial 1588 solutions use a "low"
resolution clock to timestamp things.  Low here means something like
the PCI bus clock.

Warner

---

From: imp at bsdimp.com (M. Warner Losh)
Date: Wed, 24 Oct 2007 08:27:45 -0600 (MDT)

In message: <471F357F.3040707 at meinberg.de>
            Heiko Gerstung <heiko.gerstung at meinberg.de> writes:
: UTC but "relative" synchronization to the same source. Since one of its 
: design goals (as far as I interprete it) is sub-microsecond accuracy, it 
: is not working very well over WAN links and other asymmetric network 
: infrastructures (imagine the unpredictable delay a regular 
: store-and-forward switch introduces when it is queueing packets) . NTP 
: does a much better job in such an environment, no doubt about that.

IEEE 1588 doesn't like switches that haven't been designed
specifically to pass 1588 traffic with constant latency.  It would do
much worse with routers.  At least for v1.  I've not looked at v2 at
all.

Warner

---

From: Sasha at AXERRA.com (Sasha Vainshtein)
Date: Wed, 24 Oct 2007 16:53:07 +0200

Heiko, Stewart and all,
I just wonder if the PTP "transparent clock" (TC) mechanism is not in
contradiction with the way IP routers are supposed to work?
When PTP runs over IP, at least some of the PTP traffic (DELAY_REQ and
DELAY_RESP messages ) are unicast UDP/IP Packets running between the PTP
Slave and PTP Master with the well-known UDP port assigned to PTP. Hence
the "normal" routers would simply pass these packets thru without even
looking at the UDP ports not to mention UDP payload (i.e.., PTP proper).
Of course one may add the "router alert" option to these packets (even
if, AFAIK, PTP does not specify that), but even this would hardly help
since the router would probably simply put the packet on the "slow
path". 

My 2c
                 Sasha

-----Original Message-----
From: ntpwg-bounces+sasha=axerra.com at lists.ntp.org
[mailto:ntpwg-bounces+sasha=axerra.com at lists.ntp.org] On Behalf Of Heiko
Gerstung
Sent: Wednesday, October 24, 2007 4:06 PM
To: stbryant at cisco.com
Cc: mayer at ntp.isc.org; NTP Working Group; Brad Knowles
Subject: Re: [ntpwg] Documents, slides, etc. from WG meeting

>> hardware time stamping and therefore requires specialized hardware 
>> (if you want to benefit from that superior level of accurcacy).
>>   
> There is nothing to stop NTP doing h/w timestamping per se. The 
> critical difference is that PTP (1588)  does on path correction 
> through the use of the TC mechanism. If TC were added to NTP the two 
> protocols would provide the same level of accuracy.

I agree. The TC concept introduces quite a number of security problems,
which have been addressed in V2 but obviously lead to an increased
complexity. Adding a few things like hw-timestamping and on path
correction to NTP would result in a synchronization protocol which
offers a very good performance over both a WAN link and a controlled LAN
environment (plus compatibility to millions of existing clients). Well,
I have to go and talk to my engineers now :-)

Best Regards,
  Heiko

---

From: heiko.gerstung at meinberg.de (Heiko Gerstung)
Date: Wed, 24 Oct 2007 16:57:53 +0200

> since the router would probably simply put the packet on the "slow

That is correct, a TC concept always would include special hardware which 
supports this mode. Just like hardware timestamping would require special NICs.

   Heiko

---

From: mills at udel.edu (David L. Mills)
Date: Wed, 24 Oct 2007 16:04:52 +0000

Brad,

There might be a basic misunderstanding here. IEEE 1588 uses UDP over IP 
and a protoccol which develops four timestamps used in much the same way 
as NTP. However, 1588 interfaces can capture a hardware timestamp on 
departure and arrival of an Ethernet frame.

A diagram of what is described as a two-step 1588 protocol is in the NTP 
architecture briefting on the NTP project page linked from www.ntp.org. 
Also in that briefing is a diagram of a two-step NTP protocol. Unlike 
the 1588 protocol, the two-step NTP protocol is symmetric. In the 
two-step protocol, the transmit timestamp for one packet is sent in the 
next following packet. The two-step NTP protocol uses the same packet 
format as the current one-step protocol; the existing implementation 
could in principle operate with either one- or two=step modes.

Available 1588 interfaces and servers (grandmasters) typically achieve 
accuracies in the 50-ns range. Without hardware assist but with the 
two-step variant, I would expect NTP to achieve accuracies in the 
sub-microsecond range. At the moment the accuracies with NTP and PPS 
signals hovers around 20 a few microseconds.

Dave

---

From: mills at udel.edu (David L. Mills)
Date: Wed, 24 Oct 2007 18:11:46 +0000

Guys,

I have carefully trod over the V2 spec and investigated the available 
1588-capable devices. You have to be very careful when comparing NTP and 
1588. The bottom line is that NTP is best used to synchronizes computer 
system clocks, while 1588 is best used to synchronize hardware interface 
clocks and other data acquisition or telecom devices on a LAN. To 
sustain the best accuracy, special Ethernet switches called boundary 
clocks are necessary, so 1588 would not normally be expected to transit 
a level-3 router, although in priciple it could.

The comments about 1588 V1 and V2 are interesting. All the devices I 
know about from Symmetricom, Meinberg and 1588 Ethernet interfaces are 
V1 (1588-2002). The Intel chipsets should be able to do both. The 
Symmetricom 1588 grandmaster costs several thousand bucks and doesn't 
even do NTP. The Meinberg grandmaster does 1588 and NTP and in fact does 
everything the public NTPv4 distribution does, including Autokey.

There are a few straightforward issues that might have been overlooked. 
The computer system clock is typically interpolated by the PCC/TSC with 
resolution 1 ns, but reading it via syscall typically burns 500 ns in 
modern silicon. The resolution of a 1588 timestamp is 1 ns, while the 
resolution of an NTP timestamp is 0.232 ns, although the difference is 
not significant. It is reasonable that the ioctl time to fetch a 
timestamp from a 1588 interface would be comparable to the time to read 
the system clock, 500 ns.

The TCXO rock in a 1588 interface is not likely to operate at 
frequencies much above 20 MHz, although a cold rock might be higher. In 
other words, the 1588 resolution as a practical matter will be in the 50 
ns range. The NSI Etherface and Symmetricom grandmaster claim resolution 
of 50 ns.

What the above suggests is that 1588 will likely find the most important 
application to synchronize LAN hardware interface clocks to a 
grandmaster to the order of 50 ns. Real-time devices can provide signals 
that latch the interface clock which is later read by the program. This 
has nothing to do with the computer system clock.

However, it is interesting to speculate on how 1588 and NTP could 
interoperate or support each other. In one approach 1588 can be used 
within a LAN with time exported beyond the level-3 router by NTP. In 
another, 1588 interfaces can be used to derive precision timestamps for 
use by NTP. A project on my shelf is to modify the driver for my 1588 
Etherface for use as a reference clock driver.

In answer to a previous question, 1588 V2 has the same limitations as 
NTP; both use four timestamps to compute offset and delay and assume the 
one-way delays are very nearly the same. There is a proposal for 
security extensions to 1588, although it does not address the issue of 
possibly degraded accuracy. Since 1588 uses hardware derived timestamps 
and is not expected to face seriously hostile attacks as assumed in NTP, 
this might not matter. As a very simple security provision, 1588 could 
well adopt the NTP protocol defenses used to deflect hostile replays and 
lost packets. This does not change the protocol or packet format.

Finally, the observation that 1588 multicast might become deprecated is 
interesting. I make no judgement about that, but Symmetricom promotes 
its use in networks with thousands (!) of 1588 slaves. Well, maybe; the 
SYNC message is one-to-many, but this will be followed by thousands of 
DELAY_REQ messages imploding at the grandmaster.

Dave

---

From: mills at udel.edu (David L. Mills)
Date: Wed, 24 Oct 2007 18:41:55 +0000

Martin,

The real downside of 1588 is the huge cost of the grandmaster. But, 1588 
is a really good way to synchronize interface clocks on a LAN. Assume a 
good NTP server with GPS and PPS. Expect the system clock to hold a few 
microseconds. Run 1588 on the LAN to synchronize the intervace clocks 
time and frequency, but not necessarily the exact right time and 
frequency. At periodic intervals measure the offset between the 1588 
interface clock and the system clock and broadcast that over the LAN.

Piggyback the offset in an NTP extension field for compatiblity. As 
these packets arrive, discipline the system clock to the same offset. 
1588 is used intact and interrupts don't matter.

Dave

---

From: GDowd at symmetricom.com (Greg Dowd)
Date: Wed, 24 Oct 2007 13:25:28 -0700

Costs fall.  Our first NTP server (1991) sold for almost $20k.  Alma VME
crate with Matrix CPU and double card GPS.  The big difference really
seems to reside in the implementation.  1588 is typically implemented in
hardware and hence has better possibility of sync accuracy.  NTP is
typically implemented in software, worse, even non-realtime software so
it can suffer.  The truth is you can build 1588 in software (see ixxat
site) and you can build ntp in hardware (see symmetricom site).  

As 1588 is newer, there seem to be a few additions based on what we've
learned; follow up messages allow even software based implementations
improved timestamp resolution and transparent clock definitions allow
compensation for dwell time in switching elements.

On the other hand, ntp has had a lot more system engineering built in
with more flexible concurrent modes of configuration and more rigor in
the authentication models.

As for the description of 1588 usage, that could also map to synchronous
ethernet.

-----Original Message-----
From: ntpwg-bounces+gdowd=symmetricom.com at lists.ntp.org
[mailto:ntpwg-bounces+gdowd=symmetricom.com at lists.ntp.org] On Behalf Of
David L. Mills
Sent: Wednesday, October 24, 2007 11:42 AM
To: NTP Working Group
Subject: Re: [ntpwg] NTP vs. PTP

Martin,

Dave

Martin Burnicki wrote:

> Yep. However, it depends. Basically IEEE 1588 (PTP) is capable of 
> sync'ing all clients to a server/grandmaster with high accuracy. If 
> the application does not only rely on the _same_ time for all nodes 
> but also on the _right_ time the you just have to take care that the 
> grandmaster is sync'ed e.g. to a GPS receiver.
>
> The main difference between NTP and PTP is that the statistics 
> implemented by the NTP algorithms yield quite good results even over 
> WAN connections, and just using standard hardware.
>
> PTP can achieve much higher accuracy than NTP, but this is _only_ if 
> special hardware is used which explicitely supports PTP. And this 
> basically applies to both PTP v1 and v2.
>

> another.
>
> 1.) The sending program picks up a time stamp and puts it into a 
> network packet.
>
> network drivers which partially run in kernel space, where it finally 
> ends up in a send queue. This introduces a delay which depends on the 
> CPU power and the system load (interrupt requests).
>
> starts to transmit the packet. If there's a collision on the wire then

> transmission is aborted and retried after a random delay.
> constant, depending on the length of the wire. If there's a network 
> hub between the sender and the receiver then this also introduces an 
> additional delay, which is pretty constant, though.
> If there's a router or switch between the 2 nodes then the packet may 
> be queued for an undetermined amount of time, which also results in an

> unknown delay.
> generates an interrupt request to let the packet be fetched by the 
> protocol drivers. It also takes an unknown amount of time until this 
> is done, depending on the CPU power, whether there are 
> higher-prioritized interrupts just being handled, etc.
>
> kernel space back to user space, and passed to the application which 
> then takes a time stamp of its own system time in order to compute the

> difference to the time stamp from the incoming packet.
>
> The only delay which is constant and can be measured is the 
> propagation delay on the cable.
>
> The other effects introduce a more or less random delay which can only

> be estimated by statistical methods. That's what the NTP algorithms do

> with quite good results. The advantage is that no special hardware is 
> required, but the disavantage is that you can only yield a limited 
> accuracy.
>
> In order to compensate the receive delay (i.e. when a packet comes in 
> from the wire until it arrives at the application) you have to take a 
> time stamp when a packet comes in from the wire. This is done by a 
> time stamp unit which includes a pattern matcher which has to identify

> incoming packets in the bit stream from the wire, and take a time 
> stamp if such a packet is detected.
> Both the network card driver and the application have to provide a way

> (API
> call) which lets the application retrieve that time stamp from the NIC

> driver and assign it to the associated (right) packet. This way the 
> application can compute the difference between the time it has 
> received the packet and the time the packet has arrived from the wire,

> and account for that delay.
>
> Obviously the same has to be done for outgoing packets, i.e. determine

> the time interval from when the packet is sent by the application 
> until it really goes onto the wire. The calculated delay has to be 
> passed to the receiver which has to account for that delay. 
> Unfortunately the time stamp can only be taken when the packet goes 
> out onto the wire, so when the time stamp is available the packet has 
> already been sent. The PTP protocol accounts for this situation by 
> sending a so-called follow-up packet which contains the time stamp of 
> the previous packet. The receiver then gets the original packet which 
> is time-stamped when coming in, plus the follow-up packet which 
> Using a point-to-point connection between the transmitting and the 
> receiving node you can yield an accuracy down to a couple of 
> nanoseconds by hardware timestamping.
>
> However, there's still a last delay which is not yet known by our 
> server and client. This is the propagation delay across intermediate 
> nodes like switches and routers. For example, if a switch receives an 
> incoming packet at one port, and the outgoing port is just be used by 
> another packet then the incoming packet is queued internally in a 
> FIFO. This can take up to several tens of milliseconds (!), depending 
> on the network load and the queue depth.
>
> The problem is that neither the transmitting nor the receiving node 
> can determine whether a packet has been passed on directly, or has 
> been queued, and for how long it has been queued.
>
> So even if both your endpoints provide a way for hardware time 
> stamping, a single standard switch between them can screw up the 
> accuracy. The only way to avoid this are either to use "dumb" hubs 
> which just duplicate the packets without queueing them, or to use 
> special switches which are aware of special timing packets and handle 
> them in a special way.
>
> be implemented in switches or routers in order to handle the PTP 
> packets in a special way which compensates the switch's delays. The 
> Hirschmann switch which we offer as part of our PTP starter kit 
> Since standard PC NICs don't provide hardware TSU support nowadays, 
> you must install a special PTP NIC (normally a PCI card) in each of 
> the PCs anyway.
>

> 1.) Inside the application, if a packet is sent or received. This is 
> very portable since it runs completely in user space, and this is the 
> way currently used by NTP. You can also run this on operating systems 
> where you don't have access to the source code of the NIC drivers. 
> However, this is the most inaccurate way since the time stamps depend 
> on the latency of the IP stack, and on network collisions.
>
> 2.) Inside the NIC driver, e.g. in the interrupt service routine. This

> avoids the latencies of the IP stack, but you must modify the source 
> code of the driver, or have a driver which takes time stamps if 
> certain packets are being sent or received. Also, you don't know the 
> latency due to network collisions when sending packets.
>
> 3.) By the NIC hardware. You need a time stamp unit (TSU) which 
> listens on the data lines between the MAC and the PHY. The TSU 
> contains a pattern matching unit which detects the pattern of the 
> desired packet type in the serial bit stream on those data lines. If 
> such a pattern is detected then the pattern matching unit must capture

> a time stamp of a high speed counter which is also part of the TSU. 
> Since this method takes time stamps when a packet goes on the wire or 
> arrives on the wire is also capable to eliminate the latency due to 
> network collisions.
> The driver must be able to read those time stamps from the TSU and 
> either assign them to the packets or pass them up the protocol stack 
> to the application which then assigns the time stamps to the packets. 
> The TSU can be implemented in a programmable logic device like an ASIC

> or FPGA.
>
> The latter method is the most accurate, but it requires a NIC with a 
> TSU, and a kernel driver which supports the TSU. The problem here is 
> that most commonly used NICs have the MAC and the PHY in a single 
> chip, and the data lines on which the TSU must listen is not 
> accessible outside the chip.
> However, there are also some new NIC chips available which have a 
> built-in TSU.
>
> 1.) NTP can achieve pretty good accuracy in both small and large 
> networks where you don't know which routes the packets take, and you 
> can not and don't have to rely on special hardware support for the 
> protocol.
>
> 2.) PTP can yield very high accuracy provided that the network 
> infrastructure fully supports the protocol. Obviously this is easier 
> to implement in a closed network where the administrators have full 
> control over the equipment.
>
> 3.) Once the special hardware support for PTP is not available, PTP 
> suffers from the same limitations as NTP, i.e. the unknown latencies 
> mentioned above.
> The other way round, we have made tests with NTP using the same 
> hardware timestamping methods, and we could see that NTP can yield the

> same accuracy as PTP if the basic conditions are similar.
>
> The problem here is that the current specification of the NTP protocol

> does not provide a method to send a follow-up message to the client in

> order to let the client know when the original packet really made its 
> way onto the wire. If you add such a method then it breaks 
> compatibility with existing implementations of NTP.
>
> It's an advantage for the PTP protocol that it could be implemented in

> a way which adds that option missing for NTP.
>
>
> Best regards,
>
> Martin

---

From: GDowd at symmetricom.com (Greg Dowd)
Date: Wed, 24 Oct 2007 13:45:29 -0700

At the recent ISPCS2007 plugfest, there were dozens of oem's demoing v2
of IEEE1588.  Symmetricom demonstrated v2 default profile (multicast)
and telecom profile (unicast) as did many others.  The v2 draft isn't
final yet which I imagine is why you don't see "pre-n" grandmasters
showing up as GA product.

In the unicast, multicast arena the same problem with delay requests
already applies but it's even worse.  V1 delay requests/response are
multicast (many-to-many).  In the v2 unicast model, each client
negotiates their own announce, sync and delay request flows with the
master.  More load on the master but the other ne's get the benefit.
Also, depending on the physical network implementation, some networks
don't allow multicast in both directions.

There is discussion ongoing on blended approaches such as that you
mentioned (mc sync with unicast delay) but that has the stronger
possibility of creating path asymmetry for time applications.  One thing
I haven't seen mentioned is separation of sync and servo.  In ntp, the
servo is intrinsically part of the protocol while 1588 separates the
two.

Lastly, regardless of the protocol the network itself is only capable of
passing phase and frequency to some level which is very difficult to
identify.  Google minTDEV.  

Ok, jury duty calls.  

-----Original Message-----
From: ntpwg-bounces+gdowd=symmetricom.com at lists.ntp.org
[mailto:ntpwg-bounces+gdowd=symmetricom.com at lists.ntp.org] On Behalf Of
David L. Mills
Sent: Wednesday, October 24, 2007 11:12 AM
Cc: NTP Working Group; tictoc at ietf.org
Subject: Re: [ntpwg] Documents, slides, etc. from WG meeting

Guys,

Dave

Brad Knowles wrote:

>> In discussing 1588, you need to make sure you are talking about the 
>> V2 spec. V1 is not likely to see significant deployment, whist V2 is 
>> likely to get significant hardware support and hence be widely
deployed.

>> However V2 can run over a unicast IP connection, and this is likely 
>> to find use in wide area networks. Indeed it exists largely to 
>> support the needs of the telecommunications operators for a method of

>> providing clock sync over packet.
>
>
>> I am not sure what you mean by this. How the grand master acquires 
>> and locks to an external clock reference is outside the scope of the 
>> protocol, but it's also outside the scope of the NTP.
>
>

---

From: tglassey at earthlink.net (TS Glassey)
Date: Wed, 24 Oct 2007 15:19:38 -0700

I am working on an adaptation of 1588 that reduces its messaging to a ICMP 
protocol. Anyone else interested?

T.
----- Original Message ----- 
From: "Greg Dowd" <GDowd at symmetricom.com>
To: "David L. Mills" <mills at udel.edu>; "NTP Working Group" 
<ntpwg at lists.ntp.isc.org>
Sent: Wednesday, October 24, 2007 1:25 PM
Subject: Re: [ntpwg] NTP vs. PTP

---

From: Sasha at AXERRA.com (Sasha Vainshtein)
Date: Thu, 25 Oct 2007 08:39:38 +0200

Heiko,
Thank you for a prompt response.
However, the question was not so much about special HW as about the need to modify the protocol to make the routers on the way to inspect the PTP packets they normally should not look at beyond their IP headers.
 
Regards,
                    Sasha

________________________________

From: Heiko Gerstung [mailto:heiko.gerstung at meinberg.de]
Sent: Wed 24-Oct-07 02:57
To: Sasha Vainshtein
Cc: stbryant at cisco.com; mayer at ntp.isc.org; NTP Working Group; Brad Knowles; Israel Sasson; Alon Shtern; Osnat Shasha
Subject: Re: [ntpwg] Documents, slides, etc. from WG meeting

   Heiko

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ntp.org/pipermail/ntpwg/attachments/20071025/b108efcc/attachment.html

---

From: heiko.gerstung at meinberg.de (Heiko Gerstung)
Date: Thu, 25 Oct 2007 08:56:55 +0200

Sasha,
I am sorry but my reference to "special hardware" in the context of TC 
should have been "special firmware" and was based on the assumption that 
routers are hardware devices. It might be enough to modify ntpd to work 
in a "forwarding mode" on such a router - instead of the current BC-like 
operation (get sync from one side and sync others on the other side).

Best Regards,
 Heiko

Sasha Vainshtein schrieb:
> However, the question was not so much about special HW as about the 
> need to modify the protocol to make the routers on the way to inspect 
> the PTP packets they normally should not look at beyond their IP headers.
>  
> Regards,
>                     Sasha
>  
>
> *From:* Heiko Gerstung [mailto:heiko.gerstung at meinberg.de]
> *Sent:* Wed 24-Oct-07 02:57
> *To:* Sasha Vainshtein
> *Cc:* stbryant at cisco.com; mayer at ntp.isc.org; NTP Working Group; Brad 
> Knowles; Israel Sasson; Alon Shtern; Osnat Shasha
> *Subject:* Re: [ntpwg] Documents, slides, etc. from WG meeting
>
>

---

From: Sasha at AXERRA.com (Sasha Vainshtein)
Date: Thu, 25 Oct 2007 11:26:53 +0200

Haiko and all,
I am not making distinction between SW-based and HW-based routers. 
 
My point is that IPv4 routers must comply with RFC 1812 including, among other things, the "local delivery" decisions as defined in section 5.2.3. Unicast PTP packets that must be forwarded by a compliant router clearly fall into the "queue for forwarding to the appropriate next hop" from the point of view of these requirements. The only modifications of the packet in this case would be decrement of IP TTL and the subsequent IP header checksum re-calculation.
 
Beyond that, the only standard-defined way to assure that transit routers look at the contents of a packet beyond its IP header (not to mention modifying anything in the packet) that I am aware of is the Router Alert option as defined in RFC 2113. A well-known example of a protocol that requires such look-up and modification is RSVP (RFC 2205). And it explicitly states that its "Path, PathTear, and ResvConf messages must be sent with the Router Alert option" (section 3.3).

Hence IMHO, for Unicast PTP to run over an IP network and, at the same time, to benefit from TC capabilities, the Router Alert option must be included into the IPv4 header of the PTP packets - and that in addition to any actual HW or FW that would really perform the TC function.
 
Of course, if there is no TC clocks in the IP network to be traversed by unicast PTP, the Router Alert option can be (and probably, should be) omitted. 
 
My 2c.
                    Sasha

From: Heiko Gerstung [mailto:heiko.gerstung at meinberg.de]
Sent: Wed 24-Oct-07 18:56
To: Sasha Vainshtein
Cc: stbryant at cisco.com; mayer at ntp.isc.org; NTP Working Group; Brad Knowles; Israel Sasson; Alon Shtern; Osnat Shasha
Subject: Re: [ntpwg] Documents, slides, etc. from WG meeting

Sasha,

Best Regards,
 Heiko

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ntp.org/pipermail/ntpwg/attachments/20071025/22d36b93/attachment-0001.html

---

From: melliot at symmetricom.com (Mark Elliot)
Date: Thu, 25 Oct 2007 10:56:33 -0700

Since I'm intimately familiar with Symmetricom's NTP and IEEE 1588, I'd
like to correct some observations.  First of all, Symmetricom does offer
an IEEE 1588 grandmaster.  The same unit offers NTP at the same time.
NTP is an option and it is on a second port from IEEE 1588 card.  There
is nothing to prevent anyone from placing both of these services on the
same Internet subnet.  It's true that the current IEEE 1588 grandmaster
does support IEEE 1588 version 1.  In the same platform we've also
demonstrated IEEE 1588 version 2 in the last IEEE 1588 plug fest.
Therefore, you can conclude that at some time in the future will be
offering version 2 as well.

On the subject of supporting thousands of IEEE 1588 clients this is
true.  For IEEE 1588 version 1, set at a two second sync rate, a delay
request comes in from a client approximately every 30 seconds.  This
number is approximate for two reasons.  Reason number one is I don't
have manual in front of me right now and I'm going on remembered
laboratory experience.  The second and most important reason is this
rate is randomized so that all the requests do not come in at once.  On
average, 1000 clients will generate just over 33 packets per second.  

For the most part IEEE 1588 works well in the real world.  The only
disadvantage of this scheme is when one grandmasters fails over to
another grandmaster, the 30 sec. delay request message rate can ring the
system more than some people would like.  This is a well document and
explored subject within the IEEE 1588 group.  Version 2 has ways of
mitigating this last effect as well as a host of other features.

I hope this clarifies things, Mark Elliot.  

-----Original Message-----
From: David L. Mills [mailto:mills at udel.edu] 
Sent: Wednesday, October 24, 2007 11:12 AM
Cc: NTP Working Group; tictoc at ietf.org
Subject: [TICTOC] Re: [ntpwg] Documents, slides, etc. from WG meeting

Guys,

Dave

deployed.

_______________________________________________
TICTOC mailing list
TICTOC at ietf.org
https://www1.ietf.org/mailman/listinfo/tictoc

---

From: mills at udel.edu (David L. Mills)
Date: Fri, 26 Oct 2007 12:40:46 +0000

Guys,

The 1588 provisions for switches and routers is called a boundary clock. 
It requires intricate management of various fixed latencies and queueing 
delays. 1588 NICs known to me use the Intel chipset which has an onboard 
ARM processor very definitely provisioned by firmware. As for modifying 
NTP to simulate the 1588 two-step protocol, a description on how to do 
this is in the architecture briefing on the NTP project page.

Dave

---

From: mills at udel.edu (David L. Mills)
Date: Fri, 26 Oct 2007 13:13:38 +0000

Mark,

I think we have talked before. I was told that the Symmetricom 1588 
grandmaster does not do NTP, but I was not told it could do NTP as an 
option. at the same time. I was told the customer would have to buy a 
separate server for that. The Meingerg grandmaster I have does both 
without special order option.

The delay request/response behavior is as expected and does mitigate the 
implosion hazard. However, the features noted on the web page for the 
XLI suggest the device is safe for up to thousands of clients. My 
experience and that of USNO and NIST suggest loads like this present 
special hazards independent of 1588 or NTP.

Dave

---

From: martin.burnicki at gmx.de (Martin Burnicki)
Date: Sun, 28 Oct 2007 22:50:26 +0100

Dave,

> this is in the architecture briefing on the NTP project page.

The boundary clock is just _one_ possible way to eliminate the latency of 
intermediate nodes between the grandmaster and the final client. 

In the NTP world this is comparable to an NTP server running on a company's 
firewall or router, which gets its time from the outside world and serves it 
to the internal network.

Since each boundary clock includes a control loop, a number of cascaded 
boundary clocks results in cascaded control loops, maybe with unknown 
characteristics, which may let the overall control loop between the final 
client and the grandmaster become instable.

So another approach is the transparent clock, which just adds to every packet 
how much the packet has been delayed when it has traversed an intermediate 
node. So even if there are several intermediate nodes, there's only on big 
control loop which includes the grandmaster and the final client, which can 
possibly easier be made stable than a number of cascaded loops.

For existing NTP installations the similar configuration is when clients 
behind a firewall directly send queries to a server in the outer world, and 
the queries and replies just go through the firewall, though of course a  
firewall or router would not try to compensate any latency it adds to the NTP 
packets.

The advantages and disadvantages of boundary clocks and transparent clocks for 
this purpose have been discussed on several PTP plug fests and meetings.

Martin

---

From: mills at udel.edu (David L. Mills)
Date: Mon, 29 Oct 2007 17:21:40 +0000

Martin,

Yes, I am familiar with the distinction between boundary clocks that do 
or do not involve a control loop. Of course, NTP has the same 
distiction; however, the NTP control loop has a carefully defined 
transfer function that minimized the whip effect of cascaded control 
loops. 1588 could do the same thing, but most would consider that 
outside the scope of the specification. A 1588 bis could present such a 
specification.

One thing that may have escaped mention is that 1588 synchronizes the 
NIC drivers in the network, not necessarily the computer system clock. 
This adds another control loop to the impulse response. The time purist 
might use NTP at the edges of a private network, then distribute the 
local offset of the NIC relative to the NTP time to all other hosts 
which would then run the same discipline (NTP or kernel) for the local 
system clock. So,all hosts with have the same control loop dynamics and 
very small wiggles with respec to each other.

That bit of protocol wizardry could be incorporated in 1588 bis.

Dave

---

From: burnicki at ntp.org (Martin Burnicki)
Date: Wed, 31 Oct 2007 13:34:24 +0100

Dave,

> outside the scope of the specification. A 1588 bis could present such a

Indeed. However, while there is a single implementation of NTP which us widely 
used, there are several implementations of the PTP protocol stack and the 
algorithms and control loop characteristics used.

> That bit of protocol wizardry could be incorporated in 1588 bis.

I think you have to distinguish even more. 

As I understand it the original target for NTP were computers which have a 
more or less good or bad chrystal built in, and also provide a more or less 
good implementation of the system clock. So NTP has to deal with the existing 
hardware and try to get the best results under the given conditions.

Also as I understand it, the target of PTP/IEEE1588 explicitely includes 
embedded devices, so usage of PTP for time synchronization with distributed 
instruments is part of the LXI (LAN eXtensions for Instrumentation) standard, 
the LAN-based successor to GPIB. See also
http://www.lxistandard.org/home

As opposed to a standard computer, the manufacturer of an embedded device has 
full control over the hardware, so he can provide a single oscillator of the 
quality he needs, and use that oscillator both to implement the system clock, 
and to timestamp incoming and outgoing network packets. So there is only a 
single control loop to have PTP adjust the system time.

With a standard computer things are quite different. The built-in clock is in 
most cases based on a cheap chrystal, and the NIC chips which are often on 
the mainboard nowadays don't provide a way to timestamp network packets by 
hardware. So if you want to do hardware timestamping you have to install an 
additional NIC which provides in most cases its own oscillator.

So there are several possibilities, which are basically the same for both NTP 
and PTP. Here are some thoughts:

- If you have a single NIC with its own oscillator, e.g. on a PCI card, then 
you have two different timekeepers in the system. Normally the clock based on 
the cheap chrystal is "known" by the operating system and thus controllable 
via the system API calls. It also provides the time reported to the 
applications running on that system.

The oscillator on the NIC board is something special, and you need a way to 
relate both time scales to each other, which requires an additional control 
loop. There are basically 2 ways to synchronize those 2 timekeepers: the time 
synchronization daemon (either NTP or PTP) could discipline the oscillator on 
the NIC, and the additional control loop disciplines the OS system clock, or 
vice-versa. I bet the first option yields better results than the second.

- If the operating system provided an abstraction layer for the system clock 
which could "point" to any oscillator/counter chain then the adjtime() and 
gettime() system calls could be redirected to use the better oscillator on 
the NIC PCI card instead of the built-in cheap chrystal. This would make the 
additional control loop obsolete.

- However, if you install 2 or more timestamping NIC cards in the same system 
in order to support several subnets the the original problem is back. Unless 
there's a way to synchronize the oscillators and counter chains by hardware 
(or use the oscillator of the first NIC also for the other NICs, you must 
again relate the different clock domains of the individual NICs to each 
other. 

- Anyway, you need a way for the time synchronization daemon to read the 
current time/counter of a NIC pretty fast. E.g. if the NIC is a USB device 
which supports hardware timestamping it's no problem if you read the 
timestamps with a millisecond delay. The problem arises if you want to read 
the current time counter value in a fast way in order to relate it to the 
system time, in which case the USB latency will yield some error.

We have already discussed the aspects above with Frank Kardel some time ago, 
so I'll CC: this email to Frank.

Martin
--
Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany

---

From: lmontini at cisco.com (Laurent Montini (lmontini))
Date: Wed, 31 Oct 2007 19:21:26 +0100

Hi,

I do foresee other issues to be considered with TC, related to message
flows, message processing and state in equipments.

1- When using TC one can either use one-step or two-step mode.
One-step mode would require the TC equipment to detect PTP event
messages (e.g. SYNC message) at ingress and TS (timestamp) it (TSa).
At egress, the same eqpt would have to TS the same message again, (TSb)
calculate the internal delay (Residence Time =RT = TSb-TSa) and write
this value into the PTP payload (Correction Field: CF). TC would have to
compute RT and add RT value to any other value existing into the CF.
This demands important HW function and complex design for a large
equipment with distributed architecture.

2- With two-step mode the difference is visible at TC egress.
The TC equipment would have, at ingress, to detect PTP event messages
(as frame or packet) and TS it (same as one-step above).
At egress, the eqpt would have to TS the same message again.
However the Residence Time calculation would be done after (at least
independently of) the transmission of the event message. The RT value
would be added to the Correction Field which would be sent with a
Follow-Up (FU) general message (case of SYNC message). This solves some
design issues.

However, the process (1- or 2-step) must be done for every event
messages of every PTP session (slave-master), whatever mcast or ucast
transmission.
As I'm considering ucast, this means processing every event messages of
every master-slave pairs.

3- Considering the two-step mode (and SYNC event message only for now),
the eqpt will switch the SYNC event message then process the FU messages
for every PTP sessions.
If going thru multiple serialized TCs, the event messages will be
switched multiple times and the FU messages processed same number of
times.
It may then happen that a FU message of an event message n could arrive
after the event message n+1 or n+d due to the sum of processing delay.
That is there may be a misordering due to unequal transmission delays
into TC.
This delay will depend on (non exhaustive list):
- the transmission mode
- number of PTP sessions
- rate of the event messages
- CPU power or load of each TC
- TC node design...
The slave may rearrange using the Seq ID. This implies further
processing (not sure this is expected by IEEE1588v2 slave).

4- Last potential issue of TC is the following.
TC functions (TS and RT calculation) shall be performed for event
messages SYNC and Delay_Req in order to get a delay correction for both
directions.
The sum of residence times must be known by the slave for delay
correction computation (master does not care).
For SYNC messages, the flow is in the "right" direction. That is the RT
calculations are done by TC on the from master to slave direction. The
FU will provide the master-to-slave CF value to the slave i.e. following
the flow direction.
However for Delay_Req, the flow is slave to master. 
The CF value is thus the sum of the RT for TCs from slave to master.
In order the slave to get the slave-to-master CF value, the TCs have two
options.
If one-step mode, the RT is added in HW in the CF of the Delay_Req. (cf.
1-).
If two-step mode, the TCs will not use a FU general message (which would
be sent to master). 
Actually, every TC will hold its RT value and keep state of the RT value
for every slave-master pair session.
The TC will then wait for the Delay_Resp general message sent back to
slave by the master (for the same master-slave session) and modify the
Correction Field of the Delay_Resp message.
This requires the TC to keep state of RT values for every master-slave
session.

This can be balanced, considering the rate of Delay_Req/Resp is lower
than SYNC message rate. 
Moreover using a physical (e.g. SyncE) instead of packet-based frequency
distribution for syntonization could reduce the SYNC message rate. 

An alternative approach to avoid state in TC, would be every TC to
transmit its RT directly to slave (without waiting for the Delay_Resp
back).
If there is n times TC on the path to master, the slave should receive n
time messages for every Delay_Req.
This replaces state by message generation.

Last note about servo cascading effect.
AFAIK TC has been designed to overcome clock degration due to cascading
BC with cheap servo clock on a long chain (10's of 1588 switches).
Considering telecom network and nodes (restricting the scope to operator
management domain and thus limited number of nodes), nodes may own a
better servo clock than industrial equipment. Number of nodes between
master and slaves may also be lower than in industrial environment. 
Control loop performance may be sorted out the same way SDH or SyncE
timing chain is specified.

.lm 

> From: ntpwg-bounces+lmontini=cisco.com at lists.ntp.org 
> [mailto:ntpwg-bounces+lmontini=cisco.com at lists.ntp.org] On 
> Behalf Of Martin Burnicki
> Sent: dimanche 28 octobre 2007 22:50
> To: David L. Mills
> Cc: NTP Working Group
> > It requires intricate management of various fixed latencies and 
> > queueing delays. 1588 NICs known to me use the Intel 
> chipset which has 
> > an onboard ARM processor very definitely provisioned by 
> firmware. As 
> > for modifying NTP to simulate the 1588 two-step protocol, a 
> > description on how to do this is in the architecture 
> briefing on the NTP project page.
> 
> The boundary clock is just _one_ possible way to eliminate 
> the latency of intermediate nodes between the grandmaster and 
> the final client. 
> 
> In the NTP world this is comparable to an NTP server running 
> on a company's firewall or router, which gets its time from 
> the outside world and serves it to the internal network.
> 
> Since each boundary clock includes a control loop, a number 
> of cascaded boundary clocks results in cascaded control 
> loops, maybe with unknown characteristics, which may let the 
> overall control loop between the final client and the 
> grandmaster become instable.
> 
> So another approach is the transparent clock, which just adds 
> to every packet how much the packet has been delayed when it 
> has traversed an intermediate node. So even if there are 
> several intermediate nodes, there's only on big control loop 
> which includes the grandmaster and the final client, which 
> can possibly easier be made stable than a number of cascaded loops.
> 
> For existing NTP installations the similar configuration is 
> when clients behind a firewall directly send queries to a 
> server in the outer world, and the queries and replies just 
> go through the firewall, though of course a firewall or 
> router would not try to compensate any latency it adds to the 
> NTP packets.
> 
> The advantages and disadvantages of boundary clocks and 
> transparent clocks for this purpose have been discussed on 
> several PTP plug fests and meetings.
> 
> Martin