Subject: Re: [Ntp] Robert Wilton's Discuss on draft-ietf-ntp-interleaved-modes-05: (with DISCUSS and COMMENT)

From: Miroslav Lichvar <[email-masked]>
Date: Thu, 1 Jul 2021 17:05:48 +0200

On Wed, Jun 30, 2021 at 11:46:36AM +0200, Miroslav Lichvar wrote:
> On Tue, Jun 29, 2021 at 11:09:27AM +0000, Rob Wilton (rwilton) wrote:
> > Section 2 of the draft states:
> > 
> >    The origin timestamp is a cookie, which is used to detect a packet
> >    which is not a response to the last packet sent in the other
> >    direction.  Such packets are called bogus packets in RFC 5905.
> 
> I can see how this paragraph is confusing. It was a late addition. It
> should be improved. Bogus packets are responses, not requests.

This should be now fixed in the new version of the draft, together
with the issues pointed out by Éric and Warren.

Please let me know if you think it's insufficient or you see other
issues.

https://www.ietf.org/rfcdiff?url2=draft-ietf-ntp-interleaved-modes-06

Thanks,

-- 
Miroslav Lichvar

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Fri, 16 Jul 2021 09:39:47 +0000

Hi Miroslav,

Sorry for the slow response.  I was hoping that this document would be discussed on a telechat to see whether my concerns were shared by other ADs.

Regarding the issue with the extension mechanism, another possible choice could be for the client to try and open two parallel sessions, one using a new extension field, and one without.  The client could then monitor the responses to the two sessions and choose whether to use the other with the extension or not.

However, I'm not sure whether this would really help you.  I might be misunderstanding the situation here, but it seems that this draft is partially documenting what quite a few NTP servers already do, and hence inventing a new mechanism might not really help.  But certainly if a lots of implementations are already doing this, then documenting that is helpful, and worth doing.

Hence, I propose three possible alternatives:

(1) Change this document to somehow make use of NTPs extension mechanism and continue down the path of publishing as a proposed standard.  This, I think, would probably entail taking the document back to the WG.

(2) Change the document status to Informational rather than Standards Track, modifying the text to focus on documenting how some implementations work rather than as an endorsement from IETF that this is the right way to modify the NTP protocol.

(3) Alternatively, if the authors/WG believe that publishing that document as proposed standard is definitely the right choice, then I would be willing to change my ballot position to abstain.  I.e., I don't agree with what is being done here, but neither will I block progression of the document, given that you say that this modification does not cause operational issues in practice.

Regards,
Rob

> -----Original Message-----
> From: Miroslav Lichvar <[email-masked]>
> Sent: 01 July 2021 16:06
> To: Eric Vyncke (evyncke) <[email-masked]>; Warren Kumari
> <[email-masked]>; Rob Wilton (rwilton) <[email-masked]>
> Cc: The IESG <[email-masked]>; [email-masked]; ntp-
> [email-masked]; [email-masked]; [email-masked]
> Subject: Re: [Ntp] Robert Wilton's Discuss on draft-ietf-ntp-interleaved-
> modes-05: (with DISCUSS and COMMENT)
>

---

From: Miroslav Lichvar <[email-masked]>
Date: Tue, 20 Jul 2021 13:08:14 +0200

> (2) Change the document status to Informational rather than Standards Track, modifying the text to focus on documenting how some implementations work rather than as an endorsement from IETF that th…

I'd prefer the second choice. Would anyone from the WG oppose that?

-- 
Miroslav Lichvar

---

From: "Ulrich Windl" <[email-masked]>
Date: Tue, 20 Jul 2021 13:44:10 +0200

>>> Miroslav Lichvar <[email-masked]> schrieb am 20.07.2021 um 13:08 in
Nachricht <YPaunrczI/inrtMP@localhost>:
>> (1) Change this document to somehow make use of NTPs extension mechanism
and 
> continue down the path of publishing as a proposed standard.  This, I think,

> would probably entail taking the document back to the WG.
>> 
>> (2) Change the document status to Informational rather than Standards
Track, 
> modifying the text to focus on documenting how some implementations work 
> rather than as an endorsement from IETF that this is the right way to modify

> the NTP protocol.
>> (3) Alternatively, if the authors/WG believe that publishing that document

> as proposed standard is definitely the right choice, then I would be willing

> to change my ballot position to abstain.  I.e., I don't agree with what is 
> being done here, but neither will I block progression of the document, given

> that you say that this modification does not cause operational issues in 
> practice.

Maybe NTPv5 can "lift" interleaved mode to a standard (with a fixed
implementation, maybe).

> 
> ‑‑ 
> Miroslav Lichvar
> 
> _______________________________________________
> ntp mailing list
> [email-masked] 
> https://www.ietf.org/mailman/listinfo/ntp

---

From: Miroslav Lichvar <[email-masked]>
Date: Tue, 20 Jul 2021 14:10:07 +0200

> Maybe NTPv5 can "lift" interleaved mode to a standard (with a fixed

Yes, that would be nice. My NTPv5 draft includes that. There
are only two differences between the two variants of the interleaved
mode:
- a flag is used to indicate the interleaved mode instead of checking
  equality between the origin timestamp and receive timestamp in the
  opposite directions
- a new 64-bit field is used as a server cookie instead of the receive
  timestamp

Of course, this doesn't change anything about compatibility with
existing NTPv4 implementations. There is no clean way to detect the
support.

-- 
Miroslav Lichvar

---

From: Hal Murray <[email-masked]>
Date: Tue, 20 Jul 2021 12:33:15 -0700

[email-masked] said:
>> (2) Change the document status to Informational rather than Standards Track,

> I'd prefer the second choice. Would anyone from the WG oppose that? 

I'd be happy with that.

-------

The draft should probably say something about NAT.  draft 06 says:

    The server MAY separate the timestamps by IP addresses, but it
  SHOULD NOT separate them by port numbers to support clients
  that change their port between requests, as recommended by
  [I-D.ietf-ntp-port-randomization].

If I understand things correctly, that will work if one and only one system is 
going through a NAT box and get complicated when a second system joins in.  
That might be an interesting test case.

-- 
These are my opinions.  I hate spam.

---

From: "Salz, Rich" <[email-masked]>
Date: Tue, 20 Jul 2021 19:40:30 +0000

>    I'd be happy with that.

Me too.

Although part of me wants to rear up on my hindlegs and tell the IESG they're wrong.

---

From: Miroslav Lichvar <[email-masked]>
Date: Wed, 21 Jul 2021 11:18:08 +0200

> Regarding the issue with the extension mechanism, another possible choice could be for the client to try and open two parallel sessions, one using a new extension field, and one without.  The clien…

To me that doesn't look like a very practical solution. It wastes
bandwidth. SNTP clients use only one server and might not easily be
modified to work with multiple associations. In case of a heavy packet
loss, the client would be randomly switching between the two
associations, or if it stuck to the interleaved mode once it was found
working, it wouldn't fall back to RFC5905 if the server lost the
support (e.g. after changing implementation implementation or
reconfiguration), so it would have to restart the parallel association
periodically.

Using a 28-octet extension field to convey a single bit of information
is quite wasteful anyway.

> (3) Alternatively, if the authors/WG believe that publishing that document as proposed standard is definitely the right choice, then I would be willing to change my ballot position to abstain.  I.e…

If those are the only options, the preference of the WG seems to be
the second one.

Could you please clarify why you don't see the proposal as the right
way? Do you agree with the RFC5905 interpretation that Daniel Franke
pointed out in his objection, that SNTP clients are free to put
anything in the origin timestamp because the "An SNTP client can
operate with any subset of the NTP on-wire protocol" doesn't have
a MUST?

Thanks,

-- 
Miroslav Lichvar

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Wed, 21 Jul 2021 11:19:39 +0000

Hi Miroslav,

Please see inline ...

> Sent: 21 July 2021 10:18
> To: Rob Wilton (rwilton) <[email-masked]>
> Cc: Eric Vyncke (evyncke) <[email-masked]>; Warren Kumari
> <[email-masked]>; The IESG <[email-masked]>; draft-ietf-ntp-interleaved-
> [email-masked]; [email-masked]; [email-masked]; [email-masked]
> > Regarding the issue with the extension mechanism, another possible choice
> could be for the client to try and open two parallel sessions, one using a new
> extension field, and one without.  The client could then monitor the
> responses to the two sessions and choose whether to use the other with the
> extension or not.
> 

I was assuming that a client would probably learn and maintain some historical state about whether a server at a given address has previously supported the new extension.  But I agree that it adds complexity.

> is quite wasteful anyway.

Perhaps, but that is the mechanism that is available.

> > (1) Change this document to somehow make use of NTPs extension
> mechanism and continue down the path of publishing as a proposed
> standard.  This, I think, would probably entail taking the document back to
> the WG.
> >
> Track, modifying the text to focus on documenting how some
> implementations work rather than as an endorsement from IETF that this is
> the right way to modify the NTP protocol.
> >
> as proposed standard is definitely the right choice, then I would be willing to
> change my ballot position to abstain.  I.e., I don't agree with what is being
> done here, but neither will I block progression of the document, given that
> you say that this modification does not cause operational issues in practice.
> 

In case it is not clear, I'm only representing my individual view, not any sort of consensus of the IESG (but I am also aware that Ben and Warren have balloted abstain).  Hence, a 4th choice is to wait until the doc is on the telechat, all ADs have had a chance to ballot on it and perhaps verbally discuss it at the telechat.  I think this is waiting for Karen's shepherd writeup to be updated because it was for the wrong doc.  In fact, updating the shepherd writeup and waiting for all ADs to ballot and discuss is possibly the most sensible choice of action.  I.e., I don't want you to spend time updating it to informational only for some other ADs to decide that would not be sufficient, and hence you end up wasting your time.

> way?

I think that my objections really come down to two points:

(1) The protocol has an extension mechanism, but the solution is choosing not to use it.  I appreciate that the extension mechanism is deemed to be broken and NTPv5 is meant to fix it, but then I question why standardize this particular extension now, 10 years after NTPv4 was standardized.  I.e., presumably you could bake this enhancement into NTPv5 in a very clean way.

(2) It seems to be redefining the meaning of some of the protocol fields, not in a small way, and I'm not really sure that this is the right thing to be endorsing.  I.e., balloting "No objection" wouldn't really be right, because I do have an objection to what is being done here.

However, I balance those two points with:
 - I'm not familiar with NTP (but I noted that Daniel, who presumably does know NTP, seems to making similar arguments)
 - Some implementations are clearly doing this (and seemingly have been for a long time) and pretending otherwise doesn't really help.
 - You have explained/clarified that this interoperates with existing deployments.
 - You have explained that it doesn't cause additional operational complexity in practice.

... which is why I ended up with those suggestions above.

> Do you agree with the RFC5905 interpretation that Daniel Franke

I really don't know the protocol, and I think that I would need to have a much better understanding of RFC5905 to give a properly meaningful answer here.  Looking back at Daniel's comments, I am somewhat persuaded that RFC5905 states that these fields can be ignored, and doesn't say what value field must contain if they are 'ignored'.  But equally, if the field had a non-zero value then would that not also mean that field is being used, assuming that there are no other bits in the packet to indicate whether the field has meaning?  Hopefully, clarifying this is in the plan for the next version of NTP ...
 
Regards,
Rob

---

From: Miroslav Lichvar <[email-masked]>
Date: Wed, 21 Jul 2021 14:45:25 +0200

On Wed, Jul 21, 2021 at 11:19:39AM +0000, Rob Wilton (rwilton) wrote:
> (1) The protocol has an extension mechanism, but the solution is choosing not to use it.

My view is that the fact that the protocol supports extension fields
doesn't necessarily mean they should be used for all extensions of the
protocol. So far, all specified extension fields provided additional
information that is not in the header.

With the interleaved mode we don't add information, we only switch the
reference of the transmit timestamp to correspond to the previous
packet instead of the packet in which the timestamp is contained. To
clearly indicate that we extend the protocol to allow the origin
timestamp to have a value that is not allowed by RFC5905 (in the
strict interpretation).

> I appreciate that the extension mechanism is deemed to be broken and NTPv5 is meant to fix it, but then I question why standardize this particular extension now, 10 years after NTPv4 was standardized.  I.e., presumably you could bake this enhancement into NTPv5 in a very clean way.

The interleaved mode was already included in the reference
implementation of RFC5905 when it was finalized. I wasn't following
the work of the WG at the time. I guess it was too late for the
addition. My understanding is that the WG was expected to provide a
specification for it at some point. Yes, it took a long time.

We are working on NTPv5. That may take many years. FWIW, one of the
current NTPv5 proposals has an interleaved mode. It doesn't use an
extension field for it. There is a flag in the header to select the
reference of the transmit timestamp.

> (2) It seems to be redefining the meaning of some of the protocol fields, not in a small way, and I'm not really sure that this is the right thing to be endorsing.  I.e., balloting "No objection" w…

Ok, fair enough.

I don't view it as redefining the meaning, but rather as adding new
meanings, similarly to a field which has reserved values and their
meaning is defined only later in an extension.

Per RFC5905 the origin timestamp can be either zero (meaning unknown
or unsynchronized), or a copy of the transmit timestamp from the
received packet.

In this document we add a new possibility. It can be a copy of the
receive timestamp from the received packet.

> I really don't know the protocol, and I think that I would need to have a much better understanding of RFC5905 to give a properly meaningful answer here.  Looking back at Daniel's comments, I am so…

Generally, if some field is ignored by server, does it mean that the
client is free to set it to whatever it likes? I suspect many other
protocols would be difficult to extend if that was the case.

Thanks,

-- 
Miroslav Lichvar

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 21 Jul 2021 13:05:17 +0000

I would like to push back, gently, on the IESG feedback here.

Do they have any operational experience with NTP deployments?

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Wed, 21 Jul 2021 13:37:05 +0000

Hi Rich,

Just for clarity, if you are referring to my email, as I noted on my previous email, this should be regarded solely as my individual feedback and not representing any kind of IESG consensus.  Not all ADs have balloted on this document (at least last time I checked), and I'm not aware of any IESG wide discussion of this draft, nor am I the sponsoring AD ...

No, I have no personal experience with NTP deployments, but I don't believe that is central to my objections on this document, and I have already offered to change my discuss position to abstain, which is a non-blocking position, assuming that enough other ADs ballot yes or no objection.

Regards,
Rob

> From: Salz, Rich <[email-masked]>
> Sent: 21 July 2021 14:05
> To: Miroslav Lichvar <[email-masked]>; Rob Wilton (rwilton)
> <[email-masked]>
> Cc: [email-masked]; [email-masked]; The IESG <[email-masked]>; ntp-
> [email-masked]; Eric Vyncke (evyncke) <[email-masked]>; draft-ietf-ntp-
> [email-masked]; Warren Kumari <[email-masked]>

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 21 Jul 2021 13:42:41 +0000

I was not referring just to you, but rather to the variety of comments from IESG members that said this is a hack, it won't work, etc:  Warren, Lars endorsing Warren, Theresa for gen-art, Ben Kaduk.

---

From: Erik Kline <[email-masked]>
Date: Wed, 21 Jul 2021 12:07:47 -0700

> existing NTPv4 implementations. There is no clean way to detect the

<random bad idea>
For client mode, could the client make use of the LI field to indicate it
speaks interleaved?
</>

---

From: Miroslav Lichvar <[email-masked]>
Date: Thu, 22 Jul 2021 10:51:54 +0200

> For client mode, could the client make use of the LI field to indicate it

No, there are clients that always set it to synchronized, other
clients set it always to unsynchronized, and other clients set it to
their actual status.

If we are considering unclean solutions abusing unrelated fields, the
best one would be the reference timestamp. It is 64 bits and it is
normally ignored by the server. The NTPv5 proposal uses this field to
detect whether an NTPv4 server supports NTPv5. Not a clean solution,
but I don't see a better one. What would IESG think about that?

For the interleaved mode, clients could be required to set it to a
specific value, maybe XORed with their RX and TX timestamps, to push
the false positive rate to the 1/2**128 territory. I think this would
truly make it the hack that same people already consider it.

My opinion still is that clients that set their origin timestamp to
anything else than zero or the transmit timestamp from the received
packet don't follow the NTP specification. It's not a subset of the
protocol.

In case this argument doesn't hold, I also analyzed traffic on
multiple public servers to confirm that there were no implementations
unknowingly sending requests in interleaved mode. The only instance
was the ntpsec fork of ntp, which already supported interleaved mode,
but had the support only partially removed, and this is what I think
actually prompted the Daniel's objection to the proposal.

Now, with many public servers having the support, it would be
difficult to implement a new client with such a bug and not notice it.

-- 
Miroslav Lichvar

---

From: "Ulrich Windl" <[email-masked]>
Date: Thu, 22 Jul 2021 11:19:26 +0200

>>> Miroslav Lichvar <[email-masked]> schrieb am 22.07.2021 um 10:51 in
Nachricht <YPkxqtpgzD7g8Anz@localhost>:
> On Wed, Jul 21, 2021 at 12:07:47PM ‑0700, Erik Kline wrote:
>> >

Your opinion is different from the RFC 5905:
"Origin Timestamp (org): Time at the client when the request departed
for the server, in NTP timestamp format." (page 23)

---

From: Miroslav Lichvar <[email-masked]>
Date: Thu, 22 Jul 2021 12:37:42 +0200

> for the server, in NTP timestamp format." (page 23)

If you were implementing a client and read that description out of the
context, what exactly would you put in the field? The client's
transmit timestamp?

When you read the text that follows that description:

   Receive Timestamp (rec): Time at the server when the request arrived
   from the client, in NTP timestamp format.

   Transmit Timestamp (xmt): Time at the server when the response left
   for the client, in NTP timestamp format.

what would you put in the transmit timestamp field?

That is obviously a description from the server's point of view.

To see how the timestamps are set in the client request, you need to
look at the diagram in the "On wire protocol" section, or Figure 30 in
the "Poll Process Operations" section, which has "x.org <-- p.xmt".

The document is quite confusing in some parts, but I don't see
anything that could lead you to set the origin timestamp in the
request to the server's receive timestamp (i.e. make a request in the
interleaved mode).

-- 
Miroslav Lichvar

---

From: "Ulrich Windl" <[email-masked]>
Date: Thu, 22 Jul 2021 15:40:53 +0200

>>> Miroslav Lichvar <[email-masked]> schrieb am 22.07.2021 um 12:37 in
> context, what exactly would you put in the field? The client's

Yes.

> what would you put in the transmit timestamp field?

Transmit time (current time), too.

> the "Poll Process Operations" section, which has "x.org <‑‑ p.xmt".

I thought the first is a request, while the second is a response.

> request to the server's receive timestamp (i.e. make a request in the

Me neither.

---

From: Erik Kline <[email-masked]>
Date: Wed, 21 Jul 2021 13:08:56 -0700

I remain hopeful that a reasoned back-and-forth discussion on technical and
operational points with folks across a variety of technical backgrounds is
possible here.  We ought to be able to iterate toward a net good outcome --
even if it takes a bit of time, perhaps there's a way to better communicate
the concept (among other possible paths forward).

One thing that comes to mind is that the document might benefit from a
clear "why we think this is safe" section (perhaps as the 2nd half of a
mode-agnostic Protocol Overview/Summary section).  Each individual mode
description section has some "if this is not supported by X then Y happens
and that's fine" type of text, but perhaps gathering or repeating that in
one clear place might help?  I'm not sure if that would address some of the
currently stated concerns, but it's an idea that I failed to have and
communicate during my AD Eval -- and for that I apologize.

If the agenda for the ntp meeting next week isn't already fixed, perhaps we
can discuss approaches during a slot there.

-ek  (who wrote the original IPv6 support for Google's NTP service and was
responsible for Android's SNTP code during tenure on that team)

On Wed, Jul 21, 2021 at 6:43 AM Salz, Rich <rsalz=
[email-masked]> wrote:

> I was not referring just to you, but rather to the variety of comments
> from IESG members that said this is a hack, it won't work, etc:  Warren,
> Lars endorsing Warren, Theresa for gen-art, Ben Kaduk.
>
>
>

---

From: Erik Kline <[email-masked]>
Date: Thu, 22 Jul 2021 16:54:25 -0700

> clients set it always to unsynchronized, and other clients set it to

Ah, fascinating.  =)

> but I don't see a better one. What would IESG think about that?

I'm not yet sure what would ease folks' concerns.

One thing struck me when reading your reply on another thread about using
an extension:

While I certainly see this point, it occurred to me that if the extension
where written to be a general "extra flags and fields" extension -- of
which one was defined for "I speak interleaved" and rest reserved -- then
another way to think about it is that the cost of this space would be
amortized across the other future uses of the space.

Of course, that assumes that there would be other uses of the space to come
along and make that amortization happen, and I don't know enough to
speculate with any confidence the likelihood of that...

> truly make it the hack that same people already consider it.

Operationally speaking, there ought to be one value somewhere between 1900
and ~now that could be blessed with meaning.  But I agree this seems like
"mak[ing] it the hack that same people already consider it."

---

From: "Ulrich Windl" <[email-masked]>
Date: Fri, 23 Jul 2021 09:38:55 +0200

>>> Erik Kline <[email-masked]> schrieb am 23.07.2021 um 01:54 in Nachricht
> amortized across the other future uses of the space.

Hi!

Another interesting point is what the minimum packet length is for today's high-speed networks:
If short packets are padded anyway, it's a kind of useless discussion.
The real problem is that NTP did not have a mechanism for extending the packets format in a clean way.
AFAIK NTPv0 (RFC 958) did not even have a version number.
The implementation of extension fields is a "dirty hack" IMHO.

> "mak[ing] it the hack that same people already consider it."

Another thing that is an ugly hack is the "epoch" thing (much like the GPS week number).
There should be a clean solution in NTPv5, too.

Regards,
Ulrich

---

From: Miroslav Lichvar <[email-masked]>
Date: Mon, 26 Jul 2021 11:15:03 +0200

> amortized across the other future uses of the space.

There was a draft like that
https://datatracker.ietf.org/doc/html/draft-stenn-ntp-i-do-06

But if a significant number of existing servers doesn't respond to
requests containing unknown extension fields (RFC5905 doesn't require
that), how will the client be sure whether the interleaved mode is or
is not supported? It would need to try both and see what works.

-- 
Miroslav Lichvar

---

From: "Ulrich Windl" <[email-masked]>
Date: Mon, 26 Jul 2021 11:41:58 +0200

>>> Miroslav Lichvar <[email-masked]> schrieb am 26.07.2021 um 11:15 in
Nachricht <YP59F4YqwsevWjhz@localhost>:
> On Thu, Jul 22, 2021 at 04:54:25PM ‑0700, Erik Kline wrote:
>>     Using a 28‑octet extension field to convey a single bit of information
>> where written to be a general "extra flags and fields" extension ‑‑ of
>> which one was defined for "I speak interleaved" and rest reserved ‑‑ then
> https://datatracker.ietf.org/doc/html/draft‑stenn‑ntp‑i‑do‑06 
> 

I think the current way should be deprecated as it was a bad idea (maybe good
enough for a local experiment).