Subject: [LOOPS] Timestamps and delay measurements

From: Carsten Bormann <[email-masked]>
Date: Wed, 16 Oct 2019 21:28:37 +0200

In the process of preparing the next version of the gen-info draft, there are a few items where it probably would be good to get wider input.

One of these is the use of timestamps.

Gen-info says that “block 1” reverse information (requested using the “ACK desirable” flag) has a wall clock time of reception of the forward packet that was triggering it.  This is a good way to measure one way latency if we believe the clocks on ingress and egress are synchronized; the quality of that synchronization limits the quality of the measurements.  The mechanism is still reasonable as a way to measure one way latency variance if we believe the clocks are maybe not synchronized, but do advance along the same time scale.

Question 1:
In your applications, would you expect to have reasonably synchronized clocks between ingress and egress?  Reasonable clocks at all?  Do you have data points, e.g., for random VMs in the cloud?

The prototype protocol described in appendix A uses a different form of timestamps: timestamps on a local timescale of the ingress (not further specified by the protocol) can be inserted by the ingress and are echoed back in ACKs.  (No information about time at egress is used.)  This can be used to cheaply obtain samples of round-trip time, including delays in sending back the ACK.  The timestamp is not strictly necessary if there is another correlation mechanism, as the timestamp can be stored at the ingress and retrieved if an ACK comes back.  This has two drawbacks:  (1) The timestamp needs to be stored with the cached packet, which may or may not be easy in a specific implementation, and (2) the correlation afforded by a packet sequence number has ACK ambiguity if the PSN is re-used in a retransmission (see the other message).

Question 2:
In your implementations, is it onerous to store a transmission timestamp with a cached (for retransmission) packet on the ingress?

The assumption of gen-info is that it is good to have samples of one-way delay in order to assess congestion, and that some measurement of round-trip delay is needed as an overall timing of retransmission and for ejecting cached packets.  Anything else you would expect LOOPS to measure in the way of timing?

Grüße, Carsten

---

From: "Zhouxingwang (Joe)" <[email-masked]>
Date: Mon, 21 Oct 2019 06:48:46 +0000

Hi Carsten, 

About the random VMs, some public cloud provider does not provide the synchronized time for their VMs by default, and the time difference between the VMs changes a little (found a 3750ms variation for like 40 hours roughly).
And for the cached timestamp, it will be a problem if LOOPS is implemented in high speed and memory limited device, like physical switch or router.

Thanks
Joe

-----Original Message-----
From: LOOPS [mailto:[email-masked]] On Behalf Of Carsten Bormann
Sent: Thursday, October 17, 2019 3:29 AM
To: [email-masked]
Subject: [LOOPS] Timestamps and delay measurements

Question 1:

Question 2:

Grüße, Carsten

-- 
LOOPS mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/loops

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 21 Oct 2019 23:50:07 +0200

> About the random VMs, some public cloud provider does not provide the synchronized time for their VMs by default, and the time difference between the VMs changes a little (found a 3750ms variation…

That is a lot!  I’m assuming that we want to measure one-way delay at a precision of about a 10th of the actual delay; so the target precision would often be in the single-digit ms range.  While a drift of 100 ms/h can probably still be adjusted by an adaptive algorithm, absolute measurements are not meaningful in such a situation (and we don’t know whether the 3750 ms total you saw doesn’t include wilder oscillations or even jumps).

So the one-way measurement afforded by the return of an absolute reception time may only be useful if both ingress and egress have qualified clocks.  We could make it optional, controlled by setup information.  Relying on two-way RTT measurements for congestion detection is less accurate, but doesn’t incur uncertainties from bad clocks.

> And for the cached timestamp, it will be a problem if LOOPS is implemented in high speed and memory limited device, like physical switch or router.

Let me try to get some quantitative feel about the cost of measurements.

First question is what bit rates are we going to try to achieve.
Doing LOOPS at 100 Gbit/s is interesting because of the buffer space needed to retransmit.
It seems to me that a 100 Mbit/s LOOPS tunnel is already at the high end of what many people want to do with LOOPS today, but I’m willing to be surprised.

Second, what packet rates we are looking at.

A rule of thumb for a while has been that the average packet size on the Internet is about 300 bytes; I’m sure that figure has been growing (in theory it should be 1000 bytes with an MTU of 1500 bytes and two full-size TCP data packets for one 40-byte ACK, which actually works out for elephant flows [3]; a university had ~870 bytes in 2015 [4]).  CAIDA measured ~420 bytes in 2000 [1].  Cisco rated performance figures based on 445 or 595 bytes in 2003 [2].  Is there something like an accepted rule of thumb?  I’ll assume a rounded 10000/3 bits for the below.

For a 100 Mbit/s LOOPS tunnel, we get 30000 packets per second.  At a one-way delay of 10 ms (2000 km of fiber one way), this means some 300 packets may be in flight at any one time.  Doing a measurement three times per one-way delay is probably already quite good, so we would only sample 1/100 of the packets, or 300/s.  Note that all packets already have to be stored in their entirety so the ingress can retransmit; a 4-byte timestamp is ~ 1/100 of the packet size so in terms of memory the storage need for ingress timestamps becomes 1/10000 of the buffer space.  Of course, there may be different kinds of memory in use.

In any case, there is little problem in allowing optional small data in the forward information that is to be echoed back in a block 1 acknowledgement; we wouldn’t even have to specifically dedicate these bits as a timestamp (we wouldn’t want to dictate the specific format for an echoed timestamp anyway); there may also be some use of echoed bits as a mitigation against blind attacks.

Grüße, Carsten

[1]: http://www.caida.org/research/traffic-analysis/AIX/plen_hist/
[2]: https://www.sciencedirect.com/topics/computer-science/average-packet-size
[3]: https://hal.inria.fr/hal-01497035/document
[4]: https://arxiv.org/pdf/1809.03486.pdf