Subject: [iccrg] Fwd: New Version Notification for draft-yang-tcpm-ets-00.txt

From: Yuchung Cheng <[email-masked]>
Date: Mon, 2 Nov 2020 16:25:19 -0800

Hi tcpm & iccrg,

We are proposing a new TCP timestamp option to measure the network delay
more precisely (e.g. excluding delayed ACK effects) for congestion control,
e.g. Swift published SIGCOMM 2020. The precision of the measurements can
potentially be further enhanced by NIC hardware timestamps. We are working
on a reference implementation for Linux as well.

We'll also present it in the upcoming tcpm meeting. Feedbacks are very
welcome.

---------- Forwarded message ---------
From: <[email-masked]>
Date: Mon, Nov 2, 2020 at 3:29 PM
Subject: New Version Notification for draft-yang-tcpm-ets-00.txt
To: Eric Dumazet <[email-masked]>, Yuchung Cheng <[email-masked]>,
Neal Cardwell <[email-masked]>, Kevin Yang (Yudong) <[email-masked]>

A new version of I-D, draft-yang-tcpm-ets-00.txt
has been successfully submitted by Kevin (Yudong) Yang and posted to the
IETF repository.

Name:           draft-yang-tcpm-ets
Revision:       00
Title:          TCP ETS: Extensible Timestamp Options
Document date:  2020-11-02
Group:          Individual Submission
Pages:          13
URL:            https://www.ietf.org/archive/id/draft-yang-tcpm-ets-00.txt
Status:         https://datatracker.ietf.org/doc/draft-yang-tcpm-ets/
Htmlized:       https://datatracker.ietf.org/doc/html/draft-yang-tcpm-ets
Htmlized:       https://tools.ietf.org/html/draft-yang-tcpm-ets-00

Abstract:
   This document presents ETS: an Extensible TimeStamps option for TCP.
   It allows hosts to use microseconds as the unit for timestamps to
   improve the precision of timestamps, and advertise the maximum ACK
   delay for its own delayed ACK mechanism.  Furthermore, it extends the
   information provided in the [RFC7323] TCP Timestamps Option by
   including the receiver delay in the TSecr echoing, so that the
   receiver of the ACK is able to more accurately estimate the portion
   of the RTT that resulted from time traveling through the network.
   The ETS option format is extensible, so that future extensions can
   add further information without the overhead of extra TCP option kind
   and length fields.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: Yoshifumi Nishida <[email-masked]>
Date: Sat, 14 Nov 2020 04:12:48 -0800

Hi,
Thanks for preparing the draft. I basically like the idea in the docs and
have several comments.

1: I am wondering if we really need to have 32 bits TSval and TSecr in ETS
option in SYN when TSopt is also sent?
    For example, If TSopt has 1 ms granularity, I think ETS option will
need to carry only small fractions of timestamp.

2: "To prevent a false positive PAWS rejection of a valid segment, an ETS
receiver MUST skip the PAWS check"
    -> This is one possible approach, but just skipping PAWS doesn't sound
very good to me.
         How about discarding the segment while updating TS.recent as an
alternative? If the segment is not an old one, it will be retransmitted and
accepted.
         I think this is a bit more conservative.

3: It is not very clear for me how measuring NetworkRTT can contribute to
improving RTO.
    Or, are there some other ways to utilize NetworkRTT?

4: Just out of curiosity, is intended status standard track or
experimental? ExID is usually used for exp or info docs.

Thanks,
--
Yoshi

On Mon, Nov 2, 2020 at 4:26 PM Yuchung Cheng <ycheng=
[email-masked]> wrote:

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Kevin Yang <[email-masked]>
Date: Sat, 14 Nov 2020 11:45:03 -0500

Thank you Yoshi, very good questions.

1: Our goal is to substitute the TSopt. So we design this new ETSopt not to
depend on TSopt.
    I guess this also answers your question 4, the intent is for the
standard track.

2: I think we should only update TS.recent when we assume the incoming
segment is valid.
    Otherwise, if the incoming is a real old one, we discard it but update
TS.recent;
    then a new segment arrives, we may reject this new seg
because TS.recent > seg.TSval.

3: Your understanding is correct, and in the draft we are not suggesting to
use NetworkRTT
    measurements for RTO calculation.

4: answered by 1.

Kevin

On Sat, Nov 14, 2020 at 7:13 AM Yoshifumi Nishida <[email-masked]>
wrote:

>> Neal Cardwell <[email-masked]>, Kevin Yang (Yudong) <[email-masked]
>> >
>>
>>
>>
>> URL:
>> https://www.ietf.org/archive/id/draft-yang-tcpm-ets-00.txt

---

From: Yuchung Cheng <[email-masked]>
Date: Sat, 14 Nov 2020 11:35:27 -0800

On Sat, Nov 14, 2020 at 8:45 AM Kevin Yang
<yyd=[email-masked]> wrote:
> 1: Our goal is to substitute the TSopt. So we design this new ETSopt not to depend on TSopt.
>     I guess this also answers your question 4, the intent is for the standard track.
>
> 2: I think we should only update TS.recent when we assume the incoming segment is valid.
>     Otherwise, if the incoming is a real old one, we discard it but update TS.recent;
>     then a new segment arrives, we may reject this new seg because TS.recent > seg.TSval.
>
> 3: Your understanding is correct, and in the draft we are not suggesting to use NetworkRTT
We'll clarify better in the upcoming draft revision: the draft
intends to transition from exp-id to  an official option kind number
from IANA upon becoming
an RFC. But we're following the advice of
https://tools.ietf.org/html/rfc6994 to allow larger scale experiments
for hosts not completely under control of a single entity/network. For
example guest VM TCP communications inside Cloud Data-center networks.

>
> Kevin
>
> On Sat, Nov 14, 2020 at 7:13 AM Yoshifumi Nishida <[email-masked]> wrote:
>>
>> Hi,
>> Thanks for preparing the draft. I basically like the idea in the docs and have several comments.
>>
>> 1: I am wondering if we really need to have 32 bits TSval and TSecr in ETS option in SYN when TSopt is also sent?
>>     For example, If TSopt has 1 ms granularity, I think ETS option will need to carry only small fractions of timestamp.
>>
>> 2: "To prevent a false positive PAWS rejection of a valid segment, an ETS receiver MUST skip the PAWS check"
>>     -> This is one possible approach, but just skipping PAWS doesn't sound very good to me.
>>          How about discarding the segment while updating TS.recent as an alternative? If the segment is not an old one, it will be retransmitted and accepted.
>> 3: It is not very clear for me how measuring NetworkRTT can contribute to improving RTO.
>> 4: Just out of curiosity, is intended status standard track or experimental? ExID is usually used for exp or info docs.
>>
>> Thanks,
>> --
>> Yoshi
>>
>> On Mon, Nov 2, 2020 at 4:26 PM Yuchung Cheng <ycheng=[email-masked]> wrote:
>>>
>>> Hi tcpm & iccrg,
>>>
>>> We are proposing a new TCP timestamp option to measure the network delay more precisely (e.g. excluding delayed ACK effects) for congestion control, e.g. Swift published SIGCOMM 2020. The precision of the measurements can potentially be further enhanced by NIC hardware timestamps. We are working on a reference implementation for Linux as well.
>>>
>>> We'll also present it in the upcoming tcpm meeting. Feedbacks are very welcome.
>>>
>>> To: Eric Dumazet <[email-masked]>, Yuchung Cheng <[email-masked]>, Neal Cardwell <[email-masked]>, Kevin Yang (Yudong) <[email-masked]>
>>>
>>>
>>>

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 17 Nov 2020 06:42:40 +0000

Hi, Kevin, Yuchung,

Here's the draft I referred to in the chat during your talk (and Richard 
mentioned it verbally): 
https://tools.ietf.org/html/draft-scheffenegger-tcpm-timestamp-negotiation-05

Here's the rationale behind the point both Richard and I tried to make 
in the tcpm WG just now:

You could define your new option solely with the last two negotiation 
fields.

Because the existing timestamp fields are already defined as opaque. So 
you can have a separate option just for the negotiation of what the 
existing timestamp option means (ie. its resolution).

Then, you don't force TCP to use a large option on the SYN, where space 
is critical. (But it  can use both options if there is space.)
This also gives you a fall-back if the other peer doesn't support the 
new option, but does support the old timestamp option.

The downside, is 2 extra bytes if TCP does use both options on the SYN. 
There might be other ways to arrange this, but the important thing is 
the requirements:
* fall-back during incremental deployment
* provide alternative ways to minimize space on SYN, so implementers can 
juggle priority of fitting in other options.

Cheers

Bob

On 03/11/2020 00:25, Yuchung Cheng wrote:
> Hi tcpm & iccrg,
>
> We are proposing a new TCP timestamp option to measure the network 
> delay more precisely (e.g. excluding delayed ACK effects) for 
> congestion control, e.g. Swift published SIGCOMM 2020. The precision 
> of the measurements can potentially be further enhanced by NIC 
> hardware timestamps. We are working on a reference implementation for 
> Linux as well.
>
> From: <[email-masked] <mailto:[email-masked]>>
> To: Eric Dumazet <[email-masked] <mailto:[email-masked]>>, 
> Yuchung Cheng <[email-masked] <mailto:[email-masked]>>, Neal 
> Cardwell <[email-masked] <mailto:[email-masked]>>, Kevin 
> Yang (Yudong) <[email-masked] <mailto:[email-masked]>>
>
>
>
> until the htmlized version and diff are available at tools.ietf.org 
> <http://tools.ietf.org>.
>
> The IETF Secretariat
>
>
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
                PRIVILEGED AND CONFIDENTIAL

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 17 Nov 2020 14:18:49 +0100

Just to capture what Bob and I tried to convey:

There may be a way to extend TSOpt in a backwards-compatible way,
without increasing the size of the option during the SYN handshake, nor
with the need to have both 7323 TSopt and ETSopt present simultaneously
in the SYN and SYN/ACK.

Please have a look at

Which was the signaling / handshake part to

https://tools.ietf.org/html/draft-trammell-tcpm-timestamp-interval-01

As the latter part didn't get much traction (to signal the TS
granularity and even accuracy), these two didn't move forward.

As the SYN TSopt ecr value is currently unused, and the client knows the
initial value for TSval (which should be reflected in the SYN/ACK from
the server), you can use 32 bits there for a feature declaration and
negotiation.

The high level approach there was to use the unused TSecr field (should
be 0 for 7323/1323), and overlay (xor) in the SYN/ACK's TSopt ecr field
the client's TSopt val with the servers 32-bit negotiation field... As
the client can retain knowledge of the initial TSopt value (there should
be a randomized initial value), the client can extract the server side
options by xor'ing it's known initial value when the SYN/ACK is
received. Or fall back to default 7323 operation, if the result of the
XOR is zero...

The major issue would be that TSopt with a length != 10 may be
removed/blocked by certain middleboxes, but a new TCP option has similar
deployment issues.

As Bob mentioned in another thread just now, TCP SYN option space is a
very scarce resource, and making efficient use of it should matter...

Other comments: Graph1 (page 4) has "TSval=X" rather than "ETSval=X".
Unless you plan to reuse the TSopt in an extensible way (with the same
codepoint, but differentiated by the length of the option), that may
need to be addressed to disambiguate things; You may also want to have
both ETSopt and TSopt in the SYN present, for backwards compatibility if
a separate codepoint is used.

If you want to make use of the timestamp-negotiation design, all of the
extra data shown on page 6 can be stuffed into the TSecr or ETSecr
field, alleviating the need of 4 additional SYN option bytes.

It could be used almost as a drop-in use of timestamp-negotiation, as
long as it can be guaranteed, that at least one bit is set in the 32bit
field (e.g. setting the reserved bit to 1).

EcrDelUnit: I think the codepoint assignment could be improved (but this
is really nit-picking; just that a broken implementation may send out
TSopt with len=12, but initialized-to-zero superflous bytes...
0: invalid
1: milliseconds [2^20 nanoseconds?]
2: microseconds [2^10 nanoseconds?]
3: reserved (nanoseconds)

MaxAckDel: 16 bits/microseconds gives a maximum of 65.5 ms; some (older)
stacks are using values as high as 200ms; If such a stack is updated, it
is a good reason to reduce the DelAck value used in that stack, so this
range appears properly selected (FreeBSD recently reduced from 100ms to
40ms).

Best regards,
    Richard

---

From: Kevin Yang <[email-masked]>
Date: Sun, 22 Nov 2020 20:47:06 -0500

Thank you Bob and Richard, for the nice feedback!

To Bob's feedback:
>You could define your new option solely with the last two negotiation fields
>Because the existing timestamp fields are already defined as opaque...
That's a good point. We actually did consider this in our early
iterations among the authors.
Some drawbacks from relying on TSopt:

1. It has 2 more bytes for kind and len. As you pointed out, it might
be a way to save
on established segments, but not on SYNs.

2. In addition to TSval (in old TSopt), the new ETSopt needs to
include another timestamp
to be the "ETSval" in usec. We can not fill a usec value in TSval(of
old TSopt). Otherwise,
because the passive side may not support this new protocol, it treats
TSval in msec and may trigger
false positive drops by PAWS on the passive side.

I reply Richard's feedback inline below:

> received. Or fall back to default 7323 operation, if the result of the

I agree that the TSecr in <SYN> might be used for other purposes. But I'm
not sure if we can alter the TSecr in <SYN,ACK> by XOR other info.
One specific example I came up with is that a sender may send out many
SYNs because of timeout,
then when it receives a <SYN,ACK>, it does not know which TSval should
be used for xor op.

> a separate codepoint is used.

Good point, we should revise it to ETSval for disambiguation.

> field (e.g. setting the reserved bit to 1).

In this way we need to add another "ETSval'' in usec if we use the
RFC7323/1323 TSopt,
the reason is explained above.

> 3: reserved (nanoseconds)

Thanks for the nice suggestion. I agree we probably should have
all-zero codepoint to be
"no information". For 2^k nanosec unit, I guess that is for bit-op
convenience and performance?

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 24 Nov 2020 14:22:51 +0100

> then when it receives a <SYN,ACK>, it does not know which TSval should

(E)TS opt is not necessarily the old differentiation between subsequent
SYN packets; extending some reserved bits (during the SYN handshake) may
be another easy way to have sufficient differentiation available.

E.g. The EcrDel value may be restricted to no less than 4 or 8 usec (any
lower value rounded up), yielding sufficient bits which would expected
to be zero in the SYN,ACK; The TSval offset can then be selected by the
sender in such a way, to change at these locations for each
retransmission of the SYN, allowing the sender to disambiguate which SYN
a particular SYN,ACK pairs up with...

> the reason is explained above.

As long as some more coarse time resolution is acceptable during the
(special case handling) 3WHS, all you need is a few bits to disambiguate
retransmitted SYN packets. We did spent some thought on the
retransmitted SYN problem during the TS-nego design phase. A single
reserved bit is not sufficient, though;

> convenience and performance?

Yes, simple binary shift operations instead of multiply/divide, and
internally representing timestamp values in nanoseconds
(timespec/tv_nsec). The error when using timeval/tv_usec may even be
small enough to not matter in many environments (eg. simply use a
"microsecond 2^10" value directly, without adjusting for the 2.4% error,
as you are interested in deltas on short timescales, not across long
timespans?