Subject: [tcpm] Fwd: I-D Action: draft-nishida-tcpm-agg-syn-ext-00.txt

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 2 Feb 2021 23:12:17 -0800

Hi folks,

I prepared a draft for SYN option space extensions.
I know this is a difficult issue in TCP and has been discussed for a long
time.
But, I'm thinking that it might be a good time to discuss it again.

Key ideas of the draft are the followings.
1: drastic changes in TCP's spec will not be required
    (it does not require updating TCP header format nor using multiple SYN
packets or additional SYN-like packets)
2: utilize the option negotiation schemes in mptcp for generic purposes.
so, I think it can be considered middlebox friendly.
3: it has some limitations (e.g it can only extend around 30-40 bytes for
SYN option space). But, if it's combined with EDO, we can use more option
space.
    (depends on how EDO draft will progress, though)

Please let me know if you have any questions or comments or suggestions.

Thank you so much!
--
Yoshi

---------- Forwarded message ---------
From: <[email-masked]>
Date: Tue, Feb 2, 2021 at 10:45 PM
Subject: I-D Action: draft-nishida-tcpm-agg-syn-ext-00.txt
To: <[email-masked]>

A New Internet-Draft is available from the on-line Internet-Drafts
directories.

        Title           : Aggregated Option for SYN Option Space Extension
        Author          : Yoshifumi Nishida
        Filename        : draft-nishida-tcpm-agg-syn-ext-00.txt
        Pages           : 14
        Date            : 2021-02-02

Abstract:
   TCP option space is scarce resource as its max length is limited to
   40 bytes.  This limitation becomes more significant in SYN segments
   as all options used in a connection should be exchanged during SYN
   negotiations.  This document proposes a new SYN option negotiation
   scheme that provide a feature to compress TCP options in SYN segments
   and provide more option space.  The proposed scheme does not update
   the format of TCP header nor transmit any additional SYN or SYN-like
   segments so that it has lower risks for middlebox interventions.  In
   addition, by combining another proposal for option space extension,
   it is possible to provide further option space.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-nishida-tcpm-agg-syn-ext/

There are also htmlized versions available at:
https://tools.ietf.org/html/draft-nishida-tcpm-agg-syn-ext-00
https://datatracker.ietf.org/doc/html/draft-nishida-tcpm-agg-syn-ext-00

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

_______________________________________________
I-D-Announce mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/i-d-announce
Internet-Draft directories: http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

---

From: Joseph Touch <[email-masked]>
Date: Wed, 3 Feb 2021 16:46:15 -0800

Hi, Yoshi,

> On Feb 2, 2021, at 11:12 PM, Yoshifumi Nishida <[email-masked]> wrote:
> 
> Hi folks,
> 
> I know this is a difficult issue in TCP and has been discussed for a long time. 

The proposal has two parts, which are (AFAICT) separable:
A. Aggregated options
B. Adding more option space in the 3rd packet of a 3WHS

(A) is fine, although the most significant issue is legacy interaction.

(B) is no longer extending the SYN; it’s actually just post-SYN activation of new options. 

>     (it does not require updating TCP header format nor using multiple SYN packets or additional SYN-like packets)

Post-SYN activation of new options is quite a drastic change. 

Nevermind the question of what happens to data in the SYN, which normally MUST be passed to the application at the receiver after the final ACK is received, but here MUST NOT. And the data that might go in the other direction in the 3rd message with the ACK.

Basically, you’re proposing to change the 3WHS to a 5WHS, which is a lot more significant and complex than described here (there are a lot of failure modes not addressed, for one).

> 2: utilize the option negotiation schemes in mptcp for generic purposes. so, I think it can be considered middlebox friendly.

MPTCP is negotiating variants of an option that is already active (AFAICT); that’s not the same as activating new options.

> 3: it has some limitations (e.g it can only extend around 30-40 bytes for SYN option space). But, if it's combined with EDO, we can use more option space.

If you have EDO, then just negotiate it and activate options after the connection is established. But that’s not the problem at hand - the problem is activation of options in the SYN. AFAICT, this does not do that.

Joe

> From: <[email-masked] <mailto:[email-masked]>>
> To: <[email-masked] <mailto:[email-masked]>>
> 
> 
> 
> A New Internet-Draft is available from the on-line Internet-Drafts directories.
> 
> 
> https://datatracker.ietf.org/doc/draft-nishida-tcpm-agg-syn-ext/ <https://datatracker.ietf.org/doc/draft-nishida-tcpm-agg-syn-ext/>
> 
> https://tools.ietf.org/html/draft-nishida-tcpm-agg-syn-ext-00 <https://tools.ietf.org/html/draft-nishida-tcpm-agg-syn-ext-00>
> https://datatracker.ietf.org/doc/html/draft-nishida-tcpm-agg-syn-ext-00 <https://datatracker.ietf.org/doc/html/draft-nishida-tcpm-agg-syn-ext-00>
> 
> 
> until the htmlized version and diff are available at tools.ietf.org <http://tools.ietf.org/>.
> 
> ftp://ftp.ietf.org/internet-drafts/ <ftp://ftp.ietf.org/internet-drafts/>
> 
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/i-d-announce <https://www.ietf.org/mailman/listinfo/i-d-announce>
> Internet-Draft directories: http://www.ietf.org/shadow.html <http://www.ietf.org/shadow.html>
> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt <ftp://ftp.ietf.org/ietf/1shadow-sites.txt>
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 3 Feb 2021 23:02:12 -0800

Hi Joe,

Thanks for reading the draft. I put my comments in lines.

> B. Adding more option space in the 3rd packet of a 3WHS

Right. both can be used supplementarily.

> (B) is no longer extending the SYN; it’s actually just post-SYN activation

In my view, all options are negotiated during SYN exchange. There's no post
SYN activation.
They are just additional information for the features already negotiated
during SYN exchanges.

> Nevermind the question of what happens to data in the SYN, which normally
> MUST be passed to the application at the receiver after the final ACK is
> received, but here MUST NOT. And the data that might go in the other
> direction in the 3rd message with the ACK.
>
> Basically, you’re proposing to change the 3WHS to a 5WHS, which is a lot
> more significant and complex than described here (there are a lot of
> failure modes not addressed, for one).
>

RIght. To be precise, the draft supports 3WHS, 4WHS and 5WHS. It can be
varied depending on how much extra option space is needed.
MPTCP adopts 4WHS for its negotiation. In my view, the draft just extends
the idea.
Hence, I am thinking the degree of the complexity would be more or less the
same. At least, I don't think it will be too complex to implement.

> MPTCP is negotiating variants of an option that is already active
> (AFAICT); that’s not the same as activating new options.
>

My understanding for MPTCP is similar and I think this draft does the same
things. (from my point of view)
 If you see significant differences between them, could you elaborate a bit?

> If you have EDO, then just negotiate it and activate options after the
> connection is established. But that’s not the problem at hand - the problem
> is activation of options in the SYN. AFAICT, this does not do that.
>

I think the negotiation scheme for EDO is simple. it needs to send 1 bit
information to indicate the activation of the feature.
This can be done during SYN exchange. So, I thought it's fine to use this
feature in the 3rd segment. Or, are there any risks?

Thanks for the lots of detailed comments.
--
Yoshi

---

From: Joseph Touch <[email-masked]>
Date: Thu, 4 Feb 2021 12:07:45 -0800

Hi, Yoshifumi,

> On Feb 3, 2021, at 11:02 PM, Yoshifumi Nishida <[email-masked]> wrote:
> 
> Hi Joe,
> 
> On Wed, Feb 3, 2021 at 4:46 PM Joseph Touch <[email-masked] <mailto:[email-masked]>> wrote:
> Hi, Yoshi,
> 
>> On Feb 2, 2021, at 11:12 PM, Yoshifumi Nishida <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hi folks,
>> 
> In my view, all options are negotiated during SYN exchange. There's no post SYN activation. 
> They are just additional information for the features already negotiated during SYN exchanges.

I disagree; you’re suggesting putting NEW options in packets that don’t have a SYN. In that sense, it’s similar to my proposal, which uses a segment with no SYN and no ACK bit (the out-of-band segment).

However, the problem is with the first SYN-ACK you send back. That is typically a confirmation of all options supported so the rest of the connection can commence, even for MPTCP. In your case, the exchange continues. This not only adds multiple RTTs to the open, it interferes with data sent in the SYN, data sent with the SYN-ACK, and fast-open for sure; it could also be a problem with many other options that don’t make sense in subsequent segments due to the way they need to coordinate info that is used in the SYN-ACK or other ACK segments (e.g., TCP-AO comes to mind).

At a first cut, the doc needs to address this, e.g., saying that:
	- if any user data is included before the option exchange is completed, it has to be delayed until it completes
	- how to handle options that are first seen after the SYN
		MPTCP can define its own behavior for this, but your doc cannot redefine that for other options 

> RIght. To be precise, the draft supports 3WHS, 4WHS and 5WHS. It can be varied depending on how much extra option space is needed.
> MPTCP adopts 4WHS for its negotiation. In my view, the draft just extends the idea. 

MPTCP adopts 4WHS for *its* options, which is its prerogative. You’re attempting to define 4WHS and 4WHS semantics for *all* options, which I don’t think is viable.

> Hence, I am thinking the degree of the complexity would be more or less the same. At least, I don't think it will be too complex to implement.
> 
> My understanding for MPTCP is similar and I think this draft does the same things. (from my point of view)

See above.

> I think the negotiation scheme for EDO is simple. it needs to send 1 bit information to indicate the activation of the feature. 
> This can be done during SYN exchange. So, I thought it's fine to use this feature in the 3rd segment. Or, are there any risks?

I do think that if EDO is negotiated in the SYN then you can use it in all subsequent segments (that’s already how it’s defined). I don’t think it adds any risks here that aren’t already part of EDO.

Joe

---

From: Joseph Touch <[email-masked]>
Date: Thu, 4 Feb 2021 12:34:58 -0800

FWIW, I would have called this more “delayed option negotiation” than its current title.

The core issue as I see it is atomicity of options; there are many options that interact with others that cannot be delayed this way. And if option negotiation is delayed, the issue of what to do with data in the meantime.

So overall, it could be a generally useful addition, but is quite distinct in capability and impact on both options and TCP than the previous attempted solutions.

Joe

> On Feb 4, 2021, at 12:07 PM, Joseph Touch <[email-masked]> wrote:
> 
> Hi, Yoshifumi,
> 
>> On Feb 3, 2021, at 11:02 PM, Yoshifumi Nishida <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hi Joe,
>>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 4 Feb 2021 22:50:55 -0800

Hi Joe,

> FWIW, I would have called this more “delayed option negotiation” than its

OK. It seems it makes sense. Thanks for the suggestion. I will think about
updating the title.
I also would like to hear from people if there're other thoughts.

> The core issue as I see it is atomicity of options; there are many options
> that interact with others that cannot be delayed this way. And if option
> negotiation is delayed, the issue of what to do with data in the meantime.
>
> So overall, it could be a generally useful addition, but is quite distinct
> in capability and impact on both options and TCP than the previous
> attempted solutions.
>

I basically agree with you here.
In my view, the previous approaches such as yours try to provide complete
solutions to SYN option space, which can lead to very long discussions from
my observations.
Instead, the draft tries to provide a solution that has several
limitations, but it can reduce complexity and the risk of middlebox issues.
Also, we have similar precedence for this approach.

The main focus of the draft is to support newly defined options rather than
existing options. Especially, the ones that use EXID as they tend to
consume more space.
One of my concerns is scarce SYN option space discourages people from
developing and using new features.

As you mentioned, some options are difficult to be delayed. Section 5.3
describes this limitation and explains that TFO or AO cannot be supported
with this.
But, I think this approach is still useful as some options will not be
affected even though the negotiations are delayed a bit.
For example, if you want to send special feedbacks on some events by using
a certain options, you don't need to hold data transfer for the feature.
Or, in some cases, delays can be acceptable in order to use new features.

Thanks,
--
Yoshi

> I disagree; you’re suggesting putting NEW options in packets that don’t
> have a SYN. In that sense, it’s similar to my proposal, which uses a
> segment with no SYN and no ACK bit (the out-of-band segment).
>
> However, the problem is with the first SYN-ACK you send back. That is
> typically a confirmation of all options supported so the rest of the
> connection can commence, even for MPTCP. In your case, the exchange
> continues. This not only adds multiple RTTs to the open, it interferes with
> data sent in the SYN, data sent with the SYN-ACK, and fast-open for sure;
> it could also be a problem with many other options that don’t make sense in
> subsequent segments due to the way they need to coordinate info that is
> used in the SYN-ACK or other ACK segments (e.g., TCP-AO comes to mind).
>
>>     (it does not require updating TCP header format nor using multiple
>> SYN packets or additional SYN-like packets)
>>
>>
> MPTCP adopts 4WHS for *its* options, which is its prerogative. You’re
> attempting to define 4WHS and 4WHS semantics for *all* options, which I
> don’t think is viable.
>
> Hence, I am thinking the degree of the complexity would be more or less
> the same. At least, I don't think it will be too complex to implement.
>
> I do think that if EDO is negotiated in the SYN then you can use it in all
> subsequent segments (that’s already how it’s defined). I don’t think it
> adds any risks here that aren’t already part of EDO.
>
> Joe
>

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Mon, 8 Feb 2021 22:46:45 +0100

Hi Yoshi,

Sorry to nitpick - in your draft, you mention that some hosts reflect
back unknown tcp options, which is why you are using different GID
mappings between SYN and SYN,ACK.

I did read about this behavior concerning TCP header flags / reserved
bits - but have not come across a paper where this behavior is described
for unknown TCP options (or I may have missed that aspect in the various
studies around TCP option investigations done by MPTCP and other groups).

Also, I am lost in seeing how changing the GID association helps prevent
a misinterpretation.

if the client sends out SYN [GID1, GID2] and the suggested unknown
option reflector comes back with SYN,ACK [GID2, GID3] (with the same GID
bits), you still have an overlap and a potential false negotiation, not?

The main issue about backwards compatibilty that Joe mentioned, still
persists; this approach may have some utility when two hosts have
repeated connections with each other, so that on subsequent connections,
only the abbreviation is exchanged.

Best regards,
    Richard

Am 05.02.2021 um 07:50 schrieb Yoshifumi Nishida:
> Hi Joe,
>
> On Thu, Feb 4, 2021 at 12:35 PM Joseph Touch <[email-masked]
> <mailto:[email-masked]>> wrote:
>
> OK. It seems it makes sense. Thanks for the suggestion. I will think
> about updating the title.
>     The core issue as I see it is atomicity of options; there are many
>     options that interact with others that cannot be delayed this way.
>     And if option negotiation is delayed, the issue of what to do with
>     data in the meantime.
>
>     So overall, it could be a generally useful addition, but is quite
>     distinct in capability and impact on both options and TCP than the
>     previous attempted solutions.
>
>
> In my view, the previous approaches such as yours try to provide
> complete solutions to SYN option space, which can lead to very long
> discussions from my observations.
> limitations, but it can reduce complexity and the risk of middlebox
> issues. Also, we have similar precedence for this approach.
>
> The main focus of the draft is to support newly defined options rather
> than existing options. Especially, the ones that use EXID as they tend
> to consume more space.
> describes this limitation and explains that TFO or AO cannot be
> supported with this.
> For example, if you want to send special feedbacks on some events by
> using a certain options, you don't need to hold data transfer for the
> feature.
>>     On Feb 4, 2021, at 12:07 PM, Joseph Touch <[email-masked]
>>     <mailto:[email-masked]>> wrote:
>>
>>     Hi, Yoshifumi,
>>
>>>     On Feb 3, 2021, at 11:02 PM, Yoshifumi Nishida
>>>     <[email-masked] <mailto:[email-masked]>> wrote:
>>>
>>>     Hi Joe,
>>>
>>>>         On Feb 2, 2021, at 11:12 PM, Yoshifumi Nishida
>>>>         <[email-masked] <mailto:[email-masked]>> wrote:
>>>>
>>>>         Hi folks,
>>>>
>>     I disagree; you’re suggesting putting NEW options in packets that
>>     don’t have a SYN. In that sense, it’s similar to my proposal,
>>     which uses a segment with no SYN and no ACK bit (the out-of-band
>>     segment).
>>
>>     However, the problem is with the first SYN-ACK you send back. That
>>     is typically a confirmation of all options supported so the rest
>>     of the connection can commence, even for MPTCP. In your case, the
>>     exchange continues. This not only adds multiple RTTs to the open,
>>     it interferes with data sent in the SYN, data sent with the
>>     SYN-ACK, and fast-open for sure; it could also be a problem with
>>     many other options that don’t make sense in subsequent segments
>>     due to the way they need to coordinate info that is used in the
>>     SYN-ACK or other ACK segments (e.g., TCP-AO comes to mind).
>>
>>     - if any user data is included before the option exchange is
>>     completed, it has to be delayed until it completes
>>>>             (it does not require updating TCP header format nor
>>>>         using multiple SYN packets or additional SYN-like packets)
>>>
>>>         Nevermind the question of what happens to data in the SYN,
>>>         which normally MUST be passed to the application at the
>>>         receiver after the final ACK is received, but here MUST NOT.
>>>         And the data that might go in the other direction in the 3rd
>>>         message with the ACK.
>>>
>>>         Basically, you’re proposing to change the 3WHS to a 5WHS,
>>>         which is a lot more significant and complex than described
>>>         here (there are a lot of failure modes not addressed, for one).
>>>
>>>
>>>     RIght. To be precise, the draft supports 3WHS, 4WHS and 5WHS. It
>>>     can be varied depending on how much extra option space is needed.
>>     MPTCP adopts 4WHS for *its* options, which is its prerogative.
>>     You’re attempting to define 4WHS and 4WHS semantics for *all*
>>     options, which I don’t think is viable.
>>
>>>     Hence, I am thinking the degree of the complexity would be more
>>>     or less the same. At least, I don't think it will be too complex
>>>     to implement.
>>>
>>>>         2: utilize the option negotiation schemes in mptcp for
>>>>         generic purposes. so, I think it can be considered middlebox
>>>>         friendly.
>>>
>>>         MPTCP is negotiating variants of an option that is already
>>>         active (AFAICT); that’s not the same as activating new options.
>>>
>>>
>>>>         3: it has some limitations (e.g it can only extend around
>>>>         30-40 bytes for SYN option space). But, if it's combined
>>>>         with EDO, we can use more option space.
>>>         If you have EDO, then just negotiate it and activate options
>>>         after the connection is established. But that’s not the
>>>         problem at hand - the problem is activation of options in the
>>>         SYN. AFAICT, this does not do that.
>>>
>>>
>>>     I think the negotiation scheme for EDO is simple. it needs to
>>>     send 1 bit information to indicate the activation of the feature.
>>>     This can be done during SYN exchange. So, I thought it's fine to
>>>     use this feature in the 3rd segment. Or, are there any risks?
>>
>>     I do think that if EDO is negotiated in the SYN then you can use
>>     it in all subsequent segments (that’s already how it’s defined). I
>>     don’t think it adds any risks here that aren’t already part of EDO.
>>
>>     Joe
>>
>>     [email-masked] <mailto:[email-masked]>
>>     <https://www.ietf.org/mailman/listinfo/tcpm>
>
>

---

From: Matthew Luckie <[email-masked]>
Date: Tue, 9 Feb 2021 11:15:45 +1300

Hi Richard,

> studies around TCP option investigations done by MPTCP and other groups).

There is some discussion of systems that seem to reflect TCP options here:

http://blog.multipath-tcp.org/blog/html/2018/12/19/which_servers_use_multipath_tcp.html

Matthew

---

From: Joe Touch <[email-masked]>
Date: Mon, 8 Feb 2021 14:42:49 -0800

That sounds like a very serious bug. It would be important to get it fixed before we all waste time trying to adapt to it. 

Joe

> On Feb 8, 2021, at 2:16 PM, Matthew Luckie <[email-masked]> wrote:
> 
> Hi Richard,
> 
> Matthew_______________________________________________
> tcpm mailing list
> [email-masked]

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Mon, 8 Feb 2021 23:49:10 +0100

Thanks!

Still, without additional constraints my simplistic reading of how the
GID value mapping to groups doesn't seem to address such collisions
entirely..

Also, some options are not necessarily negotiated, but only indicated -
thus a constraint that corresponding bits of the group may only be set
in a <SYN,ACK>, when they where present in the <SYN> is not sufficient.

One constraint I could come up with would be to require  the server to
reflect back "empty" (all zero option flags) blocks received from the
client, in case a bitstream collision would otherwise happen

e.g. client wants to signal option flag 7 of group 2, while server wants
to send back option flag 7 of group 1 (but not flag 7 of group 2), the
current draft would allow a "valid" exchange that looks like a reflection.

With the above, additional constraint, the server would have to return
an all-zero block for group 2 also, and may NOT skip over that...

Best regards,
    Richard

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Mon, 8 Feb 2021 23:51:06 +0100

Reading the referenced MPTCP documents and surveys, "we" already have...

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 8 Feb 2021 23:16:14 -0800

Hi Richard,

> an all-zero block for group 2 also, and may NOT skip over that...

I see your points. Thanks for pointing out an interesting case.

In such case, I might want to use the extension block (GID=3 in SYN, GID=0
in SYN/ACK) defined in the draft.
If server wants to send group IDs that didn't exist in the agg option in
SYN, it needs to attach an extension block as well.

So, in your case, sender sends back GID=1 with flag 7 and GID=3 with a
certain bit on (TBD).
I think this can be used as a proof that this is not a reflection.

Thanks,
--
Yoshi

---

From: Kangjiao <[email-masked]>
Date: Wed, 10 Feb 2021 01:14:04 +0000

Hi Yoshi,

I wrote a draft to propose an idea for MPTCP subtype capability exchange during MPTCP handshake.  We think, in MPTCP, some options are mandatory but some options can be set as optional ones. So an exchange mechanism of actual capability between both parties are needed.

URL:            https://www.ietf.org/archive/id/draft-kang-tcpm-subtype-capability-exchange-00.txt
Status:         https://datatracker.ietf.org/doc/draft-kang-tcpm-subtype-capability-exchange/
Html:           https://www.ietf.org/archive/id/draft-kang-tcpm-subtype-capability-exchange-00.html
Htmlized:       https://tools.ietf.org/html/draft-kang-tcpm-subtype-capability-exchange-00

When I read your draft, I think maybe our above work have some relevance to your suggestions. If you think it'll help with this work, I hope we can discuss this new negotiation mechanism for related options and subtypes. Thanks

Best wishes,
Jiao

From: tcpm [mailto:[email-masked]] On Behalf Of Yoshifumi Nishida
Sent: Wednesday, February 3, 2021 3:12 PM
To: [email-masked] Extensions <[email-masked]>
Subject: [tcpm] Fwd: I-D Action: draft-nishida-tcpm-agg-syn-ext-00.txt

Hi folks,

Thank you so much!
--
Yoshi

---------- Forwarded message ---------
From: <[email-masked]<mailto:[email-masked]>>
Date: Tue, Feb 2, 2021 at 10:45 PM
Subject: I-D Action: draft-nishida-tcpm-agg-syn-ext-00.txt
To: <[email-masked]<mailto:[email-masked]>>

Abstract:

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org>.

[email-masked]<mailto:[email-masked]>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 10 Feb 2021 02:28:32 -0800

Hi Kangjiao,

OK. I think it has some similarities. I will think about this further and
will get back to you.
--
Yoshi

On Tue, Feb 9, 2021 at 5:14 PM Kangjiao <[email-masked]> wrote:

> Hi Yoshi,
>
>
>
> I wrote a draft to propose an idea for MPTCP subtype capability exchange
> during MPTCP handshake.  We think, in MPTCP, some options are mandatory but
> some options can be set as optional ones. So an exchange mechanism of
> actual capability between both parties are needed.
>
>
>
> URL:
> https://www.ietf.org/archive/id/draft-kang-tcpm-subtype-capability-exchange-00.txt
>
> Status:
> https://datatracker.ietf.org/doc/draft-kang-tcpm-subtype-capability-exchange/
>
> Html:
> https://www.ietf.org/archive/id/draft-kang-tcpm-subtype-capability-exchange-00.html
>
> Htmlized:
> https://tools.ietf.org/html/draft-kang-tcpm-subtype-capability-exchange-00
>
>
>
> When I read your draft, I think maybe our above work have some relevance
> to your suggestions. If you think it'll help with this work, I hope we can
> discuss this new negotiation mechanism for related options and subtypes.
> Thanks
>
>
>
> Best wishes,
>
> Jiao
>
>
>
> *From:* tcpm [mailto:[email-masked]] *On Behalf Of *Yoshifumi
> Nishida
> *Sent:* Wednesday, February 3, 2021 3:12 PM
> *To:* [email-masked] Extensions <[email-masked]>
> *Subject:* [tcpm] Fwd: I-D Action: draft-nishida-tcpm-agg-syn-ext-00.txt
>
>
>
> Hi folks,
>
>
>

---

From: Joseph Touch <[email-masked]>
Date: Wed, 10 Feb 2021 07:54:52 -0800

Hi, all,

FWIW, my view is that:

- it MAY be reasonable for an option to negotiate parameters after the TWHS
	BUT this can be VERY difficult, esp. because the negotiation needs to account for segment loss and reordering

- a new option CANNOT provide that opportunity for existing options
	those existing options were defined assuming establishment during the TWHS

- a new option CANNOT provide that opportunity for new options UNLESS THEY EXPLICITLY ALLOW IT

So the proposed mechanism does not appear to extend the option space in a general sense, either for existing or all future options. 

Joe

> On Feb 10, 2021, at 2:28 AM, Yoshifumi Nishida <[email-masked]> wrote:
> 
> Hi Kangjiao,
> 
> OK. I think it has some similarities. I will think about this further and will get back to you.
> --
> Yoshi
> 
> On Tue, Feb 9, 2021 at 5:14 PM Kangjiao <[email-masked] <mailto:[email-masked]>> wrote:
> Hi Yoshi,
> 
>  
> 
> URL:            https://www.ietf.org/archive/id/draft-kang-tcpm-subtype-capability-exchange-00.txt <https://www.ietf.org/archive/id/draft-kang-tcpm-subtype-capability-exchange-00.txt>
> Status:         https://datatracker.ietf.org/doc/draft-kang-tcpm-subtype-capability-exchange/ <https://datatracker.ietf.org/doc/draft-kang-tcpm-subtype-capability-exchange/>
> Html:           https://www.ietf.org/archive/id/draft-kang-tcpm-subtype-capability-exchange-00.html <https://www.ietf.org/archive/id/draft-kang-tcpm-subtype-capability-exchange-00.html>
> Htmlized:       https://tools.ietf.org/html/draft-kang-tcpm-subtype-capability-exchange-00 <https://tools.ietf.org/html/draft-kang-tcpm-subtype-capability-exchange-00>
>  
> 
> From: tcpm [mailto:[email-masked] <mailto:[email-masked]>] On Behalf Of Yoshifumi Nishida
> To: [email-masked] <mailto:[email-masked]> Extensions <[email-masked] <mailto:[email-masked]>>
> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt <ftp://ftp.ietf.org/ietf/1shadow-sites.txt>_______________________________________________
> tcpm mailing list
> [email-masked]

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 11 Feb 2021 01:05:47 -0800

Hi Joe,

> - a new option CANNOT provide that opportunity for new options UNLESS THEY

I won't disagree with them, but I have a few comments on them.

- When we try to define new options, I think we cannot ignore option space
size in today's TCP.
  If there's risk that the option cannot be fit into SYN, protocol
designers can think about adopting this approach.
  Otherwise, they will have to give up using the option or sacrifice some
of existing options in some situations.
  I'm not sure if it's very difficult to negotiate parameters after TWHS
because it has already been done in MPTCP.

- Not all new options need to negotiate parameters. For
example,draft-gomez-tcpm-ack-rate-request or draft-ietf-tcpm-tcp-edo.
  For these cases, their options can be put into the aggregated option in
SYN, which can save some space without any difficulties.

- I think we can think about updating existing options to support this
approach. At least, I think it's not impossible.
  But, I guess we should focus on using this for new options at first.

Thanks,
--
Yoshi

---

From: Joseph Touch <[email-masked]>
Date: Thu, 11 Feb 2021 09:39:51 -0800

Hi, Yoshifumi,

> On Feb 11, 2021, at 1:05 AM, Yoshifumi Nishida <[email-masked]> wrote:
> 
> Hi Joe,
> 
> On Wed, Feb 10, 2021 at 7:55 AM Joseph Touch <[email-masked] <mailto:[email-masked]>> wrote:
> Hi, all,
> 
> FWIW, my view is that:
> 
> - When we try to define new options, I think we cannot ignore option space size in today's TCP.  
>   If there's risk that the option cannot be fit into SYN, protocol designers can think about adopting this approach. 

I think that applies more to the “compressed” version than the other part, but compressed options work only for initially flag (on/off) options with no parameters in the SYN.

>   Otherwise, they will have to give up using the option or sacrifice some of existing options in some situations.
>   I'm not sure if it's very difficult to negotiate parameters after TWHS because it has already been done in MPTCP.  

It has been done *within the option*, not as a general case available to option designers. Those are quite different.

> - Not all new options need to negotiate parameters. For example,draft-gomez-tcpm-ack-rate-request or draft-ietf-tcpm-tcp-edo.
>   For these cases, their options can be put into the aggregated option in SYN, which can save some space without any difficulties.

Agreed; again, that separates the compressed aspect from other aspects. Perhaps creating a single compressed option for the next N such SYN-“flag” options would be useful.

> - I think we can think about updating existing options to support this approach. At least, I think it's not impossible. 

Updating options is likely to be much more difficult than creating new ones, which is itself difficult...

>   But, I guess we should focus on using this for new options at first.

Agreed - but again, I’d suggest that all the above applies more to a single “flag options” option for new options only.

That could be done to support the next 16 such flag options options in one word-aligned option. I don’t know that it would be worth trying to optimize beyond 16 of those, though.

And I do NOT think that this option should waste space trying to game devices that ignorantly reflect SYN options in SYN-ACK. The definition in TCP and most such options stands - if an option is reflected in a SYN-ACK, it is deemed supported.

Joe

>> On Feb 10, 2021, at 2:28 AM, Yoshifumi Nishida <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hi Kangjiao,
>> 
>> https://www.ietf.org/mailman/listinfo/tcpm <https://www.ietf.org/mailman/listinfo/tcpm>
>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 12 Feb 2021 09:52:13 -0800

Hi Joe,

On Thu, Feb 11, 2021 at 9:40 AM Joseph Touch <[email-masked]> wrote:

> I think that applies more to the “compressed” version than the other part,
> but compressed options work only for initially flag (on/off) options with
> no parameters in the SYN.
>
> It has been done *within the option*, not as a general case available to
> option designers. Those are quite different.
>
> Agreed; again, that separates the compressed aspect from other aspects.
> Perhaps creating a single compressed option for the next N such SYN-“flag”
> options would be useful.
>
> Agreed - but again, I’d suggest that all the above applies more to a
> single “flag options” option for new options only.
>
> That could be done to support the next 16 such flag options options in one
> word-aligned option. I don’t know that it would be worth trying to optimize
> beyond 16 of those, though.
>
> And I do NOT think that this option should waste space trying to game
> devices that ignorantly reflect SYN options in SYN-ACK. The definition in
> TCP and most such options stands - if an option is reflected in a SYN-ACK,
> it is deemed supported.
>

OK. So, in my understanding, what you're suggesting are the followings.

- Aggregated options would be useful. but, the format can be simple 4 bytes
length option that can accomodate 16 options.
- The tricks to avoid reflecting options in SYN ACK will not be needed
because it's the receiver's responsibility.
- Delayed negotiation might be possible. But, no generic framework is
needed. Each option should be designed for its own way to support it.

But, once thing I am wondering is that we can still combine Aggregated
options and delayed negotiation even with this one.
if you are meaning options not appeared in SYN cannot be used in the
following segments, I think accecn option might be a precedent for this.

Thanks,
--
Yoshi

>>> 3: it has some limitations (e.g it can only extend around 30-40 bytes
>>> for SYN option space). But, if it's combined with EDO, we can use more
>>> option space.
>>>

---

From: Joseph Touch <[email-masked]>
Date: Fri, 12 Feb 2021 20:44:54 -0800

> On Feb 12, 2021, at 9:52 AM, Yoshifumi Nishida <[email-masked]> wrote:
> 
> - Aggregated options would be useful. but, the format can be simple 4 bytes length option that can accomodate 16 options.

Yes,

> - The tricks to avoid reflecting options in SYN ACK will not be needed because it's the receiver's responsibility.

Yes.

> - Delayed negotiation might be possible. But, no generic framework is needed. Each option should be designed for its own way to support it.

Yes.

> But, once thing I am wondering is that we can still combine Aggregated options and delayed negotiation even with this one.

Delayed negotiation is a completely separate and very complicated issue. It’s not as simple as “do it later”; in the TWHS, the sequence of messages is defined. Elsewhere, messages can come out of order - at which point the meaning of options with respect to that order has to be defined. That’s a can of worms and completely independent of whether aggregated options are supported.

> if you are meaning options not appeared in SYN cannot be used in the following segments, I think accecn option might be a precedent for this.  

Accecn may not appear in an option explicitly, but it is communicated using SYN flags. That’s very close to what the compressed options achieves.

I.e., the option value itself need not appear in the SYN, but the use of that option *is* signaled using the SYN/SYN-ACK, either way.

Again, doing that negotiaton after the TWHS is a separate and more complex issue that I believe should be defined for each option (even if you want to define it for a separate post-SYN aggregate option); it can’t be generalized for existing options.

Joe

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 17 Feb 2021 02:35:16 -0800

Hi Joe,

On Fri, Feb 12, 2021 at 8:45 PM Joseph Touch <[email-masked]> wrote:

> - Aggregated options would be useful. but, the format can be simple 4
> bytes length option that can accomodate 16 options.
>
>
> Yes,
>
> Delayed negotiation is a completely separate and very complicated issue.
> It’s not as simple as “do it later”; in the TWHS, the sequence of messages
> is defined. Elsewhere, messages can come out of order - at which point the
> meaning of options with respect to that order has to be defined. That’s a
> can of worms and completely independent of whether aggregated options are
> supported.
>
> Accecn may not appear in an option explicitly, but it is communicated
> using SYN flags. That’s very close to what the compressed options achieves.
>
> I.e., the option value itself need not appear in the SYN, but the use of
> that option *is* signaled using the SYN/SYN-ACK, either way.
>
> Again, doing that negotiaton after the TWHS is a separate and more complex
> issue that I believe should be defined for each option (even if you want to
> define it for a separate post-SYN aggregate option); it can’t be
> generalized for existing options.
>

OK. I think I understand your viewpoints. Thanks for the explanation.
However, I still have different views on them.
I've re-read the mptcp specs, but, the design of option negotiation scheme
in the doc looks to me not mptcp-specific and simply aims to exchange
additional info reliably.
Also, I still think it's possible to combine aggregate option and delayed
negotiation scheme. It can be separated, but I don't see issues even if
it's combined.

But, anyway, at this point I don't think I have enough evidence to convince
you.
I will think about this further. Also, if possible, I would like to hear
other people's options.

Thanks,
--
Yoshi

---

From: Jan Rüth <[email-masked]>
Date: Mon, 22 Feb 2021 10:45:59 +0000

Hi everyone,

maybe I can add to the measurements that Matthew cited.
We performed these only once. But we are doing other measurements that we do more frequently.

From these I can report further numbers that we never got around to actually publish.

We look for TFO support on port 80 and we send TFO cookie requests to all of IPv4.
We get roughly 3.2M valid replies with TFO cookies (as of late 2019), which rose from 500k (mid 2017).

Regarding the mirrored options, we see around 70k - 80k empty cookie replies (essentially, we get our cookie request back) that have the same behavior as outlined in Olivier’s blogpost, this number is rather stable, sometimes 10k more or less.
This behavior is focussed on IPs residing in Chinese autonomous systems.
Our assumption has always been that this mirroring is performed by some middlebox and not an end-host.

If there is interest, I can perform these measurements with an unassigned TCP option number.

Best
 Jan

> On 8. Feb 2021, at 23:15, Matthew Luckie <[email-masked]> wrote:
> 
> Hi Richard,
>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 24 Feb 2021 02:00:28 -0800

Hi Jan,

Thanks for providing very interesting information.
I am guessing the ratio is not very high, but it's a good proof that we've
still seen such nodes. (probably connection split middleboxes as you
mentioned)
Thanks for sharing!
--
Yoshi

On Mon, Feb 22, 2021 at 2:46 AM Jan Rüth <[email-masked]>
wrote:

> Regarding the mirrored options, we see around 70k - 80k empty cookie
> replies (essentially, we get our cookie request back) that have the same
> behavior as outlined in Olivier’s blogpost, this number is rather stable,
> sometimes 10k more or less.