Subject: Re: [multipathtcp] RFC6824bis edits based on implementation feedback

From: V Anil Kumar <[email-masked]>
Date: Sun, 2 Feb 2020 22:09:41 +0530 (IST)

Hi Yoshi, 

Thanks for this point. In fact, I had initially not thought of a scenario, where the map is being delivered through a retransmitted data packet while its first transmission did not include the map. Now I am just seeing the document (RFC 6824-bis) in this context. 

My understanding is that in scenarios like what I described in my previous mail, RST is likely to happen whether we explicitly state so or not. Please see the paragraph containing the below text in RFC 6824-bis. 

"If a mapping for that subflow-level sequence space does not arrive within a receive window of data, that subflow SHOULD be treated as broken, closed with a RST, and any unmapped data silently discarded." 

if we assume that the map is included while retransmitting the data (even though the first transmission did not contain the map for some reasons), we could argue that RST could be avoided provided that the retransmission is triggered within a receive window of data. But the question here would be how and when will the retransmission take place. In this case, the subflow may not initiate the retransmission of data by its own (i.e., no retransmission due to three duplicate ACKs or RTO expiry at subflow level) as there is no segment loss at subflow level sequence space. So there could be a high possibility of RST happening even before the map delivery through retransmission. 

With regards, 

Anil 

From: "Yoshifumi Nishida" <[email-masked]> 
To: "V Anil Kumar" <[email-masked]> 
Cc: "alan ford" <[email-masked]>, "multipathtcp" <[email-masked]> 
Sent: Saturday, February 1, 2020 3:39:51 AM 
Subject: Re: [multipathtcp] RFC6824bis edits based on implementation feedback 

Hi Anil, 

I have a question about your proposed text. 
I am actually wondering if we really want to terminate connection here. 

The packets without proper mappings will be treated as invalid and will be discarded. 
If an implementation failed to attach proper mapping for some reasons (e.g. option space), it might be able to attach the proper one when it retransmits the packets. This also looks ok to me. 

I don't have strong preference for this. But, do we have a reason to terminate connection? 

Thanks, 
-- 
Yoshi 

On Mon, Jan 13, 2020 at 10:28 AM V Anil Kumar < [ mailto:[email-masked] | [email-masked] ] > wrote: 

Hi, 

I have some points related to the modifications (Change 2) being proposed on data sequence map. Please see them inline. Though I am putting forward the below points, if the consensus is in favour of the proposed change for reducing implementation complexity, I am also OK with that as well. 

From: "alan ford" < [ mailto:[email-masked] | [email-masked] ] > 
To: [ mailto:[email-masked] | [email-masked] ] 
Sent: Thursday, January 2, 2020 4:21:32 AM 
Subject: [multipathtcp] RFC6824bis edits based on implementation feedback 

Hi all, 
We’d love to get this to a state of completion as soon as possible, and to this end I am starting a new thread on this topic. In discussion with the chairs, it is possible to make the desired changes in AUTH48 as long as there is WG consensus. The discussion so far has been fairly limited in terms of participation. 

I would ask the chairs please if it was possible to specify a time bound for this discussion and a default conclusion. 

Regarding the changes, in summary, there are two areas where changes have been requested by the implementation community. As we are the IETF we obviously have strong focus on “running code” and so ease of implementing standards-compliant code is strongly desirable. However, we do not wish to reduce functionality agreed by the IETF community if it is considered a required feature by the community. 

Change 1 

Change the sentence reading: 

If B has data to send first, then the reliable delivery of the ACK + MP_CAPABLE can be inferred by the receipt of this data with an MPTCP Data Sequence Signal (DSS) option (Section 3.3). 

To: 

If B has data to send first, then the reliable delivery of the ACK + MP_CAPABLE is ensured by the receipt of this data with an MPTCP Data Sequence Signal (DSS) option (Section 3.3) containing a DATA_ACK for the MP_CAPABLE (which is the first octet of the data sequence space). 

What this means: 

The current text is concerned only with ensuring a path is MPTCP capable, and so only cares that DSS option occurs on a data packet. However, the MP_CAPABLE option is defined to occupy the first octet of data sequence space and thus, if analogous to TCP, must be acknowledged. >From an implementation point of view it would make sense not to have this hanging around forever and instead define it is acknowledged at the connection level as soon as received. This change ensures the first data packet also DATA_ACKs this MP_CAPABLE octet. 

Change 2 

Change the sentence reading: 

A Data Sequence Mapping does not need to be included in every MPTCP packet, as long as the subflow sequence space in that packet is covered by a mapping known at the receiver. 

To: 

The mapping provided by a Data Sequence Mapping MUST apply to some or all of the subflow sequence space in the TCP segment which carries the option. It does not need to be included in every MPTCP packet, as long as the subflow sequence space in that packet is covered by a mapping known at the receiver. 

What this means: 

The current text does not place any restrictions on where a mapping could appear. In theory a sender could define a thousand different mappings up front, send them all, and expect a receiver to store this and reassemble data according to these mappings as it arrives. Indeed, this was never explicitly disallowed since it “might have been useful”. The implementation community, however, has expressed concerns over the difficulty of implementing this open-endedly. How many mappings is it reasonable to store? Is there a DoS risk here? Instead, it has been requested that thee specification restricts the placement of the DSS option to being within the subflow sequence space to which it applies. 

Below are my comments on this. I had shared some of these points in a previous thread that you had initiated in the same context. 

Transmitting large number of non-contiguous data sequence maps could be a misbehaviour (map-flooding), though it is not clear whether this can go to the extent of causing a potential DoS to the data receiver. So some sort of restriction on this could be useful. One approach could be to insist that the data sender should ensure that the map being transmitted is for in-window data, as per the receiver advertised window. A receiver should anyhow be willing to store the maps for in-window data to deal with packet loss. For example, when a window of data segments (say 1 to 64) is transmitted, each carrying its corresponding map, and segment-1 is lost, the maps for the remaining 63 need to be stored till the lost segment is retransmitted. Of course, in this case the maps will be stored at the receiver side along with their corresponding data. But the need to store multiple maps for in-window data would still be there. 

The problem with the proposed change (restriction) is that a data sender may find it difficult, in case a need arise to slightly delay the map delivery by few segments, i.e., sending some data first without map, and then send the corresponding map in a later segment, as shown below: 

subflow-1: segment-1 segment-3 segment-4 segment-7 

bytes:1-100 bytes:201-300 bytes:301-400 bytes:601-700 

no map map for 1-100 map for 201-400 map for 601-700 

subflow-2: segment-2 segment-5 segment-6 segment-8 

bytes: 101-200 bytes:401-500 bytes: 501-600 bytes:701-800 

map for 101-200 map for 401-600 no map map for 701-800 

In the above case, segment-1 goes without map and its map is included later in segment-3, the next data segment in the same subflow. Further, in the above scheduling pattern, the map in segment-3 cannot cover the data in segment-1 and segment-3, as some data in between (segment-2) is transmitted through another subflow. With the proposed change, the map in segment-3 will become invalid and this will eventually break subflow-1, though this could be a corner case. 

The question at this stage is why would segment-1 be transmitted without its map. In the case of bidirectional data transfer, there could be a need to pack both timestamp and SACK options in a data segment, i.e., piggybacking of SACK with data. If we consider that timestamp takes 12 bytes and SACK, even with single block, takes another 10 bytes, the remaining 18 bytes option space is not adequate to carry data sequence signal with map, especially when DSN is 64 bit long. So the delivery of either of the two (SACK or map) would be delayed. 

As far as I understand, RFC 2018 (TCP Selective Acknowledgement Options) implies that SACK should not be delayed. It states "If sent at all, SACK options SHOULD be included in all ACKs which do not ACK the highest sequence number in the data receiver's queue". It also says "If data receiver generates SACK options under any circumstance, it SHOULD generate them under all permitted circumstances". So, as part of meeting the RFC 2018 requirements, if the combination of SACK and timestamp is given preference over DSS, data segments could be transmitted without their map. 

Another case of delaying map could arise if the data sender prefers to send ADD_ADDR option, instead of map, in a data segment. It is nice that ADD_ADDR option can be delivered reliably in a pure ACK, but I think this is not the case with DSS at present. 

If we adopt the proposed change, I think it might also be helpful to spell out how the receiver is supposed to behave, if it gets maps not meeting the MUST condition in the proposed change. For example termination of the subflow with MP_TCPRST option (section 3.6 in RFC 6824-bis) with appropriate reason code and T flag value to intimate the data sender the cause for subflow termination. 
With regards, 

Anil 

Please can members of the WG express whether they are happy with these changes, or concerned. 

Best regards, 
Alan 

_______________________________________________ 
multipathtcp mailing list 
[ mailto:[email-masked] | [email-masked] ] 
[ https://www.ietf.org/mailman/listinfo/multipathtcp | https://www.ietf.org/mailman/listinfo/multipathtcp ] 

_______________________________________________ 
multipathtcp mailing list 
[ mailto:[email-masked] | [email-masked] ] 
[ https://www.ietf.org/mailman/listinfo/multipathtcp | https://www.ietf.org/mailman/listinfo/multipathtcp ]

---

From: Joe Touch <[email-masked]>
Date: Sun, 2 Feb 2020 09:39:05 -0800

RST doesn't "close" a connection; it aborts it. If a RST is sent, it 
should be sent *consistent with how TCP treats sending a RST* and not 
defined anew for this extension.

Joe

On 2/2/2020 8:39 AM, V Anil Kumar wrote:
> Hi Yoshi,
>
> Thanks for this point. In fact, I had initially not thought of a 
> scenario, where the map is being delivered through a retransmitted 
> data packet while its first transmission did not include the map. Now 
> I am just seeing the document (RFC 6824-bis) in this context.
>
> My understanding is that in scenarios like what I described in my 
> previous mail, RST is likely to happen whether we explicitly state so 
> or not. Please see the paragraph containing the below text in RFC 
> 6824-bis.
>
> "If a mapping for that subflow-level sequence space does not arrive 
> within a receive window of data, that subflow SHOULD be treated as 
> broken, closed with a RST, and any unmapped data silently discarded."
>
> if we assume that the map is included while retransmitting the data 
> (even though the first transmission did not contain the map for some 
> reasons),  we could argue that RST could be avoided provided that the 
> retransmission is triggered within a receive window of data. But the 
> question here would be how and when will the retransmission take 
> place. In this case, the subflow may not initiate the retransmission 
> of data by its own (i.e., no retransmission due to three duplicate 
> ACKs or RTO expiry at subflow level) as there is no segment loss at 
> subflow level sequence space. So there could be a high possibility of 
> RST happening even before the map delivery through retransmission.
>
> With regards,
>
> Anil
> ------------------------------------------------------------------------
> *From: *"Yoshifumi Nishida" <[email-masked]>
> *To: *"V Anil Kumar" <[email-masked]>
> *Cc: *"alan ford" <[email-masked]>, "multipathtcp" 
> <[email-masked]>
> *Sent: *Saturday, February 1, 2020 3:39:51 AM
> *Subject: *Re: [multipathtcp] RFC6824bis edits based on implementation 
> feedback
>
> Hi Anil,
>
> If an implementation failed to attach proper mapping for some reasons 
> (e.g. option space), it might be able to attach the proper one when it 
> retransmits the packets. This also looks ok to me.
>
> On Mon, Jan 13, 2020 at 10:28 AM V Anil Kumar <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi,
>
>     I have some points related to the  modifications (Change 2) being
>     proposed on data sequence map. Please see them inline. Though I am
>     putting forward the below points, if the consensus is in favour of
>     the proposed change for reducing implementation complexity, I am
>     also OK with that as well.
>
>     ------------------------------------------------------------------------
>     *From: *"alan ford" <[email-masked] <mailto:[email-masked]>>
>     *To: [email-masked] <mailto:[email-masked]>
>     *Sent: *Thursday, January 2, 2020 4:21:32 AM
>     *Subject: *[multipathtcp] RFC6824bis edits based on implementation
>     feedback
>
>     Hi all,
>     We’d love to get this to a state of completion as soon as
>     possible, and to this end I am starting a new thread on this
>     topic. In discussion with the chairs, it /is /possible to make the
>     desired changes in AUTH48 as long as there is WG consensus. The
>     discussion so far has been fairly limited in terms of participation.
>
>     I would ask the chairs please if it was possible to specify a time
>     bound for this discussion and a default conclusion.
>
>     Regarding the changes, in summary, there are two areas where
>     changes have been requested by the implementation community. As we
>     are the IETF we obviously have strong focus on “running code” and
>     so ease of implementing standards-compliant code is strongly
>     desirable. However, we do not wish to reduce functionality agreed
>     by the IETF community if it is considered a required feature by
>     the community.
>
>
>     *Change 1*
>     *
>     *
>     /   If B has data to send first, then the reliable delivery of the
>     ACK + MP_CAPABLE can be inferred by the receipt of this data with
>     an MPTCP Data Sequence Signal (DSS) option (Section 3.3).
>     /
>     To:
>
>     /   If B has data to send first, then the reliable delivery of the
>     ACK + MP_CAPABLE is ensured by the receipt of this data with an
>     MPTCP Data Sequence Signal (DSS) option (Section 3.3) containing a
>     DATA_ACK for the MP_CAPABLE (which is the first octet of the data
>     sequence space).
>     /
>     What this means:
>
>     The current text is concerned only with ensuring a path is MPTCP
>     capable, and so only cares that DSS option occurs on a data
>     packet. However, the MP_CAPABLE option is defined to occupy the
>     first octet of data sequence space and thus, if analogous to TCP,
>     must be acknowledged. >From an implementation point of view it
>     would make sense not to have this hanging around forever and
>     instead define it is acknowledged at the connection level as soon
>     as received. This change ensures the first data packet also
>     DATA_ACKs this MP_CAPABLE octet.
>
>
>     *Change 2*
>     *
>     *
>     /   A Data Sequence Mapping does not need to be included in every
>     MPTCP packet, as long as the subflow sequence space in that packet
>     is covered by a mapping known at the receiver./
>
>     To:
>
>     /   The mapping provided by a Data Sequence Mapping MUST apply to
>     some or all of the subflow sequence space in the TCP segment which
>     carries the option. It does not need to be included in every MPTCP
>     packet, as long as the subflow sequence space in that packet is
>     covered by a mapping known at the receiver.
>     /
>     What this means:
>
>     The current text does not place any restrictions on where a
>     mapping could appear. In theory a sender could define a thousand
>     different mappings up front, send them all, and expect a receiver
>     to store this and reassemble data according to these mappings as
>     it arrives. Indeed, this was never explicitly disallowed since it
>     “might have been useful”. The implementation community, however,
>     has expressed concerns over the difficulty of implementing this
>     open-endedly. How many mappings is it reasonable to store? Is
>     there a DoS risk here? Instead, it has been requested that thee
>     specification restricts the placement of the DSS option to being
>     within the subflow sequence space to which it applies.
>
>     Below are my comments on this. I had shared some of these points
>     in a previous thread that you had initiated in the same context.
>
>     Transmitting large number of non-contiguous data sequence maps
>     could be a misbehaviour (map-flooding), though it is not clear
>     whether this can go to the extent of causing a potential DoS to
>     the data receiver. So some sort of restriction on this could be
>     useful. One approach could be to insist that the data sender
>     should ensure that the map being transmitted is for in-window
>     data, as per the receiver advertised window. A receiver should
>     anyhow be willing to store the maps for in-window data to deal
>     with packet loss.. For example, when a window of data segments
>     (say 1 to 64) is transmitted, each carrying its corresponding map,
>     and segment-1 is lost, the maps for the remaining 63 need to be
>     stored till the lost segment is retransmitted. Of course, in this
>     case the maps will be stored at the receiver side along with their
>     corresponding data. But the need to store multiple maps for
>     in-window data would still be there.
>
>     The problem with the proposed change (restriction) is that a data
>     sender may find it difficult, in case a need arise to slightly
>     delay the map delivery by few segments, i.e., sending some data
>     first without map, and then send the corresponding map in a later
>     segment, as shown below:
>
>     In the above case, segment-1 goes without map and its map is
>     included later in segment-3, the next data segment in the same
>     subflow. Further,  in the above scheduling pattern, the map in
>     segment-3 cannot cover the  data in segment-1 and segment-3, as
>     some  data in between (segment-2) is transmitted through another
>     subflow..  With the proposed change, the map in segment-3 will
>     become invalid and this will eventually break subflow-1, though
>     this could be a corner case.
>
>     The question at this stage is why would segment-1 be transmitted
>     without its map. In the case of bidirectional data transfer, there
>     could be a need to pack both timestamp and SACK  options in a data
>     segment, i.e., piggybacking of  SACK with data. If we consider
>     that timestamp takes 12 bytes and SACK, even with single block,
>      takes another 10 bytes, the remaining 18 bytes option space is
>     not adequate to carry data sequence signal with map, especially
>     when DSN is 64 bit long. So the delivery of either of the two
>     (SACK or map) would be delayed.
>
>     As far as I understand, RFC 2018 (TCP Selective Acknowledgement
>     Options) implies that SACK should not be delayed. It states "If
>     sent at all, SACK options SHOULD be included in all ACKs which do
>     not ACK the highest sequence number in the data receiver's queue".
>     It also says "If data receiver generates SACK options under any
>     circumstance, it SHOULD generate them under all permitted
>     circumstances". So, as part of meeting the RFC 2018 requirements,
>     if the combination of SACK and timestamp is given preference over
>     DSS, data segments could be transmitted without their map.
>
>     Another case of delaying map could arise if the data sender
>     prefers to send ADD_ADDR option, instead of map, in a data
>     segment. It is nice that ADD_ADDR option can be delivered reliably
>     in a pure ACK, but I think this is not the case with DSS at present.
>
>     If we adopt the proposed change, I think it might also be helpful
>     to spell out how the receiver is supposed to behave, if it gets
>     maps not meeting the MUST condition in the proposed change.  For
>     example termination of the subflow with MP_TCPRST option (section
>     3.6 in RFC 6824-bis) with appropriate reason code and T flag value
>     to intimate the data sender the cause for subflow termination.
>     With regards,
>
>     Anil
>
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/listinfo/multipathtcp
>
>
>
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/listinfo/multipathtcp
>
>
>
>
>
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: <[email-masked]>
Date: Mon, 3 Feb 2020 14:24:03 +0000

<< if we assume that the map is included while retransmitting the data (even though the first transmission did not contain the map for some reasons),  >>
About “for some reason”. Is this something you’ve seen (or expect to see) regularly, or is this something that would be rare?

As you say, the current wording has a SHOULD, rather than a MUST – which would leave a bit of room to do something different if there were specific (but unlikely) circumstances that merited doing something different.

RST forces the immediate closure of an MPTCP subflow. MP_FASTCLOSE forces the immediate closure of the MPTCP connection (which can also be achieved by sending a RST on all subflows).

Best wishes
phil

From: multipathtcp <[email-masked]> On Behalf Of Joe Touch
Sent: 02 February 2020 17:39
To: V Anil Kumar <[email-masked]>; Yoshifumi Nishida <[email-masked]>
Cc: multipathtcp <[email-masked]>
Subject: Re: [multipathtcp] RFC6824bis edits based on implementation feedback

RST doesn't "close" a connection; it aborts it. If a RST is sent, it should be sent *consistent with how TCP treats sending a RST* and not defined anew for this extension.

Joe

With regards,

Anil

________________________________
From: "Yoshifumi Nishida" <[email-masked]><mailto:[email-masked]>
To: "V Anil Kumar" <[email-masked]><mailto:[email-masked]>
Cc: "alan ford" <[email-masked]><mailto:[email-masked]>, "multipathtcp" <[email-masked]><mailto:[email-masked]>
Sent: Saturday, February 1, 2020 3:39:51 AM
Subject: Re: [multipathtcp] RFC6824bis edits based on implementation feedback

Hi Anil,

Thanks,
--
Yoshi

On Mon, Jan 13, 2020 at 10:28 AM V Anil Kumar <[email-masked]<mailto:[email-masked]>> wrote:
Hi,

________________________________
From: "alan ford" <[email-masked]<mailto:[email-masked]>>
To: [email-masked]<mailto:[email-masked]>
Sent: Thursday, January 2, 2020 4:21:32 AM
Subject: [multipathtcp] RFC6824bis edits based on implementation feedback

Hi all,

Change 1

To:

Change 2

To:

What this means:

Transmitting large number of non-contiguous data sequence maps could be a misbehaviour (map-flooding), though it is not clear whether this can go to the extent of causing a potential DoS to the data receiver. So some sort of restriction on this could be useful.  One approach could be to insist that the data sender should ensure that the map being transmitted is for in-window data, as per the receiver advertised window. A receiver should anyhow be willing to store the maps for in-window data to deal with packet loss.. For example, when a window of data segments (say 1 to 64) is transmitted, each carrying its corresponding map, and segment-1 is lost, the maps for the remaining 63 need to be stored till the lost segment is retransmitted. Of course, in this case the maps will be stored at the receiver side along with their corresponding data. But the need to store multiple maps for in-window data would still be there.

In the above case, segment-1 goes without map and its map is included later in segment-3, the next data segment in the same subflow. Further,  in the above scheduling pattern, the map in segment-3 cannot cover the  data in segment-1 and segment-3, as some  data in between (segment-2) is transmitted through another subflow..  With the proposed change, the map in segment-3 will become invalid and this will eventually break subflow-1, though this could be a corner case.

With regards,

Anil

Best regards,
Alan

_______________________________________________
multipathtcp mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/multipathtcp

_______________________________________________
multipathtcp mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/multipathtcp

[email-masked]<mailto:[email-masked]>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 3 Feb 2020 23:38:45 -0800

Hi Anil,

Thanks for pointing it out. I overlooked this one.
This looks an interesting point.

It seems to me that whether RST is happen or not depends on the size of
receive window according to the text.
If the receive window size is big enough to accommodate segment 1 and
segment 3, the text "Implementations MAY hold onto such unmapped data for a
short while in the expectation that a mapping will arrive shortly. " can be
applied to the segment 1. As a result, the segment 1 won't be discarded.
However, this might be contradict with the new texts Alan proposed? Or, am
I missing something?

Thanks,
--
Yoshi

On Sun, Feb 2, 2020 at 8:42 AM V Anil Kumar <[email-masked]> wrote:

> Hi Yoshi,
>
> Thanks for this point. In fact, I had initially not thought of a scenario,
> where the map is being delivered through a retransmitted data packet while
> its first transmission did not include the map. Now I am just seeing the
> document (RFC 6824-bis) in this context.
>
> My understanding is that in scenarios like what I described in my previous
> mail, RST is likely to happen whether we explicitly state so or not. Please
> see the paragraph containing the below text in RFC 6824-bis.
>
> "If a mapping for that subflow-level sequence space does not arrive within
> a receive window of data, that subflow SHOULD be treated as broken, closed
> with a RST, and any unmapped data silently discarded."
>
> if we assume that the map is included while retransmitting the data (even
> though the first transmission did not contain the map for some reasons),
>  we could argue that RST could be avoided provided that the retransmission
> is triggered within a receive window of data. But the question here would
> be how and when will the retransmission take place. In this case, the
> subflow may not initiate the retransmission of data by its own (i.e., no
> retransmission due to three duplicate ACKs or RTO expiry at subflow level)
> as there is no segment loss at subflow level sequence space. So there could
> be a high possibility of RST happening even before the map delivery through
> retransmission.
>
> With regards,
>
> Anil
>
> ------------------------------
> *Cc: *"alan ford" <[email-masked]>, "multipathtcp" <
> [email-masked]>
> On Mon, Jan 13, 2020 at 10:28 AM V Anil Kumar <[email-masked]> wrote:
>
>> Hi,
>>
>> putting forward the below points, if the consensus is in favour of the
>> proposed change for reducing implementation complexity, I am also OK with
>> that as well.
>>
>> ------------------------------
>> *From: *"alan ford" <[email-masked]>
>> *To: [email-masked]
>> We’d love to get this to a state of completion as soon as possible, and
>> to this end I am starting a new thread on this topic. In discussion with
>> the chairs, it *is *possible to make the desired changes in AUTH48 as
>> long as there is WG consensus. The discussion so far has been fairly
>> limited in terms of participation.
>>
>> I would ask the chairs please if it was possible to specify a time bound
>> for this discussion and a default conclusion.
>>
>> Regarding the changes, in summary, there are two areas where changes have
>> been requested by the implementation community. As we are the IETF we
>> obviously have strong focus on “running code” and so ease of implementing
>> standards-compliant code is strongly desirable. However, we do not wish to
>> reduce functionality agreed by the IETF community if it is considered a
>> required feature by the community.
>>
>>
>> *Change 1*
>>
>> *   If B has data to send first, then the reliable delivery of the ACK +
>> MP_CAPABLE can be inferred by the receipt of this data with an MPTCP Data
>> Sequence Signal (DSS) option (Section 3.3). *
>> To:
>>
>>
>> *   If B has data to send first, then the reliable delivery of the ACK +
>> MP_CAPABLE is ensured by the receipt of this data with an MPTCP Data
>> Sequence Signal (DSS) option (Section 3.3) containing a DATA_ACK for the
>> MP_CAPABLE (which is the first octet of the data sequence space).*
>> What this means:
>>
>> The current text is concerned only with ensuring a path is MPTCP capable,
>> and so only cares that DSS option occurs on a data packet. However, the
>> MP_CAPABLE option is defined to occupy the first octet of data sequence
>> space and thus, if analogous to TCP, must be acknowledged. >From an
>> implementation point of view it would make sense not to have this hanging
>> around forever and instead define it is acknowledged at the connection
>> level as soon as received. This change ensures the first data packet also
>> *   A Data Sequence Mapping does not need to be included in every MPTCP
>> packet, as long as the subflow sequence space in that packet is covered by
>> a mapping known at the receiver.*
>>
>> To:
>>
>>
>> *   The mapping provided by a Data Sequence Mapping MUST apply to some or
>> all of the subflow sequence space in the TCP segment which carries the
>> option. It does not need to be included in every MPTCP packet, as long as
>> the subflow sequence space in that packet is covered by a mapping known at
>> the receiver.*
>> What this means:
>>
>> The current text does not place any restrictions on where a mapping could
>> appear. In theory a sender could define a thousand different mappings up
>> front, send them all, and expect a receiver to store this and reassemble
>> data according to these mappings as it arrives. Indeed, this was never
>> explicitly disallowed since it “might have been useful”. The implementation
>> community, however, has expressed concerns over the difficulty of
>> implementing this open-endedly. How many mappings is it reasonable to
>> store? Is there a DoS risk here? Instead, it has been requested that thee
>> specification restricts the placement of the DSS option to being within the
>> subflow sequence space to which it applies.
>>
>> Below are my comments on this. I had shared some of these points in a
>> previous thread that you had initiated in the same context.
>>
>> Transmitting large number of non-contiguous data sequence maps could be a
>> misbehaviour (map-flooding), though it is not clear whether this can go to
>> the extent of causing a potential DoS to the data receiver. So some sort of
>> restriction on this could be useful.  One approach could be to insist that
>> the data sender should ensure that the map being transmitted is for
>> in-window data, as per the receiver advertised window. A receiver should
>> anyhow be willing to store the maps for in-window data to deal with packet
>> loss. For example, when a window of data segments (say 1 to 64) is
>> transmitted, each carrying its corresponding map, and segment-1 is lost,
>> the maps for the remaining 63 need to be stored till the lost segment is
>> retransmitted. Of course, in this case the maps will be stored at the
>> receiver side along with their corresponding data. But the need to store
>> multiple maps for in-window data would still be there.
>>
>> The problem with the proposed change (restriction) is that a data sender
>> may find it difficult, in case a need arise to slightly delay the map
>> delivery by few segments, i.e., sending some data first without map, and
>> then send the corresponding map in a later segment, as shown below:
>>
>> In the above case, segment-1 goes without map and its map is included
>> later in segment-3, the next data segment in the same subflow. Further,  in
>> the above scheduling pattern, the map in segment-3 cannot cover the  data
>> in segment-1 and segment-3, as some  data in between (segment-2) is
>> transmitted through another subflow.  With the proposed change, the map in
>> segment-3 will become invalid and this will eventually break subflow-1,
>> though this could be a corner case.
>>
>> The question at this stage is why would segment-1 be transmitted without
>> its map. In the case of bidirectional data transfer, there could be a need
>> to pack both timestamp and SACK  options in a data segment, i.e.,
>> piggybacking of  SACK with data. If we consider that timestamp takes 12
>> bytes and SACK, even with single block,  takes another 10 bytes, the
>> remaining 18 bytes option space is not adequate to carry data sequence
>> signal with map, especially when DSN is 64 bit long. So the delivery of
>> either of the two (SACK or map) would be delayed.
>>
>> As far as I understand, RFC 2018 (TCP Selective Acknowledgement Options)
>> implies that SACK should not be delayed. It states "If sent at all, SACK
>> options SHOULD be included in all ACKs which do not ACK the highest
>> sequence number in the data receiver's queue". It also says "If data
>> receiver generates SACK options under any circumstance, it SHOULD generate
>> them under all permitted circumstances".   So, as part of meeting the RFC
>> 2018 requirements, if the combination of SACK and timestamp is given
>> preference over DSS, data segments could be transmitted without their map.
>>
>> Another case of delaying map could arise if the data sender prefers to
>> send ADD_ADDR option, instead of map, in a data segment. It is nice that
>> ADD_ADDR option can be delivered reliably in a pure ACK, but I think this
>> is not the case with DSS at present.
>>
>> If we adopt the proposed change, I think it might also be helpful to
>> spell out how the receiver is supposed to behave, if it gets maps not
>> meeting the MUST condition in the proposed change.  For example termination
>> of the subflow with MP_TCPRST option (section 3.6 in RFC 6824-bis) with
>> appropriate reason code and T flag value to intimate the data sender the
>> cause for subflow termination.
>>
>> With regards,
>>
>> Anil
>>

---

From: Alan Ford <[email-masked]>
Date: Tue, 4 Feb 2020 12:12:17 +0000

Hi Yoshi,

The proposed new text enforces a mapping to be sent on a packet whose subflow sequence number is included within the mapping provided. This does not directly contradict with the MAY case. Indeed this scenario could easily happen if there is a 20 byte packet followed by a 10 byte packet. The first 10 bytes of the first packet are covered by the mapping on packet 1, then the remaining 10 bytes and the following 10 bytes are covered by the mapping on packet 2. That would be valid.

To re-iterate, the primary purpose of this proposed text is to stop people sending mappings before they are needed.

Best regards,
Alan

> On 4 Feb 2020, at 07:38, Yoshifumi Nishida <[email-masked]> wrote:
> 
> Hi Anil,
> 
> It seems to me that whether RST is happen or not depends on the size of receive window according to the text.
> If the receive window size is big enough to accommodate segment 1 and segment 3, the text "Implementations MAY hold onto such unmapped data for a short while in the expectation that a mapping will arrive shortly. " can be applied to the segment 1. As a result, the segment 1 won't be discarded. 
> However, this might be contradict with the new texts Alan proposed? Or, am I missing something?
> 
> Thanks,
> --
> Yoshi
> 
> 
> 
> On Sun, Feb 2, 2020 at 8:42 AM V Anil Kumar <[email-masked] <mailto:[email-masked]>> wrote:
> Hi Yoshi,
> 
> From: "Yoshifumi Nishida" <[email-masked] <mailto:[email-masked]>>
> To: "V Anil Kumar" <[email-masked] <mailto:[email-masked]>>
> Cc: "alan ford" <[email-masked] <mailto:[email-masked]>>, "multipathtcp" <[email-masked] <mailto:[email-masked]>>
> On Mon, Jan 13, 2020 at 10:28 AM V Anil Kumar <[email-masked] <mailto:[email-masked]>> wrote:
> Hi,
> 
> From: "alan ford" <[email-masked] <mailto:[email-masked]>>
> To: [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/multipathtcp <https://www.ietf.org/mailman/listinfo/multipathtcp>
> 
> 
> https://www.ietf.org/mailman/listinfo/multipathtcp <https://www.ietf.org/mailman/listinfo/multipathtcp>
> 
> 
>

---

From: Matthieu Baerts <[email-masked]>
Date: Tue, 4 Feb 2020 14:28:44 +0100

Hi all,

First, thank you all for having contributed all these years to make 
clear MPTCP RFCs.

I'm sorry to interrupt the discussions with my message but because it 
seems there are still some opened questions related to these changes, I 
would like to express my view from the implementer side.

As a maintainer of both the MPTCP Linux out-of-tree implementation and 
the new MPTCP Linux upstream one, I fully endorse both changes.

The first one avoids confusions while the second one avoids senders to 
generate segments that are not supported by the two MPTCP Linux 
implementations and certainly by most/all existing ones. Accepting such 
Data Segment Mappings are not supported and will probably never be 
because it is complex, middleboxes don't seem to modify packets to have 
a situation like that and it can be a source of security issues. 
Avoiding senders to create such mappings would avoid issues if most 
network stacks don't implement this specific support for technical reasons.

Thank you Alan for taking care of this!

Cheers,
Matt

On 04/02/2020 13:12, Alan Ford wrote:
> Hi Yoshi,
> 
> The proposed new text enforces a mapping to be sent on a packet whose 
> subflow sequence number is included within the mapping provided. This 
> does not directly contradict with the MAY case. Indeed this scenario 
> could easily happen if there is a 20 byte packet followed by a 10 byte 
> packet. The first 10 bytes of the first packet are covered by the 
> mapping on packet 1, then the remaining 10 bytes and the following 10 
> bytes are covered by the mapping on packet 2. That would be valid.
> 
> To re-iterate, the primary purpose of this proposed text is to stop 
> people sending mappings before they are needed.
> 
> Best regards,
> Alan
> 
>> On 4 Feb 2020, at 07:38, Yoshifumi Nishida <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> Hi Anil,
>>
>> segment 3, the text "Implementations MAY hold onto such unmapped data 
>> for a short while in the expectation that a mapping will arrive 
>> shortly. " can be applied to the segment 1. As a result, the segment 1 
>> won't be discarded.
>>     data packet while its first transmission did not include the map.
>>     Now I am just seeing the document (RFC 6824-bis) in this context.
>>
>>     previous mail, RST is likely to happen whether we explicitly state
>>     so or not. Please see the paragraph containing the below text in
>>     RFC 6824-bis.
>>
>>     "If a mapping for that subflow-level sequence space does not
>>     arrive within a receive window of data, that subflow SHOULD be
>>     treated as broken, closed with a RST, and any unmapped data
>>     silently discarded."
>>
>>     if we assume that the map is included while retransmitting the
>>     data (even though the first transmission did not contain the map
>>     for some reasons),  we could argue that RST could be avoided
>>     provided that the retransmission is triggered within a receive
>>     window of data. But the question here would be how and when will
>>     the retransmission take place. In this case, the subflow may not
>>     initiate the retransmission of data by its own (i.e., no
>>     retransmission due to three duplicate ACKs or RTO expiry at
>>     subflow level) as there is no segment loss at subflow level
>>     sequence space. So there could be a high possibility of RST
>>     happening even before the map delivery through retransmission.
>>
>>     With regards,
>>
>>     Anil
>>     *From: *"Yoshifumi Nishida" <[email-masked]
>>     <mailto:[email-masked]>>
>>     *To: *"V Anil Kumar" <[email-masked] <mailto:[email-masked]>>
>>     *Cc: *"alan ford" <[email-masked]
>>     <mailto:[email-masked]>>, "multipathtcp"
>>     <[email-masked] <mailto:[email-masked]>>
>>     *Subject: *Re: [multipathtcp] RFC6824bis edits based on
>>     implementation feedback
>>
>>     Hi Anil,
>>
>>     If an implementation failed to attach proper mapping for some
>>     reasons (e.g. option space), it might be able to attach the proper
>>     one when it retransmits the packets. This also looks ok to me.
>>
>>         I have some points related to the  modifications (Change 2)
>>         being proposed on data sequence map. Please see them inline.
>>         Though I am putting forward the below points, if the consensus
>>         is in favour of the proposed change for reducing
>>         implementation complexity, I am also OK with that as well.
>>
>>         *Subject: *[multipathtcp] RFC6824bis edits based on
>>         implementation feedback
>>
>>         Hi all,
>>         topic. In discussion with the chairs, it /is /possible to make
>>         the desired changes in AUTH48 as long as there is WG
>>         consensus. The discussion so far has been fairly limited in
>>         terms of participation.
>>
>>         I would ask the chairs please if it was possible to specify a
>>         time bound for this discussion and a default conclusion.
>>
>>         changes have been requested by the implementation community.
>>         As we are the IETF we obviously have strong focus on “running
>>         code” and so ease of implementing standards-compliant code is
>>         strongly desirable. However, we do not wish to reduce
>>         functionality agreed by the IETF community if it is considered
>>         a required feature by the community.
>>
>>
>>         *Change 1*
>>         *
>>         *
>>         /   If B has data to send first, then the reliable delivery of
>>         the ACK + MP_CAPABLE can be inferred by the receipt of this
>>         data with an MPTCP Data Sequence Signal (DSS) option (Section
>>         3.3).
>>         /
>>         To:
>>
>>         /   If B has data to send first, then the reliable delivery of
>>         the ACK + MP_CAPABLE is ensured by the receipt of this data
>>         with an MPTCP Data Sequence Signal (DSS) option (Section 3.3)
>>         containing a DATA_ACK for the MP_CAPABLE (which is the first
>>         octet of the data sequence space).
>>         /
>>         What this means:
>>
>>         The current text is concerned only with ensuring a path is
>>         MPTCP capable, and so only cares that DSS option occurs on a
>>         data packet. However, the MP_CAPABLE option is defined to
>>         occupy the first octet of data sequence space and thus, if
>>         analogous to TCP, must be acknowledged. >From an
>>         implementation point of view it would make sense not to have
>>         this hanging around forever and instead define it is
>>         acknowledged at the connection level as soon as received. This
>>         change ensures the first data packet also DATA_ACKs this
>>         MP_CAPABLE octet.
>>
>>
>>         *Change 2*
>>         *
>>         *
>>         /   A Data Sequence Mapping does not need to be included in
>>         every MPTCP packet, as long as the subflow sequence space in
>>         that packet is covered by a mapping known at the receiver./
>>
>>         To:
>>
>>         /   The mapping provided by a Data Sequence Mapping MUST apply
>>         to some or all of the subflow sequence space in the TCP
>>         segment which carries the option. It does not need to be
>>         included in every MPTCP packet, as long as the subflow
>>         sequence space in that packet is covered by a mapping known at
>>         the receiver.
>>         /
>>         What this means:
>>
>>         mapping could appear. In theory a sender could define a
>>         thousand different mappings up front, send them all, and
>>         expect a receiver to store this and reassemble data according
>>         to these mappings as it arrives. Indeed, this was never
>>         explicitly disallowed since it “might have been useful”. The
>>         implementation community, however, has expressed concerns over
>>         the difficulty of implementing this open-endedly. How many
>>         mappings is it reasonable to store? Is there a DoS risk here?
>>         Instead, it has been requested that thee specification
>>         restricts the placement of the DSS option to being within the
>>         Below are my comments on this. I had shared some of these
>>         points in a previous thread that you had initiated in the same
>>         context.
>>
>>         whether this can go to the extent of causing a potential DoS
>>         to the data receiver. So some sort of restriction on this
>>         could be useful.  One approach could be to insist that the
>>         data sender should ensure that the map being transmitted is
>>         for in-window data, as per the receiver advertised window. A
>>         receiver should anyhow be willing to store the maps for
>>         in-window data to deal with packet loss. For example, when a
>>         window of data segments (say 1 to 64) is transmitted, each
>>         carrying its corresponding map, and segment-1 is lost, the
>>         maps for the remaining 63 need to be stored till the lost
>>         segment is retransmitted. Of course, in this case the maps
>>         will be stored at the receiver side along with their
>>         The problem with the proposed change (restriction) is that a
>>         data sender may find it difficult, in case a need arise to
>>         slightly delay the map delivery by few segments, i.e., sending
>>         some data first without map, and then send the corresponding
>>         map in a later segment, as shown below:
>>
>>                               bytes:1-100                
>>         bytes:201-300              bytes:301-400                
>>         bytes:601-700
>>                               no map                        map for
>>                                bytes: 101-200          
>>         bytes:401-500               bytes: 501-600              
>>          bytes:701-800
>>         segment-3 cannot cover the  data in segment-1 and segment-3,
>>         as some  data in between (segment-2) is transmitted through
>>         another subflow.  With the proposed change, the map in
>>         segment-3 will become invalid and this will eventually break
>>         subflow-1, though this could be a corner case.
>>
>>         The question at this stage is why would segment-1 be
>>         transmitted without its map. In the case of bidirectional data
>>         transfer, there could be a need to pack both timestamp and
>>         SACK  options in a data segment, i.e., piggybacking of  SACK
>>         with data. If we consider that timestamp takes 12 bytes and
>>         SACK, even with single block,  takes another 10 bytes, the
>>         remaining 18 bytes option space is not adequate to carry data
>>         sequence signal with map, especially when DSN is 64 bit long.
>>         So the delivery of either of the two (SACK or map) would be
>>         delayed.
>>
>>         As far as I understand, RFC 2018 (TCP Selective
>>         Acknowledgement Options) implies that SACK should not be
>>         delayed. It states "If sent at all, SACK options SHOULD be
>>         included in all ACKs which do not ACK the highest sequence
>>         number in the data receiver's queue". It also says "If data
>>         receiver generates SACK options under any circumstance, it
>>         SHOULD generate them under all permitted circumstances".   So,
>>         as part of meeting the RFC 2018 requirements, if the
>>         combination of SACK and timestamp is given preference over
>>         segment. It is nice that ADD_ADDR option can be delivered
>>         reliably in a pure ACK, but I think this is not the case with
>>         DSS at present.
>>
>>         If we adopt the proposed change, I think it might also be
>>         helpful to spell out how the receiver is supposed to behave,
>>         if it gets maps not meeting the MUST condition in the proposed
>>         change.  For example termination of the subflow with MP_TCPRST
>>         option (section 3.6 in RFC 6824-bis) with appropriate reason
>>         code and T flag value to intimate the data sender the cause
>>         for subflow termination.
>>         With regards,
>>
>>         Anil
>>

-- 
Matthieu Baerts | R&D Engineer
[email-masked]
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 5 Feb 2020 00:51:15 -0800

Hi Alan,

Yes, it makes sense. Thanks for the clarification.

BTW, I don't have a strong opinion about whether we should add some texts
to send RST for mapping issue case or the text in the draft Anil pointed
out is already enough.
However, if we want to add new texts, I think SHOULD will be preferable as
it won't contradict with the texts in the draft.
If we settle this, I think the doc is ready to bring it back to AUTH48.

Thanks,
--
Yoshi

On Tue, Feb 4, 2020 at 4:12 AM Alan Ford <[email-masked]> wrote:

> subflow sequence number is included within the mapping provided. This does
> not directly contradict with the MAY case. Indeed this scenario could
> easily happen if there is a 20 byte packet followed by a 10 byte packet.
> The first 10 bytes of the first packet are covered by the mapping on packet
> 1, then the remaining 10 bytes and the following 10 bytes are covered by
> the mapping on packet 2. That would be valid.
>
> segment 3, the text "Implementations MAY hold onto such unmapped data for
> a short while in the expectation that a mapping will arrive shortly. " can
> be applied to the segment 1. As a result, the segment 1 won't be discarded.
>> scenario, where the map is being delivered through a retransmitted data
>> packet while its first transmission did not include the map. Now I am just
>> seeing the document (RFC 6824-bis) in this context.
>>
>> previous mail, RST is likely to happen whether we explicitly state so or
>> not. Please see the paragraph containing the below text in RFC 6824-bis.
>>
>> within a receive window of data, that subflow SHOULD be treated as broken,
>> closed with a RST, and any unmapped data silently discarded."
>>
>>> Regarding the changes, in summary, there are two areas where changes
>>> have been requested by the implementation community. As we are the IETF we
>>> capable, and so only cares that DSS option occurs on a data packet.
>>> However, the MP_CAPABLE option is defined to occupy the first octet of data
>>> sequence space and thus, if analogous to TCP, must be acknowledged. >From
>>> an implementation point of view it would make sense not to have this
>>> hanging around forever and instead define it is acknowledged at the
>>> connection level as soon as received. This change ensures the first data
>>> packet also DATA_ACKs this MP_CAPABLE octet.
>>>
>>>
>>> *Change 2*
>>>
>>> *   The mapping provided by a Data Sequence Mapping MUST apply to some
>>> or all of the subflow sequence space in the TCP segment which carries the
>>> The current text does not place any restrictions on where a mapping
>>> could appear. In theory a sender could define a thousand different mappings
>>> up front, send them all, and expect a receiver to store this and reassemble
>>> Transmitting large number of non-contiguous data sequence maps could be
>>> a misbehaviour (map-flooding), though it is not clear whether this can go
>>> to the extent of causing a potential DoS to the data receiver. So some sort
>>> of restriction on this could be useful.  One approach could be to insist
>>> that the data sender should ensure that the map being transmitted is for

---

From: <[email-masked]>
Date: Wed, 5 Feb 2020 12:23:12 +0000

Trying to summarise – if I get the discussion right, it seems that the “Change 2” text proposed in Alan’s email is OK and that the text there and elsewhere is sufficient.

Matt – thanks for your email, it’s very good to know that the changes are important from the implementers point of view – and thanks for all you & your colleagues efforts on both the internet draft linux and the upstream one (nearly in mainline?)

Best wishes,
phil

From: multipathtcp <[email-masked]> On Behalf Of Yoshifumi Nishida
Sent: 05 February 2020 08:51
To: Alan Ford <[email-masked]>
Cc: multipathtcp <[email-masked]>
Subject: Re: [multipathtcp] RFC6824bis edits based on implementation feedback

Hi Alan,

Yes, it makes sense.. Thanks for the clarification.

BTW, I don't have a strong opinion about whether we should add some texts to send RST for mapping issue case or the text in the draft Anil pointed out is already enough.
However, if we want to add new texts, I think SHOULD will be preferable as it won't contradict with the texts in the draft.
If we settle this, I think the doc is ready to bring it back to AUTH48.

Thanks,
--
Yoshi

On Tue, Feb 4, 2020 at 4:12 AM Alan Ford <[email-masked]<mailto:[email-masked]>> wrote:
Hi Yoshi,

Best regards,
Alan

On 4 Feb 2020, at 07:38, Yoshifumi Nishida <[email-masked]<mailto:[email-masked]>> wrote:

Hi Anil,

Thanks,
--
Yoshi

On Sun, Feb 2, 2020 at 8:42 AM V Anil Kumar <[email-masked]<mailto:[email-masked]>> wrote:
Hi Yoshi,

With regards,

Anil

________________________________
From: "Yoshifumi Nishida" <[email-masked]<mailto:[email-masked]>>
To: "V Anil Kumar" <[email-masked]<mailto:[email-masked]>>
Cc: "alan ford" <[email-masked]<mailto:[email-masked]>>, "multipathtcp" <[email-masked]<mailto:[email-masked]>>
Sent: Saturday, February 1, 2020 3:39:51 AM
Subject: Re: [multipathtcp] RFC6824bis edits based on implementation feedback

Hi Anil,

Thanks,
--
Yoshi

Hi all,

I would ask the chairs please if it was possible to specify a time bound for this discussion and a default conclusion..

Change 1

To:

Change 2

   A Data Sequence Mapping does not need to be included in every MPTCP packet, as long as the subflow sequence space in that packet is covered by a mapping known at the receiver..

To:

What this means:

With regards,

Anil

Best regards,
Alan

---

From: V Anil Kumar <[email-masked]>
Date: Wed, 5 Feb 2020 20:39:25 +0530 (IST)

Hi Yoshi, 

Please see in line. 

Sent: Tuesday, February 4, 2020 1:08:45 PM 

Hi Anil, 

It seems to me that whether RST is happen or not depends on the size of receive window according to the text. 
If the receive window size is big enough to accommodate segment 1 and segment 3, the text " Implementations MAY hold onto such unmapped data for a short while in the expectation that a mapping will arrive shortly. " can be applied to the segment 1. As a result, the segment 1 won't be discarded. 

Yes. So, segment 1 may be kept in the data receiver's buffer in expectation that its mapping will arrive shortly. And in the example that we are referring to, the data sender will not be able to include the map for the data in segment 1 in segment 3 or any higher segment. 

Regards, 

Anil 

Thanks, 
-- 
Yoshi 

On Sun, Feb 2, 2020 at 8:42 AM V Anil Kumar < [ mailto:[email-masked] | [email-masked] ] > wrote: 

Hi Yoshi, 

With regards, 

Anil 

From: "Yoshifumi Nishida" < [ mailto:[email-masked] | [email-masked] ] > 
To: "V Anil Kumar" < [ mailto:[email-masked] | [email-masked] ] > 
Cc: "alan ford" < [ mailto:[email-masked] | [email-masked] ] >, "multipathtcp" < [ mailto:[email-masked] | [email-masked] ] > 
Sent: Saturday, February 1, 2020 3:39:51 AM 
Subject: Re: [multipathtcp] RFC6824bis edits based on implementation feedback 

Hi Anil, 

Thanks, 
-- 
Yoshi 

BQ_BEGIN

Hi, 

Hi all, 

Change 1 

To: 

What this means: 

Change 2 

To: 

What this means: 

Anil 

Best regards, 
Alan 

BQ_END

---

From: Alan Ford <[email-masked]>
Date: Wed, 5 Feb 2020 21:19:29 +0000

Hi Anil,

This would not be forbidden if the mapping was carried on a pure ACK with no data.

I do see the point here: if a packet of 1000 bytes contains 500 bytes of one mapping and 500 bytes of another mapping, then only one DSM would appear on one packet, leaving the mapping for the second 500 bytes to be carried somewhere else - the only option being a pure ACK. But this kind of scenario would be extremely rare and I would imagine any implementation would just split into two 500 byte segments each with the PSH flag set. I don’t think we need to spell this out in the spec however.

Regards,
Alan

> On 5 Feb 2020, at 15:09, V Anil Kumar <[email-masked]> wrote:
> 
> Hi Yoshi,
> 
> Please see in line.
>

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 05 Feb 2020 13:59:14 -0800

> I do see the point here: if a packet of 1000 bytes contains 500 bytes of one mapping and 500 bytes of another mapping, then only one DSM would appear on one packet, leaving the mapping for the seco…

Yes, an implementation would just send 2 segments each 500 bytes.

Because, if the mapping is sent on a pure ACK, the implementation needs to
retransmit that pure ACK (based on some timer) to make sure that it reaches
the receiver. Which basically means that we are making the pure ACK reliable.
Sure, everything is possible in software but the complexity would be too big.

Thus, easiest solution is to simply send 2 500 byte segments.

Christoph

---

From: V Anil Kumar <[email-masked]>
Date: Thu, 6 Feb 2020 23:37:19 +0530 (IST)

Hi Alan, 

Thank you for your reply. I have two points to clarify. Please see them in line. 

> From: "Alan Ford" <[email-masked]>
> To: "V Anil Kumar" <[email-masked]>jk
> Cc: "Yoshifumi Nishida" <[email-masked]>, "multipathtcp"
> <[email-masked]>
> Sent: Thursday, February 6, 2020 2:49:29 AM

> This would not be forbidden if the mapping was carried on a pure ACK with no

As far as I understand, delivery of DSM through pure ACK is not in the scope of the current draft. If we intend to do this, we may need an approach similar to the one proposed for delivery of ADD_ADDR option in pure ACK with reliability feature. So, packing DSM on pure ACK does not seem to be an option at this stage. 

> I do see the point here: if a packet of 1000 bytes contains 500 bytes of one
> mapping and 500 bytes of another mapping, then only one DSM would appear on one
> packet, leaving the mapping for the second 500 bytes to be carried somewhere
> else - the only option being a pure ACK. But this kind of scenario would be
> extremely rare

Yes, I do agree that the scenario you mentioned above would be extremely rare. In fact, I wonder whether such a situation (i.e., need to cover the bytes in a single packet with two different maps) would ever arise. Probably there might be some corner cases, which I don't get it rightly now. 

More importantly, the scenario that Yoshi and me are referring to is totally different: the one I had given as part of my comments in response to the proposed change 2. In this scenario, which you could see in the trailing mail, data segment-1 is transmitted without including its map, something which is permitted in the mptcp framework (6824 bis). I make this inference from the below text in 6824 bis: 

"...even if a mapping does not exist from the subflow space to the data- level space, the data SHOULD still be ACKed at the subflow (if it is in-window). This data cannot, however, be acknowledged at the data level (Section 3.3.2) because its data sequence numbers are unknown. Implementations MAY hold onto such unmapped data for a short while in the expectation that a mapping will arrive shortly." 

One option the data sender has at this stage is to include the map for data in segment-1, in segment-3 (please see the two-subflow example I gave in response to the change 2 you had proposed). But the proposed change does not permit this. What would happen to subflow-1 in this case ? 

With regards, 

Anil 

> and I would imagine any implementation would just split into two 500 byte
> segments each with the PSH flag set. I don’t think we need to spell this out in
> the spec however.

>> On 5 Feb 2020, at 15:09, V Anil Kumar < [ mailto:[email-masked] |
>> [email-masked] ] > wrote:
>> Hi Yoshi,

>> Cc: "alan ford" < [ mailto:[email-masked] | [email-masked] ] >,
>> "multipathtcp" < [ mailto:[email-masked] | [email-masked] ] >

>> If the receive window size is big enough to accommodate segment 1 and segment 3,
>> the text " Implementations MAY hold onto such unmapped data for a short while
>> in the expectation that a mapping will arrive shortly. " can be applied to the
>> segment 1. As a result, the segment 1 won't be discarded.

>> Yes. So, segment 1 may be kept in the data receiver's buffer in expectation that
>> its mapping will arrive shortly. And in the example that we are referring to,
>> the data sender will not be able to include the map for the data in segment 1
>> in segment 3 or any higher segment.

>>> Thanks for this point. In fact, I had initially not thought of a scenario, where
>>> the map is being delivered through a retransmitted data packet while its first
>>> transmission did not include the map. Now I am just seeing the document (RFC
>>> 6824-bis) in this context.

>>> My understanding is that in scenarios like what I described in my previous mail,
>>> RST is likely to happen whether we explicitly state so or not. Please see the
>>> paragraph containing the below text in RFC 6824-bis.

>>> "If a mapping for that subflow-level sequence space does not arrive within a
>>> receive window of data, that subflow SHOULD be treated as broken, closed with a
>>> RST, and any unmapped data silently discarded."

>>> if we assume that the map is included while retransmitting the data (even though
>>> the first transmission did not contain the map for some reasons), we could
>>> argue that RST could be avoided provided that the retransmission is triggered
>>> within a receive window of data. But the question here would be how and when
>>> will the retransmission take place. In this case, the subflow may not initiate
>>> the retransmission of data by its own (i.e., no retransmission due to three
>>> duplicate ACKs or RTO expiry at subflow level) as there is no segment loss at
>>> subflow level sequence space. So there could be a high possibility of RST

>>> Cc: "alan ford" < [ mailto:[email-masked] | [email-masked] ] >,
>>> "multipathtcp" < [ mailto:[email-masked] | [email-masked] ] >

>>> If an implementation failed to attach proper mapping for some reasons (e.g.
>>> option space), it might be able to attach the proper one when it retransmits
>>> the packets. This also looks ok to me.

>>>> I have some points related to the modifications (Change 2) being proposed on
>>>> data sequence map. Please see them inline. Though I am putting forward the
>>>> below points, if the consensus is in favour of the proposed change for reducing

>>>> Hi all,
>>>> We’d love to get this to a state of completion as soon as possible, and to this
>>>> end I am starting a new thread on this topic. In discussion with the chairs, it
>>>> is possible to make the desired changes in AUTH48 as long as there is WG
>>>> consensus. The discussion so far has been fairly limited in terms of
>>>> participation.

>>>> I would ask the chairs please if it was possible to specify a time bound for
>>>> this discussion and a default conclusion.

>>>> Regarding the changes, in summary, there are two areas where changes have been
>>>> requested by the implementation community. As we are the IETF we obviously have
>>>> strong focus on “running code” and so ease of implementing standards-compliant
>>>> code is strongly desirable. However, we do not wish to reduce functionality
>>>> agreed by the IETF community if it is considered a required feature by the
>>>> community.

>>>> If B has data to send first, then the reliable delivery of the ACK + MP_CAPABLE
>>>> can be inferred by the receipt of this data with an MPTCP Data Sequence Signal
>>>> (DSS) option (Section 3.3).

>>>> If B has data to send first, then the reliable delivery of the ACK + MP_CAPABLE
>>>> is ensured by the receipt of this data with an MPTCP Data Sequence Signal (DSS)
>>>> option (Section 3.3) containing a DATA_ACK for the MP_CAPABLE (which is the
>>>> first octet of the data sequence space).

>>>> The current text is concerned only with ensuring a path is MPTCP capable, and so
>>>> only cares that DSS option occurs on a data packet. However, the MP_CAPABLE
>>>> option is defined to occupy the first octet of data sequence space and thus, if
>>>> analogous to TCP, must be acknowledged. >From an implementation point of view
>>>> it would make sense not to have this hanging around forever and instead define
>>>> it is acknowledged at the connection level as soon as received. This change
>>>> ensures the first data packet also DATA_ACKs this MP_CAPABLE octet.

>>>> A Data Sequence Mapping does not need to be included in every MPTCP packet, as
>>>> long as the subflow sequence space in that packet is covered by a mapping known
>>>> at the receiver.

>>>> The mapping provided by a Data Sequence Mapping MUST apply to some or all of the
>>>> subflow sequence space in the TCP segment which carries the option. It does not
>>>> need to be included in every MPTCP packet, as long as the subflow sequence
>>>> space in that packet is covered by a mapping known at the receiver.

>>>> appear. In theory a sender could define a thousand different mappings up front,
>>>> send them all, and expect a receiver to store this and reassemble data
>>>> according to these mappings as it arrives. Indeed, this was never explicitly
>>>> disallowed since it “might have been useful”. The implementation community,
>>>> however, has expressed concerns over the difficulty of implementing this
>>>> open-endedly. How many mappings is it reasonable to store? Is there a DoS risk
>>>> here? Instead, it has been requested that thee specification restricts the
>>>> placement of the DSS option to being within the subflow sequence space to which
>>>> it applies.

>>>> Below are my comments on this. I had shared some of these points in a previous
>>>> thread that you had initiated in the same context.

>>>> misbehaviour (map-flooding), though it is not clear whether this can go to the
>>>> extent of causing a potential DoS to the data receiver. So some sort of
>>>> restriction on this could be useful. One approach could be to insist that the
>>>> data sender should ensure that the map being transmitted is for in-window data,
>>>> as per the receiver advertised window. A receiver should anyhow be willing to
>>>> store the maps for in-window data to deal with packet loss. For example, when a
>>>> window of data segments (say 1 to 64) is transmitted, each carrying its
>>>> corresponding map, and segment-1 is lost, the maps for the remaining 63 need to
>>>> be stored till the lost segment is retransmitted. Of course, in this case the
>>>> maps will be stored at the receiver side along with their corresponding data.
>>>> But the need to store multiple maps for in-window data would still be there.

>>>> The problem with the proposed change (restriction) is that a data sender may
>>>> find it difficult, in case a need arise to slightly delay the map delivery by
>>>> few segments, i.e., sending some data first without map, and then send the
>>>> corresponding map in a later segment, as shown below:

>>>> In the above case, segment-1 goes without map and its map is included later in
>>>> segment-3, the next data segment in the same subflow. Further, in the above
>>>> scheduling pattern, the map in segment-3 cannot cover the data in segment-1 and
>>>> segment-3, as some data in between (segment-2) is transmitted through another
>>>> subflow. With the proposed change, the map in segment-3 will become invalid and
>>>> this will eventually break subflow-1, though this could be a corner case.

>>>> The question at this stage is why would segment-1 be transmitted without its
>>>> map. In the case of bidirectional data transfer, there could be a need to pack
>>>> both timestamp and SACK options in a data segment, i.e., piggybacking of SACK
>>>> with data. If we consider that timestamp takes 12 bytes and SACK, even with
>>>> single block, takes another 10 bytes, the remaining 18 bytes option space is
>>>> not adequate to carry data sequence signal with map, especially when DSN is 64
>>>> bit long. So the delivery of either of the two (SACK or map) would be delayed.

>>>> As far as I understand, RFC 2018 (TCP Selective Acknowledgement Options) implies
>>>> that SACK should not be delayed. It states "If sent at all, SACK options SHOULD
>>>> be included in all ACKs which do not ACK the highest sequence number in the
>>>> data receiver's queue". It also says "If data receiver generates SACK options
>>>> under any circumstance, it SHOULD generate them under all permitted
>>>> circumstances". So, as part of meeting the RFC 2018 requirements, if the
>>>> combination of SACK and timestamp is given preference over DSS, data segments
>>>> could be transmitted without their map.

>>>> Another case of delaying map could arise if the data sender prefers to send
>>>> ADD_ADDR option, instead of map, in a data segment. It is nice that ADD_ADDR
>>>> option can be delivered reliably in a pure ACK, but I think this is not the
>>>> case with DSS at present.

>>>> If we adopt the proposed change, I think it might also be helpful to spell out
>>>> how the receiver is supposed to behave, if it gets maps not meeting the MUST
>>>> condition in the proposed change. For example termination of the subflow with
>>>> MP_TCPRST option (section 3.6 in RFC 6824-bis) with appropriate reason code and
>>>> T flag value to intimate the data sender the cause for subflow termination.
>>>> With regards,

>>>> [ https://www.ietf.org/mailman/listinfo/multipathtcp |
>>>> https://www.ietf.org/mailman/listinfo/multipathtcp ]

>>>> [ https://www.ietf.org/mailman/listinfo/multipathtcp |
>>>> https://www.ietf.org/mailman/listinfo/multipathtcp ]

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 11 Feb 2020 01:05:37 -0800

Hi folks,
I guess we might want to know the behavior of existing implementations such
as linux.
When an mptcp stack tries to send a data packet and find that there's no
enough option space for data mapping, what it will do?
Split the packet to create more option space?
--
Yoshi

On Thu, Feb 6, 2020 at 10:10 AM V Anil Kumar <[email-masked]> wrote:

> *From: *"Alan Ford" <[email-masked]>
> *To: *"V Anil Kumar" <[email-masked]>jk
> *Cc: *"Yoshifumi Nishida" <[email-masked]>, "multipathtcp" <
> [email-masked]>
> *Sent: *Thursday, February 6, 2020 2:49:29 AM
> As far as I understand, delivery of DSM through pure ACK is not in the
> scope of the current draft. If we intend to do this, we may need an
> approach similar to the one proposed for delivery of ADD_ADDR option in
> pure ACK with reliability feature.  So, packing DSM on pure ACK does not
> seem to be an option at this stage.
>
>
> I do see the point here: if a packet of 1000 bytes contains 500 bytes of
> one mapping and 500 bytes of another mapping, then only one DSM would
> appear on one packet, leaving the mapping for the second 500 bytes to be
> carried somewhere else - the only option being a pure ACK. But this kind of
> scenario would be extremely rare
>
> Yes, I do agree that the scenario you mentioned above would be extremely
> rare. In fact, I wonder whether such a situation (i.e., need to cover the
> bytes in a single packet with two different maps) would ever arise.
> Probably there might be some corner cases, which I don't get it rightly
> now.
>
> More importantly, the scenario that Yoshi and me are referring to is
> totally different: the one I had given as part of my comments in response
> to the proposed change 2. In this scenario, which you could see in the
> trailing mail, data segment-1 is transmitted without including its map,
> something which is permitted in the  mptcp framework (6824 bis). I make
> this inference from the below text in 6824 bis:
>
> "...even if a mapping does not exist from the subflow space to the data-
> level space, the data SHOULD still be ACKed at the subflow (if it is
> in-window). This data cannot, however, be acknowledged at the data level
> (Section 3.3.2) because its data sequence numbers are unknown.
> Implementations MAY hold onto such unmapped data for a short while in the
> expectation that a mapping will arrive shortly."
>
> One option the data sender has at this stage is to include the map for
> data in segment-1, in segment-3 (please see the two-subflow example I gave
> in response to the change 2 you had proposed). But the proposed change does
> not permit this. What would happen to  subflow-1 in this case ?
>
> With regards,
>
> Anil
>
> segments each with the PSH flag set. I don’t think we need to spell this
> out in the spec however.
>
> Regards,
> Alan
>
> *Sent: *Tuesday, February 4, 2020 1:08:45 PM
> *Yes.  So, segment 1 may be kept in the data receiver's buffer in
> expectation that its mapping will arrive shortly. And in the example that
> we are referring to, the data sender will not be able to include the map
> for the data in segment 1 in segment 3 or any higher segment.*
>
> *Regards,*
>
> *Anil*
>

---

From: <[email-masked]>
Date: Tue, 11 Feb 2020 15:57:03 +0000

So coming back to Anil’s (original) question.

A packet with timestamp and SACK options likely won’t have room for the DSS option, therefore the DSS mapping would be delayed. This may mean the transmitted segment is not covered by a DSS map. This would fail the MUST in Change 2 which (I think) means the subflow is closed immediately.

Some options:

  *   It’s a rare case that we don’t need to worry about.
  *   Make Change 2 a SHOULD rather than a MUST
  *   Extend Change 2 so it says something like: The mapping provided by a Data Sequence Mapping MUST apply to some or all of the subflow sequence space in the TCP segment which carries the option [+ new text] or be covered by the next segment.  Or “within the next few segments”
  *   ?

For any of these, it would be worthwhile adding an explanatory note to Change 2.

It would be really good if we could conclude on this.

Personally I’d like something that reflects the consensus implicit in the implementations, but without anything other than rare corner cases.

Thanks
phil

From: multipathtcp <[email-masked]> On Behalf Of V Anil Kumar
Sent: 06 February 2020 18:07
To: Alan Ford <[email-masked]>
Cc: multipathtcp <[email-masked]>
Subject: Re: [multipathtcp] RFC6824bis edits based on implementation feedback

Hi Alan,

________________________________
From: "Alan Ford" <[email-masked]<mailto:[email-masked]>>
To: "V Anil Kumar" <[email-masked]<mailto:[email-masked]>>jk
Cc: "Yoshifumi Nishida" <[email-masked]<mailto:[email-masked]>>, "multipathtcp" <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, February 6, 2020 2:49:29 AM
Subject: Re: [multipathtcp] RFC6824bis edits based on implementation feedback
Hi Anil,

I do see the point here: if a packet of 1000 bytes contains 500 bytes of one mapping and 500 bytes of another mapping, then only one DSM would appear on one packet, leaving the mapping for the second 500 bytes to be carried somewhere else - the only option being a pure ACK. But this kind of scenario would be extremely rare
Yes, I do agree that the scenario you mentioned above would be extremely rare. In fact, I wonder whether such a situation (i.e., need to cover the bytes in a single packet with two different maps) would ever arise. Probably there might be some corner cases, which I don't get it rightly now.

With regards,

Anil
and I would imagine any implementation would just split into two 500 byte segments each with the PSH flag set. I don’t think we need to spell this out in the spec however.

Regards,
Alan

On 5 Feb 2020, at 15:09, V Anil Kumar <[email-masked]<mailto:[email-masked]>> wrote:
Hi Yoshi,

Please see in line.

Hi Anil,

Regards,

Anil

Thanks,
--
Yoshi

With regards,

Anil

Hi Anil,

I have a question about your proposed text..
I am actually wondering if we really want to terminate connection here.

The packets without proper mappings will be treated as invalid and will be discarded.
If an implementation failed to attach proper mapping for some reasons (e..g. option space), it might be able to attach the proper one when it retransmits the packets. This also looks ok to me.

Thanks,
--
Yoshi

From: "alan ford" <[email-masked]<mailto:alan.ford@gmail..com>>

Hi all,

Change 1

To:

Change 2

To:

What this means:

As far as I understand, RFC 2018 (TCP Selective Acknowledgement Options) implies that SACK should not be delayed. It states "If sent at all, SACK options SHOULD be included in all ACKs which do not ACK the highest sequence number in the data receiver's queue".. It also says "If data receiver generates SACK options under any circumstance, it SHOULD generate them under all permitted circumstances".   So, as part of meeting the RFC 2018 requirements, if the combination of SACK and timestamp is given preference over DSS, data segments could be transmitted without their map.

If we adopt the proposed change, I think it might also be helpful to spell out how the receiver is supposed to behave, if it gets maps not meeting the MUST condition in the proposed change.  For example termination of the subflow with MP_TCPRST option (section 3..6 in RFC 6824-bis) with appropriate reason code and T flag value to intimate the data sender the cause for subflow termination.

With regards,

Anil

Best regards,
Alan

_______________________________________________
multipathtcp mailing list
[email-masked]<mailto:multipathtcp@ietf..org>
https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Christoph Paasch <[email-masked]>
Date: Tue, 11 Feb 2020 14:29:58 -0800

+MPTCP-upstreaming

> On Feb 11, 2020, at 1:05 AM, Yoshifumi Nishida <[email-masked]> wrote:
> 
> Hi folks,
> I guess we might want to know the behavior of existing implementations such as linux.
> When an mptcp stack tries to send a data packet and find that there's no enough option space for data mapping, what it will do?

iOS, multipath-tcp.org <http://multipath-tcp.org/> and Linux 5.6 implementation all favor the DSS-option over any other option (except TCP Timestamps). And TCP Timestamps and DSS comfortably fit in the TCP option-space.

If SACK-blocks should be added, we only add SACK-blocks up to the remaining space. SACK is anyways opportunistic as once there are more than 3 blocks they don't fit either.

Christoph

> On Thu, Feb 6, 2020 at 10:10 AM V Anil Kumar <[email-masked] <mailto:[email-masked]>> wrote:
> Hi Alan,
> 
> From: "Alan Ford" <[email-masked] <mailto:[email-masked]>>
> To: "V Anil Kumar" <[email-masked] <mailto:[email-masked]>>jk 
> Cc: "Yoshifumi Nishida" <[email-masked] <mailto:[email-masked]>>, "multipathtcp" <[email-masked] <mailto:[email-masked]>>
> On 5 Feb 2020, at 15:09, V Anil Kumar <[email-masked] <mailto:[email-masked]>> wrote:
> Hi Yoshi,
> 
> Please see in line.
> 
> if we assume that the map is included while retransmitting the data (even though the first transmission did not contain the map for some reasons),  we could argue that RST could be avoided provided that the retransmission is triggered within a receive window of data. But the question here would be how and when will the retransmission take place. In this case, the subflow may not initiate the retransmission of data by its own (i..e., no retransmission due to three duplicate ACKs or RTO expiry at subflow level) as there is no segment loss at subflow level sequence space. So there could be a high possibility of RST happening even before the map delivery through retransmission. 
> 
> With regards, 
> 
> Anil
>  
> In the above case, segment-1 goes without map and its map is included later in segment-3, the next data segment in the same subflow. Further,  in the above scheduling pattern, the map in segment-3 cannot cover the  data in segment-1 and segment-3, as some  data in between (segment-2) is transmitted through another subflow.  With the proposed change, the map in segment-3 will become invalid and this will eventually break subflow-1, though this could be a corner case..
>

---

From: Christoph Paasch <[email-masked]>
Date: Tue, 11 Feb 2020 14:53:29 -0800

+MPTCP-Upstreaming

> On Feb 11, 2020, at 7:57 AM, [email-masked] wrote:
> 

This packet should simply reduce the number of SACK-blocks so that the DSS-option fits in the TCP-option space. As I mentioned in my other mail, SACK is opportunistic in the sense that if there is a large number of holes in the sequence-number space today's TCP-option space limitation already limits this.

If we go along with this logic, we could argue that TCP-timestamps should be omitted in favor of even more SACK-blocks. And even then, 40 bytes might not be enough, so one might make the argument to move SACK-blocks to the TCP-payload ? (please, no! ;-) )

> Some options:
> It’s a rare case that we don’t need to worry about.

In my opinion it is rare and (AFAIK) there is no running code that implements sending or receiving early/late mappings.
Also, there is no data measuring the impact of this kind of traffic-pattern and by what percentage throughput/latency will improve when early/late mappings are allowed.

> Make Change 2 a SHOULD rather than a MUST

This means that all receivers MUST support receiving early/late mappings as they never know when a sender will follow the SHOULD or not. That means we have to clearly specify how the receiver stores these mappings, how many, how long,...

> Extend Change 2 so it says something like: The mapping provided by a Data Sequence Mapping MUST apply to some or all of the subflow sequence space in the TCP segment which carries the option [+ new text] or be covered by the next segment.  Or “within the next few segments”

This would simplify some of the problems a receiver could have with late mappings.
Nevertheless, without any data as to the prevalence of the problem and the potential benefits of the solution my strong opinion is still to not allow this for the benefit of reduced implementation complexity.

Christoph

> ?
> From: multipathtcp <[email-masked] <mailto:[email-masked]>> On Behalf Of V Anil Kumar
> To: Alan Ford <[email-masked] <mailto:[email-masked]>>
> Cc: multipathtcp <[email-masked] <mailto:[email-masked]>>
> From: "alan ford" <[email-masked] <mailto:alan.ford@gmail..com>>
> [email-masked] <mailto:multipathtcp@ietf..org>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 13 Feb 2020 01:34:29 -0800

Hi Christoph,

Thanks for the clarification. It seems that there are several ways to deal
with this situation.
I think these are not ideal, but at the same time, it won't be fatal. I
think more thorough solutions can be discussed in future versions.
Given that there're several ways to address it, I guess the draft doesn't
have to specify the behavior for this.
--
Yoshi

On Tue, Feb 11, 2020 at 2:30 PM Christoph Paasch <[email-masked]> wrote:

> iOS, multipath-tcp.org and Linux 5.6 implementation all favor the
> DSS-option over any other option (except TCP Timestamps). And TCP
> Timestamps and DSS comfortably fit in the TCP option-space.
>
> If SACK-blocks should be added, we only add SACK-blocks up to the
> remaining space. SACK is anyways opportunistic as once there are more than
> 3 blocks they don't fit either.
>
>
> Christoph
>
>> for a short while in the expectation that a mapping will arrive shortly.
>> " can be applied to the segment 1. As a result, the segment 1 won't be
>> discarded.
>>
>>> question here would be how and when will the retransmission take place. In
>>> this case, the subflow may not initiate the retransmission of data by its
>>> own (i..e., no retransmission due to three duplicate ACKs or RTO expiry at
>>> subflow level) as there is no segment loss at subflow level sequence space.
>>> So there could be a high possibility of RST happening even before the map
>>> delivery through retransmission.
>>>
>>> With regards,
>>>
>>> Anil
>>>
>>>> *   If B has data to send first, then the reliable delivery of the ACK
>>>> + MP_CAPABLE can be inferred by the receipt of this data with an MPTCP Data
>>>> *   If B has data to send first, then the reliable delivery of the ACK
>>>> + MP_CAPABLE is ensured by the receipt of this data with an MPTCP Data
>>>> sender may find it difficult, in case a need arise to slightly delay the
>>>> map delivery by few segments, i.e., sending some data first without map,
>>>> and then send the corresponding map in a later segment, as shown below:
>>>>
>>>> though this could be a corner case..
>>>>
>>>> without its map. In the case of bidirectional data transfer, there could be
>>>> a need to pack both timestamp and SACK  options in a data segment, i.e.,
>>>> Options) implies that SACK should not be delayed. It states "If sent at
>>>> all, SACK options SHOULD be included in all ACKs which do not ACK the
>>>> highest sequence number in the data receiver's queue". It also says "If
>>>> data receiver generates SACK options under any circumstance, it SHOULD
>>>> generate them under all permitted circumstances".   So, as part of meeting
>>>> the RFC 2018 requirements, if the combination of SACK and timestamp is
>>>> given preference over DSS, data segments could be transmitted without their
>>>> map.
>>>>

---

From: <[email-masked]>
Date: Tue, 18 Feb 2020 10:34:45 +0000

Alan, please go ahead and make Change 2 to the document (as well as Change 1). Please could you &/or Christoph add a small note based on the discussion we’ve had, as it’s drawn out a couple of useful clarifications.

Thanks to everyone for the discussion,
Best wishes,
Phil & Yoshi.

From: multipathtcp <[email-masked]> On Behalf Of Yoshifumi Nishida
Sent: 13 February 2020 09:34
To: Christoph Paasch <[email-masked]>
Cc: MultiPath TCP - IETF WG <[email-masked]>; mptcp Upstreaming <[email-masked]>
Subject: Re: [multipathtcp] RFC6824bis edits based on implementation feedback

Hi Christoph,

Thanks for the clarification. It seems that there are several ways to deal with this situation.
I think these are not ideal, but at the same time, it won't be fatal. I think more thorough solutions can be discussed in future versions.
Given that there're several ways to address it, I guess the draft doesn't have to specify the behavior for this.
--
Yoshi

On Tue, Feb 11, 2020 at 2:30 PM Christoph Paasch <[email-masked]<mailto:[email-masked]>> wrote:
+MPTCP-upstreaming

On Feb 11, 2020, at 1:05 AM, Yoshifumi Nishida <[email-masked]<mailto:[email-masked]>> wrote:

Hi folks,

iOS, multipath-tcp.org<http://multipath-tcp.org> and Linux 5.6 implementation all favor the DSS-option over any other option (except TCP Timestamps). And TCP Timestamps and DSS comfortably fit in the TCP option-space.

Christoph

On Thu, Feb 6, 2020 at 10:10 AM V Anil Kumar <[email-masked]<mailto:[email-masked]>> wrote:
Hi Alan,

With regards,

Anil

Regards,
Alan

Please see in line.

Hi Anil,

It seems to me that whether RST is happen or not depends on the size of receive window according to the text.
If the receive window size is big enough to accommodate segment 1 and segment 3, the text "Implementations MAY hold onto such unmapped data for a short while in the expectation that a mapping will arrive shortly.. " can be applied to the segment 1. As a result, the segment 1 won't be discarded.

Regards,

Anil

Thanks,
--
Yoshi

With regards,

Anil

Hi Anil,

Thanks,
--
Yoshi

To: multipathtcp@ietf..org<mailto:[email-masked]>

Hi all,

Change 1

To:

Change 2

To:

What this means:

With regards,

Anil

Best regards,
Alan

---

From: Alan Ford <[email-masked]>
Date: Wed, 19 Feb 2020 15:43:45 +0000

Hi Phil, all,

I have requested the following changes from the RFC Editor, as agreed:

Change 1

To:

Change 2

To:

I have been looking through the thread wondering what in particular requires clarification; most of the discussion seemed to have concluded on no additional clarifications needed, unless I misunderstood - so could you please point to what in particular you feel should be added?

Many thanks,
Alan

> On 18 Feb 2020, at 10:34, [email-masked] wrote:
> 
> On Tue, Feb 11, 2020 at 2:30 PM Christoph Paasch <[email-masked] <mailto:[email-masked]>> wrote:
> +MPTCP-upstreaming
> 
> 
> On Feb 11, 2020, at 1:05 AM, Yoshifumi Nishida <[email-masked] <mailto:[email-masked]>> wrote:
>  
> Hi folks,
> To: multipathtcp@ietf..org <mailto:[email-masked]>