Subject: Re: [multipathtcp] MPTCP implementation feedback for RFC6824bis

From: Christoph Paasch <[email-masked]>
Date: Mon, 02 Dec 2019 09:27:57 -0800

Hello Alan,

On 29/11/19 - 21:13:38, Alan Ford wrote:
> > On 28 Nov 2019, at 19:49, Christoph Paasch <[email-masked]> wrote:
> >> On Nov 28, 2019, at 8:16 AM, Alan Ford <[email-masked] <mailto:[email-masked]>> wrote:
> >>> On 27 Nov 2019, at 19:29, Christoph Paasch <[email-masked] <mailto:[email-masked]>> wrote:
> >>> Section 3.3.1, page 32 & 33, "A data sequence mapping does not need..."
> >>> 
> >>> This paragraph states that it is permissive to send a mapping in advance. Late-mapping is specified a bit higher around the sentence 
> >>> "Implementations MAY hold onto such unmapped data for a short while in the expectation that a mapping will arrive shortly"
> >>> 
> >>> This kind of early/late mapping announcements are difficult to handle in an implementation. The Linux Kernel implementation of multipath-tcp.org <http://multipath-tcp.org/> has always disallowed such kind of mappings. Meaning, whenever a DSS-option is received such that the range specified by the relative subflow-sequence number in the DSS-option and the data-length does not (partially) cover the TCP sequence number of the packet itself, the subflow will be killed with a TCP-RST. The problem around handling such early/late mappings is that it is unclear for how long the stack needs to remember these mappings (in the early-mapping case), or for how long he needs to hold on to the data (in the late-mapping case).
> >>> 
> >>> We thus suggest to change this to the following:
> >>> Whenever a DSS-option is received on a packet such that the mapping of the subflow-sequence space does not partially cover the TCP-sequence number of  the packet itself, the host MUST discard this mapping and MAY destroy the subflow with a TCP-RST. It should be noted that a DATA_FIN that does not come with data has a relative subflow-sequence number of 0 and thus should be handled separately.
> >> 
> >> This one I do have an issue with:
> >> 
> >> - It is a technical change
> >> - Wording to this effect has been in the document since pretty much the beginning
> >> - It is a MAY which might as well say “there is no guarantee this would work”
> > 
> > The problem with the MAY is that the sender can't really know if the receiver accepts it (more regarding this below)
> > 
> >> Most importantly, the replacement text seems not to address this issue at all. If I read it correctly, it says that the data sequence mapping option MUST partially cover the subflow sequence space of the packet itself. But that has nothing to do with late or early mapping, both could partially cover the subflow sequence space and preceding data.
> >> 
> >> Can you clarify exactly what you want to permit and prevent, here?
> > 
> > Let me try to clarify what exactly we mean with early/late mapping so that we are all on the same terms here:
> > 
> > Early mapping:
> > 
> > A TCP-segment with sequence-number 1 holds a DSS-option with subflow-sequence number 1001 and data-length 100. This means we need to allocate space to store this DSS-option so that when the TCP-segment with seqno 1001 arrives we can know the mapping. There may be coming more of these DSS-options which all need to be stored in allocated memory. It is unclear what the limit to this is and there is no way to communicate this limit to the sender.
> 
> I don’t think we have ever intended to support a mapping like this. If the text is not clear here then yes, we might have an issue.

Yes, I do think that the text is not very clear on that  - we should clarify
that.

> We intend only to support: A TCP segment with sequence-number 1 holds a DSS option for SSN 1 and length 10000 (so multiple segments in the future).

Sounds good!

> Or, slightly more convoluted, a TCP segment with SN 100, length 100, which holds a DSS option for 151-250, and bytes 50-150 were already covered by a previous mapping.

Also good.

> I do not believe we have ever intended mappings on data segments where the segments do not include any of the mapped data. We did, however, intend to support mappings on pure ACKs in order to avoid any option space limits.

Mappings on a pure ACK are an unlikely use-case. The problem is that in the
end the mapping needs to reliably make it to the receiver as otherwise he
needs to throw the data away. Combining it with data implicitly makes it
reliable.

In case of option-space limits, it is better to send the options that do
consume a lot of space (ADD_ADDR,...) on the pure ACK.

> 
> > Late mapping:
> > 
> > The receiver receives data without DSS-options with TCP-sequence 1 to 1001. The corresponding DSS-option however arrives with the TCP-segment with seqno 2001. Here, the receiver needs to hold on to this data, waiting for the TCP-segment with the DSS-option. At one point the receiver needs to drop the data due to memory limits. Again, the sender has no way for knowing what this limit is.
> 
> So this is slightly more problematic, and this is what the MAY in the text is designed to discuss.
> 
> I believe we had intended to support a situation where you could have a segment 1-1000 without a DSS option, and 1001-2000 also without, and then 2001-3000 with the option then providing a mapping for 1 to 2001 or higher.

I think this scenario is fine to some extend with a slight change that the
provided mapping should be for 1 to 2002 or higher (thus, including the byte
of the 2001-3000 segment).

> This would allow a sender to start pushing out data before it knew what a mapping might look like.
> It does, however, seem an unlikely situation but you as a receiver could of course reduce the subflow window size in order to limit buffering. The text does recognise the memory issue and point out that this won’t be DATA ACKed and as such a sender should soon realise and retransmit on a separate subflow and then this subflow may eventually be closed.

Yes, it seems like an unlikerly situation and I'm not sure about the
use-case for doing this.

> Another situation would involve two mappings on the same TCP segment. If the first 50 bytes of a segment are covered by the mapping provided in that segment, but there are then also another 50 bytes, then the mapping can’t be provided until the next segment.

Yes, two mappings on a single packet are problematic simply because of the
lack of option-space.

> I guess my initial thought here is that this was intended to cover a number of corner cases but if it does not work for you then there are several compliant ways of dealing with it.
> 
> > When the DSS-option comes together with the corresponding TCP-sequence it is straight-forward to store it together with the data. There are no issues with memory-allocation,... as all of this is accounted together with the announced window (yes, the memory is not counted against the window, but the receiver can foresee the DSS-option overhead when computing what window he should announce).
> > 
> > When a receiver gets data without a DSS-option, he can store it for up to 64KB of data as that is the maximum data-level length and the last segment of the 64KB-train could be holding the DSS-option. After that he has to drop the data.
> > 
> > When the mapping partially covers the segment it also isn't a problem as the unmapped part can safely be dropped and the mapped part can be passed on to the MPTCP-layer.
> > 
> > All of this does not imply that every segment of a mapping needs to hold the DSS-option. Just one of them needs to have it.
> 
> That last statement aligns with what was intended in the text. But are you really saying that, or are you saying that the first one needs to have it? Because that would be a change.

No, the first one does not need to have it. Just one of them.

Cheers,
Christoph

---

From: Alan Ford <[email-masked]>
Date: Wed, 4 Dec 2019 21:50:52 +0000

Hi Christoph,

Thank you for the clarifications. I was revisiting the text to see ways to make these clarifications, however I find myself unsure of the need; please see comments below:

Section 3.1 clarification

Towards the end of Section 3.1 we actually say the following:

   The SYN with MP_CAPABLE occupies the first octet of data sequence space, although this does not need to be acknowledged at the connection level until the first data is sent (see Section 3.3).

Which would seem to cover exactly this concern. However if you still feel further clarification is required, then we could add this also to the sentence you suggest, i.e.:

   If B has data to send first, then the reliable delivery of the ACK + MP_CAPABLE can be inferred by the receipt of this data with an MPTCP Data Sequence Signal (DSS) option (Section 3.3). 

Can change to:

   If B has data to send first, then the reliable delivery of the ACK + MP_CAPABLE can be inferred by the receipt of this data with an MPTCP Data Sequence Signal (DSS) option (Section 3.3) containing a DATA_ACK for the MP_CAPABLE (which is the first octet of the data sequence space). Furthermore, when A receives a DATA_ACK from B it is a signal of the reliable delivery of A's MP_CAPABLE.

Please confirm if you still feel this is necessary, given the quote I provide.

Early Mapping

   A Data Sequence Mapping does not need to be included in every MPTCP
   packet, as long as the subflow sequence space in that packet is
   covered by a mapping known at the receiver.  This can be used to
   reduce overhead in cases where the mapping is known in advance.  One
   such case is when there is a single subflow between the hosts, and
   another is when segments of data are scheduled in larger-than-packet-
   sized chunks.

I would suggest simply adding a sentence at the beginning saying “A Data Sequence Mapping MUST appear on a TCP segment which is covered by the mapping”.

Regarding late mapping, we say:

   Implementations MAY hold onto such unmapped data for a
   short while, in the expectation that a mapping will arrive shortly.
   Such unmapped data cannot be counted as being within the connection-
   level receive window because this is relative to the data sequence
   numbers, so if the receiver runs out of memory to hold this data, it
   will have to be discarded.  If a mapping for that subflow-level
   sequence space does not arrive within a receive window of data, that
   subflow SHOULD be treated as broken, closed with a RST, and any
   unmapped data silently discarded.

Note the last sentence. We already bound this suggestion to just a subflow receive window of data, and provide a mechanism to reject (RST and silently discard). If we add the above text re early mapping that would also apply in this case and provide your requirement that the mapping lands on a segment covered by the mapping.

Any thoughts?

Best regards,
Alan

> On 2 Dec 2019, at 17:27, Christoph Paasch <[email-masked]> wrote:
> 
> Hello Alan,
>

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 04 Dec 2019 16:33:29 -0800

Hello Alan,

> On Dec 4, 2019, at 1:50 PM, Alan Ford <[email-masked]> wrote:
> 
> Hi Christoph,
> 
> Which would seem to cover exactly this concern.

I'm not sure how it covers the concern, because the issue I raise is that the client needs to somehow find out if the MP_CAPABLE made it to the receiver. And the clarification I suggest is to spell out that it is the DATA_ACK (see my next comment).

> However if you still feel further clarification is required, then we could add this also to the sentence you suggest, i.e.:
> 

The ambiguity here is that DSS does not imply DATA_ACK. One could send a DSS without a DATA_ACK, but that would not signal to the client the reliable delivery of the MP_CAPABLE. Only the DATA_ACK will do so.

> Please confirm if you still feel this is necessary, given the quote I provide.

Yes, I still feel that this is necessary.

> I would suggest simply adding a sentence at the beginning saying “A Data Sequence Mapping MUST appear on a TCP segment which is covered by the mapping”.

Yes, adding that sentence would clarify this.

I also would remove the sentence "This can be used to..." and "One such case is..." - it adds to the confusion IMO.

> Note the last sentence. We already bound this suggestion to just a subflow receive window of data, and provide a mechanism to reject (RST and silently discard). If we add the above text re early ma…

Yes, this last sentence, combined with the above-mentioned addition should make things fine.

Thanks,
Christoph

> 
> Any thoughts?
> 
> Best regards,
> Alan
> 
>> On 2 Dec 2019, at 17:27, Christoph Paasch <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hello Alan,
>> 
>>>> On 28 Nov 2019, at 19:49, Christoph Paasch <[email-masked] <mailto:[email-masked]>> wrote:
>>>>> On Nov 28, 2019, at 8:16 AM, Alan Ford <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>>>>>> On 27 Nov 2019, at 19:29, Christoph Paasch <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>>>>>> This kind of early/late mapping announcements are difficult to handle in an implementation. The Linux Kernel implementation of multipath-tcp.org <http://multipath-tcp.org/> <http://multipath-tcp.org/ <http://multipath-tcp.org/>> has always disallowed such kind of mappings. Meaning, whenever a DSS-option is received such that the range specified by the relative subflow-sequence number in the DSS-option and the data-length does not (partially) cover the TCP sequence number of the packet itself, the subflow will be killed with a TCP-RST. The problem around handling such early/late mappings is that it is unclear for how long the stack needs to remember these mappings (in the early-mapping case), or for how long he needs to hold on to the data (in the late-mapping case).
>>>>>>

---

From: Alan Ford <[email-masked]>
Date: Thu, 5 Dec 2019 15:03:21 +0000

Hi Christoph,

> On 5 Dec 2019, at 00:33, Christoph Paasch <[email-masked]> wrote:
> 
> Hello Alan,
> 
>> On Dec 4, 2019, at 1:50 PM, Alan Ford <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hi Christoph,
>> 

OK well maybe I am not understanding your concern here. The first sentence I quote says that the MP_CAPABLE needs to be DATA_ACKed when the first data is sent.

The sentence beginning “If B has Data to send first” means:

- A->B SYN
- B->A SYN/ACK + MP_CAPABLE
- A->B ACK + MP_CAPABLE
- B->A ACK + DATA + DSS

DSS can contain one or both of DSM and DATA_ACK. However the previously quoted sentence says that the MP_CAPABLE needs to be acknowledged at the connection-level (i.e. DATA_ACKed) when the first data is sent.

However in writing this I realise that that text makes no comment about who is sending data. Because the intention is it applies to any data being sent (if B is sending, then it will include a DATA_ACK as well as a DSS; if A is sending then B will acknowledge with a DATA_ACK), but it could be read by someone to mean “only when acknowledging the first data does the MP_CAPABLE need to be included in the DATA_ACK”, which is not the desired meaning.

I am all for improving clarity so I think a clearer version of “If B has data…” would go:

    If B has data to send first, then the reliable delivery of the ACK + MP_CAPABLE is ensured by the receipt of this data with an MPTCP Data Sequence Signal (DSS) option (Section 3.3) containing a DATA_ACK for the MP_CAPABLE (which is the first octet of the data sequence space).

Would you be ok with that? It is shorter than the other edits whilst still being unambiguous.

> I also would remove the sentence "This can be used to..." and "One such case is..." - it adds to the confusion IMO.

I quite like to retain examples where possible; these seem harmless when combined with the clarification from the MUST, it ensures people behave!

Best regards,
Alan

---

From: Christoph Paasch <[email-masked]>
Date: Thu, 05 Dec 2019 09:39:46 -0800

Hello Alan,

> On Dec 5, 2019, at 7:03 AM, Alan Ford <[email-masked]> wrote:
>> On 5 Dec 2019, at 00:33, Christoph Paasch <[email-masked] <mailto:[email-masked]>> wrote:

Actually, strictly reading, the first sentence says that it must not be acknowledged until first data sent. It does not say that a MP_CAPABLE MUST be DATA_ACKed.

> Would you be ok with that? It is shorter than the other edits whilst still being unambiguous.

This however does not say that the reception fo a DATA_ACK allows A to stop sending the MP_CAPABLE. I still think that it should be spelled out somewhere.

> I quite like to retain examples where possible; these seem harmless when combined with the clarification from the MUST, it ensures people behave!

Ok, fair enough!

Thanks,
Christoph

---

From: Alan Ford <[email-masked]>
Date: Thu, 5 Dec 2019 23:30:04 +0000

> On 5 Dec 2019, at 17:39, Christoph Paasch <[email-masked]> wrote:
> 
> Hello Alan,
> 
>> On Dec 5, 2019, at 7:03 AM, Alan Ford <[email-masked] <mailto:[email-masked]>> wrote:

I really don’t read it like that - It makes no comment about when NOT to do something, simply that it needs to be acknowledged when data is sent. We could clarify it by changing it to:

   The SYN with MP_CAPABLE occupies the first octet of data sequence space, and this MUST be acknowledged at the connection level at the time the first data is sent or received (see Section 3.3).

> This however does not say that the reception fo a DATA_ACK allows A to stop sending the MP_CAPABLE. I still think that it should be spelled out somewhere.

Remember this is the case where B is sending data first. There would not be retransmitted MP_CAPABLEs from A in this case, since A has no data to send. The acknowledgement ball is entirely in B’s court, and thus if A received a data packet from B without a DSS option it would consider the connection non-MPTCP-capable.

Regards,
Alan

---

From: Alan Ford <[email-masked]>
Date: Fri, 6 Dec 2019 15:58:17 +0000

Hi all,

Following on from the discussion of implementation feedback with Christoph, I propose the following edits to RFC6824bis - which is currently in AUTH48 - as clarifications.

ADs, please can you confirm you consider these edits sufficiently editorial to fit into AUTH48.

WG participants, please speak up if you have any concerns.

Edit 1, clarifying reliability of MP_CAPABLE

Change the sentence reading:

To:

   The SYN with MP_CAPABLE occupies the first octet of data sequence space, and this MUST be acknowledged at the connection level at or before the time the first data is sent or received (see Section 3.3).

Change the sentence reading:

To:

In my personal opinion either one of these edits would be sufficient for making the point, however clearly this has caused some confusion amongst the implementor community so making both these changes should make it absolutely clear as to the expected behaviour here.

Edit 2, mapping constraint

Change the sentence reading:

   A Data Sequence Mapping does not need to be included in every MPTCP packet, as long as the subflow sequence space in that packet is covered by a mapping known at the receiver.

To:

   A Data Sequence Mapping MUST appear on a TCP segment which is covered by the mapping. It does not need to be included in every MPTCP packet, as long as the subflow sequence space in that packet is covered by a mapping known at the receiver.

Best regards,
Alan

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 9 Dec 2019 21:30:08 -0800

Hi Alan,

The texts look fine to me, but I have a few questions on them.

On Fri, Dec 6, 2019 at 7:58 AM Alan Ford <[email-masked]> wrote:

> Hi all,
>
> Following on from the discussion of implementation feedback with
> Christoph, I propose the following edits to RFC6824bis - which is currently
> in AUTH48 - as clarifications.
>
> *Edit 1, clarifying reliability of MP_CAPABLE*
>
>    The SYN with MP_CAPABLE occupies the first octet of data sequence
> space, although this does not need to be acknowledged at the connection
> level until the first data is sent (see Section 3.3).
>
> To:
>
>    The SYN with MP_CAPABLE occupies the first octet of data sequence
> space, and this MUST be acknowledged at the connection level at or before
> the time the first data is sent or received (see Section 3.3).
>

What implementations should do when they receive the first data before
MP_CAPABLE is acked?
They should terminate the connection or discard the data?

Change the sentence reading:
>
>    If B has data to send first, then the reliable delivery of the ACK +
> MP_CAPABLE can be inferred by the receipt of this data with an MPTCP Data
> Sequence Signal (DSS) option (Section 3.3).
>
> To:
>
>    If B has data to send first, then the reliable delivery of the ACK +
> MP_CAPABLE is ensured by the receipt of this data with an MPTCP Data
> Sequence Signal (DSS) option (Section 3.3) containing a DATA_ACK for the
> MP_CAPABLE (which is the first octet of the data sequence space).
>
>
> In my personal opinion either one of these edits would be sufficient for
> making the point, however clearly this has caused some confusion amongst
> the implementor community so making both these changes should make it
> absolutely clear as to the expected behaviour here.
>
>
> *Edit 2, mapping constraint*
>
>    A Data Sequence Mapping does not need to be included in every
> MPTCP packet, as long as the subflow sequence space in that packet
> is covered by a mapping known at the receiver.
>
> To:
>
>    A Data Sequence Mapping MUST appear on a TCP segment which is covered
> by the mapping. It does not need to be included in every MPTCP packet, as
> long as the subflow sequence space in that packet is covered by a mapping
> known at the receiver.
>
>
What implementations should do when a Data Sequence Mapping doesn't cover
the TCP segment that carries this option?
BTW, This is not a strong opinion, but I may prefer a text like: "A Data
Sequence Mapping MUST provide the mapping for the segment that carries this
option."
--
Yoshi

---

From: Alan Ford <[email-masked]>
Date: Tue, 10 Dec 2019 16:09:04 +0000

Hi Yoshifumi,

> On 10 Dec 2019, at 05:30, Yoshifumi Nishida <[email-masked]> wrote:
> 
> Hi Alan,
> 
> On Fri, Dec 6, 2019 at 7:58 AM Alan Ford <[email-masked] <mailto:[email-masked]>> wrote:
> Hi all,
> 
> What implementations should do when they receive the first data before MP_CAPABLE is acked?

By asking this question you have made me realise that this text is in fact incompatible with the case when A (the initiator) is also the first sender of data.

>    If B has data to send first, then the reliable delivery of the ACK + MP_CAPABLE is ensured by the receipt of this data with an MPTCP Data Sequence Signal (DSS) option (Section 3.3) containing a…

This will resolve the ambiguity in the case of B sending first.

> What implementations should do when a Data Sequence Mapping doesn't cover the TCP segment that carries this option?

There are a number of cases where the MUST does not have a consequence; it should be obvious from the text for similar failures that it can close it with a RST.

> BTW, This is not a strong opinion, but I may prefer a text like: "A Data Sequence Mapping MUST provide the mapping for the segment that carries this option.” 

OK how about: "A Data Sequence Mapping MUST provide the mapping which includes the segment that carries this option.” 

Regards,
Alan

---

From: V Anil Kumar <[email-masked]>
Date: Wed, 11 Dec 2019 01:34:12 +0530

Hi Alan,

Please see inline.

>  A Data Sequence Mapping MUST appear on a TCP segment which is covered by the mapping. It does not need to be included in every MPTCP packet, as long as the subflow sequence space in that packet is…

As far as I understand, the proposed change introduces a “MUST” to insist that the map in a segment must cover at least some data in the segment. But the document does not talk anything about the rational behind it. I guess it is purely an 
ease of implementation?

I think the design/format of the Data Sequence Mapping permits the map to stand independent of the data being carried in a segment. So, as long as an implementation is willing to deal with the complexity of storing and processing late and early mappings (with respect to the data arrival), it could be permitted provided that the received map is for an in-window data.

Anil

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 10 Dec 2019 23:55:15 -0800

Hi Alan,

On Tue, Dec 10, 2019 at 8:09 AM Alan Ford <[email-masked]> wrote:

> By asking this question you have made me realise that this text is in fact
> incompatible with the case when A (the initiator) is also the first sender
> of data.
>
> Given the problem is only with B sending data first, let us forget this
> change, and revert to Christoph’s original problem text, and use only the
> below change:
>

Thanks. this makes sense to me. But, I am wondering if we want to make the
use of DATA_ACK mandatory in this case.
How about something like: "if B has data to send first, then it MUST use an
MPTCP Data Sequence Signal (DSS) option (Section 3.3) containing a DATA_ACK
for the MP_CAPABLE (which is the first octet of the data sequence space) to
provide the reliable delivery of the ACK + MP_CAPABLE."?

> There are a number of cases where the MUST does not have a consequence; it
> should be obvious from the text for similar failures that it can close it
> with a RST.
>
> option.”
> OK how about: "A Data Sequence Mapping MUST provide the mapping which
> includes the segment that carries this option.”
>

Works for me. Thanks!
--
Yoshi

---

From: V Anil Kumar <[email-masked]>
Date: Wed, 11 Dec 2019 14:06:37 +0530

Hi Alan,

Please see inline.

>  A Data Sequence Mapping MUST appear on a TCP segment which is covered by the mapping. It does not need to be included in every MPTCP packet, as long as the subflow sequence space in that packet is…
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px}span.s1 {font-variant-ligatures: no-common-ligatures}
As far as I understand, the proposed change introduces a “MUST” to insist that the map in a segment must cover at least some data in the segment. But the document does not talk anything about the rational behind it. I guess it is purely an ease of implementation? 

Anil    

-- 

V Anil Kumar
Senior Principal Scientist
CSIR Fourth Paradigm Institute (CSIR-4PI)
NAL Belur Campus
Bangalore - 560037
E-mail: [email-masked]
Phone: +91 80 2505 1355

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 11 Dec 2019 11:22:31 -0800

Hello,

> On Dec 10, 2019, at 12:04 PM, V Anil Kumar <[email-masked]> wrote:
> 
> Hi Alan,
> 
> Please see inline.
> 
> On 12/06/19 09:28 PM, Alan Ford <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hi all,
>> 
>> 

For two reasons:

1. Ease of implementation
2. If an implementation tries to "remember" early mappings, it is not clear how many of these an implementation can hold. Thus, the sender does not know how many early mappings he can send. So, it is hard for a sender to do the right thing.

> I think the design/format of the Data Sequence Mapping permits the map to stand independent of the data being carried in a segment. So, as long as an implementation is willing to deal with the comp…

What is the concrete use-case for such early mappings? What are the benefits of it? I think that if we want to enable such implementation-complexity, we need a compelling use-case with a big benefit.

That's the reason why we (the MPTCP-upstreaming community) vouch to have this case restricted.

Cheers,
Christoph

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 11 Dec 2019 11:28:53 -0800

Hello Alan,

there is one more thing that came up from the implementation experience (thanks to Matthieu & Paolo in CC).

It is unclear in the draft (or at least, I didn't find the text ;-) ), what to do when the flags & version in the third ACK + MP_CAPABLE are inconsistent with what was negotiated in the SYN-SYN/ACK exchange.

It would be good to spell this out and say that if there are any inconsistencies, a host should simply fallback to regular TCP.

Christoph

> On Dec 10, 2019, at 8:09 AM, Alan Ford <[email-masked]> wrote:
> 
> Hi Yoshifumi,
> 
>> On 10 Dec 2019, at 05:30, Yoshifumi Nishida <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hi Alan,
>>

---

From: Alan Ford <[email-masked]>
Date: Thu, 12 Dec 2019 21:19:21 +0000

Hi Christoph (and Matthieu & Pablo),

WRT checksums, we do say:

   For example,
   if the initiator sets A=0 in the SYN but the responder sets A=1 in
   the SYN/ACK, checksums MUST be used in both directions, and the
   initiator will set A=1 in the ACK.  

   If A=1 is received by a host that does not
   want to use checksums, it MUST fall back to regular TCP by ignoring
   the MP_CAPABLE option as if it was invalid.

Which would seem to cover all these eventualities. The “C” bit is purely an indicator, and the crypto negotiation is clearly specified too.

Best regards,
Alan

> On 11 Dec 2019, at 19:28, Christoph Paasch <[email-masked]> wrote:
> 
> Hello Alan,
> 
>> On Dec 10, 2019, at 8:09 AM, Alan Ford <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hi Yoshifumi,
>>

---

From: Matthieu Baerts <[email-masked]>
Date: Fri, 13 Dec 2019 01:10:22 +0100

Hi Alan,

On Thu, Dec 12, 2019 at 10:19 PM Alan Ford <[email-masked]> wrote:

> Which would seem to cover all these eventualities. The “C” bit is purely
> an indicator, and the crypto negotiation is clearly specified too.
>

Thank you for your answer, that's clearer for the A bit.
Our concern was mainly about the version. How should we react if, as a
server supporting only v1, we have this sequence:

SYN (MP_CAPABLE: v2)  →
←  SYN+ACK (MP_CAPABLE: v1)
ACK (MP_CAPABLE v2) →

In other words, the server told the client that the maximum version it
supports is v1. But in the ACK+MP_CAPABLE, the client sends v2 again.

According to the RFC, the negotiation is done in the SYN and SYN+ACK. How
do we react if the following ACK is sending a version (e.g. v2) which is
not the expected one (e.g. v1)?

For the moment in our implementation, we fallback to regular TCP.
It means that future versions of MPTCP have to set the proper version in
the 3rd ACK -- the negotiated one -- and not a copy of what has been sent
in the SYN.

Should we add a clarification for that?

Best regards,
Matthieu

> It is unclear in the draft (or at least, I didn't find the text ;-) ),
> what to do when the flags & version in the third ACK + MP_CAPABLE are
> inconsistent with what was negotiated in the SYN-SYN/ACK exchange.
>
> It would be good to spell this out and say that if there are any
> inconsistencies, a host should simply fallback to regular TCP.
>
>
> Christoph
>

-- 
 <http://www.tessares.net> Matthieu Baerts | R&D
Engineer
[email-masked]
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

Disclaimer: https://www.tessares.net/mail-disclaimer/

---

From: V Anil Kumar <[email-masked]>
Date: Fri, 13 Dec 2019 11:22:50 +0530

Hi Christoph,

Thanks for your reply. Please see inline.

> What is the concrete use-case for such early mappings? What are the benefits of it? I think that if we want to enable such implementation-complexity, we need a compelling use-case with a big benefi…

Consider a case where a MPTCP connection consists of two subflows, and the data segments are scheduled for transmission in the order shown below below.  

Subflow-1: segment-1 segment-3 segment-5 segment-6 

 bytes:1-100 bytes:201-300 bytes:401-500 bytes:501-600 

 no map map for 1-100 map for 401-600 no map 

 

Subflow-2: segment-2 segment-4 segment-7 segment-8 

 bytes: 101-200 bytes:301-400 bytes: 601-700 bytes:701-800  

 map for 101-200 map for 301-400 map for 601-800 no map 

In the above case, the map for data in segment-1 is included in segment-3. Further, segment-3 cannot combine/cover the data in segment-1 and segment-3 in a "single map", as the data sequence space is not continuous, i.e., some in between data (segment-2) is mapped and transmitted through subflow-2. Here, the map in segment-3 does not even partially cover the data it carries. 

Both RFC 6824 and the 6824-bis do not restrict the above scenario, and I guess the change proposed now does not permit this to happen. 

Best regards, 

Anil

---

From: Alan Ford <[email-masked]>
Date: Fri, 13 Dec 2019 12:38:26 +0000

Hi Matthieu,

> On 13 Dec 2019, at 00:10, Matthieu Baerts <[email-masked]> wrote:
> 
> Hi Alan,
> 
> 
> On Thu, Dec 12, 2019 at 10:19 PM Alan Ford <[email-masked] <mailto:[email-masked]>> wrote:
> Our concern was mainly about the version. How should we react if, as a server supporting only v1, we have this sequence:
> 
> SYN (MP_CAPABLE: v2)  →
> In other words, the server told the client that the maximum version it supports is v1. But in the ACK+MP_CAPABLE, the client sends v2 again.
> 
> According to the RFC, the negotiation is done in the SYN and SYN+ACK. How do we react if the following ACK is sending a version (e.g. v2) which is not the expected one (e.g. v1)?
> 
> It means that future versions of MPTCP have to set the proper version in the 3rd ACK -- the negotiated one -- and not a copy of what has been sent in the SYN.
> 

I see your point but I don’t feel we need any clarification here. The exchange as you rightly point out is done on the SYN/SYN+ACK. So when the server sends the SYN+ACK with v1, that is the decision that has been made. Sending an ACK with v2 is not a valid response. So if the server receives this, it treats it as any other invalid option and ignores it, so it treats it as if the ACK does not have an MP_CAPABLE, and thus it will fall back to regular TCP.

Regards,
Alan

---

From: Matthieu Baerts <[email-masked]>
Date: Fri, 13 Dec 2019 14:10:41 +0100

Hi Alan,

Thank you for your reply.

On Fri, Dec 13, 2019 at 1:38 PM Alan Ford <[email-masked]> wrote:

> I see your point but I don’t feel we need any clarification here. The
> exchange as you rightly point out is done on the SYN/SYN+ACK. So when the
> server sends the SYN+ACK with v1, that is the decision that has been made.
> Sending an ACK with v2 is not a valid response. So if the server receives
> this, it treats it as any other invalid option and ignores it, so it treats
> it as if the ACK does not have an MP_CAPABLE, and thus it will fall back to
> regular TCP.
>

Even if at some point, we thought about not checking the version in the 3rd
ACK because it was a bit unclear for us, we now do fallback to regular TCP
if the version in the 3rd ACK is not the expected one.
So it seems that at the end, we correctly interpreted the RFC there. If you
feel that no clarification is needed for that, that's fine for us!

Best regards,
Matthieu

---

From: Christoph Paasch <[email-masked]>
Date: Fri, 13 Dec 2019 08:29:25 -0800

Hello,

> On Dec 12, 2019, at 9:53 PM, V Anil Kumar <[email-masked]> wrote:
> 
> Hi Christoph,
> 
>>>> On 12/06/19 09:28 PM,Alan Ford<[email-masked]> wrote:
>>>> 
>>>> Hi all,
>>>> 
>>>> 
> In the above case, the map for data in segment-1 is included in segment-3.

The question here is why would the stack not put the mapping for segment 1 on segment 1 itself. And what is the benefit of doing so?

Christoph

> Further, segment-3 cannot combine/cover the data in segment-1 and segment-3 in a "single map", as the data sequence space is not continuous, i.e., some in between data (segment-2) is mapped and transmitted through subflow-2. Here,  the map in segment-3 does not even partially cover the data it carries.
>

---

From: V Anil Kumar <[email-masked]>
Date: Fri, 13 Dec 2019 23:41:25 +0530

Hi Christoph,

Thanks again for your reply. My response is given inline.

On 12/13/19 09:59 PM, Christoph Paasch  <[email-masked]> wrote: 
> > > > On Dec 10, 2019, at 12:04 PM, V Anil Kumar <[email-masked](javascript:main.compose()> wrote:
> > > > 
> > > > 
> > > > Hi Alan, 
> > > > 
> > > > 
> > > > 
> > > > Please see inline.
> > > > 
> > > > On 12/06/19 09:28 PM,Alan Ford<[email-masked](javascript:main.compose()> wrote: 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > Hi all,
> > > > > 
> > > > > 
> > > > > 
> > > > > 

Well it could just be due to the lack of option space insegment-1. For example, the sender ofsegment-1 at that instant wants to transmit multiple TCP options (e.g.timestamp, SACK, DSS and ADD_ADDR). Obviously, they all cannot fit into optionfield of one segment, and eventually the DSS transmission got slightly delayedby a segment or two.

With regards,

Anil

---

From: Christoph Paasch <[email-masked]>
Date: Fri, 13 Dec 2019 10:24:09 -0800

On 13/12/19 - 23:41:25, V Anil Kumar wrote:
> > > > > On Dec 10, 2019, at 12:04 PM, V Anil Kumar
> > > > > <[email-masked](javascript:main.compose()> wrote:
> > > > > 
> > > > > 
> > > > > Hi Alan, 
> > > > > 
> > > > > 
> > > > > 
> > > > > Please see inline.
> > > > > 
> > > > > On 12/06/19 09:28 PM,Alan
> > > > > Ford<[email-masked](javascript:main.compose()> wrote: 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > Hi all,
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > Christoph, I propose the following edits to RFC6824bis - which
> > > > > > is currently in AUTH48 - as clarifications.
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > ADs, please can you confirm you consider these edits
> > > > > > sufficiently editorial to fit into AUTH48.
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > >  The SYN with MP_CAPABLE occupies the first octet of data
> > > > > >  sequence space, although this does not need to be acknowledged
> > > > > >  at the connection level until the first data is sent (see
> > > > > >  Section 3.3).
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > To:
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > >  The SYN with MP_CAPABLE occupies the first octet of data
> > > > > >  sequence space, and this MUST be acknowledged at the connection
> > > > > >  level at or before the time the first data is sent or received
> > > > > >  (see Section 3.3).
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > >  If B has data to send first, then the reliable delivery of the
> > > > > >  ACK + MP_CAPABLE can be inferred by the receipt of this data
> > > > > >  with an MPTCP Data Sequence Signal (DSS) option (Section 3.3). 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > To:
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > >  If B has data to send first, then the reliable delivery of the
> > > > > >  ACK + MP_CAPABLE is ensured by the receipt of this data with an
> > > > > >  MPTCP Data Sequence Signal (DSS) option (Section 3.3)
> > > > > >  containing a DATA_ACK for the MP_CAPABLE (which is the first
> > > > > >  octet of the data sequence space).
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > In my personal opinion either one of these edits would be
> > > > > > sufficient for making the point, however clearly this has caused
> > > > > > some confusion amongst the implementor community so making both
> > > > > > these changes should make it absolutely clear as to the expected
> > > > > > behaviour here.
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > >  MPTCP packet, as long as the subflow sequence space in that
> > > > > >  packet is covered by a mapping known at the receiver.
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > To:
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > >  A Data Sequence Mapping MUST appear on a TCP segment which is
> > > > > >  covered by the mapping. It does not need to be included in
> > > > > >  every MPTCP packet, as long as the subflow sequence space in
> > > > > >  that packet is covered by a mapping known at the receiver.
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > >  
> > > > > > 
> > > > > 
> > > > > As far as I understand, the proposed change introduces a “MUST” to
> > > > > insist that the map in a segment must cover at least some data in
> > > > > the segment. But the document does not talk anything about the
> > > > > rational behind it. I guess it is purely an 
> > > > > 
> > > > > ease of implementation?
> > > > > 
> > > > > 
> > > > 
> > > > 
> > > > For two reasons:
> > > > 
> > > > 1. Ease of implementation 2. If an implementation tries to
> > > > "remember" early mappings, it is not clear how many of these an
> > > > implementation can hold. Thus, the sender does not know how many
> > > > early mappings he can send. So, it is hard for a sender to do the
> > > > right thing.
> > > > 
> > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > I think the design/format of the Data Sequence Mapping permits the
> > > > > map to stand independent of the data being carried in a segment.
> > > > > So, as long as an implementation is willing to deal with the
> > > > > complexity of storing and processing late and early mappings (with
> > > > > respect to the data arrival), it could be permitted provided that
> > > > > the received map is for an in-window data.
> > > > > 
> > > > > 
> > > > 
> > > > 
> > > > What is the concrete use-case for such early mappings? What are the
> > > > benefits of it? I think that if we want to enable such
> > > > implementation-complexity, we need a compelling use-case with a big
> > > > benefit.
> > > > 
> > > > 
> > > >  
> > > > 
> > > 
> > > Consider a case where a MPTCP connection consists of two subflows, and
> > > the data segments are scheduled for transmission in the order shown
> > > below below.  
> > > 
> > > 
> > >  
> > > 
> > > 
> > > 
> > The question here is why would the stack not put the mapping for segment
> > 1 on segment 1 itself. And what is the benefit of doing so?
> > 
> > 
> > 
> >  
> > 
> 
> Well it could just be due to the lack of option space insegment-1. For
> example, the sender ofsegment-1 at that instant wants to transmit multiple
> TCP options (e.g.timestamp, SACK, DSS and ADD_ADDR). Obviously, they all
> cannot fit into optionfield of one segment, and eventually the DSS
> transmission got slightly delayedby a segment or two.

The implementation needs to enforce a strict priority of DSS over SACK and ADD_ADDR.

If the ADD_ADDR does not fit in the TCP-option space, it can send the
ADD_ADDR on a pure ACK. The echo-bit in the ADD_ADDR guarantee the reliable
delivery of it.

Sure, one could argue that favoring SACK over DSS is more important. But I
think we would need data to justify that. Only very specific traffic
patterns will fall in this use-case.

Christoph

---

From: V Anil Kumar <[email-masked]>
Date: Sat, 14 Dec 2019 19:09:54 +0530

Hi Christoph,

> The implementation needs to enforce a strict priority of DSS over SACK and ADD_ADDR.

TCP options have evolved over a period of time, and I do not think as such any document/guidelines exist on enforcing priority for one over the other, though it turns out be an interesting topic. Also, more TCP options could come up in future for implementing new features. So, it is likely that implementations would follow different strategy when it come to option priority. 

> ADD_ADDR on a pure ACK. The echo-bit in the ADD_ADDR guarantee the reliable

I noticed this new feature in RFC 6824-bis to deliver ADD_ADDR in pure ACK and still achieve the reliability. The ability to deliver MPTCP options in pure ACK will be useful, especially for options like ADD_ADDR. 

> patterns will fall in this use-case.

The bottomline is that in the event of a map being slightly delayed, i.e., delivered late with respect to the corresponding data, should it result in resetting the subflow? 

As far as the specification is concerned, it could be liberal on accepting such maps, rather than being restrictive. Even if a current implementation cannot support this, future implementations may like to, provided the specification permits this and the implementation is willing to cop up with the associated complexity. 

Best regards,

Anil

---

From: Joe Touch <[email-masked]>
Date: Sat, 14 Dec 2019 09:18:22 -0800

> On Dec 14, 2019, at 5:39 AM, V Anil Kumar <[email-masked]> wrote:
> 

FYI, this has come up as an issue before.

TCP options are individual and accepted in isolation. They are easily confirmed only during SYN exchange; anything after that requires a stateful, reliable transmission mechanism for coordination.

There is no way to retrofit any ordering constraints, or prioritization. If you need that sort of feature, it needs to be built inside a single option as “sub options”.

Joe

---

From: Yoshifumi Nishida <[email-masked]>
Date: Sun, 15 Dec 2019 03:05:25 -0800

Hi,
One concern I have here is this may mean we end up want all implementations
to support late mapping feature.
Because if an implementation that supports the feature uses it to send
data, the receiver who doesn't support it will discard the packets. To
avoid it, we need to mandate receivers to support it.
--
Yoshi

On Sat, Dec 14, 2019 at 5:40 AM V Anil Kumar <[email-masked]> wrote:

> Hi Christoph,
>
> On 12/13/19 11:54 PM, *Christoph Paasch * <[email-masked]> wrote:
>
> TCP options have evolved over a period of time, and I do not think as such
> any document/guidelines exist on  enforcing priority for one over the
> other, though it turns out be an interesting topic.  Also, more TCP options
> could come up in future for implementing  new features.  So, it is likely
> that implementations would follow different strategy when it come to option
> priority.
>
>
>
> I noticed this new feature in RFC 6824-bis to deliver ADD_ADDR in pure ACK
> and still achieve the reliability. The ability to deliver MPTCP options in
> pure ACK will be useful, especially for options like ADD_ADDR.
>
>
>
> The bottomline is that in the event of a map being slightly delayed, i.e.,
> delivered late with respect to the corresponding data, should  it result in
> resetting the subflow?
>
> As far as the specification is concerned, it could be liberal on accepting
> such maps, rather than being restrictive. Even if a current implementation
> cannot support this, future implementations may like to, provided the
> specification permits this and the implementation is willing to cop up with
> the associated complexity.
>
> Best regards,
>
> Anil
>
>
>
> Christoph
>
> >
> >
> >
> > With regards,
> >
> >
> >
> > Anil
> >
> >
> >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Christoph
> > >
> > >
> > > >
> > > >
> > > >
> > > > Further, segment-3 cannot combine/cover the data in segment-1 and
> segment-3 in a "single map", as the data sequence space is not continuous,
> i.e., some in between data (segment-2) is mapped and transmitted through
> subflow-2. Here, the map in segment-3 does not even partially cover the
> data it carries.
> > > >
> > > >
> > > >
> > > >
> > > > Both RFC 6824 and the 6824-bis do not restrict the above scenario,
> and I guess the change proposed now does not permit this to happen.
> > > >
> > > >
> > > >
> > > >
> > > > Best regards,
> > > >
> > > >
> > > >
> > > >
> > > > Anil
> > > >
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp
>

---

From: V Anil Kumar <[email-masked]>
Date: Sun, 15 Dec 2019 19:50:27 +0530

Hi Joe,

> There is no way to retrofit any ordering constraints, or prioritization. If you need that sort of feature, it needs to be built inside a single option as “sub options”.

Thank you for sharing this information.
I agree with this. 
Anil

---

From: Christoph Paasch <[email-masked]>
Date: Sun, 15 Dec 2019 18:05:31 +0100

Hello,

> On Dec 15, 2019, at 12:05 PM, Yoshifumi Nishida <[email-masked]> wrote:
> 
> Hi,
> One concern I have here is this may mean we end up want all implementations to support late mapping feature.
> Because if an implementation that supports the feature uses it to send data, the receiver who doesn't support it will discard the packets. To avoid it, we need to mandate receivers to support it.

that is indeed the problem. Once we allow the sender to do late/early mappings, a receiver has to implement it to be interoperable with all implementations.

More inline:

> On Sat, Dec 14, 2019 at 5:40 AM V Anil Kumar <[email-masked] <mailto:[email-masked]>> wrote:
> Hi Christoph,
> 
> On 12/13/19 11:54 PM, Christoph Paasch <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> 
[...]
> TCP options have evolved over a period of time, and I do not think as such any document/guidelines exist on  enforcing priority for one over the other, though it turns out be an interesting topic.…

This is purely an implementation choice. I don't think it is necessary to document this.
For example, when TCP-MD5 is used, Linux disables the TCP Timestamp option, thus enforcing priority of TCP-MD5 over Timestamps.

To re-iterate my earlier point: Unless there is real data about the performance benefits,... of enabling early/late mapping, my vote is to not allow these kind of mappings as they complexify implementations.

Christoph

> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/multipathtcp <https://www.ietf.org/mailman/listinfo/multipathtcp>

---

From: V Anil Kumar <[email-masked]>
Date: Wed, 18 Dec 2019 10:37:36 +0530

Hi,

> To re-iterate my earlier point: Unless there is real data about the performance benefits,... of enabling early/late mapping, my vote is to not allow these kind of mappings as they complexify implem…

When MPTCP is used for data transfer in both directions, the need to carry both timestamp and SACK options in a data segment may not be a rare case, as data loss or out-of-order reception can happen in both directions. On such a data segment, once timestamp and SACK options are included, packing DSS with map is not feasible. Should the data segment in such case include SACK or DSS? How does the Linux implementation of MPTCP deal with such a case?

Anil

---

From: Christoph Paasch <[email-masked]>
Date: Fri, 20 Dec 2019 09:42:35 +0100

> On Dec 18, 2019, at 6:07 AM, V Anil Kumar <[email-masked]> wrote:
> 
> Hi,
> 
> On 12/15/19 10:35 PM, Christoph Paasch <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hello,
>> 
>> 
>>> On Dec 15, 2019, at 12:05 PM, Yoshifumi Nishida <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>> Hi,
It should include a DSS and fill up the remainder of the option-space with SACK-blocks. That is also what Linux does.

Christoph