Subject: [multipathtcp] Comments on DATA FIN

From: Sébastien Barré <[email-masked]>
Date: Wed, 10 Nov 2010 14:27:03 +0100

Hi,

I have several comments about section 3.4 in the draft mptcp-multiaddressed.
If you agree with them, I can propose a rewriting of section 3.4.
I think that further thinking is needed in that area:

*/"The DATA_FIN is an optimisation to rapidly indicate the end of a data 
stream and clean up state associated with a MPTCP connection, especially 
when some subflows may have failed." :/
First, I don't see in what the DATA_FIN is an optimisation. If a subflow 
has failed, it becomes required to use the DATA_FIN. One could think 
that, anyway, if there is no datafin the failed subflow will finally 
timeout. But this has a critical consequence for the app, since in case 
of timeout, we tell the app that the pipe is broken. With the data_fin, 
instead, we can tell the app that all the data has been transferred 
successfully, since the data_fin can be transmitted on a working subflow.

*"/When DATA_FIN is sent, it should be sent on all active subflows./":
I don't see any reason for this. To me, the data_fin is just a 
particular app byte (somehow, the EOF byte). As such, it can be 
retransmitted on any subflow as per the normal retransmission policy, 
just like any app byte. Hence, I would argue to schedule the data_fin on 
any subflow. Then, if lost, it can be retransmitted on another subflow. 
Of course, it sounds like more optimized to send the data_fin on all 
subflows, but since the scheduler is supposed to select the best subflow 
for any new data to be sent, I see little benefit in doing so. Moreover, 
sending the DFIN on all subflows increases the link between the "mptcp 
layer" and the "subflow layer", architecturally speaking. This in turn 
adds some complexity to the implementations.
Hence, I would replace "it should be sent", with "it may be sent on all 
active subflows".

*"/The data sequence number included in the DATA_FIN is used to verify 
that all data has been successfully received./":
I think this is not efficient. I would instead argue for something like 
"It is mandatory to attach the DSN option in a segment containing the 
DFIN option".
We have two options for sending a FIN:
->Build a pure FIN segment (with no data in it). In that case, it is 
almost equivalent to merge the DSN option and the DFIN option, or keep 
them appart.
->Append it to the last data segment: In that case the DSN option is 
present anyway, and the including the DSN as well in the DFIN option 
would duplicate the information.
Hence, the DFIN option as I see it (and implement it currently) is just 
a two byte option.

*"/If all subflows have been closed with a FIN, that is equivalent to 
having closed the connection with a DATA_FIN.":
/I don't think this restriction is useful. I guess it is there because 
it is mentioned above that the DATA_FIN is not mandatory.
But since I explain hereabove that, IMO, the DATA_FIN should be 
mandatory, we can then afford to allow a meta-connection to survive 
temporarily with no running subflow.
This could be useful in some mobility scenarios I think. (More thought 
require for this, though).

Thoughts ?

Thanks !

regards,

Sébastien.

-- 
Sébastien Barré
Researcher,
CSE department, UCLouvain, Belgium
http://inl.info.ucl.ac.be/sbarre

---

From: "Ford, Alan" <[email-masked]>
Date: Thu, 11 Nov 2010 02:28:30 -0000

Hi Sébastien, all,

First comment: yes I agree, I shall clarify the text that the DATA_FIN provides the definitive end-of-data. You are right that timed out subflows are not a reliable solution to knowing the clean end of transmission.

Second comment:  the way I had envisaged the DATA_FIN working was that, when all data had been send, the implementation would send a FIN on each subflow, with a DATA_FIN option on each of those packets. The DATA_FIN option augments the FIN to say it applies to the whole MPTCP connection. You are right that the DATA_FIN occupies the last octet of data sequence space (like the FIN occupies the last octet of subflow sequence space), and this will be reflected in the DATA_ACK in the final FIN/ACK.

Third comment: you make a good point. The DATA_FIN combined with the DSN_MAP would allow its use with a data or non-data packet. (Combined with the last point, i.e. the last octet of subflow sequence space maps to the DATA_FIN). This final octet of data sequence space would be DATA_ACKed in the last FIN/ACK. This would make the DATA_FIN a three-octet flag option only. This is probably the correct way to do it, although we should check there are no option space concerns with this.

Fourth comment: yes, I think I agree, in line with your first comment, we should make the DATA_FIN mandatory, although a connection can still time out if all subflows have been closed with FINs.

Cheers,

Alan

From: Sébastien Barré [mailto:[email-masked]] 
Sent: 10 November 2010 13:27
To: Ford, Alan; Costin Raiciu; Mark Handley; Multipath TCP Mailing List
Subject: Comments on DATA FIN

Hi,

*"The DATA_FIN is an optimisation to rapidly indicate the end of a data stream and clean up state associated with a MPTCP connection, especially when some subflows may have failed." :
First, I don't see in what the DATA_FIN is an optimisation. If a subflow has failed, it becomes required to use the DATA_FIN. One could think that, anyway, if there is no datafin the failed subflow will finally timeout. But this has a critical consequence for the app, since in case of timeout, we tell the app that the pipe is broken. With the data_fin, instead, we can tell the app that all the data has been transferred successfully, since the data_fin can be transmitted on a working subflow.

*"When DATA_FIN is sent, it should be sent on all active subflows.":
I don't see any reason for this. To me, the data_fin is just a particular app byte (somehow, the EOF byte). As such, it can be retransmitted on any subflow as per the normal retransmission policy, just like any app byte. Hence, I would argue to schedule the data_fin on any subflow. Then, if lost, it can be retransmitted on another subflow. Of course, it sounds like more optimized to send the data_fin on all subflows, but since the scheduler is supposed to select the best subflow for any new data to be sent, I see little benefit in doing so. Moreover, sending the DFIN on all subflows increases the link between the "mptcp layer" and the "subflow layer", architecturally speaking. This in turn adds some complexity to the implementations.
Hence, I would replace "it should be sent", with "it may be sent on all active subflows".

*"The data sequence number included in the DATA_FIN is used to verify that all data has been successfully received.":
I think this is not efficient. I would instead argue for something like "It is mandatory to attach the DSN option in a segment containing the DFIN option".
We have two options for sending a FIN:
->Build a pure FIN segment (with no data in it). In that case, it is almost equivalent to merge the DSN option and the DFIN option, or keep them appart.
->Append it to the last data segment: In that case the DSN option is present anyway, and the including the DSN as well in the DFIN option would duplicate the information.
Hence, the DFIN option as I see it (and implement it currently) is just a two byte option.

*"If all subflows have been closed with a FIN, that is equivalent to having closed the connection with a DATA_FIN.":
I don't think this restriction is useful. I guess it is there because it is mentioned above that the DATA_FIN is not mandatory.
But since I explain hereabove that, IMO, the DATA_FIN should be mandatory, we can then afford to allow a meta-connection to survive temporarily with no running subflow.
This could be useful in some mobility scenarios I think. (More thought require for this, though).

Thoughts ? 

Thanks !

regards,

Sébastien.

---

From: Costin Raiciu <[email-masked]>
Date: Fri, 12 Nov 2010 22:59:18 +0800

Hi Seb,

You are completely right on  point 1,2 and 4. The draft did not mirror my thinking.

I am not sure about point three.  It makes the data FIN implicit (and subject to the previous DSN received on that subflow) rather than explicit. I see no real reason for this optimization; space should not be an issue when we\re sending the FIN anyways. It might give us headaches though with middleboxes.

Thanks,
Costin

---

From: Sébastien Barré <[email-masked]>
Date: Wed, 17 Nov 2010 11:40:11 +0100

Hi Costin,

Op 12-11-10 15:59, Costin Raiciu schreef:
> I am not sure about point three.  It makes the data FIN implicit (and subject to the previous DSN received on that subflow) rather than explicit.
Why implicit ? I was still arguing for a 2 bytes long DFIN option. Looks 
more like explicit to me. Only do I not see it as useful to explicitly 
mention
the DSN of the DFIN *inside* the option itself.
>   I see no real reason for this optimization; space should not be an issue when we\re sending the FIN anyways.
Right (for the space). But more than an optimization (I consider an 
optimization as trading off overall simplicity for increased 
efficiency), I see it as almost the opposite: small increased 
efficiency, but cleaner protocol element.
> It might give us headaches though with middleboxes.
What kind of headaches ? Both DFIN and DSNs are new options. If the DSN 
does not work through middleboxes, MPTCP is not  useable anyway.

regards,

Sébastien.

---

From: Costin Raiciu <[email-masked]>
Date: Wed, 17 Nov 2010 15:17:51 +0200

Hi,

> Why implicit ? I was still arguing for a 2 bytes long DFIN option. Looks more like explicit to me. Only do I not see it as useful to explicitly mention

I meant that it makes the last DSN of the connection implicit. It should be explicit. Note that the FINs do not occupy connection sequence space, and thus there
should be no DSN on pure FINs. 

> Right (for the space). But more than an optimization (I consider an optimization as trading off overall simplicity for increased efficiency), I see it as almost the opposite: small increased efficiency, but cleaner protocol element.
> What kind of headaches ? Both DFIN and DSNs are new options. If the DSN does not work through middleboxes, MPTCP is not  useable anyway.

When you send a DSN there must be some data in the TCP payload that this corresponds to.  When you send a data FIN there may be no data to refer to. The Data FIN itself needs a sequence number, but there is really no mapping
to the tcp payload. How would you encode the Data Fin sequence number in the dsn? Note : zero-length DSNs are already used for specifying infinite mappings and thus falling back to regular TCP.

Cheers,
Costin

---

From: "Ford, Alan" <[email-masked]>
Date: Wed, 17 Nov 2010 13:54:52 -0000

Hi Costin, all,

Hang on- are we talking about the same thing here? A FIN at the subflow level will occupy the last octet of subflow-level sequence space, and this will be acknowledged in the FIN/ACK by a regular TCP ACK.

It makes sense, therefore, that his behaviour is the same at the connection-level, i.e. a DATA_FIN occupies the last octet of connection-level sequence space, and can be acknowledged by a DATA_ACK.

What Sebastien is proposing, I believe, is that we signal the DATA_FIN by a simple two-byte option, using the DSN_MAP to map this connection-level octet to the FIN octet at the subflow-level. This one octet signal can then be acknowledged by a DATA_ACK. Whether all this will fit in the options will still need to be confirmed, however, but it makes sense to keep this connection-level behaviour the same as regular TCP.

Regards,
Alan

-----Original Message-----
From: Costin Raiciu [mailto:[email-masked]]
Sent: Wed 11/17/2010 1:17 PM
To: Sébastien Barré
Cc: Ford, Alan; Mark Handley; Multipath TCP Mailing List
Subject: Re: Comments on DATA FIN
 
Hi,

Cheers,
Costin

---

From: Sébastien Barré <[email-masked]>
Date: Wed, 17 Nov 2010 15:02:48 +0100

Hi Costin,

First, I fully aggree with the last mail from Alan, we were writing 
concurrently. I however keep sending this one, given it may clarify my 
thinking (in particular, see concrete example at the end).//

Op 17-11-10 14:17, Costin Raiciu schreef:
> I meant that it makes the last DSN of the connection implicit.
Not quite: the last DSN is part of the last mapping, since the mapping 
is one byte longer in length.
> It should be explicit. Note that the FINs do not occupy connection sequence space,
They do, or I misunderstood the following bullet in the draft:

o  There is a one-to-one mapping between the DATA_FIN and the
       subflow's FIN flag (and its associated sequence space and thus its
       acknowlegement).

> and thus there
on pure FINs, indeed (although Alan was suggesting to send the DFIN on 
all FINs). But on pure FINs+DFINs, we need to map, because putting the 
DFIN option in the segment is strictly equivalent to putting one byte of 
data in it.
> When you send a DSN there must be some data in the TCP payload that this corresponds to.
yes, the FIN is that data.
>    When you send a data FIN there may be no data to refer to. The Data FIN itself needs a sequence number, but there is really no mapping
That's not what is stated in the draft, see above. BTW, that precise 
detail was one of the only things I agreed with in the current section 
3.4 :-).
> to the tcp payload. How would you encode the Data Fin sequence number in the dsn? Note : zero-length DSNs are already used for specifying infinite mappings and thus falling back to regular TCP.
Ok so we don't have the same assumptions. Since I do assume, like the 
draft, that there is a mapping, I use a 1-byte long mapping in the case 
of a pure FIN+DFIN.
Note that this way to proceed perfectly supports the following kind of 
scenario:
*The last chunk of app data is 5000 bytes long and the app supports TSO.
*Hence, the stack builds a segment with 5000 bytes of payload, schedules 
it to some subflow, and sets the FIN flag+option DSN (mapping 
length=5001)+option DFIN.
*Due to TSO, the NIC splits the big segments into 1500 bytes chunks. 
Because it knows TCP, it will put the FIN flag on the last 1500 bytes 
segment.
Because it does not know MPTCP, it copies the DSN option as is on each 
of the smaller segments.
*When receiving the first segment, the receiver sees that a new mapping 
is present and updates its local mapping info for the subflow. It also 
sees the DFIN option, but
knows from the mapping the exact DSN of the DFIN. Hence it only needs to 
remember that it can close its receive side as soon as the last byte is 
processed.

cheers,

Sébastien.