Subject: [multipathtcp] Preparing for Prague meeting - things to delete or add to MPTCP protocol bis

From: <[email-masked]>
Date: Mon, 6 Jul 2015 18:01:21 +0000

Hi,

In the Prague meeting, we aim to move forward the protocol bis doc (to meeting our Cahrter objetive for a standards track version of RFC6824)

One topic we'd like to discuss is if there's anything that should be deleted from or added to the current draft, <https://tools.ietf.org/wg/mptcp/draft-ietf-mptcp-rfc6824bis/> https://tools.ietf.org/wg/mptcp/draft-ietf-mptcp-rfc6824bis/ - based on operational and implementation experience.

Any proposals?

Looking at the (expired) draft summarising the various implementations,

https://tools.ietf.org/html/draft-eardley-mptcp-implementations-survey-02

one thing that may be worth discussing is whether we need both 4 & 8 byte Data ack & DSS

<<   All implementations support 4 bytes "Data ACK" and "Data sequence
   number" fields, and will interoperate with an implementation sending
   8 bytes.  Implementation 1 uses only 4 bytes fields; if an
   implementation sends an 8 byte data sequence number it replies with a
   4 byte data ack.

I don't think the implementation survey suggests that there's anything else worth considering deleting - are there any other proposals?

As to things to add, there are a few proposals for some (minor?) additions in the recent set of drafts from Olivier Bonaventure and team.

Since we have a lot more requests for presentation time than it's possible to squeeze into the meeting, it would be excellent to have discussion on the list before the meeting, about anything that should be added to, or deleted from, the current rfc6824bis, before we progress it to WG last call.

thanks!

---

From: Christoph Paasch <[email-masked]>
Date: Mon, 06 Jul 2015 23:30:55 -0700

Hello,

On 06/07/15 - 18:01:21, [email-masked] wrote:
> In the Prague meeting, we aim to move forward the protocol bis doc (to
> meeting our Cahrter objetive for a standards track version of RFC6824)
> 
> 
> 
> One topic we'd like to discuss is if there's anything that should be
> deleted from or added to the current draft,
> <https://tools.ietf.org/wg/mptcp/draft-ietf-mptcp-rfc6824bis/>
> https://tools.ietf.org/wg/mptcp/draft-ietf-mptcp-rfc6824bis/ - based on
> operational and implementation experience.
> 
> 
> 
> Any proposals?

Might we consider to include something similar to draft-paasch-mptcp-syncookies-00
in RFC6824bis, to better handle MPTCP on stateless webservers?

We received offline-comments from Olivier and Alan on the draft and still
have to update it. However, neither I nor Anumita won't be able to attend the
IETF in Prague.

draft-paasch-mptcp-syncookies-00 however also depends on whether we intend
to bump the MPTCP-version number up to 1 for RFC6824bis.

> number" fields, and will interoperate with an implementation sending 8
> bytes.  Implementation 1 uses only 4 bytes fields; if an implementation
> sends an 8 byte data sequence number it replies with a 4 byte data ack.
> 
> >>

I think that both, 8 bytes Data-ACK and DSS are necessary, because it may
(in theory) be possible that a segment on one subflow gets delayed while the
other one advances the sequence numbers by 2^32 bytes. In that case this old
segment will be considered as valid. 64-bit sequence numbers will protect
against this.

> I don't think the implementation survey suggests that there's anything
> else worth considering deleting - are there any other proposals?
> 
> 
> 
> As to things to add, there are a few proposals for some (minor?) additions
> in the recent set of drafts from Olivier Bonaventure and team.

+1 on draft-bonaventure-mptcp-exp-option-00, as it allows to experiment to
figure out which other signalling mechanisms would be useful to control the
scheduling across the subflows or policies for subflow creation.

Cheers,
Christoph

---

From: Rao Shoaib <[email-masked]>
Date: Tue, 07 Jul 2015 00:12:13 -0700

While implementing MPTCP for Solaris I struggled with how the keys and 
tokens were created and used. It is not clear what needs to be in 
network byte order. I got interoperability with Linux by simply trying 
the combinations. It might be good to add some specifics unless the 
community feels that it is just my lack of understanding of cryptography.

Regards,

Shoaib

On 07/06/2015 11:01 AM, [email-masked] wrote:
> https://tools.ietf.org/wg/mptcp/draft-ietf-mptcp-rfc6824bis/ - based 
> on operational and implementation experience.
>
> Any proposals?
>
> As to things to add, there are a few proposals for some (minor?) 
> additions in the recent set of drafts from Olivier Bonaventure and team.
>
> Since we have a lot more requests for presentation time than it's 
> possible to squeeze into the meeting, it would be excellent to have 
> discussion on the list before the meeting, about anything that should 
> be added to, or deleted from, the current rfc6824bis, before we 
> progress it to WG last call.
>
> thanks!
>
>
>
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Rao Shoaib <[email-masked]>
Date: Thu, 09 Jul 2015 09:19:27 -0700

> one thing that may be worth discussing is whether we need both 4 & 8 
+1
Choosing one (8 byte ack) would make implementation as well as debugging 
easier. What is the purpose of providing two options.

Shoaib

---

From: Christoph Paasch <[email-masked]>
Date: Thu, 09 Jul 2015 09:45:37 -0700

Hello,

> easier. What is the purpose of providing two options.

I think the 4-byte option is good to reduce the TCP-option space
consumption of MPTCP.

Cheers,
Christoph

---

From: Rao Shoaib <[email-masked]>
Date: Thu, 09 Jul 2015 09:59:48 -0700

> I think the 4-byte option is good to reduce the TCP-option space
That's a good point. I suggested 8 byte option to cover any wrap around 
issues but if 4 bytes work that is fine too.

Shoaib

---

From: <[email-masked]>
Date: Fri, 10 Jul 2015 06:22:44 +0000

Hi Philip,

A change was proposed in (https://tools.ietf.org/html/draft-boucadair-mptcp-symmetric-02) to redefine IPVer field of ADD_ADDR as a set of flag bits (see below). This change applies also for ADD_ADDR2.

OLD:
        1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +---------------+---------------+--------+--------+---------------+
       |     Kind      |     Length    |ADD_ADDR| IPVer  |  Address ID   |
       +---------------+---------------+--------+--------+---------------+
       |          Address (IPv4 - 4 octets / IPv6 - 16 octets)           |
       +-------------------------------+---------------------------------+
       |   Port (2 octets)             |
       +-------------------------------+

NEW:
       1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +---------------+---------------+--------+--------+---------------+
      |     Kind      |     Length    |ADD_ADDR| Flags  |  Address ID   |
      +---------------+---------------+--------+--------+---------------+
      |          Address (IPv4 - 4 octets / IPv6 - 16 octets)           |
      +-------------------------------+---------------------------------+
      |   Port (2 octets)             |
      +-------------------------------+

                                        +-+-+-+-+
      Flags is a set of 4 flags:        |C|r|r|r|
                                        +-+-+-+-+

      C flag MUST be set to 1 when the address/port are checked.
      "rrr" are for future assignment as additional flag bits.
      r bits MUST each be sent as zero and MUST be ignored on receipt.

                                 Figure 1

Another proposal from the symmetric draft is to define one of these flag bits as C-flag to indicate to a remote peer that it can initiate subflows even if its didn't initiated the first subflow. The problem was mentioned in [I-D.ietf-mptcp-experience]:

      From a subflow viewpoint, the Multipath TCP protocol is completely

      symmetrical.  Both the clients and the server have the capability

      to create subflows.  However in practice the existing Multipath

      TCP implementations [I-D.eardley-mptcp-implementations-survey<https://tools.ietf.org/html/draft-boucadair-mptcp-symmetric-02#ref-I-D.eardley-mptcp-implementations-survey>]

      have opted for a strategy where only the client creates new

      subflows.  The main motivation for this strategy is that often the

      client resides behind a NAT or a firewall, preventing passive

      subflow openings on the client.

Cheers,
Med

De : multipathtcp [mailto:[email-masked]] De la part de [email-masked]
Envoyé : lundi 6 juillet 2015 20:01
À : [email-masked]
Objet : [multipathtcp] Preparing for Prague meeting - things to delete or add to MPTCP protocol bis

Hi,

One topic we'd like to discuss is if there's anything that should be deleted from or added to the current draft, https://tools.ietf.org/wg/mptcp/draft-ietf-mptcp-rfc6824bis/ - based on operational and implementation experience.

Any proposals?

thanks!

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 10 Jul 2015 16:30:23 +0200

Med,

>
> A change was proposed in
> (https://tools.ietf.org/html/draft-boucadair-mptcp-symmetric-02) to
> redefine IPVer field of ADD_ADDR as a set of flag bits (see below). This
> change applies also for ADD_ADDR2.

I think that replacing the IP version with a set of flags (at least in 
ADD_ADDR2 to prevent backward compatibility problems with the existing 
ADD_ADDR implementation) would make sense.

>        r bits MUST each be sent as zero and MUST be ignored on receipt.

However, I'm stil not convinced by the usefulness of using the C bit 
that you propose. If a host knows that it cannot receive a SYN over a 
given address, then the best solution is to not advertise this address. 
This will typically be the case when a host is behind a NAT or a 
firewall (I guess that at some point middlebox vendors will include 
software to parse ADD_ADDR and remove the ADD_ADDR that correspond to 
addresses that are not reachable accoring to their policies)

Olivier

---

From: <[email-masked]>
Date: Wed, 15 Jul 2015 09:05:30 +0000

Hi Olivier, 

Please see inline. 

Cheers,
Med

> -----Message d'origine-----
> De : Olivier Bonaventure [mailto:[email-masked]]
> Envoyé : vendredi 10 juillet 2015 16:30
> À : BOUCADAIR Mohamed IMT/OLN; [email-masked];
> [email-masked]
> Objet : Re: [multipathtcp] Preparing for Prague meeting - things to delete
> or add to MPTCP protocol bis
> 
> Med,
> 
> >
> > A change was proposed in

[Med] Ok.

> given address, then the best solution is to not advertise this address.

[Med] Agree. An implementation that supports the "C" flag is likely to advertise addresses/ports for which pinholes have been created in NATs/FWs and address/ports that can be used to receive incoming packets successfully. The issue with the implicit approach (i.e., no C-flag) is at the server side: should the server interpret the advertisement of an address/port as an indication that packets can be sent without experiencing reachability issues or not? The experience has shown that only the endpoint who initiated the first subflow can create new ones even if addresses/ports have been announced to the remote peer.  

The explicit approach (c-flag) explores an alternate approach that is worth to be investigated, IMHO.

> addresses that are not reachable accoring to their policies)

[Med] We can also consider some middleboxes (read CPEs) that will set the C-flag on behalf of an endpoints for a better quality of experience.

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 15 Jul 2015 16:48:40 +0200

Med,

> [Med] Agree. An implementation that supports the "C" flag is likely to advertise addresses/ports for which pinholes have been created in NATs/FWs and address/ports that can be used to receive incoming packets successfully.

> The issue with the implicit approach (i.e., no C-flag) is at the server side: should the server interpret the advertisement of an address/port as an indication that packets can be sent without experiencing reachability issues or not?

This is what RFC6824 assumes today. ADD_ADDR is only a hint about the 
possibility of trying to reach this address. Anyway it will be a SYN and 
the host will wait for a SYN+ACK to confirm the reachability

> The experience has shown that only the endpoint who initiated the first subflow can create new ones even if addresses/ports have been announced to the remote peer.

AFAIK, there has been no experience on letting the server create 
subflows on the Internet and see what happens. The existing 
implementation have assumed that this would be difficult and they have 
not implemented the code to allow the server to create subflows, but 
this could be done. It will likelly work in private networks or datacenters

> The explicit approach (c-flag) explores an alternate approach that is worth to be investigated, IMHO.

We should probably first try to create subflows from the server. We 
might be able to install a test server to analyse this on
multipath-tcp.org after the holidays. This could give use data on the
possibility of creating subflows from the server and the problems that 
such a feature could cause.
> [Med] We can also consider some middleboxes (read CPEs) that will set the C-flag on behalf of an endpoints for a better quality of experience.

I'm not sure that expecting middleboxes to change the content of the 
ADD_ADDR is a good approach in the long term. We've seen various 
problems with middleboxes that change the content of packets

Olivier

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 15 Jul 2015 08:44:13 -0700

Hello,

On 15/07/15 - 16:48:40, Olivier Bonaventure wrote:
> possibility of trying to reach this address. Anyway it will be a SYN and the
> host will wait for a SYN+ACK to confirm the reachability

I think, concerning the reachability, what is rather interesting is to
advertise the secondary address without wanting the peer to even try
establishing a new subflow to this address. This advertisement is done in
case the "primary" subflow becomes unreachable. If it becomes unreachable,
then the peer can use the advertised address to create a new subflow.

It's a little bit like the "handover"-mode we did some time ago in the Cellnet,
but the other way around (peer creates the subflows not the client).

Cheers,
Christoph

> AFAIK, there has been no experience on letting the server create subflows on
> the Internet and see what happens. The existing implementation have assumed
> that this would be difficult and they have not implemented the code to allow
> the server to create subflows, but this could be done. It will likelly work
> in private networks or datacenters
> 
> >
> We should probably first try to create subflows from the server. We might be
> able to install a test server to analyse this on
> possibility of creating subflows from the server and the problems that such
> a feature could cause.
> >
> ADD_ADDR is a good approach in the long term. We've seen various problems
> with middleboxes that change the content of packets
> 
> 
> Olivier
> 
>

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 15 Jul 2015 17:49:24 +0200

Christoph,
> then the peer can use the advertised address to create a new subflow.

You would then propose a kind of "Backup" flag in the ADD_ADDR option 
meaning that this address should only be used to create a subflow if the 
primary subflow fails ?

Olivier

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 15 Jul 2015 09:08:18 -0700

> meaning that this address should only be used to create a subflow if the

Yes, something along those lines.
When all other addresses are failing, then this "backup"-address can be used
for a new subflow.

Cheers,
Christoph

---

From: <[email-masked]>
Date: Thu, 16 Jul 2015 05:42:36 +0000

Hi Olivier, 

Please see inline.

Cheers,
Med

> Envoyé : mercredi 15 juillet 2015 16:49
> > [Med] Agree. An implementation that supports the "C" flag is likely to
> advertise addresses/ports for which pinholes have been created in NATs/FWs
> and address/ports that can be used to receive incoming packets
> successfully.
> 
> 
> > The issue with the implicit approach (i.e., no C-flag) is at the server
> side: should the server interpret the advertisement of an address/port as
> an indication that packets can be sent without experiencing reachability
> issues or not?
> 

[Med] when multiple alternate addresses are available, having the c-flag will help the remote side decide which address to pick rather than try-and-see approach that may not be optimal. c-flag can be considered an additional hint.

> 
> > The experience has shown that only the endpoint who initiated the first
> subflow can create new ones even if addresses/ports have been announced to
> the remote peer.
> 
> > [Med] We can also consider some middleboxes (read CPEs) that will set
> the C-flag on behalf of an endpoints for a better quality of experience.
>

---

From: <[email-masked]>
Date: Thu, 16 Jul 2015 05:54:53 +0000

Hi Christoph,

For this to work, ADD_ADDR should convey the external IP address/port not the internal ones.  

FWIW, the steps to be followed are listed in draft-boucadair-mptcp-symmetric:

===
   o  Each endpoint proceeds to the discovery of upstream flow-aware
      devices (e.g., NAT, Firewall).

      This can be achieved by various means, e.g., using UPnP IGD
      [IGD1][IGD2], PCP server discovery [RFC6887], PCP DHCP option
      [RFC7291], DS-Lite AFTR [RFC6334], etc.

      A NAT/firewall can be embedded in a CPE (Customer Premises
      Equipment) and/or hosted in the network provider's side.

   o  Appropriate mappings are instantiated in those discovered flow-
      aware devices.  In particular, external IP address(es) and port
      numbers are retrieved.

      This can be achieved using PCP [RFC6887].  Note, mappings created
      by PCP MAP requests are, by definition, endpoint-independent
      mappings (EIMs) with endpoint-independent filtering (EIF).
      Filters can be associated with the PCP MAP request to restrict a
      mapping to be bound to specific remote peer(s).

      PCP allows to dynamically control both NATs and firewalls.
      Furthermore, PCP allows to retrieve the lifetime associated with
      an external IP address and external port number.

      If the host is a UPnP IGD Control Point, [RFC6970] allows to relay
      UPnP IGD primitives into PCP messages.  PCP can also be used to
      control multi-layered NATs/firewall owing to the activation of
      [I-D.ietf-pcp-proxy] in intermediate NATs/firewalls.

   o  When initiating an MPTCP connection, external IP addresses and
      port numbers are communicated to the remote peer.
=====

Cheers,
Med

> De : Christoph Paasch [mailto:[email-masked]]
> Envoyé : mercredi 15 juillet 2015 18:08
> À : Olivier Bonaventure
> Cc : BOUCADAIR Mohamed IMT/OLN; [email-masked];
> [email-masked]
> > >>>[Med] Agree. An implementation that supports the "C" flag is likely
> to advertise addresses/ports for which pinholes have been created in
> NATs/FWs and address/ports that can be used to receive incoming packets
> successfully.
> > >>
> > >>
> > >>>The issue with the implicit approach (i.e., no C-flag) is at the
> server side: should the server interpret the advertisement of an
> address/port as an indication that packets can be sent without
> experiencing reachability issues or not?
> > >>