Subject: Re: [Banana] Problem Statement: Bandwidth Aggregation for Internet Access (was RE: I-D Action: draft-boucadair-mptcp-plain-mode-00.txt)

From: "Sri Gundavelli (sgundave)" <[email-masked]>
Date: Tue, 20 Oct 2015 06:51:15 +0000

> Start from the problem statement, we can design the protocols.

But, I do not see any new technical requirements in that spec. This is
stating the obvious. Unfortunately, our starting point in all the
discussions that I¹ve seen so far "appears" to be that we need to create a
new protocol and a new working group. That¹s how I¹m reading it. But,
looking at the technical consideration, I don¹t see any thing new.  The
aspect of link aggregation for realizing a larger logical pipe, discussion
around per-packet/per-flow/weighted per packet distribution or policy
rules around path selection on a flow basic ..are all the topics that IETF
had spent significant amount of time. Starting with the recent MPTCP, or
the MIP work with RFC7629, RFC6088, RFC6089 Š have all investigated this
problem. 

What we are discussing is a system/solution requirement which is for the
SDO¹s to define and state a protocol requirement, or gaps that IETF needs
to address. BBF is not willing to make a request to IETF to work on any
problem and so I wonder what are we solving ? I¹m not objecting to this
approach, but I don¹t believe we will produce any thing useful to the
community. It will be  just re-stating the existing work as a new
requirement, and if it goes forward, we will invent a new protocol which
will exactly look some other existing protocol, but won¹t be producing any
thing valuable to the industry by any measure.

Sri

On 10/19/15, 11:15 PM, "Banana on behalf of Mingui Zhang"
<[email-masked] on behalf of [email-masked]> wrote:

>Hi All,
>
>I think the discussion on-going here leads to the following problem
>statement draft which we just submitted. Start from the problem
>statement, we can design the protocols.
>
>Problem Statement: Bandwidth Aggregation for Internet Access
>
>Abstract
>
>   Bandwidth aggregation for Internet access can increase the access
>   bandwidth and reliability. It has become a desirable network
>   function, especially for those Service Providers who operates
>   multiple heterogeneous networks.
>
>   This document describes the issues with bandwidth aggregation for
>   Internet access. The required network functions to be supported are
>   specified. Several candidate solutions had been considered to support
>   bandwidth aggregation for Internet access in IETF. These techniques
>   are investigated.
>
>https://tools.ietf.org/html/draft-zhang-banana-problem-statement-00
>
>Questions? Comments?
>
>Mingui
>
>> -----Original Message-----
>> From: Mingui Zhang
>> Sent: Monday, October 19, 2015 5:38 PM
>> To: [email-masked]'; [email-masked]; BOUCADAIR
>> Mohamed IMT/OLN; [email-masked]
>> Cc: JACQUENET Christian IMT/OLN
>> Subject: RE: I-D Action: draft-boucadair-mptcp-plain-mode-00.txt
>> 
>> Hi Pierrick,
>> 
>> The classification of the BBF work and IETF work is exact and helpful.
>>I think we
>> are having a promising common view.
>> 
>> Thanks,
>> Mingui
>> 
>> > -----Original Message-----
>> > From: Banana [mailto:[email-masked]] On Behalf Of
>> > [email-masked]
>> > Sent: Monday, October 19, 2015 4:54 PM
>> > To: [email-masked]; Mingui Zhang; BOUCADAIR Mohamed
>> IMT/OLN;
>> > [email-masked]
>> > Cc: JACQUENET Christian IMT/OLN
>> > Subject: Re: [Banana] I-D Action:
>> > draft-boucadair-mptcp-plain-mode-00.txt
>> >
>> > Hi,
>> >
>> > No doubt that agreement on building blocks is a key step for a common
>> > system design... I couldn't agree more... But, AFAIK, this is
>> > currently discussed in BBF; why not starting from there (i.e. leave
>> > system design to BBF and focus on protocols)?
>> >
>> > BR,
>> > Pierrick
>> >
>> > > -----Message d'origine-----
>> > > De : Banana [mailto:[email-masked]] De la part de
>> > > [email-masked] Envoyé : lundi 19 octobre 2015 09:02 À :
>> > > [email-masked]; BOUCADAIR Mohamed IMT/OLN;
>> > [email-masked] Cc
>> > > : JACQUENET Christian IMT/OLN Objet : Re: [Banana] I-D Action:
>> > > draft-boucadair-mptcp-plain-mode-00.txt
>> > >
>> > > Hi,
>> > >
>> > > > [Mingui] It sounds more and more interesting. That logic would be
>> > > > an essential part for the bonding service arch. I think it's
>> > > > worthwhile to agree on a common model where all those necessary
>> > > > parts (load distribution,
>> > > reordering, congestion control, etc) could be integrated into one
>>system.
>> > > I agree. As far as I recall during the Bar BoF in Prague we
>> > > discussed that it would be a good approach to identify those
>>"building
>> blocks"
>> > > which are common to the existing solutions. Having such a common
>> > > model/architecture will probably help to identify the necessary
>>parts.
>> > >
>> > >
>> > >   Best regards
>> > >
>> > >      Nic
>> > >
>> > >
>> > > > -----Original Message-----
>> > > > From: Banana [mailto:[email-masked]] On Behalf Of
>> > > > [email-masked]
>> > > > Sent: Friday, October 16, 2015 2:44 PM
>> > > > To: Mingui Zhang; [email-masked]
>> > > > Cc: JACQUENET Christian IMT/OLN
>> > > > Subject: Re: [Banana] I-D Action:
>> > > > draft-boucadair-mptcp-plain-mode-00.txt
>> > > >
>> > > > Hi Mingui,
>> > > >
>> > > > Please see inline.
>> > > >
>> > > > Cheers,
>> > > > Med
>> > > >
>> > > > > -----Message d'origine-----
>> > > > > De : Banana [mailto:[email-masked]] De la part de
>> > > > > Mingui Zhang Envoyé : mercredi 14 octobre 2015 10:12 À :
>> > > > > BOUCADAIR Mohamed IMT/OLN; [email-masked] Cc : JACQUENET
>> > > > > Christian
>> > IMT/OLN Objet : Re:
>> > > > > [Banana] I-D Action: draft-boucadair-mptcp-plain-mode-00.txt
>> > > > >
>> > > > > Hi Med,
>> > > > >
>> > > > > Please see my responses inline below.
>> > > > >
>> > > > > Thanks,
>> > > > > Mingui
>> > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: [email-masked]
>> > > > > > [mailto:[email-masked]]
>> > > > > > Sent: Tuesday, October 13, 2015 11:13 PM
>> > > > > > To: Mingui Zhang; [email-masked]
>> > > > > > Cc: JACQUENET Christian IMT/OLN
>> > > > > > Subject: RE: I-D Action:
>> > > > > > draft-boucadair-mptcp-plain-mode-00.txt
>> > > > > >
>> > > > > > Hi Mingui,
>> > > > > >
>> > > > > > Thank you for sharing the comments.
>> > > > > >
>> > > > > > Please see inline.
>> > > > > >
>> > > > > > Cheers,
>> > > > > > Med
>> > > > > >
>> > > > > > > -----Message d'origine-----
>> > > > > > > De : Mingui Zhang [mailto:[email-masked]] Envoyé :
>> > > > > > > jeudi
>> > > > > > > 24 septembre 2015 09:32 À : BOUCADAIR Mohamed IMT/OLN;
>> > > > > > [email-masked] Cc
>> > > > > > > : JACQUENET Christian IMT/OLN Objet : RE: I-D Action:
>> > > > > > > draft-boucadair-mptcp-plain-mode-00.txt
>> > > > > > >
>> > > > > > > Hi Med,
>> > > > > > >
>> > > > > > > Thanks for the sharing.
>> > > > > > >
>> > > > > > > The draft details the addressing issue which is definitely
>> > > > > > > an important issue that falls in the problem space of the
>> > > > > > > bonding service. In particular, the solution for
>> > > > > > > NAT-pass-through issue is described. It assumes the
>> > > > > > > tunneling techniques cannot support NAT-pass-through
>>between the
>> CPE and the Concentrator.
>> > > > > > > This document addresses that issue. I guess this is the
>> > > > > > > major motivation for
>> > > > > proposing the
>> > > > > > "plain" mode.
>> > > > > >
>> > > > > > [Med] Yes. Other goals are get rid of the encapsulation
>> > > > > > overhead and
>> > > > > reuse the
>> > > > > > same solution for UDP-based traffic. The plain mode allows to
>> > > > > > easily
>> > > > > reuse
>> > > > > > service functions (e.g., DPI) that are deployed at the network
>> > > > > > side
>> > > > > without
>> > > > > > requiring any update to those to support the encap scheme.
>> > > > >
>> > > > > [Mingui] I guess this kind of reuse is feasible given that the
>> > > > > end systems are changed a little bit: in
>> > > > > draft-boucadair-mptcp-dhc-01, it reads
>> > > > >
>> > > > >    "This document focuses on the explicit deployment
>> > > > >    scheme where the identity of the MPTCP Concentrator(s) is
>> explicitly
>> > > > >    configured on connected hosts."
>> > > > >
>> > > >
>> > > > [Med] Yes.
>> > > >
>> > > > > >
>> > > > > > Here, I would
>> > > > > > > like to add another way to solve the same issue: the tunnel
>> > > > > > > encapsulated packet can be wrapped with a UDP header, which
>> > > > > > > forms a tunnel-over-UDP. By doing that, NAT-pass-through can
>> > > > > > > be addressed as
>> > > > > well.
>> > > > > >
>> > > > > > [Med] That's another option, indeed. Beside the encapsulation
>> > > > > > overhead,
>> > > > > there
>> > > > > > may be some side effects if intermediate functions are needed
>> > > > > > to be
>> > > > > invoked
>> > > > > > (e.g., TCP acceleration engine).
>> > > > >
>> > > > > [Mingui] Yes, there would be some gaps. If an intermediate
>> > > > > function becomes clearly required, it need be integrated into
>> > > > > the overlay
>> > model.
>> > > > >
>> > > > > The bonding service is sometimes regarded as a pure
>> > > > > load-balance-among- multipaths use case so it's natural to have
>> > > > > the idea of reusing MPTCP which is designed to handle this use
>>case.
>> > > > > However, I doubt the traditional load-balance-among-multipaths
>> > > > > use case does not fully match the bonding service. MPTCP has to
>>be
>> adapted.
>> > > > >
>> > > > > Customers of the bonding service would discriminate the
>> > > > > heterogeneous connections. For example, the alternative
>> > > > > connection would not be used unless the primary connection is
>> > > > > saturated; some application types need be recognized by the
>> > > > > bonding devices and be transported only on a primary connection;
>> > > > > load generated by home users usually contains sparse
>> > > > > applications types, therefore per-packet load distribution is
>> > > > > required to achieve a fine-grained load distribution on the
>> > > > > multiple
>> > > > connections.
>> > > > > There is no flow to be managed.
>> > > > >
>> > > > > I agree the flow & congestion control functions developed in
>> > > > > MPTCP could be reused in the bonding service architecture. The
>> > > > > question is whether it's feasible to integrated those necessary
>> > > > > functions _into_ the
>> > > > tunnel?
>> > > > > We've already seen that the Point-to-Point Tunneling Protocol
>> > > > > (PPTP) [RFC2637] has integrated quite some of them, right?
>> > > >
>> > > > [Med] I hear you. Several strategies to optimize the use of
>> > > > multiple paths can be considered in addition to the ones that were
>> > > > considered so far in
>> > > MPTCP.
>> > > > This is exactly the reason why I do think there is some value in
>> > > > separating the load distribution logic from the actual transport
>> > > > mechanism. This is particularly valid for the UDP case.
>> > > >
>> > > > >
>> > > > > >
>> > > > > > >
>> > > > > > > Traffic distribution issue is another important issue for
>> > > > > > > the bonding service. This had ever been discussed on the
>> > > > > > > list. I know MPTCP can well support flow based traffic
>>distribution.
>> > > > > > > However, I remember it was mentioned somewhere that MPTCP
>> > > > > > > can be adapted to support per-packet load sharing among the
>> multipaths.
>> > > > > > > Will this direction be
>> > > > > > considered?
>> > > > > >
>> > > > > > [Med] The per-packet distribution is not excluded by this
>>draft.
>> > > > > > We do
>> > > > > think that
>> > > > > > the load distribution logic can even be separated from the
>> > > > > > actual
>> > > > > mechanism
>> > > > > > used to transport data over multiple paths, but I must admit
>> > > > > > this is out
>> > > > > of scope
>> > > > > > of this draft.
>> > > > >
>> > > > > [Mingui] Thanks for the clarification. For home users, the
>> > > > > per-packet load distribution is desired so I hope it would be
>> > > > > covered in some other drafts.
>> > > >
>> > > > [Med] I do see interest too to have those aspects discussed in an
>>I-D.
>> > > >
>> > > > >
>> > > > >
>> > > > > >
>> > > > > > >
>> > > > > > > A clarification questions: from Figure 1, it is clear that
>> > > > > > > one peer of the MPTCP proxy will be deployed on the CPE.
>> > > > > > > What is not clear to me is where the other end is to be
>> > > > > > > deployed? Will the
>> > > > > > > L4 capable Concentrator be deployed on physical routers or
>>servers?
>> > > > > >
>> > > > > > [Med] The concentrator can be on physical routers, dedicated
>> > > > > > servers,
>> > > > > etc. This
>> > > > > > is really deployment-specific.
>> > > > > >
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Mingui
>> > > > > > >
>> > > > > > > > -----Original Message-----
>> > > > > > > > From: Banana [mailto:[email-masked]] On Behalf Of
>> > > > > > > > [email-masked]
>> > > > > > > > Sent: Tuesday, September 22, 2015 3:55 PM
>> > > > > > > > To: [email-masked]
>> > > > > > > > Cc: JACQUENET Christian IMT/OLN
>> > > > > > > > Subject: [Banana] TR: I-D Action:
>> > > > > > > > draft-boucadair-mptcp-plain-mode-
>> > > > > > > 00.txt
>> > > > > > > >
>> > > > > > > > Dear all,
>> > > > > > > >
>> > > > > > > > We submitted this I-D to discuss a proposal that aims to
>> > > > > > > > get rid of the encapsulation when aggregating multiple
>> > > > > > > > network
>> > attachments.
>> > > > > > > >
>> > > > > > > > Comments are more than welcome.
>> > > > > > > >
>> > > > > > > > Cheers,
>> > > > > > > > Med
>> > > > > > > >
>> > > > > > > > > -----Message d'origine----- De : I-D-Announce
>> > > > > > > > > [mailto:[email-masked]] De la part de
>> > > > > > > > > [email-masked] Envoyé : lundi 21 septembre
>> > > > > > > > > 2015
>> > > > > 15:31
>> > > > > > À :
>> > > > > > > > > [email-masked] Objet : I-D Action:
>> > > > > > > > > draft-boucadair-mptcp-plain-mode-00.txt
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > A New Internet-Draft is available from the on-line
>> > > > > > > > > Internet-Drafts directories.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >         Title           : An MPTCP Option for
>> > Network-Assisted
>> > > > > > MPTCP
>> > > > > > > > > Deployments: Plain Transport Mode
>> > > > > > > > >         Authors         : Mohamed Boucadair
>> > > > > > > > >                           Christian Jacquenet
>> > > > > > > > > 	Filename        :
>> draft-boucadair-mptcp-plain-mode-00.txt
>> > > > > > > > > 	Pages           : 12
>> > > > > > > > > 	Date            : 2015-09-21
>> > > > > > > > >
>> > > > > > > > > Abstract:
>> > > > > > > > >    One of the promising deployment scenarios for
>> > > > > > > > > Multipath TCP
>> > > > > > (MPTCP)
>> > > > > > > > >    is to enable a Customer Premises Equipment (CPE) that
>> > > > > > > > > is connected
>> > > > > > > to
>> > > > > > > > >    multiple networks (e.g., DSL, LTE, WLAN) to optimize
>> > > > > > > > > the usage of
>> > > > > > > its
>> > > > > > > > >    network attachments.  Because of the lack of MPTCP
>> > > > > > > > > support at
>> > > > > the
>> > > > > > > > >    server side, some service providers now consider a
>> > > > > > > > > "network-
>> > > > > > > assisted
>> > > > > > > > >    mode" that relies upon the activation of a dedicated
>> > > > > > > > > function
>> > > > > > > called
>> > > > > > > > >    MPTCP Concentrator.  This document focuses on a
>> > > > > > > > > deployment
>> > > > > > scheme
>> > > > > > > > >    where the identity of the MPTCP Concentrator(s) is
>>explicitly
>> > > > > > > > >    configured on connected hosts.
>> > > > > > > > >
>> > > > > > > > >    This document specifies an MPTCP option that is used
>> > > > > > > > > to get rid of
>> > > > > > > an
>> > > > > > > > >    encapsulation scheme between the CPE and the MPTCP
>> > > > > > Concentrator.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > The IETF datatracker status page for this draft is:
>> > > > > > > > > https://datatracker.ietf.org/doc/draft-boucadair-mptcp-p
>> > > > > > > > > lain
>> > > > > > > > > -m
>> > > > > > > > > ode/
>> > > > > > > > >
>> > > > > > > > > There's also a htmlized version available at:
>> > > > > > > > > https://tools.ietf.org/html/draft-boucadair-mptcp-plain-
>> > > > > > > > > mode
>> > > > > > > > > -0
>> > > > > > > > > 0
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Please note that it may take a couple of minutes from
>> > > > > > > > > the time of submission until the htmlized version and
>> > > > > > > > > diff are available at tools.ietf.org.
>> > > > > > > > >
>> > > > > > > > > Internet-Drafts are also available by anonymous FTP at:
>> > > > > > > > > ftp://ftp.ietf.org/internet-drafts/
>> > > > > > > > >
>> > > > > > > > > _______________________________________________
>> > > > > > > > > I-D-Announce mailing list [email-masked]
>> > > > > > > > > https://www.ietf.org/mailman/listinfo/i-d-announce
>> > > > > > > > > Internet-Draft directories:
>> > > > > > > > > http://www.ietf.org/shadow.html or
>> > > > > > > > > ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>> > > > > > > >
>> > > > > > > > _______________________________________________
>> > > > > > > > Banana mailing list
>> > > > > > > > [email-masked]
>> > > > > > > > https://www.ietf.org/mailman/listinfo/banana
>> > > > >
>> > > > > _______________________________________________
>> > > > > Banana mailing list
>> > > > > [email-masked]
>> > > > > https://www.ietf.org/mailman/listinfo/banana
>> > > >
>> > > > _______________________________________________
>> > > > Banana mailing list
>> > > > [email-masked]
>> > > > https://www.ietf.org/mailman/listinfo/banana
>> > >
>> > > _______________________________________________
>> > > Banana mailing list
>> > > [email-masked]
>> > > https://www.ietf.org/mailman/listinfo/banana
>> > >
>> > > _______________________________________________
>> > > Banana mailing list
>> > > [email-masked]
>> > > https://www.ietf.org/mailman/listinfo/banana
>> >
>> >
>> ________________________________________________________________
>> > _________________________________________________________
>> >
>> > Ce message et ses pieces jointes peuvent contenir des informations
>> > confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
>> > exploites ou copies sans autorisation. Si vous avez recu ce message
>> > par erreur, veuillez le signaler a l'expediteur et le detruire ainsi
>> > que les pieces jointes. Les messages electroniques etant susceptibles
>> > d'alteration, Orange decline toute responsabilite si ce message a ete
>> > altere, deforme ou falsifie. Merci.
>> >
>> > This message and its attachments may contain confidential or
>> > privileged information that may be protected by law; they should not
>> > be distributed, used or copied without authorisation.
>> > If you have received this email in error, please notify the sender and
>> > delete this message and its attachments.
>> > As emails may be altered, Orange is not liable for messages that have
>> > been modified, changed or falsified.
>> > Thank you.
>> >
>> > _______________________________________________
>> > Banana mailing list
>> > [email-masked]
>> > https://www.ietf.org/mailman/listinfo/banana
>
>_______________________________________________
>Banana mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/banana

---

From: <[email-masked]>
Date: Tue, 20 Oct 2015 08:02:36 +0000

Hi Sri,

Thank you for being direct to the point. 

I don't see the problem statement document advocating for new protocols but as a way to rationalize the discussion in the hope to identify a common perimeter among various initiatives in this area. As Acked in the PS ID, it is not clear at this stage whether one or multiple solutions may be required to accommodate various deployment contexts. Typically, this is one of the inputs that are expected from the IETF community.

If you consider the case of MPTCP or MIP, there are still some missing pieces (e.g., how to configure a concentrator to a CPE, the behavior to get rid of an encapsulation scheme, how to deal with the UDP traffic, etc.). This work can be endorsed by existing WGs but this is not clear at this stage. This is exactly why I see a value in unifying the voices among us.

Cheers,
Med

> De : Sri Gundavelli (sgundave) [mailto:[email-masked]]
> Envoyé : mardi 20 octobre 2015 08:51
> À : Mingui Zhang; SEITE Pierrick IMT/OLN; [email-masked]; BOUCADAIR
> Cc : JACQUENET Christian IMT/OLN
> Objet : Re: [Banana] Problem Statement: Bandwidth Aggregation for Internet
> Access (was RE: I-D Action: draft-boucadair-mptcp-plain-mode-00.txt)
>

---

From: Alexandre Petrescu <[email-masked]>
Date: Tue, 20 Oct 2015 15:23:14 +0200

> thing valuable to the industry by any measure.

So I guess we need to define that measure.

That measure could be a criterion which tells that applications are 
offered 2x more bandwidth when 2 interfaces are used.  Ideally n more 
interfaces would lead to n times more bandwidth.

And that should augment bandwidth for _any_ application (in the same 
manner as Mobile IP offers mobility for any  application not just TCP).

And that could use an intermediary gateway.

But hey, these are requirements, not problem statement :-)

Alex

---

From: "Sri Gundavelli (sgundave)" <[email-masked]>
Date: Tue, 20 Oct 2015 15:20:38 +0000

Hi Med,

Thanks for your response.

We don’t have any requirements from 3GPP/BBF or some other SDO
organization. In the absence of such input, we cannot even identify a
bounded problem statement and to further expect IETF community to
recommend one or more solutions. As such, IETF will never be in a position
to make a protocol recommendation and we have seen how painful it will be
with IPv6 transitioning.
 
Regarding gaps in existing protocols, those protocol gaps will always be
there and its a matter of some one proposing an extension to address those
gaps. Taking your MPTCP extension, do we need this group to be chartered
and have the group make a recommendation to the MPTCP working group to
work on it. It can be standardized directly there. If need be the same WG
can even publish how to use the protocol to a solve a general use-casecase
on informational track.

If the “unifying voice” effort does not result in some BOF’s/WG’s, then we
are on the same page. Last thing we want is to encourage a behavior where
some vendors-X and Vendor-Y decides to pair up and say, "lets invent our
protocol flavor and get an IETF stamp", as the existing protocol seems to
be widely used by Vendor-Z and we want market differentiation.

Regards
Sri

On 10/20/15, 1:02 AM, "[email-masked]"
<[email-masked]> wrote:

>I don't see the problem statement document advocating for new protocols
>but as a way to rationalize the discussion in the hope to identify a
>common perimeter among various initiatives in this area. As Acked in the
>PS ID, it is not clear at this stage whether one or multiple solutions
>may be required to accommodate various deployment contexts. Typically,
>this is one of the inputs that are expected from the IETF community.
>
>If you consider the case of MPTCP or MIP, there are still some missing
>pieces (e.g., how to configure a concentrator to a CPE, the behavior to
>get rid of an encapsulation scheme, how to deal with the UDP traffic,
>etc.). This work can be endorsed by existing WGs but this is not clear at
>this stage. This is exactly why I see a value in unifying the voices
>among us.
>
>Cheers,
>Med
>

---

From: "Sri Gundavelli (sgundave)" <[email-masked]>
Date: Tue, 20 Oct 2015 16:43:30 +0000

Hi Margaret,

> I don't think there is any solution that is fully documented in RFCs or
>WG Drafts, that has been implemented, deployed and found to work well.
 
If there are a set of published technical requirements from a SDO body
working on a market requirement, we could have used that for technical
evaluation of a solution and we could have made such conclusions. Given
that such requirement does not exist, its probably a subjective view that
some thing does not work well. How to demux and mix IP flows can be an
implementation consideration; or should be disallowed per earlier IETF’s
view that we don’t recommend splitting of a flow across two different
access technologies with asymmetric characteristics.

But, discussions for the purpose of cross-training/exchange of ideas
sounds great. Lets see where this goes.

Sri

On 10/20/15, 8:31 AM, "Margaret Cullen" <[email-masked]> wrote:

>
>
>Hi Sri,
>
>I do not think that there is a universal assumption that a new protocol
>or a new WG is needed.  We have started collecting information about
>solutions in this space, and there are already a number of different
>solutions that are documented in RFCs or Intenet-Drafts, or that have
>been implemented and deployed. I don't think there is any solution that
>is fully documented in RFCs or WG Drafts, that has been implemented,
>deployed and found to work well.  So, obviously there is still some work
>to do to get there.
>
>I think of our effort as an attempt to cross-train each other about the
>existing solutions, and see if there are things we can learn from each
>other that will result in an understanding of what solution(s) might work
>best for any given case where Bandwidth Aggregation is needed.  It is
>possible that different solutions will be needed in different
>environments.  Once we understand the lay of the land, though, there
>might or might not be some common work that we can do on how
>demultiplexing works, how traffic is recombined, or how these systems are
>managed, etc.  
>
>Margaret
>
>
>On Oct 20, 2015, at 2:51 AM, Sri Gundavelli (sgundave)
><[email-masked]> wrote:
>

---

From: Margaret Cullen <[email-masked]>
Date: Tue, 20 Oct 2015 11:31:14 -0400

Hi Sri,

I do not think that there is a universal assumption that a new protocol or a new WG is needed.  We have started collecting information about solutions in this space, and there are already a number of different solutions that are documented in RFCs or Intenet-Drafts, or that have been implemented and deployed. I don't think there is any solution that is fully documented in RFCs or WG Drafts, that has been implemented, deployed and found to work well.  So, obviously there is still some work to do to get there. 

I think of our effort as an attempt to cross-train each other about the existing solutions, and see if there are things we can learn from each other that will result in an understanding of what solution(s) might work best for any given case where Bandwidth Aggregation is needed.  It is possible that different solutions will be needed in different environments.  Once we understand the lay of the land, though, there might or might not be some common work that we can do on how demultiplexing works, how traffic is recombined, or how these systems are managed, etc.  

Margaret

---

From: Mingui Zhang <[email-masked]>
Date: Wed, 21 Oct 2015 03:13:04 +0000

Hi Margaret and Sri,

> From: Margaret Wasserman [mailto:[email-masked]]
> Sent: Wednesday, October 21, 2015 6:03 AM
> To: Sri Gundavelli
> Cc: Mingui Zhang; [email-masked]; [email-masked];
> BOUCADAIR Mohamed IMT/OLN; [email-masked]; JACQUENET Christian
> IMT/OLN
> Subject: Re: [Banana] Problem Statement: Bandwidth Aggregation for Internet
> On Oct 20, 2015, at 12:43 PM, Sri Gundavelli (sgundave) <[email-masked]>
> wrote:
> > How to demux and mix IP flows can be an implementation consideration;
> > or should be disallowed per earlier IETF’s view that we don’t
> > recommend splitting of a flow across two different access technologies
> > with asymmetric characteristics.
> 
> Do you know of a reference for this recommendation?  I have also heard the
> same thing, and there are good reasons for this position (some of which are
> documented in my considerations draft), but I couldn't find a reference.  There
> are also issues with per-flow methods (also discussed in my draft), because it is
> not uncommon for a home or small office to have very few flows, some of which
> are long-lived and consume a lot of bandwidth, so a flow might not be the right
> granularity for bandwidth aggregation on small networks.

I am also wondering what's the position of saying this.

Customers do want a flow to be split across heterogeneous connections. We finally see such deployments today. The "non-recommended" thing has already happened, right? Will that position be tenable?

I also saw some words about flow-splitting in the MIP proposal. Let me see...
"...but nothing prevent to use this data plane on a per packet basis."
"[RFC6088] and [RFC6089] define traffic management on a flow basis but there is no such policy on a per packet basis."

https://tools.ietf.org/html/draft-seite-dmm-rg-multihoming-00

Thanks,
Mingui

> 
> I agree that there a lot of tradeoffs, and that is why I don't believe there is one
> right answer for every case.
> 
> I hope to do some more work on my draft before the meeting, and I would be
> very interested in references that would help explain the issues I raised, as well
> as any pointers to information about what has/has not worked in different
> situations, and why.
> 
> Great!  I'm excited to hear about the experiences that other have had in this
> area!
> 
> Margaret
> 
> 
> >
> >
> >> I do not think that there is a universal assumption that a new
> >> protocol or a new WG is needed.  We have started collecting
> >> information about solutions in this space, and there are already a
> >> number of different solutions that are documented in RFCs or
> >> Intenet-Drafts, or that have been implemented and deployed. I don't
> >> think there is any solution that is fully documented in RFCs or WG
> >> Drafts, that has been implemented, deployed and found to work well.
> >> So, obviously there is still some work to do to get there.
> >>
> >> I think of our effort as an attempt to cross-train each other about
> >> the existing solutions, and see if there are things we can learn from
> >> each other that will result in an understanding of what solution(s)
> >> might work best for any given case where Bandwidth Aggregation is
> >> needed.  It is possible that different solutions will be needed in
> >> different environments.  Once we understand the lay of the land,
> >> though, there might or might not be some common work that we can do
> >> on how demultiplexing works, how traffic is recombined, or how these
> >> systems are managed, etc.
> >>
> >> Margaret
> >>
> >>
> >>> But, I do not see any new technical requirements in that spec. This
> >>> is stating the obvious. Unfortunately, our starting point in all the
> >>> discussions that I¹ve seen so far "appears" to be that we need to
> >>> create a new protocol and a new working group. That¹s how I¹m
> >>> reading it. But, looking at the technical consideration, I don¹t see
> >>> any thing new.  The aspect of link aggregation for realizing a
> >>> larger logical pipe, discussion around per-packet/per-flow/weighted
> >>> per packet distribution or policy rules around path selection on a
> >>> flow basic ..are all the topics that IETF had spent significant
> >>> amount of time. Starting with the recent MPTCP, or the MIP work with
> >>> RFC7629, RFC6088, RFC6089 Š have all investigated this problem.
> >>>
> >>> What we are discussing is a system/solution requirement which is for
> >>> the SDO¹s to define and state a protocol requirement, or gaps that
> >>> IETF needs to address. BBF is not willing to make a request to IETF
> >>> to work on any problem and so I wonder what are we solving ? I¹m not
> >>> objecting to this approach, but I don¹t believe we will produce any
> >>> thing useful to the community. It will be  just re-stating the
> >>> existing work as a new requirement, and if it goes forward, we will
> >>> invent a new protocol which will exactly look some other existing
> >>> protocol, but won¹t be producing any thing valuable to the industry
> >>> by any measure.
> >>>
> >>>
> >>> Sri
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> specified. Several candidate solutions had been considered to
> >>>> support bandwidth aggregation for Internet access in IETF. These
> >>>> techniques are investigated.
> >>>>
> >>>>>> No doubt that agreement on building blocks is a key step for a
> >>>>>> common system design... I couldn't agree more... But, AFAIK, this
> >>>>>> is currently discussed in BBF; why not starting from there (i.e.
> >>>>>> leave system design to BBF and focus on protocols)?
> >>>>>>
> >>>>>> BR,
> >>>>>> Pierrick
> >>>>>>
> >>>>>>>> [Mingui] It sounds more and more interesting. That logic would
> >>>>>>>> be an essential part for the bonding service arch. I think it's
> >>>>>>>>>> service functions (e.g., DPI) that are deployed at the
> >>>>>>>>>> network side
> >>>>>>>>> without
> >>>>>>>>>  "This document focuses on the explicit deployment  scheme
> >>>>>>>>> load-balance-among- multipaths use case so it's natural to
> >>>>>>>>> have the idea of reusing MPTCP which is designed to handle
> >>>>>>>>> this use
> >>>>> case.
> >>>>>>>>> bonding devices and be transported only on a primary
> >>>>>>>>> connection; load generated by home users usually contains
> >>>>>>>>> sparse applications types, therefore per-packet load
> >>>>>>>>> distribution is required to achieve a fine-grained load
> >>>>>>>>> distribution on the multiple
> >>>>>>>> connections.
> >>>>>>>>> question is whether it's feasible to integrated those
> >>>>>>>>> necessary functions _into_ the
> >>>>>>>> tunnel?
> >>>>>>>> multiple paths can be considered in addition to the ones that
> >>>>>>>> were considered so far in
> >>>>>>> MPTCP.
> >>>>>>>> This is exactly the reason why I do think there is some value
> >>>>>>>> in separating the load distribution logic from the actual
> >>>>>>>> transport mechanism. This is particularly valid for the UDP case.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>>  One of the promising deployment scenarios for Multipath
> >>>>>>>>>>>>> TCP
> >>>>>>>>>> (MPTCP)
> >>>>>>>>>>>>>  is to enable a Customer Premises Equipment (CPE) that is
> >>>>>>>>>>>>> connected
> >>>>>>>>>>> to
> >>>>>>>>>>>>>  multiple networks (e.g., DSL, LTE, WLAN) to optimize the
> >>>>>>>>>>>>> usage of
> >>>>>>>>>>> its
> >>>>>>>>>>>>>  This document specifies an MPTCP option that is used to
> >>>>>>>>>>>>> get rid of
> >>>>>>>>>>> an
> >>>>>>>>>>>>> Please note that it may take a couple of minutes from the
> >>>>>>>>>>>>> time of submission until the htmlized version and diff are
> >>>>>>>>>>>>> available at tools.ietf.org.
> >>>>>>>>>>>>>
> >>>>>> Ce message et ses pieces jointes peuvent contenir des
> >>>>>> informations confidentielles ou privilegiees et ne doivent donc
> >>>>>> pas etre diffuses, exploites ou copies sans autorisation. Si vous
> >>>>>> avez recu ce message par erreur, veuillez le signaler a
> >>>>>> l'expediteur et le detruire ainsi que les pieces jointes. Les
> >>>>>> messages electroniques etant susceptibles d'alteration, Orange
> >>>>>> decline toute responsabilite si ce message a ete altere, deforme ou
> falsifie. Merci.
> >>>>>>
> >>>>>> privileged information that may be protected by law; they should
> >>>>>> not be distributed, used or copied without authorisation.
> >>>>>> If you have received this email in error, please notify the
> >>>>>> sender and delete this message and its attachments.
> >>>>>> As emails may be altered, Orange is not liable for messages that
> >>>>>> have been modified, changed or falsified.
> >>>>>> Thank you.
> >>>>>>

---

From: Mingui Zhang <[email-masked]>
Date: Wed, 21 Oct 2015 07:24:42 +0000

Hi Alex, 

I know we can figure out a set of meaningful use cases, problems, requirements, gaps,.... But let me use some specific points, based on our discussion on the list, to sketch what we have to face in BANANA. 

Home users want their traffic to be split across different connections in a per-packet manner. [This is use case 1.]

The per-packet load distribution may cause the out-of-sequence issue. [This is problem 1.]

Customers require the packets to be delivered in order. [This is a requirement.] 

When the buffering mechanism is developed to handle problem 1. The packet loss and the large delay difference introduced by heterogeneous connections make the buffer overflow. [This is problem 2.]

To address problem 2, the backpressure and resending functions (especially the cross-connection resending) developed in MPTCP is helpful. But it's not clear how these functions can be used in the per-packet use case. [This is problem 3.]
Here, I don't think the per-packet use case could be handled by simply extending MPTCP. It would be a totally different story. At least, there is no flow to manage but MPTCP is designed to manage flows.

Enterprise users want per-flow traffic distribution. [This is use case 2.]

Some people argued use case 2 can be supported with some existing solutions so it is just an implementation issue. But it's not clear how to support both use case 1 and use case 2 in a single system. Some features have been supported, some are not. [There are gaps.]

Thanks,
Mingui

> From: Alexandre Petrescu [mailto:[email-masked]]
> Sent: Tuesday, October 20, 2015 9:23 PM
> To: Sri Gundavelli (sgundave); Mingui Zhang; [email-masked];
> [email-masked]; BOUCADAIR Mohamed IMT/OLN; [email-masked]
> > create a new protocol and a new working group. That¹s how I¹m reading
> > it. But, looking at the technical consideration, I don¹t see any thing
> > new.  The aspect of link aggregation for realizing a larger logical
> > pipe, discussion around per-packet/per-flow/weighted per packet
> > distribution or policy rules around path selection on a flow basic
> > ..are all the topics that IETF had spent significant amount of time.
> > Starting with the recent MPTCP, or the MIP work with RFC7629, RFC6088,
> > RFC6089 Š have all investigated this problem.
> >
> > IETF needs to address. BBF is not willing to make a request to IETF to
> > work on any problem and so I wonder what are we solving ? I¹m not
> > protocol, but won¹t be producing any thing valuable to the industry by any
> measure.
> 
> That measure could be a criterion which tells that applications are offered 2x
> more bandwidth when 2 interfaces are used.  Ideally n more interfaces would
> lead to n times more bandwidth.
> 
> And that should augment bandwidth for _any_ application (in the same manner
> as Mobile IP offers mobility for any  application not just TCP).
> 
> >>>>>>>>> The draft details the addressing issue which is definitely an
> >>>>>>>>> important issue that falls in the problem space of the bonding
> >>>>>>>>> service. In particular, the solution for NAT-pass-through
> >>>>>>>>> issue is described. It assumes the tunneling techniques cannot
> >>>>>>>>> support NAT-pass-through
> >>> between the
> >>>>>>>>> This document addresses that issue. I guess this is the major
> >>>>>>>>> motivation for
> >>>>>>> proposing the
> >>>>>>>> "plain" mode.
> >>>>>>>>
> >>>>>>>>> Traffic distribution issue is another important issue for the
> >>>>>>>>> bonding service. This had ever been discussed on the list. I
> >>>>>>>>> know MPTCP can well support flow based traffic
> >>> distribution.
> can
> >>>>>>>>> be adapted to support per-packet load sharing among the
> >>> multipaths.
> >>>>>>>>> Will this direction be
> >>>>>>>> considered?
> >>>>>>>>
> >>>>>>>>> A clarification questions: from Figure 1, it is clear that one
> >>>>>>>>> peer of the MPTCP proxy will be deployed on the CPE.
> >>>>>>>>>> We submitted this I-D to discuss a proposal that aims to get
> >>>>>>>>>> rid of the encapsulation when aggregating multiple network
> >>>> attachments.
> >>>>>>>>>>
> >>>> confidentielles ou privilegiees et ne doivent donc pas etre
> >>>> diffuses, exploites ou copies sans autorisation. Si vous avez recu
> >>>> ce message par erreur, veuillez le signaler a l'expediteur et le
> >>>> detruire ainsi que les pieces jointes. Les messages electroniques
> >>>> etant susceptibles d'alteration, Orange decline toute
> >>>> responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
> >>>>
> >>>> If you have received this email in error, please notify the sender
> >>>> and delete this message and its attachments.

---

From: <[email-masked]>
Date: Wed, 21 Oct 2015 10:17:11 +0000

Hi,

> De : Banana [mailto:[email-masked]] De la part de Mingui Zhang
> Envoyé : mercredi 21 octobre 2015 09:25
> À : Alexandre Petrescu; Sri Gundavelli (sgundave); SEITE Pierrick IMT/OLN;
> I know we can figure out a set of meaningful use cases, problems, requirements,
> gaps,.... But let me use some specific points, based on our discussion on the list,
> to sketch what we have to face in BANANA.
> 
> When the buffering mechanism is developed to handle problem 1. The packet loss
> and the large delay difference introduced by heterogeneous connections make the
> buffer overflow. [This is problem 2.]
> 
> To address problem 2, the backpressure and resending functions (especially the
> cross-connection resending) developed in MPTCP is helpful. But it's not clear how
> these functions can be used in the per-packet use case. [This is problem 3.] Here, I
> don't think the per-packet use case could be handled by simply extending MPTCP.

AFAIK, per-(TCP)packet distribution is one of the main MPTCP use-case (RFC6182). So, I'm problably missing something but can you clarify tour thougths?

> It would be a totally different story. At least, there is no flow to manage but
> MPTCP is designed to manage flows.
> 
> Some people argued use case 2 can be supported with some existing solutions so
> it is just an implementation issue. But it's not clear how to support both use case 1
> and use case 2 in a single system. 

As you said it is system design... and  I don't know what system designers (e.g. BBFF) will do but I think we can recommend protocols from the IETF toolbox. To do that, we must separate the problems, namely the way to establish the datapath (e.g. establish GRE tunnels between aggregation entities) from the way to use these datapaths (e.g. tunnels). I mean, distribution and transport functions are two different functions. Transport function is a matter of protocols while distribution is part of the system design. Once tunnels are established I can decide to distribute traffic on a per-packet or per flow basis... buffering, reordering are obviously mandatory when operating on a per-packet basis but buffering has nothing to do with protocols... what we can expect from transport protocol here, is just to add a sequence number to distributed packets (to allow reordering) and that's it... from the protocol standpoint...

Pierrick

Some features have been supported, some are
> not. [There are gaps.]
> > To: Sri Gundavelli (sgundave); Mingui Zhang;
> > [email-masked]; [email-masked]; BOUCADAIR Mohamed
> > IMT/OLN; [email-masked]
> > Subject: Re: [Banana] Problem Statement: Bandwidth Aggregation for
> > Internet Access (was RE: I-D Action:
> > draft-boucadair-mptcp-plain-mode-00.txt)
> >
> >
> >
> > > flow basic ..are all the topics that IETF had spent significant amount of time.
> > > by any

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Alexandre Petrescu <[email-masked]>
Date: Wed, 21 Oct 2015 14:07:33 +0200

Le 21/10/2015 09:24, Mingui Zhang a écrit :
> Hi Alex,
>
> I know we can figure out a set of meaningful use cases, problems,
> requirements, gaps,.... But let me use some specific points, based on
> our discussion on the list, to sketch what we have to face in
> BANANA.
>
> Home users want their traffic to be split across different
> connections in a per-packet manner. [This is use case 1.]
>

Thus going from use-case through problem to requirement makes sense.

_Some_ apps can re-order the out-of-order packets themselves, so the 
requirement should be put on the other apps or customers, not on all of 
them.

Also, the network at large (w/o considering multi-interface) offers no 
guarantee of ordered delivery of packets anyways.  I.e. packets arrive 
at my browser very much out of the order they were sent by a 
correspondent node on the other side of the Internet.  So the 
requirement should not be put on the network-at-large, but maybe on some 
entit(ies? 0..n) in the network.

> When the buffering mechanism is developed to handle problem 1. The
> packet loss and the large delay difference introduced by
> heterogeneous connections make the buffer overflow. [This is problem
> 2.]
>
> To address problem 2, the backpressure and resending functions
> (especially the cross-connection resending) developed in MPTCP is
> helpful. But it's not clear how these functions can be used in the
> per-packet use case. [This is problem 3.] Here, I don't think the
> per-packet use case could be handled by simply extending MPTCP. It
> would be a totally different story. At least, there is no flow to
> manage but MPTCP is designed to manage flows.
>
> Some people argued use case 2 can be supported with some existing
> solutions so it is just an implementation issue. But it's not clear
> how to support both use case 1 and use case 2 in a single system.
> Some features have been supported, some are not. [There are gaps.]

Another problem with MP-TCP is that it works only for TCP.  It will not 
speed up video involving UDP flows.

Another problem in general (not MP-TCP) has to do with interoperability: 
a Mobile Router implementation in my lab can not talk to a Gateway that 
is mainstream.  This prohibits the use of other very important 
reliability features of the mainstream  gateways.

Alex

> Thanks, Mingui
>
>
>> -----Original Message----- From: Alexandre Petrescu
>> [mailto:[email-masked]] Sent: Tuesday, October 20,
>> 2015 9:23 PM To: Sri Gundavelli (sgundave); Mingui Zhang;
>> IMT/OLN; [email-masked] Cc: JACQUENET Christian IMT/OLN Subject:
>> Re: [Banana] Problem Statement: Bandwidth Aggregation for Internet
>>> But, I do not see any new technical requirements in that spec.
>>> This is stating the obvious. Unfortunately, our starting point in
>>> all the discussions that I¹ve seen so far "appears" to be that we
>>> need to create a new protocol and a new working group. That¹s how
>>> I¹m reading it. But, looking at the technical consideration, I
>>> don¹t see any thing new.  The aspect of link aggregation for
>>> realizing a larger logical pipe, discussion around
>>> per-packet/per-flow/weighted per packet distribution or policy
>>> rules around path selection on a flow basic ..are all the topics
>>> that IETF had spent significant amount of time. Starting with the
>>> recent MPTCP, or the MIP work with RFC7629, RFC6088, RFC6089 Š
>>> have all investigated this problem.
>>>
>>> What we are discussing is a system/solution requirement which is
>>> for the SDO¹s to define and state a protocol requirement, or gaps
>>> that IETF needs to address. BBF is not willing to make a request
>>> to IETF to work on any problem and so I wonder what are we
>>> solving ? I¹m not objecting to this approach, but I don¹t believe
>>> we will produce any thing useful to the community. It will be
>>> just re-stating the existing work as a new requirement, and if it
>>> goes forward, we will invent a new protocol which will exactly
>>> look some other existing protocol, but won¹t be producing any
>> offered 2x more bandwidth when 2 interfaces are used.  Ideally n
>> more interfaces would lead to n times more bandwidth.
>>
>> And that should augment bandwidth for _any_ application (in the
>> same manner as Mobile IP offers mobility for any  application not
>> just TCP).
>>
>>>> I think the discussion on-going here leads to the following
>>>> problem statement draft which we just submitted. Start from the
>>>> problem statement, we can design the protocols.
>>>>
>>>> Bandwidth aggregation for Internet access can increase the
>>>> access bandwidth and reliability. It has become a desirable
>>>> network function, especially for those Service Providers who
>>>> operates multiple heterogeneous networks.
>>>>
>>>> This document describes the issues with bandwidth aggregation
>>>> for Internet access. The required network functions to be
>>>> supported are specified. Several candidate solutions had been
>>>> considered to support bandwidth aggregation for Internet access
>>>> in IETF. These techniques are investigated.
>>>>
Questions? Comments?
>>>>
>>>> Mingui
>>>>
>>>>> -----Original Message----- From: Mingui Zhang Sent: Monday,
>>>>> October 19, 2015 5:38 PM To: [email-masked]';
>>>>> [email-masked]; BOUCADAIR Mohamed IMT/OLN;
>>>>> [email-masked] Cc: JACQUENET Christian IMT/OLN Subject: RE:
>>>>> I-D Action: draft-boucadair-mptcp-plain-mode-00.txt
>>>>>
>>>>> Hi Pierrick,
>>>>>
>>>>> The classification of the BBF work and IETF work is exact and
>>>>> helpful. I think we are having a promising common view.
>>>>>
>>>>> Thanks, Mingui
>>>>>
>>>>>> -----Original Message----- From: Banana
>>>>>> [mailto:[email-masked]] On Behalf Of
>>>>>> [email-masked] Sent: Monday, October 19, 2015
>>>>>> 4:54 PM To: [email-masked]; Mingui Zhang; BOUCADAIR
>>>>>> Mohamed
>>>>> IMT/OLN;
>>>>>> [email-masked] Cc: JACQUENET Christian IMT/OLN Subject:
>>>>>> Re: [Banana] I-D Action:
>>>>>> No doubt that agreement on building blocks is a key step
>>>>>> for a common system design... I couldn't agree more... But,
>>>>>> AFAIK, this is currently discussed in BBF; why not starting
>>>>>> from there (i.e. leave system design to BBF and focus on
>>>>>> protocols)?
>>>>>>
>>>>>> BR, Pierrick
>>>>>>
>>>>>>> -----Message d'origine----- De : Banana
>>>>>>> [mailto:[email-masked]] De la part de
>>>>>>> [email-masked] Envoyé : lundi 19 octobre 2015 09:02
>>>>>>> À : [email-masked]; BOUCADAIR Mohamed IMT/OLN;
>>>>>> [email-masked] Cc
>>>>>>> : JACQUENET Christian IMT/OLN Objet : Re: [Banana] I-D
>>>>>>> Action: draft-boucadair-mptcp-plain-mode-00.txt
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>> [Mingui] It sounds more and more interesting. That
>>>>>>>> logic would be an essential part for the bonding
>>>>>>>> service arch. I think it's worthwhile to agree on a
>>>>>>>> common model where all those necessary parts (load
>>>>>>>> distribution,
>>>>>>> I agree. As far as I recall during the Bar BoF in Prague
>>>>>>> we discussed that it would be a good approach to identify
>>>>>>> those
>>>>> "building blocks"
>>>>>>> which are common to the existing solutions. Having such a
>>>>>>> common model/architecture will probably help to identify
>>>>>>> the necessary
>>>>> parts.
>>>>>>>
>>>>>>>
>>>>>>> Best regards
>>>>>>>
>>>>>>> Nic
>>>>>>>
>>>>>>>
>>>>>>>> -----Original Message----- From: Banana
>>>>>>>> [mailto:[email-masked]] On Behalf Of
>>>>>>>> [email-masked] Sent: Friday, October 16,
>>>>>>>> 2015 2:44 PM To: Mingui Zhang; [email-masked] Cc:
>>>>>>>> JACQUENET Christian IMT/OLN Subject: Re: [Banana] I-D
>>>>>>>> Action: draft-boucadair-mptcp-plain-mode-00.txt
>>>>>>>>
>>>>>>>> Hi Mingui,
>>>>>>>>
>>>>>>>> Please see inline.
>>>>>>>>
>>>>>>>> Cheers, Med
>>>>>>>>
>>>>>>>>> -----Message d'origine----- De : Banana
>>>>>>>>> [mailto:[email-masked]] De la part de Mingui
>>>>>>>>> Zhang Envoyé : mercredi 14 octobre 2015 10:12 À :
>>>>>>>>> BOUCADAIR Mohamed IMT/OLN; [email-masked] Cc :
>>>>>>>>> JACQUENET Christian
>>>>>> IMT/OLN Objet : Re:
>>>>>>>>> [Banana] I-D Action:
>>>>>>>>> Thanks, Mingui
>>>>>>>>>> -----Original Message----- From:
>>>>>>>>>> [mailto:[email-masked]] Sent:
>>>>>>>>>> Tuesday, October 13, 2015 11:13 PM To: Mingui
>>>>>>>>>> Zhang; [email-masked] Cc: JACQUENET Christian
>>>>>>>>>> IMT/OLN Subject: RE: I-D Action:
>>>>>>>>>> Cheers, Med
>>>>>>>>>>> -----Message d'origine----- De : Mingui Zhang
>>>>>>>>>>> [mailto:[email-masked]] Envoyé : jeudi 24
>>>>>>>>>>> septembre 2015 09:32 À : BOUCADAIR Mohamed
>>>>>>>>>>> IMT/OLN;
>>>>>>>>>> [email-masked] Cc
>>>>>>>>>>> : JACQUENET Christian IMT/OLN Objet : RE: I-D
>>>>>>>>>>> Action: draft-boucadair-mptcp-plain-mode-00.txt
>>>>>>>>>>>
>>>>>>>>>>> Hi Med,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the sharing.
>>>>>>>>>>>
>>>>>>>>>>> The draft details the addressing issue which is
>>>>>>>>>>> definitely an important issue that falls in the
>>>>>>>>>>> problem space of the bonding service. In
>>>>>>>>>>> particular, the solution for NAT-pass-through
>>>>>>>>>>> issue is described. It assumes the tunneling
>>>>>>>>>>> techniques cannot support NAT-pass-through
>>>>> between the CPE and the Concentrator.
>>>>>>>>>>> This document addresses that issue. I guess this
>>>>>>>>>>> is the major motivation for
>>>>>>>>> proposing the
>>>>>>>>>> "plain" mode.
>>>>>>>>>>
>>>>>>>>>> [Med] Yes. Other goals are get rid of the
>>>>>>>>>> encapsulation overhead and
>>>>>>>>> reuse the
>>>>>>>>>> same solution for UDP-based traffic. The plain mode
>>>>>>>>>> allows to easily
>>>>>>>>> reuse
>>>>>>>>>> service functions (e.g., DPI) that are deployed at
>>>>>>>>>> the network side
>>>>>>>>> without
>>>>>>>>> [Mingui] I guess this kind of reuse is feasible given
>>>>>>>>> that the end systems are changed a little bit: in
>>>>>>>>>>> like to add another way to solve the same issue:
>>>>>>>>>>> the tunnel encapsulated packet can be wrapped
>>>>>>>>>>> with a UDP header, which forms a tunnel-over-UDP.
>>>>>>>>>>> By doing that, NAT-pass-through can be addressed
>>>>>>>>>>> as
>>>>>>>>> well.
>>>>>>>>>>
>>>>>>>>>> [Med] That's another option, indeed. Beside the
>>>>>>>>>> encapsulation overhead,
>>>>>>>>> there
>>>>>>>>>> may be some side effects if intermediate functions
>>>>>>>>>> are needed to be
>>>>>>>>> invoked
>>>>>>>>> [Mingui] Yes, there would be some gaps. If an
>>>>>>>>> intermediate function becomes clearly required, it
>>>>>>>>> need be integrated into the overlay
>>>>>> model.
>>>>>>>>>
>>>>>>>>> load-balance-among- multipaths use case so it's
>>>>>>>>> natural to have the idea of reusing MPTCP which is
>>>>>>>>> designed to handle this use
>>>>> case.
>>>>>>>>> However, I doubt the traditional
>>>>>>>>> load-balance-among-multipaths use case does not fully
>>>>>>>>> match the bonding service. MPTCP has to
>>>>> be adapted.
>>>>>>>>>
>>>>>>>>> Customers of the bonding service would discriminate
>>>>>>>>> the heterogeneous connections. For example, the
>>>>>>>>> alternative connection would not be used unless the
>>>>>>>>> primary connection is saturated; some application
>>>>>>>>> types need be recognized by the bonding devices and
>>>>>>>>> be transported only on a primary connection; load
>>>>>>>>> generated by home users usually contains sparse
>>>>>>>>> applications types, therefore per-packet load
>>>>>>>>> distribution is required to achieve a fine-grained
>>>>>>>>> load distribution on the multiple
>>>>>>>> connections.
>>>>>>>>> I agree the flow & congestion control functions
>>>>>>>>> developed in MPTCP could be reused in the bonding
>>>>>>>>> service architecture. The question is whether it's
>>>>>>>>> feasible to integrated those necessary functions
>>>>>>>>> _into_ the
>>>>>>>> tunnel?
>>>>>>>>> We've already seen that the Point-to-Point Tunneling
>>>>>>>>> Protocol (PPTP) [RFC2637] has integrated quite some
>>>>>>>>> of them, right?
>>>>>>>>
>>>>>>>> [Med] I hear you. Several strategies to optimize the
>>>>>>>> use of multiple paths can be considered in addition to
>>>>>>>> the ones that were considered so far in
>>>>>>> MPTCP.
>>>>>>>> This is exactly the reason why I do think there is some
>>>>>>>> value in separating the load distribution logic from
>>>>>>>> the actual transport mechanism. This is particularly
>>>>>>>> valid for the UDP case.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Traffic distribution issue is another important
>>>>>>>>>>> issue for the bonding service. This had ever been
>>>>>>>>>>> discussed on the list. I know MPTCP can well
>>>>>>>>>>> support flow based traffic
>>>>> distribution.
>>>>>>>>>> the load distribution logic can even be separated
>>>>>>>>>> from the actual
>>>>>>>>> mechanism
>>>>>>>>>> used to transport data over multiple paths, but I
>>>>>>>>>> must admit this is out
>>>>>>>>> of scope
>>>>>>>>>> of this draft.
>>>>>>>>>
>>>>>>>>> [Mingui] Thanks for the clarification. For home
>>>>>>>>> users, the per-packet load distribution is desired so
>>>>>>>>> I hope it would be covered in some other drafts.
>>>>>>>>
>>>>>>>>>>> A clarification questions: from Figure 1, it is
>>>>>>>>>>> clear that one peer of the MPTCP proxy will be
>>>>>>>>>>> deployed on the CPE. What is not clear to me is
>>>>>>>>>>> where the other end is to be deployed? Will the
>>>>>>>>>> [Med] The concentrator can be on physical routers,
>>>>>>>>>> dedicated servers,
>>>>>>>>> etc. This
>>>>>>>>>>> Thanks, Mingui
>>>>>>>>>>>> -----Original Message----- From: Banana
>>>>>>>>>>>> [mailto:[email-masked]] On Behalf Of
>>>>>>>>>>>> [email-masked] Sent: Tuesday,
>>>>>>>>>>>> September 22, 2015 3:55 PM To: [email-masked]
>>>>>>>>>>>> Cc: JACQUENET Christian IMT/OLN Subject:
>>>>>>>>>>>> [Banana] TR: I-D Action:
>>>>>>>>>>>> We submitted this I-D to discuss a proposal
>>>>>>>>>>>> that aims to get rid of the encapsulation when
>>>>>>>>>>>> aggregating multiple network
>>>>>> attachments.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers, Med
>>>>>>>>>>>>> [mailto:[email-masked]] De la
>>>>>>>>>>>>> part de [email-masked] Envoyé :
>>>>>>>>>>>>> lundi 21 septembre 2015
>>>>>>>>> 15:31
>>>>>>>>>> À :
>>>>>>>>>>>>> A New Internet-Draft is available from the
>>>>>>>>>>>>> on-line Internet-Drafts directories.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Deployments: Plain Transport Mode Authors
>>>>>>>>>>>>> : Mohamed Boucadair Christian Jacquenet
>>>>>>>>>>>>> Filename        :
>>>>>>>>>>>>> Pages           : 12 Date            :
>>>>>>>>>>>>> 2015-09-21
>>>>>>>>>>>>> Abstract: One of the promising deployment
>>>>>>>>>>>>> scenarios for Multipath TCP
>>>>>>>>>> (MPTCP)
>>>>>>>>>>>>> is to enable a Customer Premises Equipment
>>>>>>>>>>>>> (CPE) that is connected
>>>>>>>>>>> to
>>>>>>>>>>>>> multiple networks (e.g., DSL, LTE, WLAN) to
>>>>>>>>>>>>> optimize the usage of
>>>>>>>>>>> its
>>>>>>>>>>>>> network attachments.  Because of the lack of
>>>>>>>>>>>>> MPTCP support at
>>>>>>>>> the
>>>>>>>>>>>>> server side, some service providers now
>>>>>>>>>>>>> consider a "network-
>>>>>>>>>>> assisted
>>>>>>>>>>>>> mode" that relies upon the activation of a
>>>>>>>>>>>>> dedicated function
>>>>>>>>>>> called
>>>>>>>>>>>>> MPTCP Concentrator.  This document focuses on
>>>>>>>>>>>>> a deployment
>>>>>>>>>> scheme
>>>>>>>>>>>>> This document specifies an MPTCP option that
>>>>>>>>>>>>> is used to get rid of
>>>>>>>>>>> an
lain
>>>>>>>>>>>>> -m ode/
mode
>>>>>>>>>>>>> -0 0
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please note that it may take a couple of
>>>>>>>>>>>>> minutes from the time of submission until the
>>>>>>>>>>>>> htmlized version and diff are available at
>>>>>>>>>>>>> tools.ietf.org.
>>>>>>>>>>>>>
Banana mailing list
>>>>>>>>> Banana mailing list [email-masked]
>>>>>>>> _______________________________________________ Banana
>>>>>>>> mailing list [email-masked]
>>>>>>> _______________________________________________ Banana
>>>>>>> mailing list [email-masked]
>>>>>>> _______________________________________________ Banana
>>>>>>> mailing list [email-masked]
>>>>>> informations confidentielles ou privilegiees et ne doivent
>>>>>> donc pas etre diffuses, exploites ou copies sans
>>>>>> autorisation. Si vous avez recu ce message par erreur,
>>>>>> veuillez le signaler a l'expediteur et le detruire ainsi
>>>>>> que les pieces jointes. Les messages electroniques etant
>>>>>> susceptibles d'alteration, Orange decline toute
>>>>>> This message and its attachments may contain confidential
>>>>>> or privileged information that may be protected by law;
>>>>>> they should not be distributed, used or copied without
>>>>>> authorisation. If you have received this email in error,
>>>>>> please notify the sender and delete this message and its
>>>>>> attachments. As emails may be altered, Orange is not liable
>>>>>> for messages that have been modified, changed or
>>>>>> falsified. Thank you.
>>>>>>
>>>>>> _______________________________________________ Banana
>>>>>> mailing list [email-masked]
>>>> _______________________________________________ Banana mailing
>>>> list [email-masked]
>>> _______________________________________________ Banana mailing
>>> list [email-masked]

---

From: Mingui Zhang <[email-masked]>
Date: Thu, 22 Oct 2015 08:48:02 +0000

Hi Alex,

> Sent: Wednesday, October 21, 2015 8:08 PM
> To: Mingui Zhang; Sri Gundavelli (sgundave); [email-masked];
> Access
> > our discussion on the list, to sketch what we have to face in BANANA.
> >
> > Home users want their traffic to be split across different connections
> > in a per-packet manner. [This is use case 1.]
> >
> requirement should be put on the other apps or customers, not on all of them.

[Mingui] Yes, I agree. But, I think the requirement will be put on the peering network devices uniformly for all kinds of apps or customers. Otherwise, there would be a non-negligible burden to distinguish them.

> guarantee of ordered delivery of packets anyways.  I.e. packets arrive at my
> browser very much out of the order they were sent by a correspondent node on
> the other side of the Internet.  So the requirement should not be put on the
> network-at-large, but maybe on some entit(ies? 0..n) in the network.

[Mingui] We aim to design the solution to be network based. So the requirement should be put on the network devices (named as ACC and AGG in the draft) as a nodal requirement.

> > packet loss and the large delay difference introduced by heterogeneous
> > connections make the buffer overflow. [This is problem 2.]
> >
> Another problem with MP-TCP is that it works only for TCP.  It will not speed
> up video involving UDP flows.

[Mingui] Sure. That's a gap.

> a Mobile Router implementation in my lab can not talk to a Gateway that is
> mainstream.  This prohibits the use of other very important reliability features
> of the mainstream  gateways.

[Mingui] Thanks for this information. I can see we should not pin our hope on changing the gateways. I guess this kind of interoperability issue may be resolved using tunneling technologies. At the same time, the mechanisms developed by MP-TCP are embedded into the tunnel.

Thanks,
Mingui

> >>> I¹m reading it. But, looking at the technical consideration, I don¹t
> >>> see any thing new.  The aspect of link aggregation for realizing a
> >>>>> Cc: JACQUENET Christian IMT/OLN Subject: RE:
> >>>>>>>>> [mailto:[email-masked]] De la part de Mingui Zhang
> >>>>>>>>> Envoyé : mercredi 14 octobre 2015 10:12 À :
> >>>>>>>>>> Tuesday, October 13, 2015 11:13 PM To: Mingui Zhang;
> >>>>>>>>>>> [mailto:[email-masked]] Envoyé : jeudi 24 septembre
> >>>>>>>>>>> 2015 09:32 À : BOUCADAIR Mohamed IMT/OLN;
> >>>>>>>>>> [email-masked] Cc
> >>>>>>>>> "This document focuses on the explicit deployment scheme where
> >>>>>>>>> the identity of the MPTCP
> >>>>>>>>> Concentrator(s) is
> >>>>> explicitly
> >>>>>>>>>>> the tunnel encapsulated packet can be wrapped with a UDP
> >>>>>>>>>>> header, which forms a tunnel-over-UDP.
> >>>>>>>>>>> By doing that, NAT-pass-through can be addressed as
> >>>>>>>>> well.
> >>>>>>>>>>

---

From: Mingui Zhang <[email-masked]>
Date: Thu, 22 Oct 2015 08:51:43 +0000

Hi Pierrick,

Thanks for the constructive comments. 

> AFAIK, per-(TCP)packet distribution is one of the main MPTCP use-case
> (RFC6182). So, I'm problably missing something but can you clarify tour
> thougths?

In [RFC6824], it reads 
   Once a subflow is set up, demultiplexing
   packets is done using the 5-tuple, as in traditional TCP.  The
   5-tuples will be mapped to the local connection identifier (token).
This is per-flow load balancing rather than per-packet load sharing. I believe the device needs to maintain the state about the mapping. 

In contrast, in the per-packet case, traffic would be split using random marking methods to make the splitting ratio to be quite fine-grained. It is stateless.

> As you said it is system design... and  I don't know what system designers (e.g.
> BBFF) will do but I think we can recommend protocols from the IETF toolbox. 

Yes, we would keep the system design in mind while do protocols.

> To
> do that, we must separate the problems, namely the way to establish the
> datapath (e.g. establish GRE tunnels between aggregation entities) from the
> way to use these datapaths (e.g. tunnels). I mean, distribution and transport
> functions are two different functions. Transport function is a matter of
> protocols while distribution is part of the system design. Once tunnels are
> established I can decide to distribute traffic on a per-packet or per flow basis...
> buffering, reordering are obviously mandatory when operating on a per-packet
> basis but buffering has nothing to do with protocols... what we can expect from
> transport protocol here, is just to add a sequence number to distributed
> packets (to allow reordering) and that's it... from the protocol standpoint...

It's a good idea to talk about the transport protocol.

In order to demultiplex the multiple ACC devices that are connecting to the same AGG. We need an ID field for the bonding connection. (For example, the "Key" field of the GRE header can be used for this purpose.) 

If retransmission is to be supported, we also need an ACK field. 

Thanks,
Mingui

> Sent: Wednesday, October 21, 2015 6:17 PM
> To: Mingui Zhang; Alexandre Petrescu; Sri Gundavelli (sgundave);
> > De : Banana [mailto:[email-masked]] De la part de Mingui
> > Zhang Envoyé : mercredi 21 octobre 2015 09:25 À : Alexandre Petrescu;
> > Sri Gundavelli (sgundave); SEITE Pierrick IMT/OLN;
> Cc :
> > JACQUENET Christian IMT/OLN Objet : Re: [Banana] Problem Statement:
> > Bandwidth Aggregation for Internet Access (was RE: I-D Action:
> > in a per- packet manner. [This is use case 1.]
> >
> > per-packet use case. [This is problem 3.] Here, I don't think the per-packet
> use case could be handled by simply extending MPTCP.
> 
> 
> AFAIK, per-(TCP)packet distribution is one of the main MPTCP use-case
> (RFC6182). So, I'm problably missing something but can you clarify tour
> thougths?
> 
> > It would be a totally different story. At least, there is no flow to
> As you said it is system design... and  I don't know what system designers (e.g.
> BBFF) will do but I think we can recommend protocols from the IETF toolbox. To
> do that, we must separate the problems, namely the way to establish the
> datapath (e.g. establish GRE tunnels between aggregation entities) from the
> way to use these datapaths (e.g. tunnels). I mean, distribution and transport
> functions are two different functions. Transport function is a matter of
> protocols while distribution is part of the system design. Once tunnels are
> established I can decide to distribute traffic on a per-packet or per flow basis...
> buffering, reordering are obviously mandatory when operating on a per-packet
> basis but buffering has nothing to do with protocols... what we can expect from
> transport protocol here, is just to add a sequence number to distributed
> packets (to allow reordering) and that's it... from the protocol standpoint...
> 
> Pierrick
> 
> had spent significant amount of time.
> > > > to IETF to work on any problem and so I wonder what are we solving
> > > > ? I¹m not objecting to this approach, but I don¹t believe we will
> > > > produce any thing useful to the community. It will be  just
> > > > re-stating the existing work as a new requirement, and if it goes
> > > > forward, we will invent a new protocol which will exactly look
> > > > some other existing protocol, but won¹t be producing any thing
> > > > valuable to the industry by any
> > > measure.
> > >
> support
> > > >>>> common system design... I couldn't agree more... But, AFAIK,
> > > >>>> this is currently discussed in BBF; why not starting from there (i.e.
> > > >>>>>> [Mingui] It sounds more and more interesting. That logic
> > > >>>>>> would be an essential part for the bonding service arch. I
> > > >>>>>> think it's worthwhile to agree on a common model where all
> > > >>>>>> those necessary parts (load distribution,
> > > >>>>> common model/architecture will probably help to identify the
> > > >>>>> necessary
> > > >>> parts.
> > > >>>>>
> > > >>>>>
> > > >>>>>    Best regards
> > > >>>>>
> > > >>>>>       Nic
> > > >>>>>
> > > >>>>>
> JACQUENET
> > > >>>>>>>>> [mailto:[email-masked]] Envoyé :
> > > >>>>>>>>> jeudi
> > > >>>>>>>> same solution for UDP-based traffic. The plain mode allows
> > > >>>>>>>> to easily
> > > >>>>>>> reuse
> > > >>>>>>> [Mingui] I guess this kind of reuse is feasible given that
> > > >>>>>>> the end systems are changed a little bit: in
> > > >>>>>>>>> like to add another way to solve the same issue: the
> > > >>>>>>>>> tunnel encapsulated packet can be wrapped with a UDP
> > > >>>>>>>>> header, which forms a tunnel-over-UDP. By doing that,
> > > >>>>>>>>> NAT-pass-through can be addressed as
> > > >>>>>>> well.
> > > >>>>>>>>
> > > >>>>>>>> may be some side effects if intermediate functions are
> > > >>>>>>>> needed to be
> > > >>>>>>> invoked
> > > >>>>>>> function becomes clearly required, it need be integrated
> > > >>>>>>> into the overlay
> > > >>>> model.
> > > >>>>>>>
> > > >>>>>>> load-balance-among-multipaths use case does not fully match
> > > >>>>>>> the bonding service. MPTCP has to
> > > >>> be
> > > >>> adapted.
> > > >>>>>>>
> > > >>>>>>> connection would not be used unless the primary connection
> > > >>>>>>> is saturated; some application types need be recognized by
> > > >>>>>>> the bonding devices and be transported only on a primary
> > > >>>>>>> MPTCP could be reused in the bonding service architecture.
> > > >>>>>>> The question is whether it's feasible to integrated those
> > > >>>>>>> Protocol
> > > >>>>>>>> used to transport data over multiple paths, but I must
> > > >>>>>>>> admit this is out
> > > >>>>>>> of scope
> > > >>>>>>>> of this draft.
> > > >>>>>>>
> > > >>>>>>> per-packet load distribution is desired so I hope it would
> > > >>>>>>> be covered in some other drafts.
> > > >>>>>>
> > > >>>>>>>>>> From: Banana [mailto:[email-masked]] On Behalf
> > > >>>>>>>>>> Of [email-masked]
> > > >>>>>>>>>>>     is to enable a Customer Premises Equipment (CPE)
> > > >>>>>>>>>>> that is connected
> > > >>>>>>>>> to
> > > >>>> pas etre diffuses, exploites ou copies sans autorisation. Si
> > > >>>> vous avez recu ce message par erreur, veuillez le signaler a
> > > >>>> privileged information that may be protected by law; they
> > > >>>> should not be distributed, used or copied without authorisation.
> > > >>>> As emails may be altered, Orange is not liable for messages
> > > >>>> that have been modified, changed or falsified.
> > > >>>> Thank you.
> > > >>>>
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou
> copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le
> signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration, Orange decline toute
> This message and its attachments may contain confidential or privileged
> information that may be protected by law; they should not be distributed, used
> or copied without authorisation.

---

From: Alexandre Petrescu <[email-masked]>
Date: Thu, 22 Oct 2015 12:27:42 +0200

Mingui,

Thank you for the reply.  I would like to add clarification.

Le 22/10/2015 10:48, Mingui Zhang a écrit :
[...]
>> Another problem in general (not MP-TCP) has to do with
>> interoperability: a Mobile Router implementation in my lab can not
>> talk to a Gateway that is mainstream.  This prohibits the use of
>> other very important reliability features of the mainstream
>> gateways.
>
> [Mingui] Thanks for this information. I can see we should not pin
> our hope on changing the gateways. I guess this kind of
> interoperability issue may be resolved using tunneling technologies.
> At the same time, the mechanisms developed by MP-TCP are embedded
> into the tunnel.

I would like to clarify further the interoperability need.

Tunneling is also  used in mobility management.  It is maybe not very
good to add too many levels of tunneling.  Maybe a requirement can be
put on less levels of tunnels.

But let me try to clarify the interoperability issue.

Currently we have developped a Mobile Router with protocol Mobile IP.
We have added a non-standard bandwidth aggregation feature.  This router
is put in a vehicle.  This router connects to a Home Agent fixed in the
infrastructure.  The Home Agent is also implementing protocol Mobile IP 
(the Mobile IP parts are interoperable), but has no bandwidth 
aggregation feature.  It is a highly-reliable Home Agent that we buy 
'off-the-shelf' in the market.

Because the HA does not implement our bandwidth aggregation feature we
find ourselves in a difficult position to add a new entity in the
infrastructure.  This entity is less reliable, but implements bandwidth
aggregation.

We thus end up with _two_ entities in the infra: a prototype for the
bandwidth aggregation, and the reliable HA for the Mobile IP part.

We would like the reliable HA to also implement a bandwidth aggregation
feature, that interoperates with our Mobile Router.  This will bring
back reliability and reduce the number of entities we must maintain in
the infra.

Alex

>>>> [mailto:[email-masked]] Sent: Tuesday, October
>>>> 20, 2015 9:23 PM To: Sri Gundavelli (sgundave); Mingui Zhang;
>>>> IMT/OLN; [email-masked] Cc: JACQUENET Christian IMT/OLN
>>>> Subject: Re: [Banana] Problem Statement: Bandwidth Aggregation
>>>> for Internet Access (was RE: I-D Action:
>>>>> But, I do not see any new technical requirements in that
>>>>> spec. This is stating the obvious. Unfortunately, our
>>>>> starting point in all the discussions that I¹ve seen so far
>>>>> "appears" to be that we need to create a new protocol and a
>>>>> new working group. That¹s how I¹m reading it. But, looking
>>>>> at the technical consideration, I don¹t see any thing new.
>>>>> The aspect of link aggregation for realizing a larger
>>>>> logical pipe, discussion around per-packet/per-flow/weighted
>>>>> per packet distribution or policy rules around path selection
>>>>> on a flow basic ..are all the topics that IETF had spent
>>>>> significant amount of time. Starting with the recent MPTCP,
>>>>> or the MIP work with RFC7629, RFC6088, RFC6089 Š have all
>>>>> investigated this problem.
>>>>>
>>>>> What we are discussing is a system/solution requirement
>>>>> which is for the SDO¹s to define and state a protocol
>>>>> requirement, or gaps that IETF needs to address. BBF is not
>>>>> willing to make a request to IETF to work on any problem and
>>>>> so I wonder what are we solving ? I¹m not objecting to this
>>>>> approach, but I don¹t believe we will produce any thing
>>>>> useful to the community. It will be just re-stating the
>>>>> existing work as a new requirement, and if it goes forward,
>>>>> we will invent a new protocol which will exactly look some
>>>>> other existing protocol, but won¹t be producing any thing
>>>> That measure could be a criterion which tells that
>>>> applications are offered 2x more bandwidth when 2 interfaces
>>>> are used. Ideally n more interfaces would lead to n times more
>>>> bandwidth.
>>>>
>>>> And that should augment bandwidth for _any_ application (in
>>>> the same manner as Mobile IP offers mobility for any
>>>> application not just TCP).
>>>>
>>>>>> I think the discussion on-going here leads to the
>>>>>> following problem statement draft which we just submitted.
>>>>>> access bandwidth and reliability. It has become a
>>>>>> desirable network function, especially for those Service
>>>>>> Providers who operates multiple heterogeneous networks.
>>>>>>
>>>>>> This document describes the issues with bandwidth
>>>>>> aggregation for Internet access. The required network
>>>>>> functions to be supported are specified. Several candidate
>>>>>> solutions had been considered to support bandwidth
>>>>>> aggregation for Internet access in IETF. These techniques
>>>>>> are investigated.
>>>>>>
Questions? Comments?
>>>>>>
>>>>>> Mingui
>>>>>>
>>>>>>> -----Original Message----- From: Mingui Zhang Sent:
>>>>>>> Monday, October 19, 2015 5:38 PM To:
>>>>>>> [email-masked]'; [email-masked];
>>>>>>> BOUCADAIR Mohamed IMT/OLN;
>> [email-masked]
>>>>>>> Cc: JACQUENET Christian IMT/OLN Subject: RE: I-D Action:
>>>>>>> The classification of the BBF work and IETF work is
>>>>>>> exact and helpful. I think we are having a promising
>>>>>>> common view.
>>>>>>>
>>>>>>> Thanks, Mingui
>>>>>>>
>>>>>>>> [email-masked] Sent: Monday, October 19,
>>>>>>>> 2015 4:54 PM To: [email-masked]; Mingui Zhang;
>>>>>>>> BOUCADAIR
>> Mohamed
>>>>>>> IMT/OLN;
>>>>>>>> [email-masked] Cc: JACQUENET Christian IMT/OLN
>>>>>>>> No doubt that agreement on building blocks is a key
>>>>>>>> step for a common system design... I couldn't agree
>>>>>>>> more... But, AFAIK, this is currently discussed in
>>>>>>>> BBF; why not starting from there (i.e. leave system
>>>>>>>> design to BBF and focus on protocols)?
>>>>>>>>
>>>>>>>> BR, Pierrick
>>>>>>>>
>>>>>>>>> [email-masked] Envoyé : lundi 19 octobre 2015
>>>>>>>>> 09:02 À : [email-masked]; BOUCADAIR Mohamed
>>>>>>>>> IMT/OLN;
>>>>>>>> [email-masked] Cc
>>>>>>>>> : JACQUENET Christian IMT/OLN Objet : Re: [Banana]
>>>>>>>>>> service arch. I think it's worthwhile to agree on
>>>>>>>>>> a common model where all those necessary parts
>>>>>>>>>> (load distribution,
>>>>>>>>> I agree. As far as I recall during the Bar BoF in
>>>>>>>>> Prague we discussed that it would be a good approach
>>>>>>>>> to identify those
>>>>>>> "building blocks"
>>>>>>>>> which are common to the existing solutions. Having
>>>>>>>>> such a common model/architecture will probably help
>>>>>>>>> to identify the necessary
>>>>>>> parts.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best regards
>>>>>>>>>
>>>>>>>>> Nic
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> [email-masked] Sent: Friday, October
>>>>>>>>>> 16, 2015 2:44 PM To: Mingui Zhang; [email-masked]
>>>>>>>>>> Cc: JACQUENET Christian IMT/OLN Subject: Re:
>>>>>>>>>> [Banana] I-D Action:
>>>>>>>>>>> Mingui Zhang Envoyé : mercredi 14 octobre 2015
>>>>>>>>>>> 10:12 À : BOUCADAIR Mohamed IMT/OLN;
>>>>>>>>>>> [email-masked] Cc : JACQUENET Christian
>>>>>>>> IMT/OLN Objet : Re:
>>>>>>>>>>> [Banana] I-D Action:
>>>>>>>>>>>>> jeudi 24 septembre 2015 09:32 À : BOUCADAIR
>>>>>>>>>>>>> Mohamed IMT/OLN;
>>>>>>>>>>>> [email-masked] Cc
>>>>>>>>>>>>> The draft details the addressing issue which
>>>>>>>>>>>>> is definitely an important issue that falls
>>>>>>>>>>>>> in the problem space of the bonding service.
>>>>>>>>>>>>> In particular, the solution for
>>>>>>>>>>>>> NAT-pass-through issue is described. It
>>>>>>>>>>>>> assumes the tunneling techniques cannot
>>>>>>>>>>>>> support NAT-pass-through
>>>>>>>>>>>>> This document addresses that issue. I guess
>>>>>>>>>>>>> this is the major motivation for
>>>>>>>>>>> proposing the
>>>>>>>>>>>> "plain" mode.
>>>>>>>>>>>>
>>>>>>>>>>>> same solution for UDP-based traffic. The plain
>>>>>>>>>>>> mode allows to easily
>>>>>>>>>>> reuse
>>>>>>>>>>>> service functions (e.g., DPI) that are
>>>>>>>>>>>> deployed at the network side
>>>>>>>>>>> without
>>>>>>>>>>> [Mingui] I guess this kind of reuse is feasible
>>>>>>>>>>> given that the end systems are changed a little
>>>>>>>>>>> bit: in draft-boucadair-mptcp-dhc-01, it reads
>>>>>>>>>>>
>>>>>>>>>>> "This document focuses on the explicit
>>>>>>>>>>> deployment scheme where the identity of the MPTCP
>>>>>>>>>>>  Concentrator(s) is
>>>>>>> explicitly
>>>>>>>>>>>>> like to add another way to solve the same
>>>>>>>>>>>>> issue: the tunnel encapsulated packet can be
>>>>>>>>>>>>> wrapped with a UDP header, which forms a
>>>>>>>>>>>>> tunnel-over-UDP. By doing that,
>>>>>>>>>>>> may be some side effects if intermediate
>>>>>>>>>>>> functions are needed to be
>>>>>>>>>>> invoked
>>>>>>>>>>> intermediate function becomes clearly required,
>>>>>>>>>>> it need be integrated into the overlay
>>>>>>>> model.
>>>>>>>>>>>
>>>>>>>>>>> The bonding service is sometimes regarded as a
>>>>>>>>>>> pure load-balance-among- multipaths use case so
>>>>>>>>>>> it's natural to have the idea of reusing MPTCP
>>>>>>>>>>> which is designed to handle this use
>>>>>>> case.
>>>>>>>>>>> load-balance-among-multipaths use case does not
>>>>>>>>>>> fully match the bonding service. MPTCP has to
>>>>>>> be adapted.
>>>>>>>>>>>
>>>>>>>>>>> Customers of the bonding service would
>>>>>>>>>>> discriminate the heterogeneous connections. For
>>>>>>>>>>> example, the alternative connection would not be
>>>>>>>>>>> used unless the primary connection is saturated;
>>>>>>>>>>> some application types need be recognized by the
>>>>>>>>>>> bonding devices and be transported only on a
>>>>>>>>>>> primary connection; load generated by home users
>>>>>>>>>>> usually contains sparse applications types,
>>>>>>>>>>> therefore per-packet load distribution is
>>>>>>>>>>> required to achieve a fine-grained load
>>>>>>>>>>> service architecture. The question is whether
>>>>>>>>>>> it's feasible to integrated those necessary
>>>>>>>>>>> functions _into_ the
>>>>>>>>>> tunnel?
>>>>>>>>>>> We've already seen that the Point-to-Point
>>>>>>>>>>> Tunneling Protocol (PPTP) [RFC2637] has
>>>>>>>>>>> integrated quite some of them, right?
>>>>>>>>>>
>>>>>>>>>> [Med] I hear you. Several strategies to optimize
>>>>>>>>>> the use of multiple paths can be considered in
>>>>>>>>>> addition to the ones that were considered so far
>>>>>>>>>> in
>>>>>>>>> MPTCP.
>>>>>>>>>> This is exactly the reason why I do think there is
>>>>>>>>>> some value in separating the load distribution
>>>>>>>>>> logic from the actual transport mechanism. This is
>>>>>>>>>> particularly valid for the UDP case.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Traffic distribution issue is another
>>>>>>>>>>>>> important issue for the bonding service.
>>>>>>>>>>>>> This had ever been discussed on the list. I
>>>>>>>>>>>> the load distribution logic can even be
>>>>>>>>>>>> separated from the actual
>>>>>>>>>>> mechanism
>>>>>>>>>>>> used to transport data over multiple paths,
>>>>>>>>>>>> but I must admit this is out
>>>>>>>>>>> of scope
>>>>>>>>>>>> of this draft.
>>>>>>>>>>>
>>>>>>>>>>> users, the per-packet load distribution is
>>>>>>>>>>> desired so I hope it would be covered in some
>>>>>>>>>>> other drafts.
>>>>>>>>>>
>>>>>>>>>>>>> A clarification questions: from Figure 1, it
>>>>>>>>>>>>> is clear that one peer of the MPTCP proxy
>>>>>>>>>>>>> will be deployed on the CPE. What is not
>>>>>>>>>>>>> clear to me is where the other end is to be
>>>>>>>>>>>>> deployed? Will the L4 capable Concentrator
>>>>>>>>>>>>> be deployed on physical routers or
>>>>>>> servers?
>>>>>>>>>>>>
>>>>>>>>>>>> [Med] The concentrator can be on physical
>>>>>>>>>>>> routers, dedicated servers,
>>>>>>>>>>> etc. This
>>>>>>>>>>>>>> [mailto:[email-masked]] On Behalf
>>>>>>>>>>>>>> Of [email-masked] Sent:
>>>>>>>>>>>>>> Tuesday, September 22, 2015 3:55 PM To:
>>>>>>>>>>>>>> [email-masked] Cc: JACQUENET Christian
>>>>>>>>>>>>>> IMT/OLN Subject: [Banana] TR: I-D Action:
>>>>>>>>>>>>>> that aims to get rid of the encapsulation
>>>>>>>>>>>>>> when aggregating multiple network
>>>>>>>> attachments.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [mailto:[email-masked]]
>>>>>>>>>>>>>>> De la part de [email-masked]
>>>>>>>>>>>>>>> Envoyé : lundi 21 septembre 2015
>>>>>>>>>>> 15:31
>>>>>>>>>>>> À :
>>>>>>>>>>>>>>> A New Internet-Draft is available from
>>>>>>>>>>>>>>> the on-line Internet-Drafts directories.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> is to enable a Customer Premises
>>>>>>>>>>>>>>> Equipment (CPE) that is connected
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> multiple networks (e.g., DSL, LTE, WLAN)
>>>>>>>>>>>>>>> to optimize the usage of
>>>>>>>>>>>>> its
>>>>>>>>>>>>>>> network attachments.  Because of the
>>>>>>>>>>>>>>> lack of MPTCP support at
>>>>>>>>>>> the
>>>>>>>>>>>>>>> mode" that relies upon the activation of
>>>>>>>>>>>>>>> a dedicated function
>>>>>>>>>>>>> called
>>>>>>>>>>>>>>> MPTCP Concentrator.  This document
>>>>>>>>>>>>>>> focuses on a deployment
>>>>>>>>>>>> scheme
>>>>>>>>>>>>>>> This document specifies an MPTCP option
>>>>>>>>>>>>>>> that is used to get rid of
>>>>>>>>>>>>> an
lain
mode
>>>>>>>>>>>>>>>  minutes from the time of submission
>>>>>>>>>>>>>>> until the htmlized version and diff are
>>>>>>>> informations confidentielles ou privilegiees et ne
>>>>>>>> doivent donc pas etre diffuses, exploites ou copies
>>>>>>>> sans autorisation. Si vous avez recu ce message par
>>>>>>>> erreur, veuillez le signaler a l'expediteur et le
>>>>>>>> detruire ainsi que les pieces jointes. Les messages
>>>>>>>> electroniques etant susceptibles d'alteration, Orange
>>>>>>>> decline toute responsabilite si ce message a ete
>>>>>>>> This message and its attachments may contain
>>>>>>>> confidential or privileged information that may be
>>>>>>>> protected by law; they should not be distributed, used
>>>>>>>> or copied without authorisation. If you have received
>>>>>>>> this email in error, please notify the sender and
>>>>>>>> delete this message and its attachments. As emails may
>>>>>>>> be altered, Orange is not liable for messages that
>>>>>>>> have been modified, changed or falsified. Thank you.
>>>>>>>>

---

From: Margaret Cullen <[email-masked]>
Date: Thu, 22 Oct 2015 06:48:32 -0400

Hi Mingui,

> Home users want their traffic to be split across different connections in a per-packet manner. [This is use case 1.]

I don't think that most home users are thinking in these terms.  In fact, I don't think that most home users know the difference between per-packet and per-flow demultiplexing.  I would state this more like:

Home users want faster and more reliable Internet service.  When they are paying for two (or more) connections, they would like the bandwidth of both connections to be used to their advantage.  They would like to realize this advantage for applications that use a single, long-lived, high bandwidth connection (such as video streaming), as well as for applications that use many shorter-lived connections (i.e. web browsing).
> Customers require the packets to be delivered in order. [This is a requirement.] 

By "Customers", are you talking about service providers or end-users?  Either way, I am not sure that "packets must be delivered in order" is a first order requirement.  I think the first order requirement would be something like "Customers would like the bandwidth of both connections to be used as efficiently as possible".  Packet ordering derives from that first-order requirement, because out-of-order packets cause retransmissions and delays, thus reducing the efficiency of bandwidth use.

Margaret

---

From: Mingui Zhang <[email-masked]>
Date: Fri, 23 Oct 2015 07:58:17 +0000

Hi Alex, 

Thanks for the clarifications. 

The group aims to add bandwidth aggregation features into network devices. In the scenario you described, I guess you map the HA to AGG while the Mobile Router is mapped to ACC. If the HA has implemented AGG features, you need not install the extra prototype in the network infrastructure. I think such deployment makes sense. 

By the way, does your Mobile Router use both LTE and DSL to connect to the HA? 

Thanks,
Mingui

> Sent: Thursday, October 22, 2015 6:28 PM
> To: Mingui Zhang
> Cc: Sri Gundavelli (sgundave); [email-masked];
> [email-masked]; BOUCADAIR Mohamed IMT/OLN; [email-masked];
> JACQUENET Christian IMT/OLN
> >> other very important reliability features of the mainstream gateways.
> >
> > [Mingui] Thanks for this information. I can see we should not pin our
> > hope on changing the gateways. I guess this kind of interoperability
> > issue may be resolved using tunneling technologies.
> > At the same time, the mechanisms developed by MP-TCP are embedded into
> > the tunnel.
> 
> Tunneling is also  used in mobility management.  It is maybe not very good to
> add too many levels of tunneling.  Maybe a requirement can be put on less
> levels of tunnels.
> 
> We have added a non-standard bandwidth aggregation feature.  This router is
> put in a vehicle.  This router connects to a Home Agent fixed in the
> infrastructure.  The Home Agent is also implementing protocol Mobile IP (the
> Mobile IP parts are interoperable), but has no bandwidth aggregation feature.
> It is a highly-reliable Home Agent that we buy 'off-the-shelf' in the market.
> 
> find ourselves in a difficult position to add a new entity in the infrastructure.
> This entity is less reliable, but implements bandwidth aggregation.
> 
> We thus end up with _two_ entities in the infra: a prototype for the bandwidth
> aggregation, and the reliable HA for the Mobile IP part.
> 
> feature, that interoperates with our Mobile Router.  This will bring back
> reliability and reduce the number of entities we must maintain in the infra.
> 
> Alex
> 
> 
> >
> > Thanks, Mingui
> >
> >>
> >> Alex
> >>
> >>> Thanks, Mingui
> >>>
> >>>
> >>>>> don¹t see any thing new.
> >>>>> The aspect of link aggregation for realizing a larger logical
> >>>>> ..are all the topics that IETF had spent significant amount of
> >>>>> time. Starting with the recent MPTCP, or the MIP work with
> >>>>>> problem statement draft which we just submitted.
> >>>>>> Internet access. The required network functions to be supported
> >>>>>> are specified. Several candidate solutions had been considered to
> >>>>>>> [email-masked]'; [email-masked]; BOUCADAIR
> >>>>>>> Mohamed IMT/OLN;
> >> [email-masked]
> >>>>>>>> this is currently discussed in BBF; why not starting from there
> >>>>>>>> (i.e. leave system design to BBF and focus on protocols)?
> >>>>>>>>
> >>>>>>>> BR, Pierrick
> >>>>>>>>
> >>>>>>>>>> [email-masked] Sent: Friday, October 16, 2015
> >>>>>>>>>> 2:44 PM To: Mingui Zhang; [email-masked]
> >>>>>>>>>>> Envoyé : mercredi 14 octobre 2015
> >>>>>>>>>>> 10:12 À : BOUCADAIR Mohamed IMT/OLN; [email-masked] Cc :
> >>>>>>>>>>> JACQUENET Christian
> >>>>>>>> IMT/OLN Objet : Re:
> >>>>>>>>>>> [Banana] I-D Action:
> >>>>>>>>>>>> RE: I-D Action:
> >>>>>>>>>>>>> jeudi 24 septembre 2015 09:32 À : BOUCADAIR Mohamed
> >>>>>>>>>>>>> IMT/OLN;
> >>>>>>>>>>>> [email-masked] Cc
> >>>>>>>>>>>>> bonding service.
> >>>>>>>>>>>>> In particular, the solution for NAT-pass-through issue is
> >>>>>>>>>>>>> described. It assumes the tunneling techniques cannot
> >>>>>>>>>>>>> support NAT-pass-through
> >>>>>>>>>>> the end systems are changed a little
> >>>>>>>>>>>>> issue: the tunnel encapsulated packet can be wrapped with
> >>>>>>>>>>>>> a UDP header, which forms a tunnel-over-UDP. By doing
> >>>>>>>>>>>>> that, NAT-pass-through can be addressed as
> >>>>>>>>>>> well.
> >>>>>>>>>>>>
> >>>>>>>>>>> Protocol (PPTP) [RFC2637] has integrated quite some of them,
> >>>>>>>>>>> right?
> >>>>>>>>>>
> >>>>>>>>>>>>> the bonding service.
> >>>>>>>>>>>>> This had ever been discussed on the list. I know MPTCP can
> >>>>>>>>>>>>> well support flow based traffic
> >>>>>>> distribution.
> >>>>>>>>>>>>> deployed? Will the L4 capable Concentrator be deployed on
> >>>>>>>>>>>>> physical routers or
> >>>>>>> servers?
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> [email-masked] Sent:
> >>>>>>>>>>>>>>> De la part de [email-masked] Envoyé : lundi 21
> >>>>>>>>>>>>>>> septembre 2015
> >>>>>>>>>>> 15:31
> >>>>>>>>>>>> À :
> >>>>>>>>>>>>>>> Abstract: One of the promising deployment scenarios for
> >>>>>>>>>>>>>>> Multipath TCP
> >>>>>>>>>>>> (MPTCP)

---

From: Mingui Zhang <[email-masked]>
Date: Fri, 23 Oct 2015 08:06:14 +0000

Hi Margaret,

Appreciate the enrichments and adjustments! I think the text should be documented somewhere. 

Thanks,
Mingui

> From: Margaret Cullen [mailto:[email-masked]]
> Sent: Thursday, October 22, 2015 6:49 PM
> To: Mingui Zhang
> Cc: Alexandre Petrescu; Sri Gundavelli (sgundave); [email-masked];
> I don't think that most home users are thinking in these terms.  In fact, I don't
> think that most home users know the difference between per-packet and
> per-flow demultiplexing.  I would state this more like:
> 
> Home users want faster and more reliable Internet service.  When they are
> paying for two (or more) connections, they would like the bandwidth of both
> connections to be used to their advantage.  They would like to realize this
> advantage for applications that use a single, long-lived, high bandwidth
> connection (such as video streaming), as well as for applications that use many
> shorter-lived connections (i.e. web browsing).
> >
> By "Customers", are you talking about service providers or end-users?  Either
> way, I am not sure that "packets must be delivered in order" is a first order
> requirement.  I think the first order requirement would be something like
> "Customers would like the bandwidth of both connections to be used as
> efficiently as possible".  Packet ordering derives from that first-order
> requirement, because out-of-order packets cause retransmissions and delays,
> thus reducing the efficiency of bandwidth use.
> 
> Margaret
> 
> >
> >
> >

---

From: Alexandre Petrescu <[email-masked]>
Date: Fri, 23 Oct 2015 16:56:22 +0200

Le 23/10/2015 09:58, Mingui Zhang a écrit :
> The group aims to add bandwidth aggregation features into network
> devices. In the scenario you described, I guess you map the HA to AGG
> while the Mobile Router is mapped to ACC. If the HA has implemented
> AGG features, you need not install the extra prototype in the network
> infrastructure. I think such deployment makes sense.
>

It uses two LTE interfaces.

But we should be able to make it  independent of the interface kind.  It
should work the same over any kind, and ADSL would be a valuable
addition for homenet users.

Alex

>> [mailto:[email-masked]] Sent: Thursday, October 22,
>> 2015 6:28 PM To: Mingui Zhang Cc: Sri Gundavelli (sgundave);
>> IMT/OLN; [email-masked]; JACQUENET Christian IMT/OLN Subject: Re:
>> [Banana] Problem Statement: Bandwidth Aggregation for Internet
>> Access
>>
>> Mingui,
>>
>> Le 22/10/2015 10:48, Mingui Zhang a écrit : [...]
>>>> interoperability: a Mobile Router implementation in my lab can
>>>> not talk to a Gateway that is mainstream.  This prohibits the
>>>> use of other very important reliability features of the
>>>> mainstream gateways.
>>>
>>> interoperability issue may be resolved using tunneling
>>> technologies. At the same time, the mechanisms developed by
>>> MP-TCP are embedded into the tunnel.
>>
>> Tunneling is also  used in mobility management.  It is maybe not
>> very good to add too many levels of tunneling.  Maybe a requirement
>> can be put on less levels of tunnels.
>>
>> Currently we have developped a Mobile Router with protocol Mobile
>> IP. We have added a non-standard bandwidth aggregation feature.
>> This router is put in a vehicle.  This router connects to a Home
>> Agent fixed in the infrastructure.  The Home Agent is also
>> implementing protocol Mobile IP (the Mobile IP parts are
>> interoperable), but has no bandwidth aggregation feature. It is a
>> highly-reliable Home Agent that we buy 'off-the-shelf' in the
>> market.
>>
>> Because the HA does not implement our bandwidth aggregation feature
>> we find ourselves in a difficult position to add a new entity in
>> the infrastructure. This entity is less reliable, but implements
>> bandwidth aggregation.
>>
>> We thus end up with _two_ entities in the infra: a prototype for
>> the bandwidth aggregation, and the reliable HA for the Mobile IP
>> part.
>>
>> We would like the reliable HA to also implement a bandwidth
>> aggregation feature, that interoperates with our Mobile Router.
>> This will bring back reliability and reduce the number of entities
>> we must maintain in the infra.
>>
>> Alex
>>
>>
>>>
>>> Thanks, Mingui
>>>
>>>>
>>>> Alex
>>>>
>>>>> Thanks, Mingui
>>>>>
>>>>>
>>>>>> [mailto:[email-masked]] Sent: Tuesday,
>>>>>> October 20, 2015 9:23 PM To: Sri Gundavelli (sgundave);
>>>>>> Mingui Zhang; [email-masked];
>>>>>> [email-masked]; BOUCADAIR
>>>> Mohamed
>>>>>> Subject: Re: [Banana] Problem Statement: Bandwidth
>>>>>> Aggregation for Internet Access (was RE: I-D Action:
>>>>>>> starting point in all the discussions that I¹ve seen so
>>>>>>> far "appears" to be that we need to create a new protocol
>>>>>>> and a new working group. That¹s how I¹m reading it. But,
>>>>>>> looking at the technical consideration, I don¹t see any
>>>>>>> thing new. The aspect of link aggregation for realizing a
>>>>>>> larger logical pipe, discussion around
>>>>>>> per-packet/per-flow/weighted per packet distribution or
>>>>>>> policy rules around path selection on a flow basic ..are
>>>>>>> all the topics that IETF had spent significant amount of
>>>>>>> time. Starting with the recent MPTCP, or the MIP work
>>>>>>> with RFC7629, RFC6088, RFC6089 Š have all investigated
>>>>>>> this problem.
>>>>>>>
>>>>>>> requirement, or gaps that IETF needs to address. BBF is
>>>>>>> not willing to make a request to IETF to work on any
>>>>>>> problem and so I wonder what are we solving ? I¹m not
>>>>>>> objecting to this approach, but I don¹t believe we will
>>>>>>> produce any thing useful to the community. It will be
>>>>>>> just re-stating the existing work as a new requirement,
>>>>>>> and if it goes forward, we will invent a new protocol
>>>>>>> which will exactly look some other existing protocol, but
>>>>>>> won¹t be producing any thing valuable to the industry by
>>>>>>> any
>>>>>> measure.
>>>>>>
>>>>>> applications are offered 2x more bandwidth when 2
>>>>>> interfaces are used. Ideally n more interfaces would lead
>>>>>> to n times more bandwidth.
>>>>>>
>>>>>>> On 10/19/15, 11:15 PM, "Banana on behalf of Mingui
>>>>>>> Zhang" <[email-masked] on behalf of
>>>>>>> [email-masked]> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> following problem statement draft which we just
>>>>>>>> submitted. Start from the problem statement, we can
>>>>>>>> design the protocols.
>>>>>>>>
>>>>>>>> Bandwidth aggregation for Internet access can increase
>>>>>>>> the access bandwidth and reliability. It has become a
>>>>>>>> desirable network function, especially for those
>>>>>>>> Service Providers who operates multiple heterogeneous
>>>>>>>> networks.
>>>>>>>>
>>>>>>>> functions to be supported are specified. Several
>>>>>>>> candidate solutions had been considered to support
>>>>>>>> bandwidth aggregation for Internet access in IETF.
>>>>>>>> These techniques are investigated.
>>>>>>>>
>>>>>>>> https://tools.ietf.org/html/draft-zhang-banana-problem-statement-
>>>>>>>>
>>>>>>>>
00
>>>>>>>>> Cc: JACQUENET Christian IMT/OLN Subject: RE: I-D
>>>>>>>>>> [email-masked] Sent: Monday, October
>>>>>>>>>> 19, 2015 4:54 PM To: [email-masked]; Mingui
>>>>>>>>>> Zhang;
>> BOUCADAIR
>>>> Mohamed
>>>>>>>>> IMT/OLN;
>>>>>>>>>>> [email-masked] Envoyé : lundi 19 octobre
>>>>>>>>>>> 2015 09:02 À : [email-masked]; BOUCADAIR
>>>>>>>>>>> Mohamed
>> IMT/OLN;
>>>>>>>>>> [email-masked] Cc
>>>>>>>>>>>> [Mingui] It sounds more and more interesting.
>>>>>>>>>>>> That logic would be an essential part for the
>>>>>>>>>>>> bonding service arch. I think it's worthwhile
>>>>>>>>>>>> to agree on a common model where all those
>>>>>>>>>>>> necessary parts (load distribution,
>>>>>>>>>>> Prague we discussed that it would be a good
>>>>>>>>>>> approach to identify those
>>>>>>>>> "building blocks"
>>>>>>>>>>> which are common to the existing solutions.
>>>>>>>>>>> Having such a common model/architecture will
>>>>>>>>>>> probably help to identify the necessary
>>>>>>>>> parts.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best regards
>>>>>>>>>>>
>>>>>>>>>>> Nic
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> [email-masked] Sent: Friday,
>>>>>>>>>>>> October 16, 2015 2:44 PM To: Mingui Zhang;
>>>>>>>>>>>>> [mailto:[email-masked]] De la part
>>>>>>>>>>>>> de Mingui Zhang Envoyé : mercredi 14 octobre
>>>>>>>>>>>>> 2015 10:12 À : BOUCADAIR Mohamed IMT/OLN;
>>>>>>>>>>>>>> To: Mingui Zhang; [email-masked] Cc:
>>>>>>>>>>>>>> JACQUENET Christian IMT/OLN Subject: RE:
>>>>>>>>>>>>>> I-D Action:
>>>>>>>>>>>>>>> -----Message d'origine----- De : Mingui
>>>>>>>>>>>>>>> Zhang [mailto:[email-masked]]
>>>>>>>>>>>>>>> Envoyé : jeudi 24 septembre 2015 09:32 À
>>>>>>>>>>>>>>> : BOUCADAIR Mohamed IMT/OLN;
>>>>>>>>>>>>>> [email-masked] Cc
>>>>>>>>>>>>>>> The draft details the addressing issue
>>>>>>>>>>>>>>> which is definitely an important issue
>>>>>>>>>>>>>>> that falls in the problem space of the
>>>>>>>>>>>>>>> bonding service. In particular, the
>>>>>>>>>>>>>>> solution for NAT-pass-through issue is
>>>>>>>>>>>>>>> described. It assumes the tunneling
>>>>>>>>>>>>>>> This document addresses that issue. I
>>>>>>>>>>>>>>> guess this is the major motivation for
>>>>>>>>>>>>> proposing the
>>>>>>>>>>>>>> "plain" mode.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> same solution for UDP-based traffic. The
>>>>>>>>>>>>>> plain mode allows to easily
>>>>>>>>>>>>> reuse
>>>>>>>>>>>>> [Mingui] I guess this kind of reuse is
>>>>>>>>>>>>> feasible given that the end systems are
>>>>>>>>>>>>> changed a little bit: in
>>>>>>>>>>>>> deployment scheme where the identity of the
>>>>>>>>>>>>> MPTCP Concentrator(s) is
>>>>>>>>> explicitly
>>>>>>>>>>>>>>> like to add another way to solve the
>>>>>>>>>>>>>>> same issue: the tunnel encapsulated
>>>>>>>>>>>>>>> packet can be wrapped with a UDP header,
>>>>>>>>>>>>>>> which forms a tunnel-over-UDP. By doing
>>>>>>>>>>>>>> [Med] That's another option, indeed. Beside
>>>>>>>>>>>>>> the encapsulation overhead,
>>>>>>>>>>>>> there
>>>>>>>>>>>>> intermediate function becomes clearly
>>>>>>>>>>>>> required, it need be integrated into the
>>>>>>>>>>>>> overlay
>>>>>>>>>> model.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The bonding service is sometimes regarded as
>>>>>>>>>>>>> a pure load-balance-among- multipaths use
>>>>>>>>>>>>> case so it's natural to have the idea of
>>>>>>>>>>>>> reusing MPTCP which is designed to handle
>>>>>>>>>>>>> this use
>>>>>>>>> case.
>>>>>>>>>>>>> load-balance-among-multipaths use case does
>>>>>>>>>>>>> not fully match the bonding service. MPTCP
>>>>>>>>>>>>> has to
>>>>>>>>> be adapted.
>>>>>>>>>>>>>
>>>>>>>>>>>>> discriminate the heterogeneous connections.
>>>>>>>>>>>>> For example, the alternative connection would
>>>>>>>>>>>>> not be used unless the primary connection is
>>>>>>>>>>>>> saturated; some application types need be
>>>>>>>>>>>>> recognized by the bonding devices and be
>>>>>>>>>>>>> transported only on a primary connection;
>>>>>>>>>>>>> load generated by home users usually
>>>>>>>>>>>>> contains sparse applications types, therefore
>>>>>>>>>>>>> per-packet load distribution is required to
>>>>>>>>>>>>> achieve a fine-grained load distribution on
>>>>>>>>>>>>> the multiple
>>>>>>>>>>>> connections.
>>>>>>>>>>>>> I agree the flow & congestion control
>>>>>>>>>>>>> functions developed in MPTCP could be reused
>>>>>>>>>>>>> in the bonding service architecture. The
>>>>>>>>>>>>> question is whether it's feasible to
>>>>>>>>>>>>> integrated those necessary functions _into_
>>>>>>>>>>>>> the
>>>>>>>>>>>> tunnel?
>>>>>>>>>>>> [Med] I hear you. Several strategies to
>>>>>>>>>>>> optimize the use of multiple paths can be
>>>>>>>>>>>> considered in addition to the ones that were
>>>>>>>>>>>> considered so far in
>>>>>>>>>>> MPTCP.
>>>>>>>>>>>> This is exactly the reason why I do think there
>>>>>>>>>>>> is some value in separating the load
>>>>>>>>>>>> distribution logic from the actual transport
>>>>>>>>>>>>>>> This had ever been discussed on the list.
>>>>>>>>>>>>>>> I know MPTCP can well support flow based
>>>>>>>>>>>>>>> traffic
>>>>>>>>> distribution.
>>>>>>>>>>>>>>> However, I remember it was mentioned
>>>>>>>>>>>>>>> somewhere that
>> MPTCP
>>>>>> can
>>>>>>>>>>>>> [Mingui] Thanks for the clarification. For
>>>>>>>>>>>>> home users, the per-packet load distribution
>>>>>>>>>>>>> is desired so I hope it would be covered in
>>>>>>>>>>>>> some other drafts.
>>>>>>>>>>>>
>>>>>>>>>>>>>>> A clarification questions: from Figure 1,
>>>>>>>>>>>>>>> it is clear that one peer of the MPTCP
>>>>>>>>>>>>>>> proxy will be deployed on the CPE. What
>>>>>>>>>>>>>>> is not clear to me is where the other end
>>>>>>>>>>>>>>> is to be deployed? Will the L4 capable
>>>>>>>>>>>>>>> Concentrator be deployed on physical
>>>>>>>>>>>>>>> routers or
>>>>>>>>> servers?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Banana [mailto:[email-masked]]
>>>>>>>>>>>>>>>> On Behalf Of
>>>>>>>>>>>>>>>> Tuesday, September 22, 2015 3:55 PM
>>>>>>>>>>>>>>>> To: [email-masked] Cc: JACQUENET
>>>>>>>>>>>>>>>> Christian IMT/OLN Subject: [Banana] TR:
>>>>>>>>>>>>>>>> I-D Action:
>>>>>>>>>>>>>>>> We submitted this I-D to discuss a
>>>>>>>>>>>>>>>> proposal that aims to get rid of the
>>>>>>>>>>>>>>>> encapsulation when aggregating
>>>>>>>>>>>>>>>> multiple network
>>>>>>>>>> attachments.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Authors : Mohamed Boucadair Christian
>>>>>>>>>>>>>>>>> Jacquenet Filename        :
>>>>>>>>>>>>>>>>> Pages           : 12 Date
>>>>>>>>>>>>>>>>> : 2015-09-21
>>>>>>>>>>>>>>>>> Abstract: One of the promising
>>>>>>>>>>>>>>>>> deployment scenarios for Multipath
>>>>>>>>>>>>>>>>> TCP
>>>>>>>>>>>>>> (MPTCP)
>>>>>>>>>>>>>>>>> multiple networks (e.g., DSL, LTE,
>>>>>>>>>>>>>>>>> WLAN) to optimize the usage of
>>>>>>>>>>>>>>> its
>>>>>>>>>>>>>>>>> server side, some service providers
>>>>>>>>>>>>>>>>> now consider a "network-
>>>>>>>>>>>>>>> assisted
>>>>>>>>>>>>>>>>> mode" that relies upon the activation
>>>>>>>>>>>>>>>>> of a dedicated function
>>>>>>>>>>>>>>> called
>>>>>>>>>>>>>>>>> This document specifies an MPTCP
>>>>>>>>>>>>>>>>> option that is used to get rid of
>>>>>>>>>>>>>>> an
lain
mode
>>>>>>>>>>>>>>>>> -0 0
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Please note that it may take a couple
>>>>>>>>>>>>>>>>> of minutes from the time of
>>>>>>>>>>>>>>>>> submission until the htmlized version
>>>>>>>>>>>>>>>>> and diff are available at
>>>>>>>>>>>>>>>>> tools.ietf.org.
>>>>>>>>>>>>>>>>>
>>>>>>>>>> doivent donc pas etre diffuses, exploites ou
>>>>>>>>>> copies sans autorisation. Si vous avez recu ce
>>>>>>>>>> message par erreur, veuillez le signaler a
>>>>>>>>>> l'expediteur et le detruire ainsi que les pieces
>>>>>>>>>> jointes. Les messages electroniques etant
>>>>>>>>>> protected by law; they should not be distributed,
>>>>>>>>>> used or copied without authorisation. If you have
>>>>>>>>>> received this email in error, please notify the
>>>>>>>>>> As emails may be altered, Orange is not liable for
>>>>>>>>>> messages that have been modified, changed or
>>>>>>>>>> falsified. Thank you.
>>>>>>>>>>

---

From: <[email-masked]>
Date: Mon, 26 Oct 2015 09:52:49 +0000

> Envoyé : jeudi 22 octobre 2015 10:52
> À : SEITE Pierrick IMT/OLN; Alexandre Petrescu; Sri Gundavelli (sgundave);
> > (RFC6182). So, I'm problably missing something but can you clarify
> > tour thougths?
> 
> In [RFC6824], it reads
> This is per-flow load balancing rather than per-packet load sharing. I believe the
> device needs to maintain the state about the mapping.
> 
> In contrast, in the per-packet case, traffic would be split using random marking
> methods to make the splitting ratio to be quite fine-grained. It is stateless.
> 

Ok, I understand. Thanks for clarification. Now, the MPTCP stacks we have tested support different traffic scheduling schemes; some of them operate on a packet basis with various distribution algorithms, e.g. round robin, weighted round robin, giving precedence to a given access, checking QoS on accesses, etc.... I'm not a MPTCP expert and these schemes are possibly implementation specific. Anyway, AFAIU, traffic scheduling is a major work item for MPTCP guys (for Banana as well...BTW) and I think we should clarify this point with them to avoid misunderstanding.

> > do that, we must separate the problems, namely the way to establish
> > the datapath (e.g. establish GRE tunnels between aggregation entities)
> > from the way to use these datapaths (e.g. tunnels). I mean,
> > distribution and transport functions are two different functions.
> > Transport function is a matter of protocols while distribution is part
> > of the system design. Once tunnels are established I can decide to distribute
> traffic on a per-packet or per flow basis...
> > buffering, reordering are obviously mandatory when operating on a
> > per-packet basis but buffering has nothing to do with protocols...
> > what we can expect from transport protocol here, is just to add a
> > sequence number to distributed packets (to allow reordering) and that's it... from
> the protocol standpoint...
> 
> In order to demultiplex the multiple ACC devices that are connecting to the same
> AGG. We need an ID field for the bonding connection. (For example, the "Key"
> field of the GRE header can be used for this purpose.)
> 
> > > Zhang Envoyé : mercredi 21 octobre 2015 09:25 À : Alexandre
> > > Petrescu; Sri Gundavelli (sgundave); SEITE Pierrick IMT/OLN;
> > > requirements, gaps,.... But let me use some specific points, based
> > > on our discussion on the list, to sketch what we have to face in BANANA.
> > >
> > > connections in a per- packet manner. [This is use case 1.]
> > >
> > > per-packet
> > (RFC6182). So, I'm problably missing something but can you clarify
> > tour thougths?
> >
> > BBFF) will do but I think we can recommend protocols from the IETF
> > toolbox. To do that, we must separate the problems, namely the way to
> > establish the datapath (e.g. establish GRE tunnels between aggregation
> > entities) from the way to use these datapaths (e.g. tunnels). I mean,
> > distribution and transport functions are two different functions.
> > Transport function is a matter of protocols while distribution is part
> > of the system design. Once tunnels are established I can decide to distribute
> traffic on a per-packet or per flow basis...
> > buffering, reordering are obviously mandatory when operating on a
> > per-packet basis but buffering has nothing to do with protocols...
> > what we can expect from transport protocol here, is just to add a
> > sequence number to distributed packets (to allow reordering) and that's it... from
> the protocol standpoint...
> >
> > Pierrick
> >
> > > > > This is stating the obvious. Unfortunately, our starting point
> > > > > in all the discussions that I¹ve seen so far "appears" to be
> > > > > that we need to create a new protocol and a new working group.
> > > > > That¹s how I¹m reading it. But, looking at the technical
> > > > > consideration, I don¹t see any thing new.  The aspect of link
> > > > > aggregation for realizing a larger logical pipe, discussion
> > > > > around per-packet/per-flow/weighted per packet distribution or
> > > > > policy rules around path selection on a flow basic ..are all the
> > > > > topics that IETF
> > > > > for the SDO¹s to define and state a protocol requirement, or
> > > > > gaps that IETF needs to address. BBF is not willing to make a
> > > > > request to IETF to work on any problem and so I wonder what are
> > > > > we solving ? I¹m not objecting to this approach, but I don¹t
> > > > > believe we will produce any thing useful to the community. It
> > > > > will be  just re-stating the existing work as a new requirement,
> > > > > and if it goes forward, we will invent a new protocol which will
> > > > > exactly look some other existing protocol, but won¹t be
> > > > > producing any thing valuable to the industry by any
> > > > measure.
> > > >
> > > > That measure could be a criterion which tells that applications
> > > > are offered 2x more bandwidth when 2 interfaces are used.  Ideally
> > > > n more interfaces would lead to n times more bandwidth.
> > > >
> > > > same manner as Mobile IP offers mobility for any  application not just TCP).
> > > >
> > > > >>>> No doubt that agreement on building blocks is a key step for
> > > > >>>> a common system design... I couldn't agree more... But,
> > > > >>>> AFAIK, this is currently discussed in BBF; why not starting from there
> (i.e.
> > > > >>>>>>>>> definitely an important issue that falls in the problem
> > > > >>>>>>>>> space of the bonding service. In particular, the
> > > > >>>>>>>>> solution for NAT-pass-through issue is described. It
> > > > >>>>>>>>> assumes the tunneling techniques cannot support
> > > > >>>>>>>>> NAT-pass-through
> > > > >>> between the
> > > > >>>>>>> I agree the flow & congestion control functions developed
> > > > >>>>>>> in MPTCP could be reused in the bonding service architecture.
> > > > >>>>>> multiple paths can be considered in addition to the ones
> > > > >>>>>> that were considered so far in
> > > > >>>>> MPTCP.
> > > > >>>>>> value in separating the load distribution logic from the
> > > > >>>>>> actual transport mechanism. This is particularly valid for the UDP
> case.
> > > > >>>>>>
> > > > >>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> Traffic distribution issue is another important issue
> > > > >>>>>>>>> for the bonding service. This had ever been discussed on
> > > > >>>>>>>>> the list. I know MPTCP can well support flow based
> > > > >>>>>>>>> traffic
> > > > >>> distribution.
> > > > >>>>>>>> the load distribution logic can even be separated from
> > > > >>>>>>>> the actual
> > > > >>>>>>> mechanism
> > > > >>>>>>>>> A clarification questions: from Figure 1, it is clear
> > > > >>>>>>>>> that one peer of the MPTCP proxy will be deployed on the CPE.
> > > > >>>>>>>>>> We submitted this I-D to discuss a proposal that aims
> > > > >>>>>>>>>> to get rid of the encapsulation when aggregating
> > > > >>>>>>>>>> multiple network
> > > > >>>> attachments.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>>     This document specifies an MPTCP option that is
> > > > >>>>>>>>>>> used to get rid of
> > > > >>>>>>>>> an
> > > > >>>> veuillez le signaler a l'expediteur et le detruire ainsi que
> > > > >>>> les pieces jointes. Les messages electroniques etant
> > d'alteration, Orange decline toute responsabilite si ce message a ete altere,
> deforme ou falsifie. Merci.
> >