Subject: Re: [Banana] diff (banana load distribution, next-hop selection of multipath)

From: Margaret Cullen <[email-masked]>
Date: Fri, 3 Feb 2017 10:49:15 -0500

> On Dec 23, 2016, at 5:00 AM, Mingui Zhang <[email-masked]> wrote:
> 
> Hi,
> 
> It seems worthwhile to sort out the differences between the next hop selection issue in the routing area and the banana load distribution issue. 

I agree.  This is not about choosing a next hop for each flow, this is about allowing a single flow to share the bandwidth of more than one access link.
> 
> 
> An advantage over the next-hop selection is that banana offers a small reordering buffer at the egress. For this reason, a certain level of packet disorder can be absorbed so that it does not screw up the TCP. With this buffer, banana can perform per-packet load distribution, which means packets from the same flow will be transmitted over multiple different paths.

This paragraph is specific to a particular solution — the bonding tunnel solution.   And MPTCP-based solution would handle flow control and recombination differently.

Margaret

---

From: Joe Touch <[email-masked]>
Date: Fri, 3 Feb 2017 08:46:10 -0800

Hi, all,

So am I to understand that the "problem statement" is:

    - given two cooperating devices, how to manage a set of bonded links
between them?

Isn't that what routers already do?

(yes, I know that I've glossed over the fact that these "links" might be
tunnels - because we have dozens of ways of doing that already)

Joe

---

From: Mingui Zhang <[email-masked]>
Date: Sat, 4 Feb 2017 06:51:24 +0000

Hi Joe, all,

> -----Original Message-----
> From: Joe Touch [mailto:[email-masked]]
> Sent: Saturday, February 04, 2017 12:46 AM
> To: Margaret Cullen; Mingui Zhang
> Cc: [email-masked]
> Subject: Re: [Banana] diff (banana load distribution, next-hop selection of
> multipath)
> 
> Hi, all,
> 

There is one key difference. What is routers nowadays doing is per-flow load balancing using hashing rather than flow splitting across this set of bonded links and then reordering. 

Thanks,
Mingui

> 
> (yes, I know that I've glossed over the fact that these "links" might be tunnels -
> because we have dozens of ways of doing that already)
> 
> Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 3 Feb 2017 23:47:42 -0800

> On Feb 3, 2017, at 10:51 PM, Mingui Zhang <[email-masked]> wrote:
> 
> Hi Joe, all,
> 

True for many commercial routers but channel bonding isn't new. 

I'm asking what the goal here is - new channel bonding algs? Thats IRTF space. What's left?

Joe

---

From: Mingui Zhang <[email-masked]>
Date: Sat, 4 Feb 2017 08:51:17 +0000

Hi Joe,

But channel bonding is not applicable to heterogeneous links (wired & wireless). It's not applicable to tunnels which are built up on multi-hop paths either. 

So, according to my understanding the goal is to sequence incoming packets and distribute them onto multiple paths (heterogeneous yet multi-hops) in a per-packet manner and then restore their order according to their sequence number. 

Thanks,
Mingui

> Sent: Saturday, February 04, 2017 3:48 PM
> To: Mingui Zhang
> Cc: Margaret Cullen; [email-masked]
> >>    - given two cooperating devices, how to manage a set of bonded
> >> links between them?
> >>
> > There is one key difference. What is routers nowadays doing is per-flow load
> balancing using hashing rather than flow splitting across this set of bonded links
> and then reordering.
> 
> >> (yes, I know that I've glossed over the fact that these "links" might
> >> be tunnels - because we have dozens of ways of doing that already)
> >>
> >> Joe

---

From: Margaret Cullen <[email-masked]>
Date: Sat, 4 Feb 2017 12:49:31 -0500

Hi Joe,

> On Feb 3, 2017, at 11:46 AM, Joe Touch <[email-masked]> wrote:

No that is not what we are talking about here.  We are talking about distributing a single flow across non-bonded links, potentially provided by different Internet Service Providers, and recombining that traffic at a convergence point for both paths (whether that is a tunnel endpoint, or a proxy is solution specific) to arrive at the remote destination as it was originally sent by the local end-point.

> Isn't that what routers already do?

No, routers don’t do this.  Routers choose a next hop for each IP packet based on <destination IP address>, <source/dest IP addresses> or <source/dest IP addresses, transport protocol, and protocol source/dest ports>, depending on what type of router you are talking about.  

There are all sorts of devices that are routers AND include other sorts of traffic shaping or loadsharing.  There are even some routers deployed today that can share a single flow across multiple access links, recombining the traffic in a provider network.   However, there is no standard that I am aware of for doing this, so both ends need to be provided by a single vendor.

Margaret

---

From: Joe Touch <[email-masked]>
Date: Sat, 4 Feb 2017 09:54:15 -0800

On 2/4/2017 12:51 AM, Mingui Zhang wrote:
> Hi Joe,
>
> But channel bonding is not applicable to heterogeneous links (wired & wireless).
That doesn't seem like that's new - that's ECMP (where you are defining
the "cost" of the links as equivalent, even when they aren't identical).

>  It's not applicable to tunnels which are built up on multi-hop paths either. 
A tunnel is a link, when implemented properly. There should never be a
difference.

(e.g., the multihop part is exactly equivalent to multiple L2 hops)

> So, according to my understanding the goal is to sequence incoming packets and distribute them onto multiple paths (heterogeneous yet multi-hops) in a per-packet manner and then restore their order…
Sure - getting that right is an issue of picking the right algorithm,
but again that seems like an IRTF issue at this point (unless you're
just identifying an existing ECMP algorithm, at which point BANANA
should specify "no new algs").

Joe

---

From: Joe Touch <[email-masked]>
Date: Sat, 4 Feb 2017 09:56:56 -0800

On 2/4/2017 9:49 AM, Margaret Cullen wrote:
> ,,,

OK, so this would be about picking a standard for coordinating the
endpoints, tunneling with sequence numbers, and (maybe) the "ECMP"
sharing alg.

I'm just trying to scope the problem. Right now, it looks a lot like it
might also include trying to use multiple heterogeneous links without a
cooperating party upstream where the traffic merges, which is quite a
different problem.

IMO, it'd be useful to be clear about that scope.

Joe

---

From: Margaret Cullen <[email-masked]>
Date: Sat, 4 Feb 2017 13:06:17 -0500

> endpoints, tunneling with sequence numbers, and (maybe) the "ECMP"

Picking/developing one (or more) standard(s), yes.  A few different solutions have been proposed, and we quickly reviewed them during the BOF.  The two major classes of solution under discussion on the BANANA list have been bonded tunnels (with packet resequencing and flow control at the bonded tunnel endpoints), or something MPTCP Proxy-based (some with various extensions for handling UDP/QUIC traffic).

Because there are significant trade-offs between these two types of solutions that make them applicable for different scenarios, it is possible that we might end up deciding that it makes sense to standardize one of each class.  Or, maybe one of them can be improved to become the preferred solution for all scenarios.  I don’t think we know how that will come out yet.

> cooperating party upstream where the traffic merges, which is quite a

No, that has been tried in the past with very poor results.

> IMO, it'd be useful to be clear about that scope.

Thanks!  I will try to make this clearer in the problem statement.

Margaret

---

From: Margaret Cullen <[email-masked]>
Date: Sat, 4 Feb 2017 13:21:24 -0500

> On Feb 4, 2017, at 12:54 PM, Joe Touch <[email-masked]> wrote:
> 
> should specify "no new algs”).

For the purpose of this message, I am going to refer to the end points of a BANANA flow as “BANANA Boxes”.  Depending on what solution(s) we develop, these could be bonded tunnel endpoints or proxies.  I am sure we can come up with a better name for these things when we write any sort of architecture/solution documents, but I am failing to come up with a better name at the moment.

I think we could/should specify:  how the traffic is sent between the “BANANA Boxes” (addressing, ports, encapsulations, etc.), how traffic is recombined into a single flow on the remote end, and any signaling that is needed between the “BANANA Boxes” (for channel set-up, flow control, etc.).  I think those are the things that need to be standardized in order for BANANA-aware CPE from multiple vendors to talk to a centralized “BANANA Box” (or perhaps to BANANA-aware CPE boxes on the other end), which is something that real-life vendors have indicated an interest in being able to do.

Personally, I would consider algorithms that run within a single box to decide whether to aggregate a particular flow, or how to distribute traffic between the local access links, to be implementation-specific.  This is an area where one vendor could differentiate from another, by coming up with algorithms that provide better performance, faster failover, or the like.  

Margaret

---

From: Margaret Cullen <[email-masked]>
Date: Sat, 4 Feb 2017 13:27:53 -0500

> On Feb 4, 2017, at 1:21 PM, Margaret Cullen <[email-masked]> wrote:
> 
> Personally, I would consider algorithms that run within a single box to decide whether to aggregate a particular flow...

Actually, there might be a couple of exceptions to my statement that this could be implementation-specific.

We might want to specifically state how "BANANA Boxes" would detect a BANANA flow and avoid aggregating it again.  We might also want to define how they should detect some other flows that already have multi-path capability, like end-to-end MPTCP and exempt those flows from BANANA processing. 

But, I think the performance/reliability-related decisions about whether to split a given flow (that is not specifically called out in the standard) across multiple available access networks should be implementation-specific.

Does that makes sense?

Margaret

---

From: Margaret Cullen <[email-masked]>
Date: Sat, 4 Feb 2017 13:32:09 -0500

Note:  While this is interesting, we are talking about solutions requirements at this point (or perhaps charter language)…

Joe, with the explanation you’ve received, do you now think you understand the problem?  I’ll try to update the problem description to make this clearer.  Could you help to make sure my new language makes this clear?

Thanks,
Margaret

> On Feb 4, 2017, at 1:27 PM, Margaret Cullen <[email-masked]> wrote:
> 
> 
> _______________________________________________
> Banana mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/banana

---

From: Joe Touch <[email-masked]>
Date: Sat, 4 Feb 2017 10:37:41 -0800

On 2/4/2017 10:32 AM, Margaret Cullen wrote:
> Joe, with the explanation you’ve received, do you now think you understand the problem? 
Yes, it's much more clear now.
>  I’ll try to update the problem description to make this clearer.  Could you help to make sure my new language makes this clear?
Sure - send an update off-list and I'll be happy to check.

Thanks,
Joe

---

From: "Zuojing (2012 Laboratories)" <[email-masked]>
Date: Fri, 10 Feb 2017 02:10:27 +0000

Dear Margaret and all,

It's true that we need to exempt end-to-end MPTCP flows from BANANA processing in case the sub-flows of MPTCP are split by BANANA once again. 

I find we exposed a promising item that deserves our effort: we may consider how to integrate end-to-end MPTCP with BANANA. For example, the IP addresses used as tunnel endpoints could be reused by the MPTCP stack hence the address exchanging time introduced by the MPTCP stack could be reduced. I remember this kind of transition strategy was somehow touched upon in the BOF discussion.

BR,
Jing

-----邮件原件-----
发件人: Banana [mailto:[email-masked]] 代表 Margaret Cullen
发送时间: 2017年2月5日 2:28
收件人: Joe Touch
抄送: Mingui Zhang; [email-masked]
主题: Re: [Banana] diff (banana load distribution, next-hop selection of multipath)

Does that makes sense?

Margaret

---

From: Joe Touch <[email-masked]>
Date: Fri, 10 Feb 2017 09:41:33 -0800

Hi, Jing (et al.),

> It's true that we need to exempt end-to-end MPTCP flows from BANANA processing in case the sub-flows of MPTCP are split by BANANA once again.

I disagree.

MPTCP flows are no different than a single TCP connection; both assume a
single set of "path" properties, and both could be affected by use of
multipath.

However, both may need to use multipath to have the resources needed,
e.g., to complete a transfer quickly.

AFAICT, both should be treated the same for the purposes of BANANA, and
neither one should have a blanket exemption.

Joe

---

From: "Zuojing (2012 Laboratories)" <[email-masked]>
Date: Mon, 13 Feb 2017 09:23:59 +0000

Hi Joe,

Right. BANANA needs not to distinguish subflows from TCP flows only if Service Providers require BANANA boxes to do such exemption For example, Service Providers may purposely let those flows already being conducted by other multipath technologies be “bypassed”. 

There is a scenario that comes into my mind: the BANANA box may simultaneously be an MPTCP proxy device. The MPTCP proxy function handles the TCP flow and generates MPTCP subflows. In this case, the generated subflows should not to go through the traffic distribution function of BANANA once again. 

I can imagine another scenario: BANANA is supported by the end device. Assume this end device supports MPTCP. It should exempt its MPTCP subflows from being processed by the traffic distribution function of BANANA.

BR, 
Jing

-----邮件原件-----
发件人: Joe Touch [mailto:[email-masked]] 
发送时间: 2017年2月11日 1:42
收件人: Zuojing (2012 Laboratories); Margaret Cullen
抄送: Mingui Zhang; [email-masked]
主题: Re: [Banana] diff (banana load distribution, next-hop selection of multipath)

Hi, Jing (et al.),

I disagree.

MPTCP flows are no different than a single TCP connection; both assume a single set of "path" properties, and both could be affected by use of multipath.

However, both may need to use multipath to have the resources needed, e.g., to complete a transfer quickly.

AFAICT, both should be treated the same for the purposes of BANANA, and neither one should have a blanket exemption.

Joe

---

From: Jim Reid <[email-masked]>
Date: Mon, 13 Feb 2017 09:32:05 +0000

> On 13 Feb 2017, at 09:23, Zuojing (2012 Laboratories) <[email-masked]> wrote:
> 
> BANANA needs not to distinguish subflows from TCP flows only if Service Providers require BANANA boxes to do such exemption For example, Service Providers may purposely let those flows already being conducted by other multipath technologies be “bypassed”. 

Surely these sorts of issues are for the WG to discuss once it has been created? They shouldn’t be relevant to the matter at hand, namely the new WG’s charter.

---

From: Michael Menth <[email-masked]>
Date: Mon, 13 Feb 2017 10:48:06 +0100

Hi Zuojing,

interesting concept. However, I guess it will face at least two
challenges: detection of mptcp flows and overall state maintenance.
Whatever banana does, it must be simple as one major goal is high
bandwidth. Cooperation/coexistence/transparency of mptcp and banana is
certainly an issue to be clarified.

Regards,

Michael

-- 
Prof. Dr. habil. Michael Menth
University of Tuebingen
Faculty of Science
Department of Computer Science
Chair of Communication Networks
Sand 13, 72076 Tuebingen, Germany
phone: (+49)-7071/29-70505
fax: (+49)-7071/29-5220
mailto:[email-masked]
http://kn.inf.uni-tuebingen.de

---

From: Joe Touch <[email-masked]>
Date: Mon, 13 Feb 2017 09:47:00 -0800

+1

I raised the point more to indicate that MPTCP is not relevant to the
charter.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 13 Feb 2017 09:52:26 -0800

First: I agree with Jim that this is out of scope for the charter.

Second: to queue up some observations IF MPTCP handling is part of a
solution (which I remain unconvinced is necessary):

On 2/13/2017 1:23 AM, Zuojing (2012 Laboratories) wrote:
> Hi Joe,
>
> Right. BANANA needs not to distinguish subflows from TCP flows only if Service Providers require BANANA boxes to do such exemption
Which I believe should NOT be required or even suggested...

>  For example, Service Providers may purposely let those flows already being conducted by other multipath technologies be “bypassed”. 
Again, an MPTCP subflow should be treated identically to a TCP flow. If
TCP flows can be split across connections, then so can MPTCP subflows.
There is no reason to treat them differently.

> There is a scenario that comes into my mind: the BANANA box may simultaneously be an MPTCP proxy device.
The MPTCP proxy rules would govern its behavior, not BANANA (IMO).

>  The MPTCP proxy function handles the TCP flow and generates MPTCP subflows. In this case, the generated subflows should not to go through the traffic distribution function of BANANA once again. 
>
> I can imagine another scenario: BANANA is supported by the end device. Assume this end device supports MPTCP.

Again, the MPTCP system needs to be the one to accommodate things, not
BANANA (IMO).

Joe

>  It should exempt its MPTCP subflows from being processed by the traffic distribution function of BANANA.
>
> BR, 
> Jing
>
> -----邮件原件-----