Subject: Re: [Banana] BANANA BOF in Prague?

From: "Zhangmingui (Martin)" <[email-masked]>
Date: Wed, 28 Jun 2017 10:01:11 +0000

Hi Dave,

The ECN capability is a good thing for bonding tunnels. The ECN capability for a specific tunnel has been specified by 3.3.2 of a recently updated draft-ietf-tsvwg-rfc6040update-shim-03. Basically, the ECN bits will be carried within the outer IP header that is immediately visible to the forwarding element [RFC6040]. BANANA boxes may support this capability as well if each tunnel supports it. 

There is one additional thing can be done by this group. 

   "GRE tunnels do not support dynamic configuration based on capability
   negotiation, so the ECN capability has to be manually configured,
   which is specified in Section 4.3 of RFC 6040."

With the signaling protocol, this capability can also be automatically negotiated by the peering BANANA boxes other than manually configured. 

Thanks,
Mingui

> -----Original Message-----
> From: Banana [mailto:[email-masked]] On Behalf Of Dave Dolson
> Sent: Tuesday, May 23, 2017 10:45 PM
> To: Margaret Cullen; [email-masked]
> Subject: Re: [Banana] BANANA BOF in Prague?
> 
> Margaret,
> Thanks for submitting the request.
> I think the charter needs to include scope for the algorithm deciding what
> proportion of traffic the sender must put on each link.
> 
> In my mind, the sender must model the congestion window for each link, and
> therefore know the loss (or ECN) experienced on each link.
> There may be other approaches. But I don't think this can be left to the
> implementers.
> 
> -Dave
> 
> 
> 
> 
> -----Original Message-----
> From: Banana [mailto:[email-masked]] On Behalf Of Margaret Cullen
> Sent: Monday, May 22, 2017 8:11 PM
> To: [email-masked]
> Subject: [Banana] BANANA BOF in Prague?
> 
> Just FYI —
> 
> I have submitted a request for a WG Forming BANANA BOF in Prague with the
> information included below.  This information has also been added to the BOF
> Wiki.
> 
> Please let me know if you notice any errors or omissions, or if you have any
> questions or suggestions.
> 
> Thanks,
> Margaret
> 
> BOF Name:  Bandwidth Aggregation for Network Access (BANANA)
> 
> Description:
> 
> Bandwith Aggregation consists of splitting local traffic across multiple Internet
> links on a per-packet basis, including the ability to split a single flow across
> multiple links when necessary.
> 
> It is the goal of this WG to produce a Bandwidth Aggregation solution that will
> provide the following benefits:
> 
> - Higher Per-Flow Bandwidth: Many Internet links available to homes
>   and small offices (DSL, Cable, LTE, Satellite, etc.) have relatively
>   low bandwidth.  Users may wish to run applications (such as
>   streaming video, or content up/downloads) that require (or could
>   benefit from) more bandwidth for a single traffic flow than is
>   available on any of the local links.  A Bandwidth Aggregation
>   solution could supply the needed bandwidth by splitting a single
>   traffic flow across multiple Internet links.
> 
> - Reduced Cost: Traffic sharing on a per-packet basis allows the full
>   bandwidth of the lowest-cost link to be used first, only using a
>   higher-cost link when the lowest-cost link is full.
> 
> - Increased Reliability: When one Internet link goes down, ongoing
>   application flows can be moved to another link, preventing service
>   disruption.
> 
> Agenda
> - Agenda bash, scribe, minute taker [5min]
> - Review of proposed charter text (see below) [10 mins]
> - Charter discussion [45 mins]
> - Questions: [30 mins]
>     - Is the charter text clear and understandable?
>     - Should the IETF do this work?
>     - Are you willing to contribute (write, review, email, etc.)
> 
> Status: WG Forming
> Responsible AD: Suresh Krishnan
> BoF proponents: Margaret Cullen / Brian Trammel / Mingui Zhang BoF chairs:
> Margaret Cullen / Brian Trammel Number of people expected to attend: 100
> Length of session (1, 1.5, 2, or 2.5 hours): 1.5 hours Conflicts to avoid (whole
> Areas and/or WGs): Internet area, Homenet, TRILL, MPTCP, QUIC
> 
> -     Mailing List: ​https://www.ietf.org/mailman/listinfo/banana
> -     Draft charter: see below
> -     Relevant drafts: [[BR]]
> https://datatracker.ietf.org/doc/draft-leymann-banana-signalling
> https://datatracker.ietf.org/doc/draft-leymann-banana-data-encap
> https://datatracker.ietf.org/doc/draft-leymann-banana-integration
> https://datatracker.ietf.org/doc/draft-leymann-banana-discard-load-rebalance
> https://datatracker.ietf.org/doc/draft-leymann-banana-discard-timer
> https://tools.ietf.org/html/draft-kanugovi-intarea-mams-protocol-04
> https://tools.ietf.org/html/draft-zhu-intarea-mams-control-protocol-01
> 
> 
> Proposed Charter Text and Milestones:
> 
> The BANdwidth Aggregation for Network Access (BANANA) Working Group is
> chartered to develop solution(s) to support dynamic path selection on a
> per-packet basis in networks that have more than one point of attachment to
> the Internet.
> 
> Bandwith Aggregation consists of splitting local traffic across multiple Internet
> links on a per-packet basis, including the ability to split a single flow across
> multiple links when necessary.
> 
> It is the goal of this WG to produce a Bandwidth Aggregation solution that will
> provide the following benefits:
> 
> - Higher Per-Flow Bandwidth: Many Internet links available to homes
>   and small offices (DSL, Cable, LTE, Satellite, etc.) have relatively
>   low bandwidth.  Users may wish to run applications (such as
>   streaming video, or content up/downloads) that require (or could
>   benefit from) more bandwidth for a single traffic flow than is
>   available on any of the local links.  A Bandwidth Aggregation
>   solution could supply the needed bandwidth by splitting a single
>   traffic flow across multiple Internet links.
> 
> - Reduced Cost: Traffic sharing on a per-packet basis allows the full
>   bandwidth of the lowest-cost link to be used first, only using a
>   higher-cost link when the lowest-cost link is full.
> 
> - Increased Reliability: When one Internet link goes down, ongoing
>   application flows can be moved to another link, preventing service
>   disruption.
> 
> Proposed BANANA solutions use different approaches (e.g. tunnels, proxies,
> etc.) to split and recombine traffic, but at an abstract level, they involve a local
> (hardware or software) component on the multi-access network, a remote
> component within the Internet, and mechanisms for those components to find
> each other, exchange signalling information, and direct traffic to each other.
> We refer to these functional components as the Local and Remote "BANANA
> Boxes", and we refer to the method they use to direct traffic to each other as a
> "BANANA Encapsulation".
> 
> The Bandwidth Aggregation solutions developed in this group will work
> whether the attached links are provided by a single Internet Service Provider or
> multiple Providers.
> 
> The BANANA WG will have the following work items:
> 
> - Determine how Local and Remote BANANA Boxes find each other.
> 
> - Specify a signalling protocol that can be used to send configuration
>   and control information between BANANA boxes, including:
>     -  IP Prefixes of local links
>     -  Information about link properties & status
>     -  Information needed by the encapsulations
> 
> - Select (and extend, if necessary) an existing tunneling
>   encapsulation for sending traffic between BANANA Boxes.
> 
> - Work with other IETF WGs defining BANANA encapsulations
>   (if any) to ensure that the discovery mechanism and signalling
>   protocol will meet their needs.
> 
> BANANA Boxes will determine if a specific flow is eligible for Bandwith
> Aggregation. If a flow is not eligible, it will not be split across multiple attached
> links.
> 
> For this initial charter, we will focus on how Local BANANA Boxes communicate
> with Remote BANANA Boxes.  We will not address the topic of cooperation
> between multiple Local BANANA Boxes.
> 
> MILESTONES
> Apr 2018 Adopt WG draft for discovery/configuration mechanism Apr 2018
> Adopt WG draft for signalling proocol Apr 2018 Adopt WG draft for tunnel
> encapsulation Feb 2019 WGLC on discovery/configuration mechanism Feb 2019
> WGLC on signalling protocol Feb 2019 WGLC on tunnel encapsulation Aug 2019
> Send discovery/configuration mechanism to the IESG Aug 2019 Send signalling
> protocl to the IESG Aug 2019 Send tunnel encapsulation to the IESG
> 
> 
> _______________________________________________
> Banana mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/banana
> _______________________________________________
> Banana mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/banana

---

From: Dave Dolson <[email-masked]>
Date: Wed, 28 Jun 2017 11:41:50 +0000

Mingui,
Negotiation? Is it not reasonable to require support at both BANANA tunnel end-points?

David Dolson
Sandvine
  Original Message
From: Zhangmingui (Martin)
Sent: Wednesday, June 28, 2017 6:01 AM
To: Dave Dolson; Margaret Cullen; [email-masked]
Subject: RE: [Banana] BANANA BOF in Prague?

Hi Dave,

Thanks,
Mingui

---

From: "Zhangmingui (Martin)" <[email-masked]>
Date: Thu, 29 Jun 2017 02:51:06 +0000

Hi Dave,

For bonding tunnels to be specified by the BANANA group we may well make this feature be mandatory. So I think you propose a fair point. 

That said, a BANANA box might communicate with a legacy box. For this scenario, negotiation seems still necessary.

Thanks,
Mingui 

> From: Dave Dolson [mailto:[email-masked]]
> Sent: Wednesday, June 28, 2017 7:42 PM
> To: Zhangmingui (Martin); Margaret Cullen; [email-masked]
> The ECN capability is a good thing for bonding tunnels. The ECN capability for a
> specific tunnel has been specified by 3.3.2 of a recently updated
> draft-ietf-tsvwg-rfc6040update-shim-03. Basically, the ECN bits will be carried
> within the outer IP header that is immediately visible to the forwarding
> element [RFC6040]. BANANA boxes may support this capability as well if each
> tunnel supports it.
> 
> With the signaling protocol, this capability can also be automatically negotiated
> by the peering BANANA boxes other than manually configured.
> 
> Thanks,
> Mingui
> 
> > I think the charter needs to include scope for the algorithm deciding
> > what proportion of traffic the sender must put on each link.
> >
> > In my mind, the sender must model the congestion window for each link,
> > and therefore know the loss (or ECN) experienced on each link.
> > There may be other approaches. But I don't think this can be left to
> > the implementers.
> >
> > -Dave
> >
> >
> >
> >
> > I have submitted a request for a WG Forming BANANA BOF in Prague with
> > the information included below.  This information has also been added
> > to the BOF Wiki.
> >
> > Please let me know if you notice any errors or omissions, or if you
> > have any questions or suggestions.
> >
> > Thanks,
> > Margaret
> >
> > Bandwith Aggregation consists of splitting local traffic across
> > multiple Internet links on a per-packet basis, including the ability
> > to split a single flow across multiple links when necessary.
> >
> > It is the goal of this WG to produce a Bandwidth Aggregation solution
> > that will provide the following benefits:
> >
> > Margaret Cullen / Brian Trammel Number of people expected to attend:
> > 100 Length of session (1, 1.5, 2, or 2.5 hours): 1.5 hours Conflicts
> > to avoid (whole Areas and/or WGs): Internet area, Homenet, TRILL,
> > MPTCP, QUIC
> >
> > chartered to develop solution(s) to support dynamic path selection on
> > a per-packet basis in networks that have more than one point of
> > attachment to the Internet.
> >
> > Bandwith Aggregation consists of splitting local traffic across
> > multiple Internet links on a per-packet basis, including the ability
> > to split a single flow across multiple links when necessary.
> >
> > It is the goal of this WG to produce a Bandwidth Aggregation solution
> > that will provide the following benefits:
> >
> > etc.) to split and recombine traffic, but at an abstract level, they
> > involve a local (hardware or software) component on the multi-access
> > network, a remote component within the Internet, and mechanisms for
> > those components to find each other, exchange signalling information, and
> direct traffic to each other.
> > We refer to these functional components as the Local and Remote
> > "BANANA Boxes", and we refer to the method they use to direct traffic
> > to each other as a "BANANA Encapsulation".
> >
> > whether the attached links are provided by a single Internet Service
> > Provider or multiple Providers.
> >
> > BANANA Boxes will determine if a specific flow is eligible for
> > Bandwith Aggregation. If a flow is not eligible, it will not be split
> > across multiple attached links.
> >
> > For this initial charter, we will focus on how Local BANANA Boxes
> > communicate with Remote BANANA Boxes.  We will not address the topic
> > of cooperation between multiple Local BANANA Boxes.
> >
> > MILESTONES
> > Adopt WG draft for signalling proocol Apr 2018 Adopt WG draft for
> > tunnel encapsulation Feb 2019 WGLC on discovery/configuration
> > mechanism Feb 2019 WGLC on signalling protocol Feb 2019 WGLC on tunnel
> > encapsulation Aug 2019 Send discovery/configuration mechanism to the
> > IESG Aug 2019 Send signalling protocl to the IESG Aug 2019 Send tunnel
> > encapsulation to the IESG
> >
> >

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Thu, 29 Jun 2017 13:06:10 +0200

There's also an issue of the network carrying the tunnel having impaired support for the ECN codepoints -- this is often because of networks configured to use the entire ToS byte for forwarding decisions, rather than the 6 dscp bits. So even if both BANANA endpoints support ECN, they might need to do cooperative network capability sensing to deal with these networks. Though this isn't negotiation per se, it uses equivalent mechanisms.

Cheers,

Brian (as individual)

> On 29 Jun 2017, at 04:51, Zhangmingui (Martin) <[email-masked]> wrote:
> 
> Hi Dave,
>

---

From: "Zhangmingui (Martin)" <[email-masked]>
Date: Fri, 30 Jun 2017 08:58:32 +0000

Right. Only if all routers carrying the tunnel support ECN. This is a prerequisite.

One issue that might be also interesting: who is the "sender"? We know ECN is to indicate the status of the path so that the TCP sender, which is actually the host, can adjust its sending rate accordingly. But, the sender mentioned here is one of the two BANANA boxes. The load balancer of the BANANA box can determine proportion of the traffic transmitted on each tunnel rather that the overall traffic sending rate. The delivery and operations with the ECN bits are protocol issues while the load balancing strategy based on the ECN bits is local and seems more of an implementation issue. 

Thanks,
Mingui

> From: Brian Trammell (IETF) [mailto:[email-masked]]
> Sent: Thursday, June 29, 2017 7:06 PM
> To: Zhangmingui (Martin)
> Cc: Dave Dolson; Margaret Cullen; [email-masked]
> There's also an issue of the network carrying the tunnel having impaired
> support for the ECN codepoints -- this is often because of networks configured
> to use the entire ToS byte for forwarding decisions, rather than the 6 dscp bits.
> So even if both BANANA endpoints support ECN, they might need to do
> cooperative network capability sensing to deal with these networks. Though
> this isn't negotiation per se, it uses equivalent mechanisms.
> 
> Cheers,
> 
> Brian (as individual)
> 
> 
> > For bonding tunnels to be specified by the BANANA group we may well make
> this feature be mandatory. So I think you propose a fair point.
> >
> > That said, a BANANA box might communicate with a legacy box. For this
> scenario, negotiation seems still necessary.
> >
> > Thanks,
> > Mingui
> >
> >> The ECN capability is a good thing for bonding tunnels. The ECN
> >> capability for a specific tunnel has been specified by 3.3.2 of a
> >> recently updated draft-ietf-tsvwg-rfc6040update-shim-03. Basically,
> >> the ECN bits will be carried within the outer IP header that is
> >> immediately visible to the forwarding element [RFC6040]. BANANA boxes
> >> may support this capability as well if each tunnel supports it.
> >>
> >> With the signaling protocol, this capability can also be
> >> automatically negotiated by the peering BANANA boxes other than manually
> configured.
> >>
> >> Thanks,
> >> Mingui
> >>
> >>> I think the charter needs to include scope for the algorithm
> >>> deciding what proportion of traffic the sender must put on each link.
> >>>
> >>> In my mind, the sender must model the congestion window for each
> >>> link, and therefore know the loss (or ECN) experienced on each link.
> >>> I have submitted a request for a WG Forming BANANA BOF in Prague
> >>> with the information included below.  This information has also been
> >>> added to the BOF Wiki.
> >>>
> >>> It is the goal of this WG to produce a Bandwidth Aggregation
> >>> solution that will provide the following benefits:
> >>>
> >>> - Reduced Cost: Traffic sharing on a per-packet basis allows the
> >>> full  bandwidth of the lowest-cost link to be used first, only using
> >>> a  higher-cost link when the lowest-cost link is full.
> >>>
> >>> https://datatracker.ietf.org/doc/draft-leymann-banana-discard-load-r
> >>> eb
> >>> alance
> >>> The BANdwidth Aggregation for Network Access (BANANA) Working Group
> >>> is chartered to develop solution(s) to support dynamic path
> >>> selection on a per-packet basis in networks that have more than one
> >>> point of attachment to the Internet.
> >>>
> >>> It is the goal of this WG to produce a Bandwidth Aggregation
> >>> solution that will provide the following benefits:
> >>>
> >>> - Reduced Cost: Traffic sharing on a per-packet basis allows the
> >>> full  bandwidth of the lowest-cost link to be used first, only using
> >>> a  higher-cost link when the lowest-cost link is full.
> >>>
> >>> "BANANA Boxes", and we refer to the method they use to direct
> >>> traffic to each other as a "BANANA Encapsulation".
> >>>
> >>> The Bandwidth Aggregation solutions developed in this group will
> >>> work whether the attached links are provided by a single Internet
> >>> Service Provider or multiple Providers.
> >>>
> >>> - Specify a signalling protocol that can be used to send
> >>> configuration  and control information between BANANA boxes, including:
> >>> - Work with other IETF WGs defining BANANA encapsulations  (if any)
> >>> to ensure that the discovery mechanism and signalling  protocol will
> >>> meet their needs.
> >>>
> >>> Bandwith Aggregation. If a flow is not eligible, it will not be
> >>> split across multiple attached links.
> >>>
> >>> Apr 2018 Adopt WG draft for discovery/configuration mechanism Apr
> >>> 2018 Adopt WG draft for signalling proocol Apr 2018 Adopt WG draft
> >>> for tunnel encapsulation Feb 2019 WGLC on discovery/configuration
> >>> mechanism Feb 2019 WGLC on signalling protocol Feb 2019 WGLC on
> >>> tunnel encapsulation Aug 2019 Send discovery/configuration mechanism
> >>> to the IESG Aug 2019 Send signalling protocl to the IESG Aug 2019
> >>> Send tunnel encapsulation to the IESG
> >>>
> >>>

---

From: Dave Dolson <[email-masked]>
Date: Fri, 30 Jun 2017 11:58:58 +0000

I don't think that's true that all (or any) routers must support ECN in order for ECN to be used. I think the only requirement is that the routers do not bleach or use the ECN bits for something else.

The Congestion Experienced signal needs to be propagated to the actual sender of the traffic. The BANANA boxes also need to observe the Congestion Experienced (and packet loss) ‎per path, to determine the relative capacities of each path.

That's how it seems to me anyhow.

David Dolson
Sandvine
  Original Message
From: Zhangmingui (Martin)
Sent: Friday, June 30, 2017 4:58 AM
To: Brian Trammell (IETF)
Cc: Margaret Cullen; [email-masked]; Dave Dolson
Subject: Re: [Banana] BANANA BOF in Prague?

Thanks,
Mingui

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Fri, 30 Jun 2017 16:58:28 +0200

hi Dave,

Yeah, "support" is a ternary state at layer 3:

(1) runs AQM and marks CE on ECT-enabled traffic according to AQM algorithm

(2) leaves IP ECN bits alone

(3) bleaches IP ECN bits, or drops packets with other than the non-ECN codepoint.

Capability sensing involves ensuring that the packets carrying the BANANA tunnel aren't in state 3. States 1 and 2 are, from the capability sensing standpoint, equivalent.

Cheers,

Brian

> On 30 Jun 2017, at 13:58, Dave Dolson <[email-masked]> wrote:
>

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 30 Jun 2017 18:08:06 +0200

Actually, you can potentially use ECN for the tunnel only, as long as you have a way to feedback the congestion information to the tunnel ingress and the ingress node has a way to react to the feedback appropriately, e.g. moving traffic from the congestion path to another path. If you use this for the tunnel, you probably even don’t want the endpoint to react as well because you just moved to a bigger link and then the endpoint reducing it’s rate doesn’t make any sense anymore. However, I case all available link are congestion and you can’t react at the ingress anymore, you need to provide feedback to the sending endpoint. In this case the tunnel ingress could mark the inner IP packets if ECN-enabled or drop some packets. So it’s not super easy but probably worth looking into. 

Mirja

> Am 30.06.2017 um 10:58 schrieb Zhangmingui (Martin) <[email-masked]>:
>