Subject: Re: [Banana] Updated Charter

From: "Zhangmingui (Martin)" <[email-masked]>
Date: Sun, 1 Oct 2017 10:06:05 +0000

Hi Sri,

So you are suggesting to combine several control protocols with this or that feature in order to meet the requirements? I don't believe this strategy works. Moreover, these requirements are not exhaustive. For example, the ECN feature recently poped up. The realistic way is to consider these features in one control protocol.

Please see my comments inline below.

________________________________

From: Sri Gundavelli (sgundave) [[email-masked]]
Sent: Saturday, September 30, 2017 13:27
To: Zhangmingui (Martin); Margaret Cullen
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Banana] Updated Charter

Hi Mingui,

You talked about three new technical requirements for “Banana” solution. I thought I will provide my comments on those three requirements and how they are realized today. See inline …

From: "Zhangmingui (Martin)" <[email-masked]<mailto:[email-masked]>>
Date: Friday, September 29, 2017 at 4:49 AM
To: Margaret Cullen <[email-masked]<mailto:[email-masked]>>, Microsoft Office User <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: [Banana] Updated Charter

Hi Margaret,

> [With the caveat that I don’t think anything should be DSL/LTE-specific, though, or assume that there are exactly two links, or that there is only one “N2” that can reach a given end-node]

This is a good point. It’s common that providers use multiple aggregation BANANA boxes to achieve load balance. N2 would be a branch router (not the aggregation BANANA box) that adopts anycast mechanism. Suppose the aggregation BANANA boxes behind N2 are N3 and N4, then the tunnel endpoints would be N1-N3 or N1-N4 rather than N1-N2. For this reason, N3 needs to let N1 know he is the real tunnel endpoint. To me, this feature has to be realized by a BANANA control protocol. DNS does not support this feature.

[Sri] Your ingress prefix comes from an anchor. Now, if you have 10 remote sites with an anchor in each site and if you want the CPE to use all the anchors, you need prefixes from each of those anchors. So, its not clear how the source address selection works, or many prefixes will you host on the network behind the CPE. May be you meant some thing else, but this requirement is not clear to me.

[Mingui] To make it clear, please take a look at the following figure. The real tunnel endpoint is H instead of G. This anycast mechanism was presented in the BOF of IETF 97. You can see the IP address (G) given by the DNS server will not be used as the tunnel endpoint's IP address.  The IP address (H) of the tunnel endpoint is not exposed to the CPE till the request from the CPE is received.

But, if you are pointing to an anchor-less model, probably LISP is the protocol for you. There, encap selection is based on destination address.  LISP has no concept of an anchor, but uses tunnels and encap selection is based on destination address. LISP  also has multipath support and so that would be a great protocol for this solution

[Mingui] LISP will not be chosen unless LISP experts would like to write a draft to clearly specify how it works for BANANA. Except the multipath and anchor-less feature, there is a large gap to be filled.

We can also identify several other features that require BANANA control protocol(s).
Bypass is another important feature that requires BANANA control protocol(s). Basically, this requirement requires the two BANANA boxes to use the control protocol to timely exchange the traffic types that need to bypass the two bonding tunnels.

[Sri] This property is there in mobile architecture and typically referred to as SIPTO. Look at RFC 6909 (https://tools.ietf.org/html/rfc6909)
is a good starting point.  Note that the ingress prefixes are topologically anchored on the anchor and so for that traffic to skip the tunnel, they need to be NAT translated.

[Mingui] It's a no-doubt thing that RFC6909 can be a good starting point. If you would like to tweak it to meet the bypass requirement of BANANA, that would be great. For exmaple, you can resolve how to make it also works for IPv6 traffic and how to offload a flow from a set of tunnels. A new draft is needed.

Providers want performance measurement probes to be transmitted along the same path as data packets. These probes will be defined as control messages. Think about the above anycast mechanism. Providers do not expect these probes to be exchanged between N1 and N2. In order to make the probes be exchanged between N1 and N3 (or N4), these control messages (better all the control messages) need to take the same tunnels as data packets. I do not know an existing standard control protocol that meets this requirement and can be used in the BANANA scenario.

If the quality of one of the tunnels downgrades below a certain level (e.g., the measured RTT exceeds 300ms), the two BANANA boxes have to agree on to disable this tunnel instantly while remain the other tunnel. This action has to be executed relying on the control protocol. Is there an existing standard control protocol already supports this feature. I don’t think so.

[Sri] As Florin pointed out, there are many performance measurement tools/probes that are out there.  IPPM guys have published a number of RFC’s in this space.

[Mingui] Of course I know there are lots of measurement tools, but the point is not about the measurement itself.  The first point is that the probes have to be delivered "in-band" with data packets. And, guarantee them to take exactly the same data path as data packets. The second point is that the disabling action has to be conducted by the control protocol.

https://datatracker.ietf.org/wg/ippm/documents/

These tools can measure various aspects at a very granular level. You can send probes over L3/L4  tunnels, or non-tunnel path. You can send probes of certain protocol type, frequency, with certain DSCP markings, certain packet sizes .. and you can measure various performance aspects (jitter, MOS, packet loss …etc). You can configure these tools independent of the multipath feature, but the reports from these measurements influence the protocol states of other protocols.

You may want the tunnel to be torn down when the latency exceeds 300 milli-secs, but in some other deployment they want the tunnel to be torn down when the voice MOS measurement falls below certain value. So, this is not about a “standard", its more about configuration and a deployment knob.

[Mingui] No, it's not a torn down. If you can imagine, this tunnel will be enabled again after its quality is fine again. All these actions need to be handled by the control protocol.

We do use these tools in cisco products supporting hybrid access solutions.

[Mingui] I do not doubt those tools are being used in your products. But I cannot believe hybrid access solutions are supported by using those tools alreay. There are clear gaps.  I remember you specified your proposal to target hybrid access in the following draft.  It does contain extensions to MIP.
https://www.ietf.org/archive/id/draft-seite-dmm-rg-multihoming-00.txt

[Mingui] The final BANANA charter says "Select or specify protocol(s) that can be used to send control information between BANANA Boxes". Through the discussions over the last two weeks, we can sense MIP is a good candidate. Sincerely hope you can write a draft to make people be clear how the MIP can be extended to be used between BANANA boxes.

Thanks,
Mingui

From: Banana [mailto:[email-masked]] On Behalf Of Margaret Cullen
Sent: Friday, September 29, 2017 2:49 AM
To: Sri Gundavelli (sgundave)
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Banana] Updated Charter

Hi Sri,

On Sep 28, 2017, at 12:18 PM, Sri Gundavelli (sgundave) <[email-masked]<mailto:[email-masked]>> wrote:

We cannot solve market fragmentation issue around protocol use and at least that cannot be the reason for defining a new protocol, or WG creation.  If that is the PS, Wim’s suggestion to focus on an information elements is a great proposal to get all solutions towards a common approach, but not by mandating a specific protocol use.

The point of standardization, at least IETF standardization, is so that there can be products from multiple vendors that _interoperate_.  In my opinion, this is a case where interoperability would be particularly desirable, especially for situations where bandwidth aggregation is performed over-the-top (as Dave Sinicrope would say), rather than terminating in an operator's network.  The good news is that we already have multiple options for bandwidth aggregation out there, with running code, and they work.  The downside is that they are all proprietary, so they don’t work _together_.  In your terminology, you can only get the benefits of bandwidth aggregation if N1 and N2 were bought from the same vendor, and the goal of this WG would be to fix that.

N1 wants to discover N2; It can be based on DNS FQDN, DNS SRV lookup, DHCP option, an attribute in AAA that comes to the device as part of the access authentication, or a static IP address configured locally.

Most of these choices require some sort of standards development work:  a DHCP option, defining a AAA attribute, defining a new DNS SRV type, etc…  None of these things can just magically be used to find “N2” in an interoperable fashion, without some standards work.  Having 6 options is about as useful, for interoperability, as having no option at all.  So, it would be good to define one way that N1 finds N2 (or a limited set of ways, and an order for trying them), so that boxes from different vendors can actually find each other without users having to populate multiple databases, deal with multiple boxes that do different things with the same configuration information,  or configure the same information into half-a-dozen places.

N1 wants to register with N2. N1 wants to presents its identity, authorize it self. N2 needs to use the identify and check AAA/local DB to authorize N1

Or it could use TLS, or DTLS, or…  There are numerous ways to authenticate between boxes, and we won’t get any interoperability until we pick exactly one.  If we wanted to use AAA (RADIUS or Diameter), we would also (at the very least) need to define a service name, and agree to at least one “mandatory to implement” method.  AAA might not be the best choice, though, given that we probably don’t want to limit bandwidth aggregation to the case where N1 and N2 have a single authority or overarching federation to run the AAA infrastructure.  So, a certificate-based or PKI system might be better?

N1 wants some flexible way to generate its identify, based on access network, hardware identifiers ..etc
N1 wants to register its reachability with N2. I have IP1 from LTE and IP2 from DSL
N1 wants a ask N2 for a set of prefixes for its ingress network from the anchor. It can obtain/register/negotiate Prefix P1, P2 and P3.
N1 wants an overlay tunnel to N2 over LTE. and another tunnel over DSL So, N2 can forward the traffic bound to N1’s ingress prefixes over an overlay path. Hiding the ingress prefix traffic from the transport topology
N1/N2 wants to continuously check path-reachability/peer-loss over DSL and also over LTE
[…]

These are the sorts of things that require the exchange of control information.  [With the caveat that I don’t think anything should be DSL/LTE-specific, though, or assume that there are exactly two links, or that there is only one “N2” that can reach a given end-node]  Perhaps you are saying that MIP already has a control-information-exchange mechanism that we could use for this?  If so, then great, you should propose it, and we should consider it!  We might need to extend a MIP control-information-exchange to include the information that is needed for per-packet load-balancing and recombination.  If we also used MIP as the tunnel-based Bandwidth Aggregation mechanism, we might also need to extend the MIP tunnel support to include per-packet load balancing and recombination functions — perhaps adding a packet sequence number, for example.  If the proposed BANANA WG decides this is the best approach, perhaps we could work with the DMM WG to standardize those extensions?  There may be other protocols we should consider for this role, though, that don’t have existing IETF WGs.  In those cases, we might do the extensions in the proposed BANANA WG.

I would also add something that you didn’t include:

There might be more than one Bandwidth Aggregation mechanism available on N1.  For example, we might have an MPTCP-Proxy-based mechanism available that can be used for Bandwidth Aggregation of TCP traffic, and a tunneling mechanism that can handle any IP traffic.  N1 might want to know which mechanisms are available on N2 (or potentially on N3 or N4 if they can be used to reach the same end node), and what types of traffic they can handle.  We might need some mechanism to resolve what traffic will be sent where, etc.  If we envision a world with multiple standard Bandwidth Aggregation mechanisms, we may need to choose at least one mandatory-to-implement Bandwidth Aggregation mechanism that all of N1…Nn will support, so that we can ensure interoperability.

Margaret

---

From: "Henderickx, Wim (Nokia - BE/Antwerp)" <[email-masked]>
Date: Sun, 1 Oct 2017 12:22:04 +0000

Margeret, will look at the proposed charter again but I am traveling non-stop these days so sorry for my delayed responses. This is idd an improvement afais.
The main issue is that there is a set of protocols that could benefit from the banana work and hence I suggested the information model as a target. The reason is to allow the different protocols to adopt the banana behvior in their work. At least this was my view on how we could move forward.

On 26/09/2017, 22:20, "Margaret Wasserman" <[email-masked]> wrote:

    Hi Wim,
    
    I have removed the word signaling in the most recently distributed version, as proposed by Dave Allan below.  Does that address your concern, as well?
    
    Would people also find it clearer if we used the term protocol(s) or mechanism(s) or something similar where we are currently using the word "solutions"?  I think that word may mean different things to different people.
    
    Margaret
    
    Sent from my iPhone
    
    > On Sep 26, 2017, at 1:36 AM, Henderickx, Wim (Nokia - BE/Antwerp) <[email-masked]> wrote:
    > 
    > My 2 cents on this and I believe this is where a lot of controversy is coming from, is why would we not charter Banana as a WG which is defining an information model that is required to execute banana. So banana defines which information needs to be exchanged and what the banana entities do with it to perform banana
    > As such the actual protocol works can be done in different WG(s) and we can leverage existing work as Sri was pointing out.
    > 
    > Would this work for people? My 2 cents, just a try to help.
    > 
    > On 25/09/2017, 22:13, "Banana on behalf of David Allan I" <[email-masked] on behalf of [email-masked]> wrote:
    > 
    >    I'd probably be happier losing signaling. IMO it is an overloaded term...
    > 
    >    "specify protocol(s) that can be used..." 
    > 
    >    WDYT?
    >    Dave
    > 
    >    -----Original Message-----
    >    From: Margaret Cullen [mailto:[email-masked]] 
    >    Sent: Monday, September 25, 2017 12:57 PM
    >    To: David Allan I <[email-masked]>
    >    Cc: [email-masked]
    >    Good point, Dave!
    >    I am a little concerned about the overuse of the term “mechanism”, though (since I define Bandwidth Aggregation mechanisms in the text).  So how about just changing “protocol” to “protocol(s)"?:
    > 
    >    OLD:
    > 
    >        Select or specify a signalling protocol that can be used to send 
    >        control information between BANANA Boxes, including:
    > 
    >    NEW:
    > 
    >        Select or specify signaling protocol(s) that can be used to send 
    >        control information between BANANA Boxes, including:
    > 
    >    Or is theres something more that you were trying to capture by changing from “protocols” to “mechanisms”?
    > 
    >    In addition to what you mentioned, this might allow us to reuse an existing protocol to do part of this job, even if that protocol could not be extended to cover everything we need for BANANA.
    > 
    >    Margaret
    > 
    > 
    > 
    > 
    > 
    >> On Sep 25, 2017, at 3:42 PM, David Allan I <[email-masked]> wrote:
    >> 
    >> HI Margaret
    >> 
    >> An aspect that concerns me for a while is the notion that there will be a single signaling protocol to satisfy a laundry list of requirements. On first blush this seems to suggest a solution is already in the wings that needs the laundry list, or that we will end up with  a bloated superset god protocol. Neither of which is IMO a totally desirable outcome.
    >> 
    >> Easiest fix for me would be to replace “a signaling protocol” with “mechanisms”.
    >> 
    >> Cheers
    >> Dave
    >> 
    >> Sent: Monday, September 25, 2017 12:33 PM
    >> To: [email-masked]
    >> Cc: [email-masked]
    >> No problem, Philip.  I have included the latest text below.  This does not yet include the changes I am currently discussing with Dave Sinicrope.
    >> 
    >> Margaret
    >> 
    >> 
    >> Charter: BANdwidth Aggregation for Network Access WG
    >> 
    >> The BANdwidth Aggregation for Network Access (BANANA) Working Group is chartered to develop solution(s) to support dynamic path selection on a per-packet basis in networks that have more than one point of attachment to the Internet.
    >> 
    >> Bandwidth Aggregation consists of splitting local traffic across multiple Internet links on a per-packet basis, including the ability to split a single flow across multiple links when necessary.
    >> 
    >> It is the goal of this WG to produce a Bandwidth Aggregation solution that will provide the following benefits:
    >> 
    >>    • Higher Per-Flow Bandwidth: Many Internet links available to homes and small offices (DSL, Cable, LTE, Satellite, VPNs, etc.) have relatively low bandwidth. Users may wish to run applications (such as streaming video, or content up/downloads) that require (or could benefit from) more bandwidth for a single traffic flow than is available on any of the local links. A Bandwidth Aggregation solution could supply the needed bandwidth by splitting a single traffic flow across multiple Internet links.
    >>    • Reduced Cost: Traffic sharing on a per-packet basis allows the full bandwidth of the lowest-cost link to be used first, only using a higher-cost link when the lowest-cost link is full.
    >>    • Increased Reliability: When one Internet link goes down, ongoing application flows can be moved to another link, preventing service disruption.
    >> 
    >> Proposed BANANA solutions use different mechanisms (e.g. tunnels, proxies, etc.) to split and recombine traffic, but at an abstract level, they involve a local (hardware or software) component on the multi-access network, a remote component within the Internet or at the remote end, and mechanisms for those components to find each other, exchange signalling information, and direct traffic to each other.   We refer to the functional components at each end as the Local and Remote “BANANA Boxes”, and we refer to the mechanisms they use to direct traffic to each other as “Bandwidth Aggregation mechanisms”.  
    >> 
    >> [Note:  Despite our use of the term “Boxes”, it should be understood 
    >> that a “BANANA Box” might be a software component running on a piece 
    >> of hardware with another primary purpose (e.g. a CPE router).]
    >> 
    >> The Bandwidth Aggregation solutions developed in this group will work 
    >> in true multi-provider scenarios (i.e. they will not depend on all of 
    >> the aggregated links being provided by a single Internet access 
    >> provider nor by a group of cooperating providers).  Any protocols 
    >> defined by this group will be IP-based, link-layer-independent 
    >> solutions, and they will be designed to work across NATs and other middle boxes, as needed.
    >> 
    >> The BANANA WG is chartered to complete the following  work items:
    >>    • Informally document/discuss BANANA problem statement and usage scenarios in a non-archival document (e.g. Wiki, Google Doc, etc.)
    >>    • Determine how Local and Remote BANANA Boxes find each other (i.e. describe how BANANA boxes will be provisioned/configured.)
    >>    • Select or specify a signalling protocol that can be used to send control information between BANANA Boxes, including:
    >>    • IP Prefixes of access  links
    >>    • Information about link status and properties (including congestion)
    >>    • Information needed by the Bandwidth Aggregation mechanism(s) in use
    >>    • Determining which flows are/are not eligible for Bandwidth Aggregation
    >>    • Select (and extend, if necessary) a tunneling encapsulation for sending traffic between BANANA Boxes.
    >> 
    >> When applicable, the BANANA WG will use existing IETF protocols, or extensions to existing IETF protocols, as the basis for the work items listed above.  When an existing protocol is used, the WG deliverable will be a document describing the use of that protocol for Bandwidth Aggregation and/or a set of options or extensions to an existing IETF protocol to make it useful for Bandwidth Aggregation.
    >> 
    >> The BANANA WG will also work with other IETF WGs (and other SDOs, as requested) that define additional Bandwidth Aggregation mechanisms (if any)  to ensure that the protocols defined by the BANANA WG will meet the needs of those additional mechanisms.
    >> 
    >> Milestones
    >>    • Apr 2018 Adopt WG draft for provisioning/configuration mechanism
    >>    • Apr 2018 Adopt WG draft for signaling protocol
    >>    • Apr 2018 Adopt WG draft(s) for tunnel encapsulation(s)
    >>    • Feb 2019 WGLC on provisioning/configuration mechanism
    >>    • Feb 2019 WGLC on signaling protocol
    >>    • Feb 2019 WGLC on tunnel encapsulation(s)
    >>    • Aug 2019 Send provisioning/configuration mechanism to the IESG
    >>    • Aug 2019 Send signalling protocol to the IESG
    >>    • Aug 2019 Send tunnel encapsulation(s) to the IESG
    >> 
    >> On Sep 25, 2017, at 12:37 PM, <[email-masked]> <[email-masked]> wrote:
    >> 
    >> Margaret,
    >> Please could you post the text on the mailing list, as our firewall 
    >> blocks google docs Thanks!
    >> phil
    >> 
    >> Sent: 22 September 2017 20:19
    >> To: [email-masked]
    >> Subject: [Banana] Updated Charter
    >> 
    >> I have updated the charter text in an attempt to reflect all of the feedback to date.  You can find the new version here:
    >> https://docs.google.com/document/d/1byOJ_To6eL1ZBxKSYpTafQbngTBiNwxaK7
    >> ReIsld9Ek/edit
    >> 
    >> Thoughts?  Comments?
    >> 
    >> Do folks think this is ready to send to the IESG?  Or are there other changes that it would make it clearer or better?
    >> 
    >> Margaret
    > 
    >    _______________________________________________
    >    Banana mailing list
    >    [email-masked]
    >    https://www.ietf.org/mailman/listinfo/banana
    > 
    >

---

From: "Sri Gundavelli (sgundave)" <[email-masked]>
Date: Sun, 1 Oct 2017 17:24:49 +0000

Hi Mingui,

> [Mingui] To make it clear, please take a look at the following figure. The real tunnel endpoint is H instead of G. This anycast mechanism was presented in the BOF of IETF 97. You can see the IP add…

Ok! Take a look at Runtime LMA assignment for PMIPv6. That has support for terminating tunnels on different line cards.

https://tools.ietf.org/html/rfc6463

> It's a no-doubt thing that RFC6909 can be a good starting point. If you would like to tweak it to meet the bypass requirement of BANANA, that would be great. For exmaple, you can resolve how to make it also works for IPv6 traffic and how to offload a flow from a set of tunnels. A new draft is needed.

We did not send IPv6 offload Rule to CPE, as IESG did not wanted that at that time. That support amounts to NATng IPv6 traffic at the CPE which they wanted to avoid. Text from the RFC below.

   RFC6909:

   "There are
   better ways to address the offload problem for IPv6, and with the
   goal not to create a NAT66 requirement, this specification therefore
   does not address traffic offload support for IPv6 flows."

Also, skipping traffic from certain access/tunnel is already supported as part of the flow spec.

> Through the discussions over the last two weeks, we can sense MIP is a good candidate. Sincerely hope you can write a draft to make people be clear how the MIP can be extended to be used between BANANA boxes.

Very good. But, Its not the MIP guys, or LISP guys to come to Banana group and prove they have support for the solution you are defining. The burden is on you guys to prove what you are asking for 1.) make sense and reasonable  2.) there is no IETF protocol has support for it, 3.) current IETF protocols cannot be extended to support that 4.) prove that the use-cases and the laundry list of requirements make sense and generally applicable to broad set of deployments.

> So you are suggesting to combine several control protocols with this or that feature in order to meet the requirements? I don't believe this strategy works. Moreover, these requirements are not exh…

We don’t bundle all functions into a single Master protocol that does every thing. It always starts as no “not exhaustive list”, but very soon you will find it to be incomplete and at that point you need to start aiming for a new master protocol. That’s the reason why IPPM guys use separate protocol for performance measurement and with tons of capabilities. The idea of emulating some or all features from those 20 RFC’s that IPPM guys have published into a single mobility protocol signaling is not a good idea, IMO.

Sri

From: Banana <[email-masked]<mailto:[email-masked]>> on behalf of "Zhangmingui (Martin)" <[email-masked]<mailto:[email-masked]>>
Date: Sunday, October 1, 2017 at 3:06 AM
To: Margaret Cullen <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Banana] Updated Charter

Hi Sri,

From: Sri Gundavelli (sgundave) [[email-masked]<mailto:[email-masked]>]

Hi Mingui,

Hi Margaret,

Thanks,
Mingui

Hi Sri,

Margaret