Subject: [Tsvwg] sctp multihoming for load balancing between multiple associations

From: Manikanda Prabhu <[email-masked]>
Date: Sun, 9 Jan 2005 14:49:22 +0530

hi
    we are working on sctp multihoming feature to load balance between
multiple associations as our undergraduate project. (say if i have 3
associations for http, ftp, smtp and access to 4 networks, to
efficiently utilize the available total bw)
we have some ideas towards the load balancing policy to use

1. User specified
      load balancing policy specified by the user using some command
like sctp- config to set up the priority level of the applications
using sctp ( say ftp > http > smpt > ...)
 
2. Transparent to the user
      here the sctp module analyze the traffic pattern of the user
(using  some machine learning techniques ?) and effictively set the
priority level of the applications using sctp

say if i have access to 4 networks, using some policy, sctp has
assigned 2 for ftp (dedicated) and share the remaining 2 to all other
associations.

we have some doubts.

--is it advisable to go for  load balancing between between multiple
associations ? or can sctp specified by rfc2960  effectively utilize 
the total bw without a explicit load balancing policy ?
--is it advisable to use AI techniques to analyze the user traffic (i
case if the policy is not specified by the user) at the transport
layer ? will that be a big over head ?

any thoughts about this ?

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 9 Jan 2005 02:44:13 -0700

Manikanda,

You do not need multiple associations to load balance over multiple
networks.  A single association with CMT will do.

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Caitlin Bester <[email-masked]>
Date: Sun, 09 Jan 2005 10:22:42 -0800

> networks.  A single association with CMT will do.

Sharing a single association across multiple applications is
generally not a good idea unless you want to implement very
complex fairness algorithms. Even if privileged software keeps
any one application from sending bad chunks, you still need
to guard against one application taking all the available
transmit credits. Load-balancing across applications does
not work if you throttle the responsible and glutton
applications equally.

You also have to build your data structures so that they can
be shared without risk even when separate applications crash
separately.

And if you do want to implement very complx fairness algorithms,
why confine them to SCTP. Wouldn't it be more effective to
load-balance at the IP layer? You don't want to provide
incentive for a hog application to remain UDP or TCP
precisely so they can avoid the fairness algorithms.

Using SCTP can greatly reduce the overhead of control traffic
by combining thousands of what would have been connections into
streams within an association -- and that can be quite valuable
for applications that can easily have a large number of "connections".
But do you really have as large a numbr of applications?

---

From: Manikanda Prabhu <[email-masked]>
Date: Mon, 10 Jan 2005 11:16:27 +0530

> not work if you throttle the responsible and glutton

i didnt mean single associations across multiple applications. what i
thought was multiple associations one for each application and a load
balancing technique to effectively switch the traffic of different
associations between the available networks.
 
> precisely so they can avoid the fairness algorithms.

i thought of doing it in transport layer because transport layer is
accessible to the user and the user can specify a load balancing
policy (if he wants to) using sockets api. a sctp aware application
can specify the policy if the user wants to allocate high bandwidth
and reliable network to his high priority application. say if i want
my ftp transfer to be reliable and fast, i can allocate my high bw
network to it using dynamic ip reconfiguration and my low bw link to
my http transfer
(both ftp and http using seperate associations)

> But do you really have as large a numbr of applications?

yes. this makes me to think..

what i initially thought was tcp doesnt support multihoming and single
link is shared across multiple connections. with sctps inherent
support to multihoming, the user has an option to route traffic across
multiple links

please correct me if iam entirely wrong

Thank You
Manik

---

From: Randall Stewart <[email-masked]>
Date: Mon, 10 Jan 2005 06:10:41 -0500

Commenting to all 3 posts :-D

First and formost RFC2960 specifically prohibits loadbalancing
with multi-homing at the SCTP layer. The reason this is
prohibited is there is NOT enough research on how
to do it properly... little problems show up such
as two destionations with slightly different
delay propagations can make false fast retransmits
and other oddities begin to happen.. Sure if
you put two hosts with four networks all connected
via a couple of switches you probably will NOT
see these oddities.. but put that same setup in
a inter-net or larger intra-net setting and strange
things will possible occur....

There is some research going on at the University of Deleware
in the PEL lab on CMT however... when complete we may get
an idea on how to fix or compensate for these issues.

However shared bottleneck detection is NOT being examined
by this work (Janardhan's work has been scoped NOT to
look into this area). So basically 1/2 of the worked needed
to truely do CMT will be in place at the end of Jana's
research :->

As to application sharing and fairness there are two
levels to look at..

Fair amongst some number of app's on a host

and

Fair to the network..

The first, from a network presepective, is not our concern. The second
however is... And in order to solve that we need to have solved
the shared bottleneck detection problem :-D

R

-- 
Randall Stewart
803-345-0369 <or> 815-342-5222(cell)

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 10 Jan 2005 16:36:49 -0700

Randall,

On Mon, 10 Jan 2005, Randall Stewart wrote:

> with multi-homing at the SCTP layer.

Missed that: could you point to it in the RFC?  What page was that
on?

--brian

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 10 Jan 2005 16:31:47 -0800

>>with multi-homing at the SCTP layer.
Well you could "load balance" between multiple associations by having
the initial connections be load-balanced over multiple associations for 
different
applications between the same SCTP endpoints.

And I can't think of any reason why that would be invalid.

I can't think of any reason why it would be particularly effective or 
useful though.

Any discussion of the usefullness of "multiple association CMT" probably 
should
be deferred until after CMT itself has been kicked around a bit. The 
applications
that would benefit most from load-balancing would almost always do so within
a single association. We need to convince ourselves as to when it is 
truly an
advantage to do this and when it just risks trashing all congestion 
management.

In addition to the congestion management questions, there is the far 
trickier question
of endpoint context. Having your peer "load balance" the association may 
force
a receiver to do a lot more work, and potentially might undo the benefits of
the load balancing. There will also be the question as to whether it is 
possible
to do this load balancing in a manner that minimizes application impact. For
example, with RDMA, it would be highly beneficial to the application for any
load balancing to be done such that each stream was still handled on one 
path.
Is that unique to RDMA, or will other applications have similar issues?

These will all be useful and interesting discussions, but in terms of 
tsvwg it
really shows that single association CMT needs to be discussed first.

---

From: giuseppe <[email-masked]>
Date: Tue, 11 Jan 2005 10:43:50 +0900

Randall Stewart wrote:

>>>> associations ? or can sctp specified by rfc2960  effectively 
>>>> utilize the total bw without a explicit load balancing policy ?
I would inform all of the interested person of this matter that also me 
and my colleagues are working on shared bottleneck detection.
We had three works, one of them is on going:
1) Overview of fast retransmit and congestion control problems in SCTP 
load balancing (originally setupped by Andrea Senatore at University of 
Salerno) 
(http://www.diiie.unisa.it/aree/telecomunicazioni/persone/collab/gdm/SCTPLoadSharing.pdf)
2) Indipendence test based on Gaussian traffic assumption, 
(http://www.diiie.unisa.it/aree/telecomunicazioni/persone/collab/gdm/SPOCdetection.ps)
3) Entropic detector, currently I am developping the basic idea at the 
Fukuoka Institute of Technology

Furthermore, with Salvatore Loreto we are building an intercontinental 
testbed to measure performances of load sharing.
Any collaboration will be appreciated.

-- 
----------------------
----------------------
Giuseppe De Marco, Ph.D.

Dipartimento di Ingegneria 
dell'Informazione e Ingegneria Elettrica
DIIIE, University of Salerno
via ponte don Melillo 1, Fisciano, Salerno
tel: +39 089 964012
email: [email-masked]

Department of Information and 
Communication Engineering 
Faculty of Information Engineering 
Fukuoka Institute of Technology (FIT)
3-30-1 Wajiro-Higashi, Higashi-ku, Fukuoka 811-0295 
Japan
email: [email-masked]
--------------------

---

From: Randall Stewart <[email-masked]>
Date: Tue, 11 Jan 2005 06:03:59 -0500

> Missed that: could you point to it in the RFC?  What page was that

Good point... we did have load balancing in the early
drafts.. but the AD's had us remove them and we
evidently failed to capture the stricture against it
in the RFC... and actually.. unless the AD's object we
should leave it that way.. With the ongoing research
it won't be to much longer and we can reverse all the
negatives... and for those implementations that do it
without the research to correct problems with F-R etc..
They will have to live with the side effects....

R

---

From: Randall Stewart <[email-masked]>
Date: Tue, 11 Jan 2005 06:09:24 -0500

> And I can't think of any reason why that would be invalid.

Actually no one in sigtran at the time objected to much
to load-balancing being taken out of SCTP. Since most
all figured they would just do it at the layer above

> I can't think of any reason why it would be particularly effective or 

Either is legal... multiple assocs OR the ULP in a single asoc.

And as Brian points out.. we did not add a stricture against
doing it at the SCTP layer.. however there are penalty's
that will be paid without the corrective algorithms that Jana
is working on...
> advantage to do this and when it just risks trashing all congestion 

I agree.. and I think that with what Jana is doing and Giuseppe on
shared bottle neck detection.. we may have a reasonable way to
turn it on and off and avoid the down sides to it... a bit
more time and work should tell :-D

> Is that unique to RDMA, or will other applications have similar issues?

Don't know.. you probably will have preferences on an application
by application basis. If we work out the CMT kinks their will
of course need to be a switch for the app to disable it.
> really shows that single association CMT needs to be discussed first.

Yep.. and hopefully in a few months Jana will provide us some
ground work in that area :-D

R

---

From: Janardhan Iyengar <[email-masked]>
Date: Tue, 11 Jan 2005 10:30:37 -0500 (EST)

Manikanda,

> networks.  A single association with CMT will do.

I agree. To elaborate, CMT (Concurrent Multipath Transfer) enables load
balancing over multiple network paths within an SCTP association. Please
see our work at:

http://www.cis.udel.edu/~iyengar/publications/2004.spects.iyengar.pdf
http://www.cis.udel.edu/~iyengar/publications/2004.icon.iyengar.pdf
http://www.cis.udel.edu/~iyengar/publications/2005.icdcs.iyengar.pdf

> > associations ? or can sctp specified by rfc2960  effectively utilize=20

SCTP specified by RFC2960 will not be sufficient to utilize the total
bandwidth due to reordering that will be introduced by any load balancing
policy. But in the work mentioned above, we append/modify SCTP mechanisms
to allow load balancing within an association.

As far as load balancing among multiple associations, we have shown that
CMT outperforms "ideal" load balancing using multiple associations
singnificantly. There is information sharing that happens within an
association that allows a single CMT association to perform *much* better
than multiple-association-loadbalancing (what we call AppStripe or
Application Striping).

regards,
jana

---------------------------------------------------------------
Janardhan R. Iyengar           http://www.cis.udel.edu/~iyengar
Protocol Engineering Lab  --   CIS   --  University Of Delaware
---------------------------------------------------------------

---

From: Janardhan Iyengar <[email-masked]>
Date: Tue, 11 Jan 2005 11:06:14 -0500 (EST)

> that will be paid without the corrective algorithms that Jana

This is an important point. Even though RFC 2960 does not prevent load
balancing explicitly, the mechanisms are just not there to handle the
side-effects of concurrently sending data over multiple network paths -
these side-effects can cause significant performance degradation.
See
http://www.cis.udel.edu/~iyengar/publications/2003.spects.iyengar.pdf
http://www.cis.udel.edu/~iyengar/publications/2004.spects.iyengar.pdf

> > advantage to do this and when it just risks trashing all congestion

Yes. I should also point out that we have seen how information sharing
allows a CMT association to significantly outperform "multiple association
CMT" (what we call application striping).

> > Is that unique to RDMA, or will other applications have similar issues?

Agreed. For your example though, with CMT mechanisms, you should be able
to add a simple scheduler to pin streams onto paths.. that hook should be
simple to add. CMT mechanisms use and follow only TSNs, nothing is assumed
about SSNs or SIDs.

regards,
jana

---

From: Qiaobing Xie <[email-masked]>
Date: Thu, 27 Jan 2005 19:20:50 -0600

> associations between the available networks.

An application that uses a SCTP association to send data to a peer can influence which 
network the majority of the traffic would travel by using the set-primary addr function in 
the SCTP API. Imagine that you have an overall load/traffic manager that knows the existence 
of multiple networks, their characteristics, load status, etc, and that knows the 
policy/requirements of you applications (ftp, http), then it should not be hard to have this 
traffic manager (which is outside of SCTP stack of cause) instruct each application when and 
how to set the primary addr of their SCTP association. I don't see a problem here but am not 
sure if this is what you want.

regards,
-Qiaobing