Subject: [Tsvwg] TCP vs SCTP throughput

From: "Ramaiah, Chaitra M (STSD)" <[email-masked]>
Date: Wed, 5 Nov 2008 03:17:42 +0000

Hi All,

  Not sure whether the question is appropriate for this forum. Posting it anyway. Just want to know if there are any performance benchmarks been done comparing the throughput of TCP vs. SCTP? I know the results vary across implementations, but any pointers will be appreciated. Considering ideal conditions, i.e. similar implementation of both TCP and SCTP, end hosts not too busy and high bandwidth network, does the byte stream mode of TCP have an edge over the throughput in comparison to message passing of SCTP?

Different scenarios would perhaps be:
1) Single byte exchange
2) Small request/small response
3) small request/big response
4) file transfer
5) web server/client

Regards,
Chaitra

---

From: Fred Baker <[email-masked]>
Date: Tue, 4 Nov 2008 21:52:08 -0800

There was a recent study in a workshop alongside SIGCOMM; you can look  
at the work at http://www.cis.udel.edu/~amer/PEL/poc/index.html.

In short, TCP and SCTP have the same congestion avoidance algorithms,  
retransmission algorithms, and etc. Moving one file, they behave about  
the same. Moving multiple files, such as http would in a typical web  
page, SCTP works around head-of-line blocking better than TCP, as data  
from another file can move when one of the streams blocks. Also, fast- 
retransmit can happen with SCTP in cases where it can't for TCP  
because packet arrivals in another stream can trigger fast recovery  
for a blocked stream. And oh by the way, if I have separate TCP  
sessions doing the sawtooth at the same time, they are likely to hit  
the paek at the same time, while in SCTP, congestion control across  
the session and all the streams in it manages that effect.

If you are moving multiple files, SCTP has an edge over TCP. If you  
are moving one file, they're about the same.

On Nov 4, 2008, at 7:17 PM, Ramaiah, Chaitra M (STSD) wrote:

> Hi All,
>
>  Not sure whether the question is appropriate for this forum.  
> Posting it anyway. Just want to know if there are any performance  
> benchmarks been done comparing the throughput of TCP vs. SCTP? I  
> know the results vary across implementations, but any pointers will  
> be appreciated. Considering ideal conditions, i.e. similar  
> implementation of both TCP and SCTP, end hosts not too busy and high  
> bandwidth network, does the byte stream mode of TCP have an edge  
> over the throughput in comparison to message passing of SCTP?
>

---

From: Randy Stewart <[email-masked]>
Date: Wed, 5 Nov 2008 06:32:37 -0500

> over the throughput in comparison to message passing of SCTP?

Chaitra:

One of the big problems when you do these types of comparisons is
the fact that SCTP uses the CRC32c checksum and TCP uses
a simple additive sum. Add to that the fact that most modern
NIC cards support the TCP-checksum offload and this poses a
huge performance difference.

Quite some time ago when I was performance tuning the BSD stack (and  
it has
been a long time) I was seeing most of the CPU time (looking at a  
profile of
the kernel) be used by the CRC32c algorithm.. in fact all other CPU  
useage
in the SCTP stack was noise :-)

I have done some 10Gig testing with a pair of 8 core 1.8Gig Xeon's and
seen SCTP get up to 3Gig per second as compared to TCP topping out
at 5Gig per second... this was quite some time ago and I know the
bottleneck for TCP (and SCTP) was a particular network thread which
may well be improved i.e. allow multiple of the threads to be
servicing the NIC or handing off the mbuf to a service thread so the
service thread can service the NIC while the stack (TCP or SCTP)  
continues
to run in a different processor.

One of these days I will circle back and do some more performance  
testing/improvement
in the BSD stack..  but not for probably a while.. :-)

As to byte stream mode verses messages... to do a fair comparison you  
would either
need to run SCTP in its explicit EOR mode (which can get you a similar  
mode as TCP) or
make sure TCP's application time framing the messages is also included  
in the tests..

Michael Tuexen also did some studies a while back but his were  
comparing the
various SCTP stacks... (looking for performance bugs)...

I don't know of no direct comparison between the two mainly I think  
due to
the apples to oranges nature.

One could dream up a comparison test that could be done... i.e.
1) Have TCP frame its messages (or use SCTP's EOR mode)
2) Have TCP run a CRC32C checksum in user space and validate
    the data.

Then compare the transfer rates... this would give one a very similar  
benchmarking methodology
and would discount the CRC32c and message framing...

Hmm.. maybe if I get some time at the IETF I will write such a test ;-)

R

---

From: Qiaobing Xie <[email-masked]>
Date: Wed, 05 Nov 2008 09:20:59 -0600

Maybe we should reopen the discussion on allowing the ends to choose 
between CRC32c or simple checksum at the association setup?

-Qiaobing

Randy Stewart wrote:
>>  Not sure whether the question is appropriate for this forum. Posting 
>> it anyway. Just want to know if there are any performance benchmarks 
>> been done comparing the throughput of TCP vs. SCTP? I know the 
>> results vary across implementations, but any pointers will be 
>> appreciated. Considering ideal conditions, i.e. similar 
>> bandwidth network, does the byte stream mode of TCP have an edge over 
>> the throughput in comparison to message passing of SCTP?
>
> Chaitra:
>

---

From: Rick Jones <[email-masked]>
Date: Wed, 05 Nov 2008 10:14:42 -0800

> huge performance difference.

Indeed.  Other than disabling CKO - and thus layered offloads like TSO 
and LRO to get a different "apples to apples" comparison, one thing to 
do would be to compare something like transaction rate and service 
demand for a netperf single-byte [SCTP|TCP]_RR test (or an "omni" test 
in the current top-of-trunk with suitable settings, selecting the 
protocol of choice).  Then you don't really care if CKO/TSO/LRO are 
enabled because they don't do squat for a single-byte ping-pong test :)

I'm not sure how segmentation strategies might play into it, but a 
netperf ./configured with --enable-burst could be used to put lots of 
single-byte transactions out onto a single connection when combined with 
the test-specific -D option to set NODELAY.

rick jones

---

From: Fred Baker <[email-masked]>
Date: Wed, 5 Nov 2008 10:58:56 -0800

> between CRC32c or simple checksum at the association setup?

I think there would be value. The selected algorithm is pretty much a  
non-starter.

---

From: Michael Tüxen <[email-masked]>
Date: Wed, 5 Nov 2008 20:27:37 +0100

Dear all,

message throughput for SCTP depends also on the size if the user  
messages.

For example, the receiver will perform one receiver call per message. So
if you use 1 byte messages, you do one system call per byte. Using TCP
you get 1400 or more bytes per system call. There is also some  
additional
overhead.

I did some testing of SCTP only (not comparing it with TCP) between two
platforms and you can see the results at
http://sctp.fh-muenster.de
The measurements are old (I actually should redo them with recent kernel
versions) and I focused on interoperability.

All curves show that the throughput depends on the message...

Best regards
Michael

On Nov 5, 2008, at 7:14 PM, Rick Jones wrote:

>>> know the results vary across implementations, but any pointers  
>>> will  be appreciated. Considering ideal conditions, i.e. similar   
>>> implementation of both TCP and SCTP, end hosts not too busy and  
>>> high  bandwidth network, does the byte stream mode of TCP have an  
>>> edge  over the throughput in comparison to message passing of SCTP?
>> Chaitra:
> Indeed.  Other than disabling CKO - and thus layered offloads like  
> TSO and LRO to get a different "apples to apples" comparison, one  
> thing to do would be to compare something like transaction rate and  
> service demand for a netperf single-byte [SCTP|TCP]_RR test (or an  
> "omni" test in the current top-of-trunk with suitable settings,  
> selecting the protocol of choice).  Then you don't really care if  
> CKO/TSO/LRO are enabled because they don't do squat for a single- 
> byte ping-pong test :)
>
> netperf ./configured with --enable-burst could be used to put lots  
> of single-byte transactions out onto a single connection when  
> combined with the test-specific -D option to set NODELAY.
>
> rick jones
>

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 5 Nov 2008 13:27:12 -0700

Michael,

I did some bottleneck comparison tests between Native Sockets
TCP, XTI over Sockets TCP and Native XTI SCTP summarized in a
paper here:

  http://www.openss7.org/strsctp_paper.html

I also did some older Native Sockets TCP vs. Native Sockets SCTP
comparisons here including dual rails:

  http://www.openss7.org/sctp_perf.html

The large body of research done captures a number of aspects of
performance:

1. Although SCTP has a more intensive checksum algorithm, CPU
   performance is more sensistive to precise implementation and
   interaction between the implementation and the system
   scheduler.  An implementation that fights the system
   scheduler (i.e. that causes more context switches than
   necessary) experiences far poorer CPU performance than one
   that works with the scheduler.

   An implementation that has good memory speed and keep good
   cache-heat on the buffers is not significantly affected
   by copies nor by checksum algorithm.

2. One the wire through a bottleneck, SCTP experiences poorer
   throughput than TCP under zero-loss conditions simply because
   SCTP has more overhead bits per packet.  The additional
   overhead bits for SCTP are, of course,  less noticable at
   larger write sizes.

3. When not operating through a bottleneck (e.g. on dual network
   rails) and using CMT SCTP can exhibit throughputs exceeding
   TCP by the factor of the number of available paths for CMT.
   So, for example, over dual 1G rails, SCTP CMT can easily exhibit 
   1.8Gbps throughput whereas TCP only exhibits .9Gbps
   throughput.  There has been some controversy over whether
   SCTP CMT is fair to TCP in such circumstances.

4. When operating under noise or disruption and particularly
   where multiple network paths are available, SCTP exhibits
   dramatically better throughput and latency than TCP.  There
   are a number of papers exhibiting this.  Google is your
   friend here.

Largely most implementations are to the point where single path,
zero-loss throughput differences between SCTP and TCP are not
significant.  SCTP indeed shines in comparison for latency (HOL
blocking) over noisy noisy networks and shines in throughput for
CMT over multiple path networks.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Ramaiah, Chaitra M (STSD)" <[email-masked]>
Date: Thu, 6 Nov 2008 06:26:51 +0000

Dear All,

  Thanks for the responses. It helped a lot!

We had done some performance comparison between TCP and SCTP some time back. Not an exhaustive test setup and results but just to get a gist on how SCTP throughput may fare with TCP.

Two 4 way IA64 systems connected back-to-back 1 GBPS link, NODELAY turned on, for both, features like CKO, TSO and other optimizations turned off for TCP.

The only optimization done was cache prefetching in the data path for both TCP and SCTP. Test used was the venerable netperf (TCP_RR and SCTP_RR) for various data sizes, request/response sizes being equal.

SCTP fared at 70% that of TCP when averaged out. A close observation revealed that for small sizes, TCP outperformed almost yielding double TPS and the gap narrowed as the size increased with almost being equal at larger sizes. Of course, a point to be factored in was that the implementation slightly differed where TCP implementation seems to be better.

I know that we cannot draw any conclusions with such minimal tests, trivial setup and especially with differences in implementations, but just wanted to share the observations.

Regards,
Chaitra

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian F. G. Bidulock
Sent: Thursday, November 06, 2008 1:57 AM
To: Michael Tüxen
Cc: Jones, Richard (Mr. Netperf); Randy Stewart; [email-masked]
Subject: Re: [Tsvwg] TCP vs SCTP throughput

Michael,

--brian

--
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/