Subject: Comments on your SCPS-TP testing report

From: "Adrian J. Hooke" <[email-masked]>
Date: Fri, 20 Sep 2002 11:07:01 -0700

Hi William:

The SCPS team has been alerted that you are asking for comments on your 
SCPS-TP testing report by 21 September:

http://roland.grc.nasa.gov/~ivancic/papers_presentations/SCPS_TCP_RATE_FINAL_0903.pdf

Unfortunately we don't have the resources to do an in-depth technical 
evaluation of your work. However, we have taken a quick glance at the 
results and have two recommended changes and some comments. In particular, 
we would strongly suggest that (for consistency with the rest of the 
technical data) you should re-write the Abstract, Executive Summary 
(section 1) and Conclusions (Section 11) along the lines suggested below.

Best regards
Adrian Hooke, NASA-JPL

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

RECOMMENDED CHANGE:

1 ABSTRACT

Tests were performed at Glenn Research Center (GRC) to compare the 
performance  of the Space Communications Protocol Standard - Transport 
Protocol (SCPS-TP,otherwise known as "TCP Tranquillity") relative to other 
variants of TCP, and to
determine the implementation maturity level of these protocols - 
particularly for higher speeds. The testing was performed over reasonably 
high data rates of up to 100 Mbps with delays that are  characteristic of 
near-planetary environments. The tests were run for a fixed packet size, 
but for variously errored environments. This report documents the testing 
performed to date.

2 EXECUTIVE SUMMARY

There have been numerous  discussions  relative to the role of the SCPS-TP 
in the  ever-evolving TCP family, particularly with respect to other known 
TCP features that accommodate long bandwidth-delay products (e.g., large 
windows, selective acknowledgements). In order to gain a better 
understanding of the  performance of the SCPS-TP relative to other TCP 
variants and to determine the maturity of the various  options for use on 
higher-rate space links, the NASA Space and Data Communications Systems 
(SCDS) Office requested Glenn Research Center to perform a comprehensive 
set of tests. Tests were
performed at Glenn Research Center to validate the operation of the 
Reference Implementation of the SCPS-TP relative to its controlling CCSDS 
specification, and to perform a comprehensive comparison of SCPS-TP 
protocol options relative to other TCP options.

We have studied the effect of delay and BER on the performance of 
congestion-friendly and rate-based protocols in emulated space links, both 
under uncongested conditions and with limited congestion. The results 
correlate well with previous testing of TCP options, including the SCPS-TP:
    * The single stream and multi-stream test results clearly illustrate 
that the TCP-Vegas enhancements to TCP (that are implemented by the 
SCPS-TP) provide measurable performance improvements over a TCP SACK 
implementation tested in  high bandwidth-delay product environments. Since 
performance considerations are subjective, the operational value of these 
performance increases can best be assessed by the users of specific 
applications hosted within those environments.
    * Very small transactions such as command and control will probably see 
little difference in performance for any variant of TCP.
    * In extremely error-prone environments with high RTT latencies, use of 
a rate-based TCP variant such as that provided by SCPS-TP is advisable, 
assuming that the network implementation is properly engineered. Users are 
cautioned  against using rate-based protocols on networks where resource 
allocation policies are based on contention.
    * The tested SCPS-TP implementation was the protocol Reference 
Implementation, which exists as a user application rather than as a more 
generalized protocol service provider. For some deployments, an in-kernel 
protocol service provider may prove more desirable than the deployment of 
application level protocol service providers. The tradeoff is between the 
more efficient use of resources and possibly higher performance for 
in-kernel services versus the ease of maintainability and deployment of 
application level implementations. Commercial in-kernel and application 
level implementations of SCPS-TP exist, but have not been tested as part of 
this effort.
    * Even with equal performance, the SCPS-TP version of a rate-based 
protocol may be more desirable to implement than other rate-based protocols 
(such as the Multicast Dissemination Protocol) since the SCPS-TP is capable 
of requiring only sending-side modification. This feature of the SCPS-TP 
eliminates many of the risks associated with requiring universal deployment 
of new software.
    * Existing TCP implementations (drawn from a variety of communities, 
and including the SCPS-TP) appear to satisfy all currently-known space 
mission needs; however, the space mission community should maintain an 
awareness of current and future TCP research that is being performed by 
other communities.
11 CONCLUSIONS

(Repeat the same changes that are noted above in the Executive Summary.)

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

RECOMMENDED CHANGE:

Sec 4.2.2, "Modified Slow Start", there should be a bullet that reads:

    *  The Vegas Fast Retransmit algorithm is *not* part of SCPS-TP.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

COMMENT:

Section 9.1.1.3:  the "optimum receiving buffer size" is mentioned, which 
is defined to be in a fairly narrow range centered around one Bandwidth 
Delay Product (BDP) of the long-delay test case.  With SCPS's 
Vegas-Corruption and Rate configurations, this is not optimal.  Your 
Solaris implementation might not have shown it due to OS-imposed penalties 
on outside-the-kernel protocols and buffer management, but in high-error 
conditions, the SCPS protocols are best configured with greater than one 
BDP (e.g., two) of storage, so that the time spent waiting for a 
retransmission to be received does not cause the sender to become window 
limited.  This isn't typically an issue with native TCP, as the error 
recovery regime in the congestion control mechanisms limits the protocol's 
ability to consume buffer space.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

COMMENT:

You appear to have missed the most important "Multiple Flow" test to 
conduct.  In Section 9.3, you state that "In addition, 
SCPS-Vegas-Corruption was not tested in this environment, as it would have 
been a misapplication of the protocol."  This is seriously incorrect.  The 
attached MILCOM paper specifically examines the issue of use of the 
Vegas-Corruption configuration for congestion control in the presence of 
errors.  It's absolutely the *RIGHT* application of the SCPS-TP protocol, 
and is a significant omission from your paper.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

COMMENT:

Also, in that same section 9.3, you seem to have missed the opportunity to 
draw some relatively important conclusions:

a) SCPS Vegas Congestion proved to be *more fair* than TCP with the Van 
Jacobson congestion control algorithm.

b) Because there were fewer retransmissions required, the SCPS Vegas 
algorithm was also *more efficient* than TCP with the Van Jacobson 
congestion control algorithm.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

COMMENT:

In Section 10, you cite some problems with TCP Vegas in general (the last 
three bullets before Section 11).  It should be noted that the issue cited 
in the second bullet is easily addressed and anticipates the deployment of 
Explicit Congestion Notification (ECN).  The other two bullets cite issues 
in which Vegas algorithms are less aggressive than the Van Jacobson 
algorithm, and will, no doubt, receive attention should these become real 
problems.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

-end-

---

From: William Ivancic <[email-masked]>
Date: Mon, 23 Sep 2002 09:13:36 -0400

Adrian,

Thanks for the input.  It has been received in sufficient time to get into 
the final report.

We will consider you input.  I do have some questions, however, regarding 
some of the suggestions.

We tested and set the optimum receiver buffer size starting a twice the 
Bandwidth-Delay product (BDP) and found for our test setup with the OS used 
(which appeared to be the most stable OS) that the optimal setting was 
slightly less the 1 BDP.  So, we could say that "in theory" the receive 
buffer should be 2 x BDP, but the data did not support that of Solaris.

IMHO, protocol tuning doesn't scale well - particularly in non-gateway 
implementations and for systems such as NASA's planned sensorWeb.   I would 
really like to see better auto-tuning features developed.

The paper on performance for "SCPS-TP assumed corruption", is for a gateway 
implementations.  All data is received at the gateway and buffered.  Thus, 
all data is buffered at one source and ECN type mechanisms can be 
implemented on the point-to-point link.  This is fine, but is a very 
limited case.  I would not be comfortable extrapolating such performance to 
general deployments of "SCPS-TP assumed corruption" over fully meshed 
networks.  Am I missing something?  Gateway testing is great, but assuming 
everything will use a gateway is ..... maybe not to good.   If one encrypts 
prior to the gateway (very common in a military environment), your limited 
as to what you can do.  I'm always a bit fearful when I see testing 
performed using gateways in that I fear decision makers may not understand 
that those results may only be valid when used in an architecture that 
allows for deployment of a gateway.

Assuming ECN in a fully meshed network may be wishful thinking.  Although 
ECN concepts have been around and experimentally show to be advantageous in 
an ideal world,  I believe, to date, deployment of ECN is limited at 
best.   I'm  may be incorrect here as I do not have the compete  knowledge 
of what is deployed in the backbone.  A quick breeze through the QoS 
commands on Cisco routers did not show ECN settings, but that was a very 
quick search.

Best Regards,

Will

At 11:07 AM 9/20/02 -0700, Adrian J. Hooke wrote:
> implementation tested in  high bandwidth-delay product environments. 
> Since performance considerations are subjective, the operational value of 
> these performance increases can best be assessed by the users of specific 
>    * Very small transactions such as command and control will probably 
> see little difference in performance for any variant of TCP.
>    * In extremely error-prone environments with high RTT latencies, use 
> of a rate-based TCP variant such as that provided by SCPS-TP is 
> advisable, assuming that the network implementation is properly 
> engineered. Users are cautioned  against using rate-based protocols on 
> networks where resource allocation policies are based on contention.
> level implementations of SCPS-TP exist, but have not been tested as part 
> of this effort.
> protocol may be more desirable to implement than other rate-based 
> protocols (such as the Multicast Dissemination Protocol) since the 
> SCPS-TP is capable of requiring only sending-side modification. This 
> feature of the SCPS-TP eliminates many of the risks associated with 
> requiring universal deployment of new software.