Subject: RE: [Tsvwg] Concerns about SCTP

From: "Williams, Jim" <[email-masked]>
Date: Thu, 9 Jan 2003 11:52:08 -0500

> -----Original Message-----
> From: Randall Stewart [mailto:[email-masked]]
> 
> Some comments for you to think upon... :>
> 
> Williams, Jim wrote:
> > [...]
> > 
> > I regard the following as two basic
> > rules of protocol design.
> > 
> > Rule 1:  Two implementations that are fully
> >          compliant with the protocol definition
> >          must interoperate.  If they don't,
> >          then that is a defect in the protocol
> >          definition.
> > 
> > Rule 2:  One end of an association should not
> >          be able to consume an unbounded amount
> >          of resources on the other end without
> >          negotiation and agreement from the other
> >          end.
> > 
> > I believe in this respect, the multi-homing feature
> > of SCTP comes up short.  Each remote address that
> > is supported consumes resources on the local end.
> > SCTP places no limit and requires no negotiation
> > as to the number addresses that can be included
> > in an INIT.  The receiver must either allocate
> > the resources needed to support this number of
> > addresses (violating rule 2), or must abort
> > the association if too many are requested
> > (violating rule 1).
> > 
> 
> I don't see how this violates rule 1. SCTP provides a
> basic "I can't do that .. not enough resources" in response
> to something that can't be done. Yes the association will
> not come up until the peer reduces the resources request...

If one views it from a system perspective, a user may
be running a file system over RDDP over SCTP.  If the
file system fails to mount, the user would regard this
a failure to interoperate.  However this failure can
be prevented in multiple ways.

1.  At the RDDP over SCTP adaption layer, specify that
    if an endpoint specifies multiple IP addresses and
    the association setup is aborted with a resource error,
    the endpoint MUST retry the association with fewer
    IP addresses. (The draft could also remain silent
    on this and trust the good sense of the upper layers
    to do this anyway, but that seems like asking
    for interoperability woes unnecessarily.)

    Although this works, I regard it as more of a 
    work around than a well designed negotiation.
    An example of a well designed negotiation is
    the mechanism use by SCTP to determine the number
    of streams per association.

2.  Treat this as a flaw in SCTP and fix it.  Set
    aside practical and political considerations for
    the moment.  I regard this a flaw (perhaps minor
    but real) that affects not only RDDP, but all
    low end, high speed, embedded implementations of
    SCTP.  If the goal of SCTP is to become a
    universal transport, not just a high end
    transport, then this may make sense.

    I acknowledge that responders to my original
    post do not agree that there is any flaw.  But
    my own opinion is not yet changed by reading
    the responses.

3.  Acknowledge that SCTP is not an optimum transport
    for RDDP, and use a different transport.  If one
    were to design a transport optimized for high
    speed hardware accelerated applications, it would
    look quite different from SCTP.

    The counter arguments here, though, are that there
    is advantage to standardizing on a transport across
    many applications even if not optimum for each
    application.  And also that transports should
    be designed to optimize data flow through the
    network, not to optimize end point implementation,
    (at least that seems to be the IETF philosophy).

Anyway, this is the direction I was hoping to steer
the discussion.  For now, #1 seems like the preferred
direction.  But I though #2 and #3 were worth at least
brief discussion.

> I know, for me, I truly have
> a hard time understanding how ECN would work better for
> flow control than the rwnd. aka
> 
> ------DATA-----> (rcver realizes its is running low
>                    on resources (chunks/structures/?)
>   <------SACK(rwnd=0)

I agree with this.  But I was unaware that this is allowed.
TCP specifically forbids retracting a previously advertised
window.

I am not clear on where SCTP stands on this.  It appears
that in RFC2960 there is no prohibition on this, but
it does seem contrary to a few SHOULDs regarding rwnd
calculation.  Anyway, this may be the right
answer unless the SHOULDs evolve into de facto MUSTs.

Interestingly, though, RDDP has the property that all
received data can be immediately processed.  There is
never any need to stop the remote sender.  The only
problem occurs when data comes in faster than it
can be processed.  In this case the desired behavior
is to slow down the sender a bit, not stop it.
Therefore ECN might be a better functional fit
for this requirement than rwnd.  

If the receiver knew how much
data the sender currently had outstanding (usually
equal to its cwnd) then it could set rwnd to
a half (or some other fraction) of that and 
accomplish the same thing, but the receiver
does not have this knowledge.  This creates
a problem using rwnd to throttle the sender's
rate.  But this problem doesn't appear
insurmountable.

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 9 Jan 2003 12:13:47 -0600

>for this requirement than rwnd.  
That property is intrinsic to RDDP no matter what transport
it runs over. The ULP is responsible for pacing untagged
messages, but only the LLP paces tagged messages.

RDDP releases all LLP buffers "immediately", meaning as
quickly as the placement of the tagged/untagged data can be
completed. Given that characteristic it is highly unlikely
that there would be buffering resources dedicated to
specific assocation or connection.

Therefore the challenge, whether over MPA/TCP and/or SCTP,
is to ensure that the internal bandwidth required to support
DDP placement will not be swamped by all incoming data.

Specifically the windows SHOULD be set so that the risk of
exhausting the placement bandwith is as low as possible.

However those windoes CANNOT beset so that said risk is
non-existent. LLP Credits are delivered asynchronously,
uncertainty on placemnt speed because of potential host bus
contention or retries, and more importantly in integral
amounts. Those integral amounts, spread over hundreds of
thousands of connections, either force a severe
under-allocation or an over-allocation.

And certainly the available to-host bandwidth over a
conventional host bus is statistically reliable but
defintely not guaranteed. Limiting your RNIC to the host bus
*guaranteed* rates would be a severe limitation.

I believe most implementers will properly choose to take a
slight risk of over-allocation, knowing that in the worst
case the result is that a packet is dropped. A properly
credited LLP packet arrives and there is no available buffer
for it. No LLP ack is generated, and it is eventually
retransmitted. The same thing can happen at any number of
routers and switches along the route for the very same
reason.

Also consider that in some cases the host memory placement
capacity may actually exceed the capacity of the external
link layer. In that case LLP flow control deals only with
network congestion, RNIC and host buffering is irrelevant.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Randall Stewart <[email-masked]>
Date: Fri, 10 Jan 2003 06:52:19 -0600

Comments in-line

> direction.  But I though #2 and #3 were worth at least

As you may have guessed I to am in camp 1 :> I don't beleive
it is a flaw... and even if we did (which I know I don't) the
solution would require multiple message round trips to setup
the addresses... this is quite a bit unlike the streams issue.

I just don't see this as a problem.. the DDP adaption is very
unlikely to allow multiple IP addresses... if so the NIC's
will need to all be related and cross-configured together...

DDP clients will just have to be aware of this issue...

On top of that I also think multi-homing is not as numerous
as you think for clients... two maybe three addresses is what
most clients will have.. not the 100's you seem to invision.. in
fact we could even add to the SCTP-DDP adaption something like
"A user of SCTP MUST NOT bind more than N addresses"... and then
put in verbage to say how the receiver an association with more
than N addresses aborts it... very simple.. and I believe there
is a user defined reason for abort that I know the socket API
supports... so we define a SCTP-DDP adatption ABORT reason that
comes in with the abort.. and your problem is solved...maybe not
what you want but I don't see anything wrong with this..

> TCP specifically forbids retracting a previously advertised

I suggest you take a close look at the code snippet below:
"
#define	sbspace(sb) \
     ((long) imin((int)((sb)->sb_hiwat - (sb)->sb_cc), \
	 (int)((sb)->sb_mbmax - (sb)->sb_mbcnt)))
"

This is one of the key macro's used out of sys/socketvar.h which
is used to caclulate space and the windows.. notice the use
of sb_mbmax and sb_mbcnt... these track the amount of mbuf
useage...

The reason I know this is I have been bitten by the above when we were
doing our SCTP implementation... Since the mbcnt also comes in to play I
was seeing the rwnd drop sharply at the end unexpectedly.. this is
because the mbuf limits were being hit before the sb_cc limits...

I won't go in to the details of how I worked around this ... my point
is that TCP has factored into its rwnd (at least in bsd) the amount
of mbuf's (structures to track things) used for years...

R
-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: Qiaobing Xie <[email-masked]>
Date: Tue, 21 Jan 2003 16:44:52 -0600

...
>     for interoperability woes unnecessarily.)

As you indicated above that you are essentially talking about system level
inter-operability (ie. what a user may see), "the good sense of the upper layers"
is not only desirable, but is fundamental. One can not expect system level
inter-operability without "the good sense" in system engineering at all the
layers.

>     of streams per association.

As a matter of fact, the SCTP design team had discussion on this and deliberately
choose to use *minimal negotiation* at the association setup. The reason is that,
nothing comes free and lower setup latency was considered more important to most
applications. Moreover, the stream negotiation mechanism (that you seem to agree
with) also need "the good sense of the upper layers" to work well - e.g., the
initiator side's upper layer will need to allow its SCTP stack to re-try with a
lowered number of outbound streams.

-Qiaobing