Subject: [Tsvwg] SCTP API (draft-ietf-tsvwg-sctpsocket-11)

From: Mark Butler <[email-masked]>
Date: Mon, 21 Nov 2005 06:50:06 -0700

I have a few comments on the proposed SCTP API in the current sctpsocket 
draft:

1. Stack-level message reassembly  should be guaranteed for messages 
below some minimum size so that applications are not forced to implement 
message reassembly in all cases. Otherwise even simple UDP style code 
has to support manually reassembling short messages (if fragmentation is 
enabled).  Perhaps MSG_WAITALL could be used to request this if necessary.

2. The current draft specifies "Only one message at a time can be 
partially delivered."  This seems to guarantee head of line blocking 
where one large partially received message might block all other streams 
in the same association for a considerable time.

I suggest this problem be solved by adding the ability to perform 
receive operations against a particular stream, so that applications are 
not forced to de-multiplex the streams themselves.  That would eliminate 
inter-stream head of line blocking, simplify the application programming 
model, allow different threads to easily process different streams, and 
so on.

The ideal way of doing this would be to have a function call to get an 
associated socket that is bound to a specified stream  pair (inbound and 
outbound).  A typical use for this would be to easily port a large body 
of existing code that uses multiple parallel TCP connections, giving 
unified congestion control.

Something like:

   sctp_peeloff_stream(int sd, int assoc_id, int stream_no):

ARGUMENTS
   sd is the descriptor for an existing one-to-one or one-to-many 
association socket.
   assoc_id specifies the association (if sd is one-to-many), ignored if 
sd is one to one.
   stream_no is the stream number to peel off.
  
RETURN VALUE
   Returns a new stream socket descriptor, or -1 on error. 

BEHAVIOR
   This function creates a stream socket from a primary one-to-one or 
one-to-many SCTP association socket.  A stream socket is used to branch 
off traffic for a specific association stream to an independent socket.

All messages for the specified stream number are delivered to the stream 
socket instead of the primary socket.  All send operations on a stream 
socket are delivered on an identically numbered outbound stream.  Once a 
stream socket (and all its aliases) are closed, messages for the 
specified stream are again receivable on the primary socket.  When the 
parent association ends, transfer operations on the stream socket fail 
with an EPIPE error.

========================

   The sctp_sendmsg function should accept a -1 for the stream_no 
argument when sending a message on a stream socket, and return an error 
if any other stream number not equal to socket stream number is specified. 

Stream sockets should be able to have independent send and receive 
buffers adjustable using the standard setsockopt SO_SNDBUF and SO_RCVBUF 
options. They should also have independent SCTP_NODELAY settings.

By the way, I am curious why SCTP protocol does not specify offsets in 
fragmented DATA chunks so that zero copy hardware accelerated message 
reassembly would be practical, etc.

 - Mark Butler

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 21 Nov 2005 23:30:55 +0100

Hi Mark,

please find some comments in-line.

Best regards
Michael

On Nov 21, 2005, at 2:50 PM, Mark Butler wrote:

> I have a few comments on the proposed SCTP API in the current  
> sctpsocket draft:
>
> 1. Stack-level message reassembly  should be guaranteed for  
> messages below some minimum size so that applications are not  
> forced to implement message reassembly in all cases. Otherwise even  
> simple UDP style code has to support manually reassembling short  
> messages (if fragmentation is enabled).  Perhaps MSG_WAITALL could  
> be used to request this if necessary.
Sounds reasonable.
> partially delivered."  This seems to guarantee head of line  
> blocking where one large partially received message might block all  
> other streams in the same association for a considerable time.
This is true in particular on the sending side. All fragments have to  
have
consecutive TSNs and not sending increasing TSN results in other  
problems.

So yes, if you send large messages this streams blocks the other  
streams.
This is not related to the API issue.
> receive operations against a particular stream, so that  
> applications are not forced to de-multiplex the streams  
> themselves.  That would eliminate inter-stream head of line  
> blocking, simplify the application programming model, allow  
> different threads to easily process different streams, and so on.
I do not think that this is a good idea.
Assume that you have one assoc with two incoming streams. The  
receiver calls
a blocking recv for stream 0. But only messages for stream 1 arrive. The
receive buffer gets filled, zero window probing starts and we have a  
deadlock.

I think it is important to see that having one association with two
streams in each direction is different from having two associations with
one stream in each direction.
>
> The ideal way of doing this would be to have a function call to get  
> an associated socket that is bound to a specified stream  pair  
> (inbound and outbound).  A typical use for this would be to easily  
> port a large body of existing code that uses multiple parallel TCP  
> connections, giving unified congestion control.
>
> Something like:
>
>   assoc_id specifies the association (if sd is one-to-many),  
> ignored if sd is one to one.
>   This function creates a stream socket from a primary one-to-one  
> or one-to-many SCTP association socket.  A stream socket is used to  
> branch off traffic for a specific association stream to an  
> independent socket.
>
> All messages for the specified stream number are delivered to the  
> stream socket instead of the primary socket.  All send operations  
> on a stream socket are delivered on an identically numbered  
> outbound stream.  Once a stream socket (and all its aliases) are  
> closed, messages for the specified stream are again receivable on  
> the primary socket.  When the parent association ends, transfer  
> operations on the stream socket fail with an EPIPE error.
>
> ========================
>
> argument when sending a message on a stream socket, and return an  
> error if any other stream number not equal to socket stream number  
> is specified.
> buffers adjustable using the standard setsockopt SO_SNDBUF and  
> SO_RCVBUF options. They should also have independent SCTP_NODELAY  
> settings.
>
> By the way, I am curious why SCTP protocol does not specify offsets  
> in fragmented DATA chunks so that zero copy hardware accelerated  
> message reassembly would be practical, etc.
Possibly because no one suggested it?

---

From: Mark Butler <[email-masked]>
Date: Mon, 21 Nov 2005 20:38:43 -0700

Mark Butler wrote and Michael Tuexen replied:

>> where one large partially received message might block all  other 
>> streams in the same association for a considerable time.
>

I understand that SCTP currently requires fragments to have consecutive 
TSNs, which implies transmission level HOL blocking on large messages.  
However, this should not force the API to block delivery of a message 
from a different stream in the presence of packet loss on a partially 
delievered message on a separate stream.  Perhaps the documentation 
should specify that only one message at a time may be received per stream.

>> receive operations against a particular stream, so that  applications 
>> are not forced to de-multiplex the streams  themselves.  That would 
>> eliminate inter-stream head of line  blocking, simplify the 
>> application programming model, allow  different threads to easily 
>> process different streams, and so on.
>

This problem is eliminated if you use O_NONBLOCKING or multiple threads 
with stream specific sockets as I described.

---

From: Randall Stewart <[email-masked]>
Date: Tue, 22 Nov 2005 08:58:19 -0500

> should specify that only one message at a time may be received per stream.

I don't see that an implemenation cannot make the choice
to allow a read of a partial delivery message and then
still allow another message to be sent up from a different
stream.. as long as the application realizes that it
would need to pay careful attention to the SID in the
message as it receives things..

In fact thinking on this.. it would only work if you
were getting the snd_rcvinfo structure. For applications
that are NOT getting the snd_rcvinfo structure and only
looking at the MSG_EOR in the recvmsg() flags.. this would
mean you would do:

rcvmsg(...) <---No MSG_EOR (partial delivery p1of5)
rcvmsg(...) <---No MSG_EOR (partial delivery p2of5)
rcvmsg(...) <---MSG_EOR (but from another stream)
rcvmsg(...) <---No MSG_EOR (partial delivery p3of5)

And at the 4th read the app would be confused...

I think the salient point here is that without
getting the sndrecv_info every time you read then
the above would be broken.

Thats probably why we did make it seem that a PD-API
message blocks all others..

And thinking about it the transport level consequences...
If the stack invokes the PD-API then that means the
stack is running out of space and still in the process
of receiving the message. This means that the sender
(unless its sending out-of-order TSN's) is going to
send nothing but this message you are getting up the
PD-API...

Only at the tail of the senders HOL that is going on
could another message slip in..

Hmm.. overall it seems to me that this minor possiblity
is not worth doing especially when you consider
the consequneces to the app that does not get
the sndrcv_info stuff..

R

-- 
Randall Stewart
803-345-0369 <or> 815-342-5222(cell)

---

From: Mark Butler <[email-masked]>
Date: Tue, 22 Nov 2005 10:09:36 -0700

> the above would be broken.

I would say that any application that uses only recvmsg() on a multiple 
stream assocation is probably fundamentally broken.  That is why I 
suggested a peeloff stream socket API, so that multi-stream applications 
could be programmed in the traditional manner, one socket per stream on 
the same association.

> could another message slip in..

Say you are transmitting over a connection with a large bandwidth delay 
product, the receiver is running out of buffer space, and you send a 
message with three fragments, followed by a series of other messages 
that are all in-flight.

Somewhere along the line the packet carrying the second fragment of the 
first message is  lost.  On the arrival of the third fragment a SACK is 
returned, and the stack partially delivers the first fragment.

The series of following in-flight messages arrive unharmed.  Following 
the API requirement, the application issues a sctp_recvmsg() to get the 
remainder of the first message.

But this call cannot succeed until the SACK arrives at the sender, and 
fragment 2 is redelivered to the receiver, typically until a minimum RTT 
has passed.  Meanwhile, messages from other streams (up to RTT*BW worth) 
fill the receive buffers in a manner unable to be remedied through 
partial delivery and every other deliverable message is either blocked 
or dropped due to buffer overflow.

Basically, partial delivery as specified causes the whole association to 
degrade to single connection TCP behavior in the presence of packet 
loss, with no significant upside.   Applications that use sendmsg() and 
recvmsg() calls alone can only effectively use one stream, so they do 
not need this behavior - they only need the weaker condition that 
partial deliveries on any given stream are done in order.  Might that be 
acceptable?

 - Mark

---

From: Vlad Yasevich <[email-masked]>
Date: Tue, 22 Nov 2005 14:44:25 -0500

Hi Mark

I am a little confused by this scenario.  Most of my comments are
based on the few implementations I am familiar with.

> message with three fragments, followed by a series of other messages

The receiver generally starts running out of buffer space if the
messages aren't consumed, right?

> returned, and the stack partially delivers the first fragment.

OK (assuming messages aren't consumed and the buffere space is filling up)

> remainder of the first message.

At this point, the application will be consuming data that's been
backing up and releasing the buffer space...  That doesn't mean that it
will try to consume the partial message.

> has passed. 

The app will start consuming messages that have been backing up and
freeing up the buffer space.  There is no way to expicitely say "I want
some particular message".

> Meanwhile, messages from other streams (up to RTT*BW worth)

Other messages might be delivered if backlog is beeing drained.

-vlad

---

From: Mark Butler <[email-masked]>
Date: Tue, 22 Nov 2005 17:26:35 -0700

>messages aren't consumed, right?

Yes.

>will try to consume the partial message.

It has no other choice once a partial delivery has been initiated.

>freeing up the buffer space.  
>
The application cannot drain other messages from the buffer once a 
partial delivery is in progress. This is a problem in the presence of 
packet loss.

>There is no way to expicitely say "I want
Of course.

>Other messages might be delivered if backlog is beeing drained.
My point is that the API forces the implementation to stall delivery of 
all other messages until the remainder of partially delivered message 
arrives, making the backlog worse after a lost packet, in a manner that 
SCTP itself does not require.  This is a particularly bad problem on a 
many-to-many style socket.  If a peer goes down or becomes unreachable, 
the socket would stall until the implementation timed out and delivered 
a partial delivery aborted event.  This might be several seconds during 
which all other clients are blocked.

As an additional problem, partial delivery isn't thread safe without 
application level mutexes - mutexes that have to be held for an 
arbitrarily long time. Even without any packet loss, a mutex protecting 
the socket would have to be held until the message is delivered 
completely. 

A better option would be to allow the application to specify a maximum 
message size and disable partial delivery completely. The implementation 
would then be required to reserve enough space to receive a maximum size 
message before acknowledging chunks from fragmented messages.

Perhaps something like setsockopt(...,SCTP_MAXMSGSIZE,....) and 
setsockopt(...,SCTP_PARTIAL_DELIVERY,[0 | 1]).

The ability to disable partial delivery would drastically simplify the 
process of converting current UDP server code to SCTP.  Just change the 
protocol family, set a socket option, and it just works.

 - Mark

---

From: Randall Stewart <[email-masked]>
Date: Mon, 28 Nov 2005 08:24:57 -0500

> or dropped due to buffer overflow.

Hmm.. in order for me to believe this scenario you would
then have to have, added to it, a really small rwnd. Thats
the only way I could see your scenario working. For the BSD
implementation the default rwnd is aroudn 233k... So
in the above example the app would have had to have 220k or
so of unread data... this seems a stretch...

I will concede that it can happen, but your scenario seems
unlikely to me unless the app shrinks the rwnd to 4k or so
which on a large BW/Delay connection is the last thing
an APP should be doing.. if nothing else it should be
raising it higher...

R

---

From: Randall Stewart <[email-masked]>
Date: Mon, 28 Nov 2005 08:39:42 -0500

>> Other messages might be delivered if backlog is beeing drained.

Mark:

One thing here is that I think that you will find
that most implemenations put OFF starting a PD-API until
the last possible moment...

For example.. the BSD implementation will NOT start a partial
unless:

The queue of un-delivered data (aka small messages
that have B&E bits set) is empty.

AND
[
the entire message is present (aka no pd-api)
OR
the data on the reasm queue is larger than
the hiwater mark (sock recv buf) minus 6000 bytes
and we have a portion of the message deliverable.
]

This means that almost all the recv window must be
dedicated to this large message(s)...aka in the
default case you must have fragmented messages
totaling about 226,000 bytes of data on the
re-assembly queue.... Your example you gave
would not work unless you get the socket rwnd
down below 10k... (which would be silly on
a high bw/delay socket)...

> the socket would have to be held until the message is delivered completely.

I would guess the "SCTP_MAXMSGSIZE" would have to be executed
on the sender side? Right now there is a way to turn off
message fragmentation... but thats a different thing all
together... I have no problem adding the above in the
next draft... it sound reasonable..

and the SCTP_PARTIAL_DELIVERY 0/1 would have to be executed on
the receiver side?

And so does turning off the pd-api ... of course that
could cause problems for the app if the sender side
did not do the SCTP_MAXMSGSIZE though :-0

R

...

> ------------------------------------------------------------------------
>

---

From: Mark Butler <[email-masked]>
Date: Mon, 28 Nov 2005 17:20:48 -0700

> the last possible moment...

Yes - Lksctp is the same way.  The problem is that a partial delivery 
serializes all other associations on the same one-to-many socket until 
it completes, which may be never.  Inter-stream serialization also bad, 
but not nearly as serious - most applications can tolerate that on 
occasion.  Stalling every association on a one-to-many socket should not 
be allowed, certainly not by default.

> ....

For protection against one-to-many socket stalling attacks, it should 
not have to.  However, normally, it should be set on the sending and 
receiving side to the same value, to allow immediate rejection on the 
sending side, and truncation on the receiving side.

> and the SCTP_PARTIAL_DELIVERY 0/1 would have to be executed on

It would be preferable to have partial delivery off by default, 
_especially_ for one-to-many sockets.  Current applications would only 
need to enable partial delivery if they needed to support very large 
messages.

> did not do the SCTP_MAXMSGSIZE though :-0

Maximum message size should normally be specified at the ULP level, so a 
compliant sender should not have to do anything (with partial delivery 
off by default, and adequate receive buffer sizes). I propose that 
truncation (flagged w/ MSG_TRUNC) be used to handle messages from 
non-conforming senders.  (Please refer to my other message)

 Thanks,

   Mark