Subject: [Tsvwg] SCTP SAR limitations

From: Mark Butler <[email-masked]>
Date: Mon, 21 Nov 2005 21:53:30 -0700

The SCTP protocol currently requires that message fragments be assigned 
consecutive TSNs, which causes large messages to block the transmission 
of short messages because the transmitter cannot interleave message 
fragments.

This makes it inadvisable to generate SCTP messages larger than a small 
multiple of the PMTU in many applications.  If an application needs 
larger message sizes,  message segmentation and reassembly must be 
performed by a higher level protocol, such as DDP.  However, the "cell" 
tax of SCTP + DDP is excessive in applications with predominantly small 
message sizes.  Upper layer techniques generally require applications to 
implement an independent message segmentation and reassembly layer, 
complicating the programming model and causing unnecessary context switches.

The latency/jitter problem could be solved by adding a SCTP protocol 
option that adds a fragment offset field to the header of fragmented 
message chunks, allowing transmitters to interleave message fragments 
from different streams as advisable, typically round robin fashion 
switching streams after each DATA chunk, rather than switching only 
after each message.

Does anyone else think such a protocol option would be a good idea?

 - Mark Butler

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 22 Nov 2005 02:55:21 -0700

Mark,

We discussed this in detail, oh, about a year or so ago.  I am still
not convinced that it is necessary to block other streams and that
other messages cannot be interleaved anyway.  Our implementation does
this without problems for years.  Noone seems to be able to point to
a concrete reason why a sender should be restricted from intermingling
fragments from different streams and the logic at the receiver required
to properly piece the messages together is trivial (if not already
present).

Gosh, sorry, that was 4 years ago.  See the ladders under the subject
"SCTP and fragmentation" posted to both the tsvwg and sigtran mailing
lists in the archives starting from November 11, 2001.

I asked that the MUST be downgraded to a SHOULD and was told that the
I-G could not change the protocol...

Hah!  That is quite funny in retrospect.  Now its a PS.

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Mark Butler <[email-masked]>
Date: Tue, 22 Nov 2005 05:56:26 -0700

>to properly piece the messages together is trivial (if not already
Thanks for your feedback, Brian.

As I understand it, since message fragments are without consecutive 
fragment sequence numbers or fragment offsets, reconstructing a 
fragmented message can only be done once DATA chunks have arrived for 
all TSNs from the message start TSN to the message end TSN inclusive, 
because any intervening TSN is a candidate to be part of the message.

I agree this can be implemented, but it appears it creates an 
inter-stream dependency for fragmented messages that could be avoided 
with the appropriate offset or fragment sequence header.  This 
completion dependency for large messages is probably better than 
preempting all other streams, but a fragment offset / sequence number 
option would eliminate both problems and be much more economical to 
implement.

I am concerned about the current API requirement for only one message to 
be deliverable on a socket at a time.  In the presence of packet loss 
and partial delivery, this could block the socket for at least an RTT, 
stalling deliverable messages on other streams.  It would be highly 
desirable for an SCTP association not to behave worse than a comparable 
bundle of TCP connections.  The API is still fixable right?

  - Mark Butler

---

From: Randall Stewart <[email-masked]>
Date: Tue, 22 Nov 2005 09:33:42 -0500

Mark:

A couple of comments in-line...

> complicating the programming model and causing unnecessary context 

As Caitlin pointed out... I don't really think that a
socket API is going to be the most ideal for SCTP+DDP
(or TCP+DDP for that matter). It becomes very difficult
to pre-post buffers and do all the things that DDP/RDMA
expect. Now that being said... as to adding a new DATA
type to SCTP.. for that is what you would be doing ..
SCTP is extensible.. write a draft that defines what
you want it to do with any new feilds (offset etc)..

> Does anyone else think such a protocol option would be a good idea?

I don't see that it is necessary to change the base protocol
but I have no problem with you writting a new draft that
proposes a new type of data chunk ;-)

R

-- 
Randall Stewart
803-345-0369 <or> 815-342-5222(cell)

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 22 Nov 2005 12:10:16 -0700

Mark,

> because any intervening TSN is a candidate to be part of the message.

Well, there cannot be any gaps in the TSNs received up to the point that
the message is being assembled, and partial delivery therefore cannot
overtake a gap: but that is the always the case, whether other chunks
are intermingled or not.

> option would eliminate both problems and be much more economical to 

If you mean that retransmission of the little message that fits in a TSN
getting lost blocks the big hunking message, that is not true.  The big
hunking message would have gotten blocked by the lost of a TSN anyway.

> bundle of TCP connections.  The API is still fixable right?

The result from the last discussion was that SCTP addresses HOL blocking
on the receive side, but fails on the sending side: the sender must send
smaller messages to compensate for SCTP's shortcomings.

--brian

---

From: Mark Butler <[email-masked]>
Date: Tue, 22 Nov 2005 17:59:02 -0700

>overtake a gap: but that is the always the case, whether other chunks
A fragment sequence or offset would provide the ability to overtake 
gaps, something that is necessary for robust behavior with fragment 
interleaving.

>hunking message would have gotten blocked by the lost of a TSN anyway.

Again, a fragment sequence/offset option would fix this.

>smaller messages to compensate for SCTP's shortcomings.
The receive side problem here is with the API, not the protocol.

 - Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 22 Nov 2005 19:37:16 -0700

Mark,

>    Again, a fragment sequence/offset option would fix this.

Yes.  If you look back at the DDP discussion, though, placing an offset
in the Data Chunk in support of DDP was nixed by most of those involved
at the time.  The view was that if the application (or an adaptation
layer, as was intended for SCTP) wishes to place an offset in the
payload and send payloads unordered and perform its own placement (and
jumping around gaps) it is welcome, but within the transport user, or
adaptation layer, and not SCTP.  I believe that is how DDP arrived at
where it is today.

I don't think that those involved with SCTP in the WG will be any more
receptive today to including an offset in the Data Chunk than they were
4 years ago.  Some simple and elegant proposals were put forward for
simple segmentation marking method for Data Chunks.  It wasn't for lack
of a straightforward technical solution, but the belief within the WG
that this was a mechanism that should be placed in the application
rather than the transport protocol.

I agreed that offset should be handled by the application layer at the
time, particularly for DDP.  DDP is quite heavily tied to an API
implementation and attempting to accomodate all possible API
implementations at the transport layer seemed to me to be asking for
trouble.

>    The receive side problem here is with the API, not the protocol.

True, but the send side problem is with the protocol, not the API.  That
is what we were focusing in on our discussions before.

What is your worry with the API?  IETF does not standardize
implementation specific APIs.  The document is INFORMATIONAL.  It only
represents one way of doing things.

You can find another approach to an SCTP sockets interface here:

  http://www.openss7.org/man/man2html?7+sctp

which some have commented is closer to POSIX.  And the outline of the
TPI version of the interface here:

  http://www.openss7.org/man/man2html?7+tpi_sctp

Also a detailed XTI interface here:

  http://www.openss7.org/man/man2html?3+xti_sctp

(There is also an NPI interface, but the manpage is incomplete on the
web server.)

Communications frameworks that are neither Sockets nor STREAMS will have
a completely different interpretation.

If I wanted to (which I don't), a pretty simple library could translate
between the sockets interface above and the limited one in the draft.

I could write an INFORMATIONAL RFC for either of both of these too, and
get them published as such.  The IETF probably wouldn't endorse any of
them, and by generating an INFORMATIONAL RFC have not really endorsed
Randall, et al.'s either.

If your concern is that as a POSIX user you want a standardized way of
doing things for portability and all those other reasons, why not wait
and raise your concerns in the IEEE ballot?  (That is, if it ever makes
it that far, or if we haven't retired by then.)

The INFORMATIONAL RFC approach (IMO) is primarily advanced by implementers
of the protocol and is based more on several protocol implementaion
views of the world, rather than what users expect from the interface.

--brian

---

From: Mark Butler <[email-masked]>
Date: Tue, 22 Nov 2005 23:22:44 -0700

>adaptation layer, and not SCTP.  I believe that is how DDP arrived at
That is interesting.  Thanks for the information. 

>implementations at the transport layer seemed to me to be asking for
I would say that they are independent issues.  Message fragmentation at 
the SCTP layer affects a variety of existing ULPs that do not use DDP.  
Fragment interleaving support (and better loss recovery) could 
transparently benefit them.  DDP adaptation layers for a variety of 
common ULPs may be a long time in coming, where ULP multi-streaming over 
SCTP is relatively trivial by comparison.

 ATM may not be the world's most successful protocol, but surely they 
had a point doing interleaving as low as layer 2.  The opportunity may 
be past for SCTP, but it seems unfortunate when nearly all the 
infrastructure is already there.

>represents one way of doing things.
My primary concern with the API is setting a de facto standard with 
behavior that is potentially difficult to program or forces undesirable 
behavior, such as stalling all threads of a directory server while a 
partial delivery times out. Portability is a concern as well.

With SCTP it should be possible to specify byte stream vs message 
orientation, reliability, and ordering on a stream by stream, post 
initiation basis . If you could peel off stream specific sockets, than 
an SCTP association could substitute (at runtime) for the bundle of 
"connections" typical of media protocols, without gratuitous code 
changes.  Any comments on that idea?

 - Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 23 Nov 2005 00:52:53 -0700

Mark,

> partial delivery times out. Portability is a concern as well.

I don't see how it can become a de facto standard when a formal standard
(POSIX/SUSv3) exists for the sockets interface, and especially if it is
unusable in some respect.

> changes.  Any comments on that idea?

In the DDP discussion there was talk of splitting an "SCTP-stream"
socket using some kind of peel-off operation as well.

An SCTP-stream, however, is not a full-duplex concept.  It only goes in
one direction.  As far as the SCTP transport protocol is concerned,
outboudn stream #5 at one peer has no relationship to any outbound
stream at the other peer.  Relationships between outbound streams at
either end of the association is left completely for the ULP to decide.
And most do.  SIGTRAN UAs, for example, specify how messages are to be
placed on streams, however, under SIGTRAN there is still no mapping
between an outbound stream at one end of the association and an outbound
stream at the other.  DDP is a little different.  iSCSI/SCTP is
different too.  So is SIP/SCCP.  The list goes on.

So, to peel off this stream-socket, it would proabably become necessary
to specify some how in the peel off process which inbound and outbound
streams belong to the socket.  Also, because there may be no full-duplex
mapping between an inbound and outbound stream, its applicability would
be severely limited to those ULPs that try to make a "connection" out of
a pair of streams.  Many ULPs do not. SIGTRAN does not.  My primary
interest area for SCTP is SIGTRAN.  So, I'm kind of neutral on the idea.

I certainly don't see a problem in receiving intermingled messages from
different streams taking a POSIX sockets approach on SOCK_SEQPACKET
sockets.  Using the API I mentioned[1], handling this is a simple matter
of setting SCTP_RECVID so that SCTP_SID will be delivered as anciliary
data on all successful recvmsg(2) calls.  The returned MSG_EOR flag
indicates whether some subsequent read will return more data for the same
stream.  The SCTP_SID delivered in anciliary control part indicates
which stream the message (or message fragment) the actual received data
belongs to.  Yes this means that the application is performing some
functions of reassembly (or really just demultiplexing) when messages
are really big or highly variable in size.

I don't know how Randall's draft handles this, but the restriction of
reading one message from one stream at a time until that message is
completely received doesn't make much sense to me.  It is, IMO, an
unnecessary restriction, unless one is treating messages from all
streams like a single stream (e.g., for a socket compatible with an
application written for TCP SOCK_STREAM socket that ignores message
boundaries, treats messages like they are a byte stream, interleaves
complete messages into this single byte stream, and does not report
stream-id or MSG_EOR to recvmsg(2)).  Or, if one is using the read(2),
readv(2), recv(2) or recvfrom(2) interface that cannot receive the
necessary stream-id information.  But then, those interfaces are
seldomly used for SOCK_SEQPACKET sockets anyway (case in point OSI) and
are tuned to treat the socket like a terminal or byte stream (they don't
even return MSG_EOR).

So, I suppose I don't see the need for a peeled off stream-socket, can't
quite figure out how it would work for something like SIGTRAN, and have
the facilities within the POSIX/SUSv3 socket API already to handle the
situation.  Am I missing something? (BOTF)

--brian

[1] http://www.openss7.org/man/man2html?7+sctp

---

From: Mark Butler <[email-masked]>
Date: Wed, 23 Nov 2005 07:24:29 -0700

>unusable in some respect.
I can see that partial delivery is helpful in some cases (single 
threaded, one-to-one, very large messages) , I am just trying to explain 
why it would be a good idea to be able to turn it off.

>And most do.  
Since sockets are generally a full duplex abstraction, some allowance 
must be made.  If it would be helpful, the stream peeloff interface 
could allow separate inbound and outbound stream numbers, or leave 
either unbound for strictly half duplex operation.

>interest area for SCTP is SIGTRAN.  So, I'm kind of neutral on the idea.
Half duplex peeloff solves this problem, I believe.

>are really big or highly variable in size.
The problem with partial delivery using MSG_EOR is stalling other 
threads or processes receiving messages on the same socket, particularly 
one-to-many style sockets in a common producer-consumer or request 
dispatch model.  Directory servers are a latency sensitive application 
where partial delivery is unacceptable behavior because it stalls 
dispatch of requests from all other clients.

 - Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 23 Nov 2005 11:21:58 -0700

Mark,

>    Half duplex peeloff solves this problem, I believe.

As I said, I am neutral because I have no use for it...

>    threads   or   processes   receiving  messages  on  the  same  socket,
>    particularly  one-to-many  style sockets in a common producer-consumer
>    or  request dispatch model.  Directory servers are a latency sensitive
>    application where partial delivery is unacceptable behavior because it
>    stalls dispatch of requests from all other clients.
>     - Mark

I have no use for 1:many either.  But regarding partial delivery, you
are using the wrong design pattern for the job.  If you attempt to have
your entire thread poll accessing the same TCP SOCK_STREAM socket, your
performance will degrade.  Did you think SCTP with large messages would
be any different?  Use a dedicated set of worker threads to read the
socket and demultiplex messages (just as one would for SOCK_STREAM) to
act as the producer pool, pass them to the consumer pool from there.
Promoting multiple readers is going to generate lock contention anyway.
Think poll or select or another asyncrhonous event driver interface
such as STREAMS, or POSIX asyncrhonous IO.  Or, better still, use an
existing communications framework (ACE comes to mind) where this low
level work is done for you, allowing portability across SOCK_DGRAM,
SOCK_STREAM and SOCK_SEQPACKET.

--brian

---

From: Mark Butler <[email-masked]>
Date: Wed, 23 Nov 2005 14:59:20 -0700

>be any different?  
No.  Stalling during partial delivery is not caused by threading.  After 
a partial delivery of a message, no other messages can be delivered 
until the rest of the original message arrives and is delivered to a 
reader.  The problem is that this can be an arbitrary amount of time - 
most likely the RTT to replace a lost packet, but perhaps 60 seconds or 
more if contact is lost with the sender.

Here is an extract from the Linux kernel SCTP implementation 
(2.6.10,sctp/ulpqueue.c):

199         /* If we are in partial delivery mode, post to the lobby until
200          * partial delivery is cleared, unless, of course _this_ is
201          * the association the cause of the partial delivery.
202          */
203
204         if (!sctp_sk(sk)->pd_mode) {
205                 queue = &sk->sk_receive_queue;
206         } else if (ulpq->pd_mode) {
207                 if (event->msg_flags & MSG_NOTIFICATION)
208                         queue = &sctp_sk(sk)->pd_lobby;
209                 else {
210                         clear_pd = event->msg_flags & MSG_EOR;
211                         queue = &sk->sk_receive_queue;
212                 }
213         } else
214                 queue = &sctp_sk(sk)->pd_lobby;

Whenever the socket has a partial delivery pending, the ULP queue 
function has to hold other incoming messages in a separate partial 
delivery wait queue, here called "pd_lobby".  When the partial delivery 
is cleared,  messages can again be queued on the socket receive queue.

225         /* Did we just complete partial delivery and need to get
226          * rolling again?  Move pending data to the receive
227          * queue.
228          */
229         if (clear_pd)
230                 sctp_ulpq_clear_pd(ulpq);
231
232         if (queue == &sk->sk_receive_queue)
233                 sk->sk_data_ready(sk, 0);
234         return 1;

And of course, after a partial (multi-phase) delivery is completed, any 
pending messages have to be returned to the receive queue as well.

This is not an implementation specific issue - any implementation that 
does partial delivery on a shared socket must block other messages until 
the original message is delivered completely.

This delay is probably at most an RTT or so on a healthy network, but if 
an endpoint disappears silently due to a routing failure or system 
crash, the socket will block indefinitely while the stack waits for the 
rest of the partially delivered message to arrive.   It might be thirty 
to sixty seconds before the absence of a heartbeat is noticed, the 
association is shutdown, a partial delivery event is posted, and inbound 
messages are again readable on the socket.

On a one-to-one style socket, partial delivery just causes the 
association to degrade to single connection TCP behavior, a tolerable 
albeit unnecessary short term delay if the path is up, and unavoidable 
hanging if the the path is down.  However, on a one-to-many style 
socket, either type of delay stalls all the other associations, which is 
bad for 100 ms delays, and practically catastrophic for 30000 ms delays.

Worse partial delivery makes denial of service attacks against such 
servers trivial - just establish an association and send a series of 
messages that contain missing fragments.  As the receive window fills 
up, the stack will attempt to do a partial delivery on one of your 
messages, after which the socket will be unusable because the rest of 
the message will never arrive. The stack will think you are still alive, 
indefinitely stalling the server process.

Comments anyone?

 - Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 24 Nov 2005 03:24:18 -0700

Mark,

On Wed, 23 Nov 2005, Mark Butler wrote:

>    No.   Stalling  during  partial  delivery  is not caused by threading.
>    After  a  partial  delivery  of  a  message,  no other messages can be
>    delivered  until  the  rest  of  the  original  message arrives and is
>    delivered  to  a reader.  The problem is that this can be an arbitrary
>    amount  of  time  -  most likely the RTT to replace a lost packet, but
>    perhaps 60 seconds or more if contact is lost with the sender.

--X-snip-X--

I have no doubt that you are correct with regard to the draft API and
the lksctp implementation.  I am rather surprised to see the amount of
trouble that was taken in implementation to introduce unnatural
blocking.

My point was that, if an implementation uses the MSG_EOR in the POSIX
way, there is not a problem.  Here's how I implemented it:

DATA Chunks are held in the receive buffer awaiting delivery.  When it
becomes possible to deliver a data chunk to the ULP, whether a fragment
or not, in accordance with gap processing and TSN sequence, that data
chunk is delivered to one of two receive queues: one for unordered
chunks (from all streams), the other for ordered chunks (from all
streams).  Fragmented chunks are delivered to the queue as soon as it is
possible to deliver them.

In sctp_recvmsg() processing, chunks are read from the ordered receive
queue if the MSG_OOB flag is not set on call.  Chunks are read from the
unordered receive queue if the MSG_OOB flag is set.  This permits
unordered data to be expedited, which is important for SIGTRAN
applications.

During sctp_recvmsg() processing, data is delivered to the user buffer
until the count is satisfied, a chunk has been completely read that had
the E flag set,  or a chunk is encountered on the queue that belongs to
a different stream than the data transferred.  If data remains in a
chunk after the count is satisfied, for SOCK_SEQPACKET the remainder is
placed back on the queue for a subsequent read.  Whenever a chunk is
read that has an E bit set (terminating the read), the MSG_EOR flag is
set on return.  Clear otherwise.

Now this is not precisely POSIX.  POSIX says that only a message at a
time is read, so a strict interpretation would mean that the read should
terminate at the end of the chunk.  The user can't really tell, so big
deal.  Coalescing adjacent chunks in this way is more efficient.  OTOH,
treatment of the MSG_EOR flag is POSIX.

Different logic is used for a TCP compatible byte-stream mode
(SOCK_STREAM) an RUDP compatible unordered message mode (SOCK_RDM).

So, in your case here is what happens.  The receive receives the first
of three chunks in a fragmented record on stream #1, and the second is
lost, and then the third (last) chunk is received.  Then, say, 50 chunks
from other streams.  The gap logic delivers the first chunk to the
receive queue. Then, because the gap does not block other streams, the
other 50 chunks are delivered to the receive queue.  The third chunk in
the sequence is blocked from hitting the receive queue by the missing
second chunk.

On the first recvmsg() call, the ULP gets the first fragment.  Because
the next chunk in the receive queue belongs to a different stream, the
read terminates.  The MSG_EOR flag is clear on return, indicating to the
caller that a subsequent recvmsg() call will be required to retrieve the
remainder of the message.  The SCTP_SID ancillary data returned
indicates stream #1.  The ULP calls recvmsg() again.  When the call
returns, the caller can find that MSG_EOR is set, but the SCTP_SID
indicates stream #77, representing a complete record received from
stream #77.  Each recvmsg() call will continue to deliver data that
arrived before the second fragment is filled in, until chunks 2 and 3 in
the sequence finally arrive at the head of the receive queue.  Then a
call to recvmsg() will finally return SCTP_SID indicating stream #1 and
MSG_EOR indicating the end of a record.

This type of approach is done for OSI protocols on SOCK_SEQPACKET all
the time (SOCK_SEQPACKET and MSG_EOF was added to support OSI
protocols).

It is to this interface that I meant to say the other design patterns
were applicable.

I agree with you, the draft API is broken and should be fixed, but I
don't know that this is really the right fora in which to fix it.

--brian

---

From: Randall Stewart <[email-masked]>
Date: Mon, 28 Nov 2005 08:14:08 -0500

One comment below...

> pending messages have to be returned to the receive queue as well.

I don't see that to be true in the general case... basically there
IMO should be no reason that you could not read another message
as long as the message includes the "ancillary data" to tell you
that this new message is from a different stream...

Now that being said, I think the problem is more related to
the framework of the implementation.. at least it is for BSD. The
sockets interface is limited in that ALL the recv/recvmsg/recvfrom
funnel into the same set of protocol reads. This means that if
an implemenation did this, then any application that used the
recv/recvfrom call and was sent a big message with a small
one interleaved somewhere (on a different stream) would have
be broken.

So, its not that it cannot be done.. but I think that
many of the socket framework within the various stack
makes it hard to do.. without major changes to the socket
side of things...

For us, in working with the BSD stack, we were asked to
limit the amount of things we touched on that side of the
code... so we have the same issue that the linux folks have.

There is a way to turn off the PD-API though... basically
it must be done on the sender side... Can't remember the
option but it was added back in the DDP discussion era..

And, as Brian said, the sockets draft will be informational and
only that.. it is shaping up to be the defacto standard... since
solaris, linux, bsd, and HP all use it... not sure about
AIX....

> the original message is delivered completely.

If we state that in the draft (not sure we may do this).. we should
not.. and I will have a look and change it if its so.. it really
becomes an implementation choice as to if this is so...

> indefinitely stalling the server process.

As I said, this is a implemenation choice, not a mandate... so I don't
see it as an issue... And I may address this if I can figure out how
in the BSD implemenation without major changes to the sockets code
itself that is :-0

R

> Comments anyone?
> 
> - Mark
> 
> 
> 
> ------------------------------------------------------------------------
>

---

From: Mark Butler <[email-masked]>
Date: Mon, 28 Nov 2005 14:50:23 -0700

>unordered data to be expedited, which is important for SIGTRAN
I can see where this would be useful.  However, using MSG_OOB makes it 
hard to block for the receipt of either in band or out of band data.  
i.e one would have to use select and then issue a pair of non-blocking 
reads, one in band and one out, right?

>...
Yes, this would prevent stalling.  The problem is that this technique 
moves demux responsibility one level higher. Atomic message delivery 
semantics make the application developers job much easier.

 - Mark

---

From: Mark Butler <[email-masked]>
Date: Mon, 28 Nov 2005 16:53:08 -0700

Randall, I have a suggested api change to solve this problem (and add 
considerable other benefits) detailed below. First a couple of comments.

>> This is not an implementation specific issue - any implementation 
>> that does partial delivery on a shared socket must block other 
>> messages until the original message is delivered completely.
>
>
>

It is in the second to last paragraph of section 3.1.4 - "Only one 
message at a time can be partially delivered.".

This requirement is a good one. Removing it would solve the one-to-many 
stalling problem, but would make application coding much more complex - 
no simple recvmsg wrappers, for example. 

A one-to-one socket application that needs partial delivery, e.g. to 
handle very long messages, can probably live with inter-stream 
serialization.  One-to-many sockets are where partial delivery is a 
serious problem.

The ability to control partial delivery would solve the problem 
completely for most applications - no wrappers, no manual stream demux.  
Of course, without partial delivery, the stack must do something if sent 
a message larger than SO_RCVBUF. 

Proposal:

Add a SCTP_PARTIAL_DELIVERY option. 1 on, zero off.  It should be 
disabled by default, at least for one-to-many sockets, where partial 
delivery is positively dangerous. Disabling it by default for one-to-one 
sockets would simplify the programming model and increase portability - 
no more MSG_EOR loops everywhere.  The only applications that should 
need to enable partial delivery are ones with large message sizes (>64K) 
or serious memory constraints.

Add a SCTP_MAXMSGSIZE socket option, such that any larger message 
received over the wire be delivered in truncated form (with MSG_TRUNC 
set), relieving buffer pressure. Further chunks from the same message 
would be discarded.  The same option would also be enforced on the 
sending side, by rejecting larger message sends with EMSGSIZE.

Applications should be required (by default) to use a receive buffer at 
least twice as large as the required message size. On the implementation 
side, by default (partial delivery disabled) the effective maximum 
message size should be clamped to one half the size of the receive 
buffer. Setting a SCTP_MAXMSGSIZE larger than SO_RCVBUF/2 should return 
an error.  Messages sent over the wire larger than 
minimum(SO_MAXMSGSIZE, SO_RCVBUF/2) should be delivered in truncated 
form (again with MSG_TRUNC set) by the receiver.  This guarantees that 
the stack will not stall. 

When partial delivery is enabled, the SCTP_MAXMSGSIZE restriction should 
be enforced (for consistency), but not the receive buffer related 
message size limitation.

Applications with proper buffer sizes would always get atomic delivery, 
never see truncated messages from conforming clients, be resilent to 
network problems and malicious clients, never stall across different 
associations, and have reduced inter-stream serialization by default.  
Applications needing very large messages (generally on a one-to-one 
socket) can enable partial delivery where required.

How does that sound?

 - Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 28 Nov 2005 17:09:42 -0700

Mark,

> reads, one in band and one out, right?

If you want to do that, just set SO_OOBINLINE with setsockopt(2).

> semantics make the application developers job much easier.

Yes, but beneath a limit.  You can't send more than 4096 bytes through a
POSIX pipe without interleaving either...

If your application cannot handle interleaving, send smaller records.
On the send side I do not fragment a record for transmission to smaller
than 1/2 the maximum data chunk payload for the MTU.  That means that
messages sent smaller than about half an MTU will not be interleaved.

You have to do this to avoid HOL blocking on the send side anyway.
Remember?--a protocol problem, not an API problem.

--brian

---

From: Mark Butler <[email-masked]>
Date: Mon, 28 Nov 2005 18:13:22 -0700

Brian,

>If you want to do that, just set SO_OOBINLINE with setsockopt(2).
Thanks, I didn't know that.

>POSIX pipe without interleaving either...
If you have more than one stream, yes.  Of course UDP applications like 
NFS do larger sizes - 8 or 32K. NFS over SCTP ought to be practical, as 
should atomic delivery for anything up to at least 64K.  In draft api 
implementions like LKSCTP, there is the full infrastructure to make this 
happen almost all the time, so I suggest that users would be well served 
by making it guaranteed (by default).

>Remember?--a protocol problem, not an API problem.
The protocol problem is only per association.  The dangerous stalling 
problem is only with one-to-many sockets, which is a draft API problem, 
not a protocol problem.  Partial delivery in the api does serialize 
streams unnecessarily in the presence of packet loss as well.  Small 
messages should be deliverable while recovering from packet loss on 
larger messages, providing the former are unordered or on a different 
stream.

- Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 28 Nov 2005 20:40:19 -0700

Mark,

On Mon, 28 Nov 2005, Mark Butler wrote:

>    problem  is  only  with  one-to-many  sockets,  which  is  a draft API
>    problem,  not  a  protocol  problem.  Partial delivery in the api does
>    serialize  streams  unnecessarily  in  the  presence of packet loss as
>    well.   Small  messages  should  be  deliverable while recovering from
>    packet  loss on larger messages, providing the former are unordered or
>    on a different stream.

Even though I am not interested in 1:many, I have no use for it,
sending smaller records solves both.

It is not a problem guaranteeing no partial deliveries beneath a
certain size; however, I am really lithe to make it a socket option:
SCTP has too many socket options to be usable aready.

--brian

---

From: Mark Butler <[email-masked]>
Date: Mon, 28 Nov 2005 23:59:00 -0700

>SCTP has too many socket options to be usable aready.
Too many options are not a problem if people rarely have to use them.  
Just pick the right defaults.  Partial delivery by default makes the 
protocol hard to use.  If it is disabled by default, then you get the 
simple programming model of SOCK_DGRAM with reliabilty and ordering 
guarantees.

However, you have a good point. Along the lines of your earlier 
suggestion, perhaps it would be preferable to align the socket types 
with Posix. It would be cleaner to use SOCK_DGRAM instead of 
SOCK_SEQPACKET for one-to-many sockets, as well as sockets that require 
atomic message delivery.  SOCK_DGRAM does not have to mean unreliable or 
unordered. This is what Posix (1003.1-2004) says:

       The SOCK_DGRAM socket type supports connectionless data transfer
    which is not necessarily acknowledged or reliable. Datagrams may be
    sent to the address specified (possibly multicast or broadcast) in
    each output operation, and incoming datagrams may be received from
    multiple sources. The source address of each datagram is available
    when receiving the datagram. An application may also pre-specify a
    peer address, in which case calls to output functions shall send to
    the pre-specified peer. If a peer has been specified, only datagrams
    from that peer shall be received. A datagram must be sent in a
    single output operation, and must be received in a single input
    operation. The maximum size of a datagram is protocol-specific; with
    some protocols, the limit is implementation-defined. Output
    datagrams may be buffered within the system; thus, a successful
    return from an output function does not guarantee that a datagram is
    actually sent or received. However, implementations should attempt
    to detect any errors possible before the return of an output
    function, reporting any error by an unsuccessful return value.

SOCK_DGRAM is the only Posix standard SOCK_xxx type that specifies 
one-to-many support and atomic, length restricted message operations. No 
partial delivery allowed.  Of course SOCK_DGRAM can be used in 
one-to-one mode as well. Passive socket support would be necessary.  
Listen()/accept() would behave like (and could share implementation 
with) sctp_peeloff().

Here is what Posix says about SOCK_STREAM:

    The SOCK_STREAM socket type provides reliable, sequenced,
    full-duplex octet streams between the socket and a peer to which the
    socket is connected. A socket of type SOCK_STREAM must be in a
    connected state before any data may be sent or received. Record
    boundaries are not maintained; data sent on a stream socket using
    output operations of one size may be received using input operations
    of smaller or larger sizes without loss of data. Data may be
    buffered; successful return from an output function does not imply
    that the data has been delivered to the peer or even transmitted
    from the local system. If data cannot be successfully transmitted
    within a given time then the connection is considered broken, and
    subsequent operations shall fail. A SIGPIPE signal is raised if a
    thread sends on a broken stream (one that is no longer connected).
    Support for an out-of-band data transmission facility is
    protocol-specific.

Note: "Record boundaries are not maintained".  So partial deliveries do 
not matter. A read should be able to return any number of messages, 
including partial messages. Writes should combine. This would allow 
proper drop in compatibilty with TCP, and more efficient operation for 
such applications as well.  Connection oriented mode only, just like TCP.

Finally, here is what Posix says about SOCK_SEQPACKET:

    The SOCK_SEQPACKET socket type is similar to the SOCK_STREAM type,
    and is also connection-oriented. The only difference between these
    types is that record boundaries are maintained using the
    SOCK_SEQPACKET type. A record can be sent using one or more output
    operations and received using one or more input operations, but a
    single operation never transfers parts of more than one record.
    Record boundaries are visible to the receiver via the MSG_EOR flag
    in the received message flags returned by the recvmsg()
    <../functions/recvmsg.html> function. It is protocol-specific
    whether a maximum record size is imposed.

Here Posix explicitly allows partial deliveries and requires 
applications to check the MSG_EOR flag to get message boundaries.  This 
behavior would also be useful for applications generating messages 
incrementally, and signalling completion at end of record, presumably by 
setting MSG_EOR.  This category is particularly appropriate for large 
messages.  Connection oriented, one-to-one mode only, of course.

What about streams? SOCK_DGRAM doesn't have a problem - no partial 
deliveries allowed. SOCK_SEQPACKET could serialize partial deliveries 
across streams the way it is specified now - no one-to-many stalling 
problem.  SOCK_STREAM would only be used with one stream at a time.

That would then eliminate the need for a SCTP_PARTIAL_DELIVERY option. 
Any application needing atomic message delivery or one-to-many semantics 
would use SOCK_DGRAM instead.  SCTP_MAXMSGSIZE would only be needed for 
SOCK_DGRAM and could be ignored otherwise.

This would give much improved Posix compliance and proper protocol 
substitutability, very much in the spirit
of the three design objectives listed in the draft. There are probably a 
lot of UDP and TCP applications where SCTP could be a true drop in 
replacement, where the draft api interface only properly substitutes for 
SOCK_SEQPACKET.

Comments, anyone?

 - Mark

Note: Posix information from 
http://www.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_10.html
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 29 Nov 2005 00:36:30 -0700

Mark,

On Mon, 28 Nov 2005, Mark Butler wrote:

>    Any   application  needing  atomic  message  delivery  or  one-to-many
>    semantics would use SOCK_DGRAM instead.  SCTP_MAXMSGSIZE would only be
>    needed for SOCK_DGRAM and could be ignored otherwise.
>    of the three design objectives listed in the draft. There are probably
>    a  lot  of UDP and TCP applications where SCTP could be a true drop in
>    replacement,  where  the draft api interface only properly substitutes
>    for SOCK_SEQPACKET.
>    Comments, anyone?
>     - Mark

You pretty much pegged my sockets interface.  The only difference is I
use SOCK_RDM, not SOCK_DGRAM for reliable datagram (yes, BSD, not POSIX)
that was used for RUDP, one of the early ancestors of SCTP.  Also
SOCK_RDM is connection oriented, not connectionless.  Regardless of the
definition of SOCK_DGRAM, many implementations expect it to be
connectionless and not connection oriented (i.e, no listen() or accept()
allowed, even though connect() works fine for connectionless sockets).

I think the current API is so far from POSIX (and BSD), it should use
SOCK_SCTP_*.

(More options, more features, more is better: how about a whole set of
SO_SCTP_* options just for SCTP sockets?  Let's define sctp_listen,
sctp_bind, sctp_connect, sctp_sendmsg, sctp_recvmsg, sctp_sendto, ...
Far be it for SCTP to use tried, true, portable, and standard interfaces.
Its some much _more_ than that  ;).

But, likely nobody, authoring this API agrees with me (or you) on any of
these points.  The industry has spoken: or at least a very narrow part
of it.

--brian

---

From: Randall Stewart <[email-masked]>
Date: Tue, 29 Nov 2005 07:33:09 -0500

> message at a time can be partially delivered.".

Actually, I scanned the document yesterday and found this..

I think that in reality it needs to state..
"
Only one message at a time per stream can be partially delivered.
"

which is what the next version has been editted to say ;-D

> serialization.  One-to-many sockets are where partial delivery is a 

As I stated earlier in my email.. making this work is not hard to
do (i.e. making it so that you can get a pdapi message in one
stream while receiving others).. the problem is that the
stack cannot tell if you are receiving from the recvfrom().. which
means you are toast... or you are doing a recvmsg() with ancillary
data.. or a sctp_recvmsg()...

That is the real issue.....

> or serious memory constraints.

I have no problem with that.. and if no one objects I will add it to
the next version.. it makes sense to me to be able to turn
it off :-D

As to the MSG_EOR loop.. not sure if you really will get rid
of this.. unless you want message truncation..

If the stack delivers a whole message up to the sockets
layer... aka say a 32k message... then the user does
a 16k read... you now have a partial read.. not a partial
delivery... which means the user must be aware of the EOR...

> sending side, by rejecting larger message sends with EMSGSIZE.

So on sending size.. reject.. makes sense..

On the receiving side you want to truncate the message? This
is what the MSG_TRUNC flag does now.. basically (at least
in theory) if you have set MSG_TRUNC and read a message
as I described above.. the rest of the message is thrown
away... so what would SCTP_MAXMSGSIZE do on the receive
side without MSG_TRUNC being set?

Seems to me we should just be using MSG_TRUNC and not having
this active on the recevier side.. only the sender side???

> an error.

So basically you are saying that SCTP_MAXMSGSIZE should be on
by default and set to SO_RCVBUF/2...

And setting SO_RCVBUF to anything automagically sets
SCTP_MAXMSGSIZE = SO_RCVBUF/2 ...

   Messages sent over the wire larger than
> the stack will not stall.

The above I need to think about...

> be enforced (for consistency), but not the receive buffer related 

This seems odd to me.. since if I am enabling PD-API then
I should be able to have whatever size message the stack
will let me send through... (in bsd's case its the SO_RCVBUF
size)... After all, if you enable it you should understand
the concept and the fact that the sender needs to make
adjustments for sending large messages...

> socket) can enable partial delivery where required.

Some I of this I want to think about..

Most of it sounds fine to me :-D

R

---

From: Mark Butler <[email-masked]>
Date: Tue, 29 Nov 2005 05:40:32 -0700

>allowed, even though connect() works fine for connectionless sockets).
Do you mean kernels currently enforce that in the socket layer?

If SOCK_DGRAM is ambiguous, I suggest SOCK_CONN_DGRAM - which is what 
AIX uses for connection orriented datagram sockets.  The advantage over 
using SOCK_RDM is that it is easier to cross reference the former to 
Posis semantics.

>I think the current API is so far from POSIX (and BSD), it should use
DCCP implementations currently use SOCK_DCCP, but changing to 
SOCK_CONN_DGRAM has been discussed recently. Gratuitous protocol 
specific socket types are a violation of the spirit of the socket api.

It is misleading to use standard types without delivering standard 
semantics. However, the draft one-to-one socket type almost matches 
Posix SOCK_STREAM - all it needs is write combining and cork-ability for 
efficiency.  It also almost matches Posix SOCK_SEQPACKET - all it needs 
is write combining and requiring callers to set MSG_EOR to end a message.

The problem area is the draft one-to-many socket type, which is 
currently a dysfunctional hybrid of Posix SOCK_DGRAM and Posix 
SOCK_SEQPACKET.  Disabling partial delivery with oversize message 
truncation would almost make a draft one-to-many socket into a proper 
SOCK_DGRAM, and a draft one-to-one socket almost into a proper 
SOCK_CONN_DGRAM.  The remaining differences are minor.

>Its some much _more_ than that  ;).
Those interfaces would generally be library functions that wrap common 
kernel interfaces.  I think they are a good thing, generally speaking.  
A separate call to switch streams requires per thread state, or worse a 
separate syscall.

>these points.  The industry has spoken: or at least a very narrow part
The main argument I might anticipate for sticking with 
SOCK_STREAM/SOCK_SEQPACKET is backward compatibility.  I think that is a 
poor one at this stage. SCTP just barely made it into Linux and Solaris 
kernels, so the socket types can still be fixed easily.  A years later 
standardization process would break the ABI, forcing ugly 
LD_ASSUME_KERNEL style tricks for older applications.

 - Mark

---

From: Randall Stewart <[email-masked]>
Date: Tue, 29 Nov 2005 07:49:04 -0500

I will make only a couple of comments to this..

Brian F. G. Bidulock wrote:
> that was used for RUDP, one of the early ancestors of SCTP.

RUDP is NOT an early ancestor of SCTP... In fact I don't think
that most of us (execept maybe Ken) had read the RUDP drafts.
SCTP started from MDTP... and then evolved from the input
of the sigtran group that worked on it...

  Also
> I think the current API is so far from POSIX (and BSD), it should use

Gee.. it sure works and fits well into the BSD framework.. kinda strange
that you should say that :>

R

---

From: Mark Butler <[email-masked]>
Date: Tue, 29 Nov 2005 06:43:31 -0700

> which is what the next version has been editted to say ;-D

That's fine with me, as long as there is a way to disable partial delivery.

> That is the real issue.....

It doesn't matter as long as partial delivery is disabled.  All three of 
those interfaces would always read a single (possibly truncated) 
message, just as in SOCK_DGRAM.

>> delivery is positively dangerous. Disabling it by default for 
>> one-to-one sockets would simplify the programming model and increase 
>> portability - no more MSG_EOR loops everywhere.  The only 
>> applications that should need to enable partial delivery are ones 
>> with large message sizes (>64K) or serious memory constraints. ...
>
>

Yes, messages should definitely be truncated if oversize when partial 
delivery is enabled.  Otherwise either the stack would stall.  Message 
truncation flagged w/ MSG_TRUNC is standard SOCK_DGRAM semantics. There 
is no reliable way to get an arbitrarily sized message or do partial 
delivery on a one-to-many socket, which is why SOCK_DGRAM truncates 
messages larger than the read buffer.

Since we do not have the option of dropping messages larger than 
SO_RCVBUF (as UDP does), delivering them in truncated form w/ MSG_TRUNC 
set is the friendliest reliable behavior.  All current one-to-many 
socket ULPs have message length limitations, so if a normal application 
saw MSG_TRUNC set, it would be an indication of a bug or a malicious 
client.  Any data beyond the allowed message size is useless.

> delivery... which means the user must be aware of the EOR...

When partial delivery is disabled, partial reads should be disabled as 
well.  Otherwise you cannot have multiple threads reliably read from the 
same socket, a relatively common SOCK_DGRAM usage. (It is worth noting 
that Linux uses "wake one" semantics for efficiency in such cases).

> On the receiving side you want to truncate the message?

Yes.

> This

No - MSG_TRUNC is set after the read, to signal a truncated message, not 
before.

SCTP_MAXMSGSIZE would truncate messages larger than the specified size, 
and set MSG_TRUNC. When partial delivery is disabled, reads larger than 
the read buffer size, should be truncated and flagged the same way, just 
like SOCK_DGRAM.

> this active on the recevier side.. only the sender side???

MSG_TRUNC is meaningless on the sender side.

If we do sender side only, anyone can stall the stack when partial 
delivery is disabled by sending a sufficiently large message over the 
wire.  It would be better to get the message and be able to abort the 
association if it is garbage than to add and handle more exotic 
notifications.

>> Applications should be required (by default) to use a receive buffer 
>> at least twice as large as the required message size. On the 
>> implementation side, by default (partial delivery disabled) the 
>> effective maximum message size should be clamped to one half the size 
>> of the receive buffer. Setting a SCTP_MAXMSGSIZE larger than 
>> SO_RCVBUF/2 should return an error.
>
>
>

Yes.  However, it would be cleaner, and avoid setsockopt ordering 
problems, to store and return the original SCTP_MAXMSGSIZE, and have the 
implementation use both constraints together or with a derived variable 
as it sees fit.  There is precedent for this behavior.  Then you can 
_allow_ implementations to deliver messages larger than than SO_RCVBUF / 
2 and less than the maximum message size without truncation (provided 
they do not stall), but only _guarantee_ non-truncated message delivery 
when the message size is less than SO_RCVBUF / 2 and SCTP_MAXMSGSIZE.

UDP behaves similarly with a fixed maximum message size of 64K, except 
it silently drops messages instead of truncating them when they do not 
fit in the receive buffer.

> When partial delivery is enabled, the SCTP_MAXMSGSIZE restriction 
> should be enforced (for consistency), but not the receive buffer 
> related message size limitation.
>

It is fine with me if SCTP_MAXMSGSIZE only applies when partial delivery 
is disabled.

>> Applications with proper buffer sizes would always get atomic 
>> delivery, never see truncated messages from conforming clients, be 
>> resilent to network problems and malicious clients, never stall 
>> across different associations, and have reduced inter-stream 
>> serialization by default.  Applications needing very large messages 
>> (generally on a one-to-one socket) can enable partial delivery where 
>> required.
>
>
>

I am glad to hear it.

 - Mark

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 29 Nov 2005 07:22:51 -0800

On Nov 29, 2005, at 5:43 AM, Mark Butler wrote:

> It doesn't matter as long as partial delivery is disabled.  All  
> three of those interfaces would always read a single (possibly  
> truncated) message, just as in SOCK_DGRAM.
>
>>> Proposal:
>>>
>>> disabled by default, at least for one-to-many sockets, where  
>>> partial delivery is positively dangerous. Disabling it by default  
>>> for one-to-one sockets would simplify the programming model and  
>>> increase portability - no more MSG_EOR loops everywhere.  The  
>>> only applications that should need to enable partial delivery are  
>>> ones with large message sizes (>64K) or serious memory  
>>> constraints. ...
>>
>>
> Yes, messages should definitely be truncated if oversize when  
> partial delivery is enabled.  Otherwise either the stack would  
> stall.  Message truncation flagged w/ MSG_TRUNC is standard  
> SOCK_DGRAM semantics. There is no reliable way to get an  
> arbitrarily sized message or do partial delivery on a one-to-many  
> socket, which is why SOCK_DGRAM truncates messages larger than the  
> read buffer.
>
> SO_RCVBUF (as UDP does), delivering them in truncated form w/  
> MSG_TRUNC set is the friendliest reliable behavior.  All current  
> one-to-many socket ULPs have message length limitations, so if a  
> normal application saw MSG_TRUNC set, it would be an indication of  
> a bug or a malicious client.  Any data beyond the allowed message  
> size is useless.

I see a more fundamental conflict between the sockets interface and  
interleaved messages.

With an RDMA style QP/CQ interface all buffers are pre-posted by the  
consumer, and there are methods
to limit the number of buffers that are allocated but not delivered  
on either a global or per-connection
basis.

A socket interface pretty fundamentally assumes that buffer space is  
provided in the call (in which
case it is not pre-associated with any message) or is system  
allocated if the message arrives before
the caller asks for it.

Any scheme I can imagine to solve this problem using a socket  
interface is either going to:

a) arbitrarily deliver partial messages, forcing the application to  
do reassembly. Under the
     current sockets API this can only happen in limited  
circumstances that the consumer is
    fully capable of preventing.
b) hold an arbitrary number of partial messages waiting for their  
completion with no mechanism
    to monitor these resources in order to prevent a Denial of  
Service attack.

One simple mechanism that the layer above SCTP can use is to simply  
place the message
sequence number and message offset in each SCTP message, size the  
SCTP messages
so that they fit in a single chunk, and then use unordered delivery.  
The middleware receiver
then places the segments as they arrive, and delivers only complete  
messages to the ULP.
but the middleware is fully aware of how many messages are pending  
and can take
appropriate action. A segment sequence number is also required if  
interleaving is going
to be allowed, so that complete messages can be detected.

That should sound familiar to some readers, it's how RDDP/SCTP  
processes untagged
messages (RDMA Sends).

The reason that approach was taken, as opposed to modifying ordered  
streams to guarantee
unordered delivery to the RDMA layer, was to ensure that existing  
SCTP stacks could be
left intact. The conclusion with the RDDP/SCTP adaptation was that  
the penalty for solving
it above the SCTP layer ("wasting" the Stream Sequence Number and  
having to put in
an alternate field that had the same semantics but was legal for an  
unordered stream)
was minor enough for the benefit of enabling this new feature over  
*existing* SCTP stacks.

I believe that the same rationale applies to solving the large- 
message problem. In fact
using the RDDP/SCTP adaptation and only implementing untagged  
messages would
be an excellent solution. It can be done easily in a user mode  
library over any SCTP stack.

---

From: Randall Stewart <[email-masked]>
Date: Tue, 29 Nov 2005 11:33:05 -0500

One comment... below.. (maybe two :-D)

> A separate call to switch streams requires per thread state, or worse a 

Yes, but for efficency they are best to be real system calls.. librarys
that wrap ancilliary data functions only to have the kernel unwrap
the ancillary data functions will cost a LOT performance wise
when one starts trying to make a stack hit the 1Gig or higher
type network...

As to the interfaces... I think that this just shows that the
writters of first sockets aka the bsd folks did NOT think
about anything but TCP and UDP... SCTP with its multi-homing
was NOT in their thought processes... as far as posix goes.. they
have just standardized what was already standard... aka the
way everyone was doing it... (yes they may have done some
minor tweaks.. but not many.. it was the BSD guys that
came up with it all :->)

> LD_ASSUME_KERNEL style tricks for older applications.

I am not too keen on changing the socket types at this point. The
code has been out there much longer than "a year".. yes it is
finally showing up in a disk now.. but that is just a reflection
on how a O/S development cycle works..

If we had a lot of interested folks demanding this.. I will
gladly bend with consensus... but right now... Brian tells
us he does not care about it.. and so you are the only
voice that is asking for changes..

Anyone else want to have different socket types???

R

---

From: Randall Stewart <[email-masked]>
Date: Tue, 29 Nov 2005 11:44:18 -0500

> delivery is enabled.  Otherwise either the stack would stall.  Message 
^^^^^^^^^^^^^^^^^^^^^^

Did you mean disabled here?  If I want partial delivery and turn
it on why would the stack truncate a message?

> client.  Any data beyond the allowed message size is useless.

I agree, in principle, if we have pdapi turned off.. NOT on :-D

> that Linux uses "wake one" semantics for efficiency in such cases).

Hm.. so you are equating pdapi=off to MSG_TRUNC=on then...

> No - MSG_TRUNC is set after the read, to signal a truncated message, not 

ahh.. sorry.. I was thinking it as being a receive option :-D... which
then means it only makes sense that if you turn PD-API off then
you get a truncation of a message..

But I am not sure the value of SCTP_MAXMSGSIZE... would not
this then just be a matter of the sender setting this.. since
on the receive side.. even if you take it all the way down
at the reassembly.. you will get a blocking behavior at the
receiver... For example..

My SMMS = 3000bytes

<<<<TSN1(1440) B=1 E=0
<<<<TSN2(1440) B=0 E=0
<<<<TSN3(1440) B=0 E=0 ******A
<<<<TSN4(1440) B=0 E=0

I know at point A, that the user is getting a truncated
message... the rest of the TSN's are still going to
roll in even if I am dumping them (since I sent the
3000 byte message up the socket buffer with the MSG_TRUNC
flag set to on).

Seems to me that the MSG_TRUNC gets applied based on the buffer
size of the recv() call..

Unless you want to just have it so it can discard data
early (as I illustrate above)... note that this gets
a bit tricky to do to :-0

> MSG_TRUNC is meaningless on the sender side.

No, not msg_trun SCTP_MAXMSGSIZE...
> It is fine with me if SCTP_MAXMSGSIZE only applies when partial delivery 

I glad we agree here :-D

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 29 Nov 2005 11:48:03 -0700

Caitlin,

Well put.  I agree 100%.

--brian

---

From: Mark Butler <[email-masked]>
Date: Tue, 29 Nov 2005 23:00:34 -0700

Caitlin Bestler wrote:

>     current sockets API this can only happen in limited  circumstances 
> that the consumer is
>    to monitor these resources in order to prevent a Denial of  Service 
> attack.

On a one-to-one socket, this is not a problem - no more than TCP 
anyway.  Each connection must be accepted, and resources are bound by 
SO_RCVBUF.

One-to-many sockets are trickier. SO_RCVBUF should be used per socket 
(not per assocation) just like SOCK_DGRAM.  The stalling problem 
originally motivating HAVE_SCTP_MULTIBUF is solved by truncating 
arriving messages over SCTP_MAXMSGSIZE.  Atomic delivery and message 
size limits must go together.

Somewhere there should be a limit on associations per one-to-many socket 
as well, such that idle (including pending message non-responding) 
associations are shutdown in LRU order whenever the limit is exceeded.  
This can be done at the application level with association change 
events, and the special one-to-many listen() behavior.  This would be a 
good thing to do in the stack, however, perhaps with a socket option 
called SCTP_MAXASSOC.   SCTP_AUTOCLOSE timeouts are helpful, but would 
cause gratuitous assocation shutdowns in many applications, which is why 
the option is disabled by default.  Keeping an MRU association cache 
would perform better, be more effective against DoS attacks, and scale 
better under load (not that it should be required of course).

[Randall: sendmsg() on one-to-many sockets is transparent to peer 
association shutdowns, right?   i.e. sendmsg() should wait for a known 
pending shutdown on the association to complete, and then reinitiate a 
new association rather than dropping or re-ordering messages, right?  
Perhaps this behavior should be specified - clients should not have to 
handle such things, especially when it is trivial for the stack to do it.

Also the SCTP_AUTOCLOSE section does not specify whether it does 
shutdown or an abort. I am guessing it should be the former.]

> That should sound familiar to some readers, it's how RDDP/SCTP  

I agree that SCTP is not suited for atomic delivery of very large 
messages (greater than 64K or so). Nor does it support end-to-end 
message delivery failure or completion notifications the way RDMA does.

> was minor enough for the benefit of enabling this new feature over  

I thought that decision was made for efficiency.  If all the messages 
are unordered and unfragmented, as in the DDP/SCTP adaptation draft, 
they should flow through any SCTP stack without delay, no matter what 
changes are made to the way ordered and fragmented messages are 
handled.  If you relied on the stream sequence number instead, that 
would unnecessarily complicate hardware implementations, right?

> be an excellent solution. It can be done easily in a user mode library 
> over any SCTP stack.
>
I would like to see an open source RDMA over SCTP / TCP socket 
implementation with an IT-API or perhaps uDAPL interface (any comments 
on the relative advantages?).  However, I also think reliable, ordered 
SOCK_DGRAM message delivery for reasonable message sizes would be 
independently useful.

- Mark

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 29 Nov 2005 22:14:14 -0800

On Nov 29, 2005, at 10:00 PM, Mark Butler wrote:

>
>>
>> The reason that approach was taken, as opposed to modifying  
>> ordered  streams to guarantee
>> left intact. The conclusion with the RDDP/SCTP adaptation was  
>> that  the penalty for solving
>> an alternate field that had the same semantics but was legal for  
>> an  unordered stream)
>> was minor enough for the benefit of enabling this new feature  
>> over  *existing* SCTP stacks.
>
> I thought that decision was made for efficiency.  If all the  
> messages are unordered and unfragmented,
> as in the DDP/SCTP adaptation draft, they should flow through any  
> SCTP stack without delay, no matter
> what changes are made to the way ordered and fragmented messages  
> are handled.  If you relied on the
> stream sequence number instead, that would unnecessarily complicate  
> hardware implementations, right?

If there were no existing SCTP stacks to deal with the ideal would  
have been to define the Stream Sequence
Number as an opaque pass-thru for streams designated to carry only  
unordered data chunks. Essentially,
just let the application tag them, pass that tag through unmolested,  
and let the application put them back
in order.

But the consensus was that the value of an evolutionary solution as  
opposed to a correctional one, was
far greater than the savings of a single sixteen bit field.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 30 Nov 2005 00:20:48 -0700

Caitlin,

> far greater than the savings of a single sixteen bit field.

Rather than concensus, I recall it as more of a statement: that the I-G
could not change the protocol.  Hah!

--brian

---

From: Mark Butler <[email-masked]>
Date: Wed, 30 Nov 2005 00:25:33 -0700

Randall,

>> delivery is enabled.  Otherwise either the stack would stall.  Message 

Yes.  Partial delivery enabled mode should not truncate messages.  I 
agree SCTP_MAXMSGSIZE should only be effective when partial delivery is 
_disabled_.

By the way, I suggest that SCTP_PARTIAL_DELIVERY be renamed to 
SCTP_PARTIAL_READ. This would be more obvious and indicate the mandatory 
message at a time semantics more clearly.  Someday, SCTP_PARTIAL_WRITE 
might be helpful, for proper TCP / Posix SOCK_STREAM semantics.

>
>
>> truncation flagged w/ MSG_TRUNC is standard SOCK_DGRAM semantics. 
>> There is no reliable way to get an arbitrarily sized message or do 
>> partial delivery on a one-to-many socket, which is why SOCK_DGRAM 
>> truncates messages larger than the read buffer.
>>
>> normal application saw MSG_TRUNC set, it would be an indication of a 
>> bug or a malicious client.  Any data beyond the allowed message size 
>> is useless.
>
>

Yes.

>
>>
>> When partial delivery is disabled, partial reads should be disabled 
>> as well.  Otherwise you cannot have multiple threads reliably read 
>> from the same socket, a relatively common SOCK_DGRAM usage. (It is 
>> worth noting that Linux uses "wake one" semantics for efficiency in 
>> such cases).
>
>

Yes.  Partial delivery off should make a socket adopt SOCK_DGRAM 
semantics with respect to reads.  That is why SCTP_PARTIAL_READ would be 
a better name than SCTP_PARTIAL_DELIVERY (which by itself is really an 
implementation detail, not a behavior guarantee).

>>
>>
>> No - MSG_TRUNC is set after the read, to signal a truncated message, 
>> not before.
>
>

SCTP_MAXMSGSIZE is helpful because applications get notified of oversize 
messages immediately, so they can dump the association before it wastes 
an arbitrary amount of buffer space on a shared buffer one-to-many 
socket.  If you have a 16 MB shared buffer and a thousand clients, no 
single client should be able to fill up a large fraction of the shared 
buffer before the application has any idea what is going on.

The combination of SCTP_MAXMSGSIZE and per-association stream limits 
places an upper bound on how much memory a single association can 
consume in a shared buffer with a series of partially completed messages 
(as from a non-conforming or malicious sender).  Shared buffers (or 
shared resource bounds) are a good thing, and this would make them reliable.

There should be an effective SO_RCVBUF/2 message size limit (when 
partial reads are disabled) as well, to prevent stack level inter-stream 
or inter-association stalling, regardless of whether SCTP_MAXMSGSIZE is 
set or not. 

> size of the recv() call..

Yes.  We should extend that behavior - no need to have two independent 
message truncation flags.

> a bit tricky to do to :-0

To get SOCK_DGRAM semantics, this has to be done somewhere.  Most 
naturally in the same place as SAR.

Just keep tombstone records in the partially completed message queue, 
and expire them occasionally based on cumulative stream sequence numbers.

 - Mark

---

From: Mark Butler <[email-masked]>
Date: Wed, 30 Nov 2005 03:39:07 -0700

I wrote:

> SCTP_PARTIAL_READ. This would be more obvious and indicate the 
> mandatory message at a time semantics more clearly.  Someday, 
> SCTP_PARTIAL_WRITE might be helpful, for proper TCP / Posix 
> SOCK_STREAM semantics.

On second thought, as a general principle, when a stack does partial 
delivery should be an implementation detail, completely opaque to the 
user.  Rather we should control whether partial reads are allowed, and 
whether that fragments from different streams are interleaved.

SCTP_FRAGMENT_INTERLEAVE would enable interleaving fragments from 
different streams.
It is an unexpected, non-Posix compliant behavior, and should be 
disabled by default.

SCTP_PARTIAL_READ would enable partial message reads.  It should default 
on under the current SOCK_xxx regime, I suppose.  However, it really 
ought to be determined by socket type: Posix SOCK_STREAM / 
SOCK_SEQPACKET on, SOCK_DGRAM / SOCK_CONN_DGRAM off.   Then the option 
would be largely redundant. 

Please note that partial reads do not imply partial delivery. 
One-to-many sockets should never, ever do partial deliveries, but it can 
be useful to allow partial reads.  Partial delivery should only be done 
when fragment interleaving is enabled, with true Posix SOCK_STREAM 
sockets, or under buffer pressure (with locking) on a proper one-to-one 
Posix SOCK_SEQPACKET socket.  This is all implementation stuff that 
doesn't belong in the API specification, however.

Partial delivery events are now superfluous.  Fragment interleaved 
streams do not need them, because partial delivery is a way of life for 
them.  They were only previously needed to abort records on one-to-many 
sockets, but since partial delivery should never be done on one-to-many 
sockets in any case, they should be excised.

Any problems with using SCTP_PARTIAL_READ and SCTP_FRAGMENT_INTERLEAVE 
like that?  We wouldn't really need the former with proper Posix socket 
types, of course.

- Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 30 Nov 2005 04:26:10 -0700

Mark,

> It is an unexpected, non-Posix compliant behavior, and should be 

POSIX does not say that fragments from different streams can't be
interleaved.

--brian

---

From: Randall Stewart <[email-masked]>
Date: Wed, 30 Nov 2005 08:34:29 -0500

Mark:

I like Brian, agree with Caitlin on the problems
inherit in the socket interface.. amazing.. usually
Brian and I don't agree so often ;-D

Other comments below...

> better under load (not that it should be required of course).

Autoclose was not really meant to control resources but instead to
make it more appear udpish to the user.. aka it does not
have to worry about connection "state".... as one does in
the one-2-one model...

Interesting that you mention MAXASSOC.. since the IOS version
of SCTP has this very option in its setsockopt().. calls :-D
Maybe we should have such an addition... not sure..

Others ???

> shutdown or an abort. I am guessing it should be the former.]

Yes, autoclose is supposed to do a graceful close NOT an abort.. that
wording I will fix :-D

As to sendmsg() on top of a association that is in shutdown pending,
this makes me wonder.. why would you expect an application to
initiate a shutdown and then send data to it?

As to making the stack hang on to the association and restart it up..
it is easy for you to say "it is trivial for the stack to do"... but
I don't think its so trivial... yes its just code (as Kacheong always
says)... but before adding code or being even for adding code :)
I would like to know the scenario you have in mind ...

> message delivery failure or completion notifications the way RDMA does.
Interesting.. the MPI folks at the UBC have had some good luck
with large message.. in fact there smallest message (if I remember
right) is way greater than 64k :-D

But I tend to agree that if you want to do high speed transfer
of large messages the stack may not be the most efficent place
to do it :0

R

> independently useful

---

From: Randall Stewart <[email-masked]>
Date: Wed, 30 Nov 2005 08:46:14 -0500

> like that?  We wouldn't really need the former with proper Posix socket 
So, if we disallow partial delivery how does one avoid a deadlock
scenario? This was the whole reason Vern put it in if I remember
right...

I don't have a problem with SCTP_PARTIAL_READ or SCTP_FRAGMENT_INTERLEAVE
and having it off by default (interleave) seems natural
to me..

But I don't see how partial delivery can be avoided unless
one has some sort of negotiated max message size that
the sender can enforce based on say the advertised initial
rwnd...

And even at that.. a malicious sender that ignores that
limit could cause a deadlock.. I suppose you could
abort the association that mis-behaves.. but this
seem rather drastic ...

R

---

From: Sridhar Samudrala <[email-masked]>
Date: Wed, 30 Nov 2005 11:33:56 -0800

> whether that fragments from different streams are interleaved.

I just started following this thread. 
I think the term SCTP_PARTIAL_READ here is confusing. 
Currently in lksctp, partial reads are always on and happen when the user
passes a buffer that is smaller than the message. A partial read is 
indicated by not setting MSG_EOR flag. The remaining part of the message
is returned in the later recv calls.

You seem to be suggesting a new option to follow UDP behavior
where a message is always passed to the user in a single recv call by
truncating it and setting MSG_TRUNC if the user buffer is smaller than
the message size.

Is my interpretation correct? If so, doesn't this violate the principle of
reliable data delivery? 

Thanks
Sridhar

---

From: Mark Butler <[email-masked]>
Date: Wed, 30 Nov 2005 13:15:48 -0700

>POSIX does not say that fragments from different streams can't be
But it does specify that records may not be interleaved, which is the 
same thing.  Adding streams shouldn't be an excuse to break existing 
semantics - principle of least surprise, at a minimum. Streams should be 
transparent to readers.

 - Mark

---

From: Mark Butler <[email-masked]>
Date: Wed, 30 Nov 2005 14:15:27 -0700

> have to worry about connection "state".... as one does in

SCTP_AUTOCLOSE does a passable job at resource control, whatever the 
intent.   If there was no need for resource control, you could leave 
associations open indefinitely. 

SCTP_MAXASSOC would be better - no need to close idle assocations except 
under pressure.

> Maybe we should have such an addition... not sure..
....

> Yes, autoclose is supposed to do a graceful close NOT an abort.. that

Good.

> initiate a shutdown and then send data to it?

An SCTP_AUTOCLOSE shutdown is not initiated by the application on either 
end. Message operations should be transparent to autoclose.

> I would like to know the scenario you have in mind ...

I don't think the stack should hang on to the association, nor does it 
need to hang on to the association id.  It just needs to serialize 
auto-start with auto-stop.  The checks for association state are there 
already - there just need to be shutdown completion waits added, similar 
to that used in socket close().

> of large messages the stack may not be the most efficent place

Yes - that should be "not generally well suited".  DDP/SCTP wouldn't 
have a problem.

MPI is a controlled environment. Inter-client serialization on large 
messages is not a problem in MPI - normally high bandwidth, bounded 
number of clients all working together on the same job,  nodes not 
dissapearing at random, and shared barrier semantics. 

- Mark

---

From: Mark Butler <[email-masked]>
Date: Wed, 30 Nov 2005 15:05:39 -0700

Randall Stewart wrote:

>> Any problems with using SCTP_PARTIAL_READ and 
>> SCTP_FRAGMENT_INTERLEAVE like that?  We wouldn't really need the 
>> former with proper Posix socket types, of course.
>

Partial delivery should be allowed whenever the higher level semantics 
allow it, rather than being independently specified by the user.

Deadlock is avoided in the cases where partial reads are disallowed by 
using message truncation.  If you disable partial reads to get 
SOCK_DGRAM semantics, message size limitations come as part of the package.

> the sender can enforce based on say the advertised initial

No transport level negotiation necessary.  That is the ULPs 
responsibility.  SOCK_DGRAM ULPs generally have hard limits.  32K for 
NFS (UDP), 512 bytes for DNS, ~256 bytes for VoIP, and so on.

The stack can guarantee any message less than (SO_RCVBUF / 2) gets 
through unharmed.  The application implementing the ULP just sets 
SO_RCVBUF to twice the ULPs maximum segment size, uses appropriately 
sized read buffers, and forgets about it.

> abort the association that mis-behaves.. but this

That is the reason for (SO_RCVBUF / 2) truncation semantics (when 
partial reads are disabled).  No need for the stack to abort the 
association.  The application can take whatever application appropriate 
by checking the MSG_TRUNC flag.

- Mark

---

From: Mark Butler <[email-masked]>
Date: Wed, 30 Nov 2005 15:58:00 -0700

>I think the term SCTP_PARTIAL_READ here is confusing. 
I would like to see SOCK_DGRAM / SOCK_CONN_DGRAM specified directly. It 
is much more obvious. If you agree, please say so.

>is returned in the later recv calls.

Yes, standard Posix SOCK_SEQPACKET semantics.  The serious problem, is 
with partial deliveries on a one-to-many socket, because they serialize 
communication from all other associations, potentially hanging the 
socket for a very long time.  Lksctp is currently vulnerable to this 
problem. 

>Is my interpretation correct? If so, doesn't this violate the principle of
Yes, your interpretation is correct. It doesn't violate the principle of 
reliable delivery because it would only be used with ULPs that specify 
(or possibly negotiate) maximum message sizes.

 - Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 30 Nov 2005 16:04:21 -0700

Mark,

> socket for a very long time.  Lksctp is currently vulnerable to this 

Well, no.  POSIX SOCK_SEQPACKET sets MSG_TRUNC if the message (not the
record) is larger that the supplied buffer, just as does SOCK_DGRAM.

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 30 Nov 2005 16:01:41 -0700

Mark,

On Wed, 30 Nov 2005, Mark Butler wrote:

>    semantics  - principle of least surprise, at a minimum. Streams should
>    be transparent to readers.
>     - Mark

Not the same thing.  That is interleaving records from the SAME stream.

--brian

---

From: Mark Butler <[email-masked]>
Date: Wed, 30 Nov 2005 17:51:36 -0700

>record) is larger that the supplied buffer, just as does SOCK_DGRAM.
LKSCTP is vulnerable as  implemented.  Partial delivery locks 
one-to-many sockets by association.

Regrettably, the Posix documentation for SOCK_SEQPACKET is inconsistent. 
This is the description from section 2.10.6:

No difference from SOCK_STREAM except record boundaries, means that 
message boundaries are either invisible or identical with record 
boundaries.  Not two independent sets of boundaries.

However the documentation for recvmsg() implies that records and 
messages are different:

    The recvmsg() function shall return the total length of the message.
    For message-based sockets, such as SOCK_DGRAM and SOCK_SEQPACKET,
    the entire message shall be read in a single operation. If a message
    is too long to fit in the supplied buffers, and MSG_PEEK is not set
    in the flags argument, the excess bytes shall be discarded, and
    MSG_TRUNC shall be set in the msg_flags member of the msghdr
    structure. For stream-based sockets, such as SOCK_STREAM, message
    boundaries shall be ignored. In this case, data shall be returned to
    the user as soon as it becomes available, and no data shall be
    discarded.
    ...
    MSG_EOR

        End-of-record was received (if supported by the protocol).

    MSG_EOR

        End-of-record was received (if supported by the protocol).

MSG_EOR

    End-of-record was received (if supported by the protocol).

MSG_EOR

    End-of-record was received (if supported by the protocol).

MSG_EOR

    End-of-record was received (if supported by the protocol).

This is a complete contradiction. It only makes sense if there are 
protocols that support two sets of boundaries, lower level message 
boundaries embedded in higher level record boundaries.  Are there any?

More likely there are two conceptions of SOCK_SEQPACKET that made their 
way into the standard - One is SOCK_DGRAM with ordering and reliability 
added.  The other is a byte stream with occasional boundary markers.  
The draft api uses the latter conception, consistent with Posix XSH 
2.10.6.  Which conception do existing SOCK_SEQPACKET protocols use?

I guess I can't say standard SOCK_SEQPACKET semantics anymore. Sigh.

- Mark

POSIX references:

 http://www.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_10.html
 http://www.opengroup.org/onlinepubs/009695399/functions/recvmsg.html
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Mark Butler <[email-masked]>
Date: Wed, 30 Nov 2005 18:19:59 -0700

Brian,

>Not the same thing.  That is interleaving records from the SAME stream.
We will have to disagree here. 

- Mark

---

From: Mark Butler <[email-masked]>
Date: Wed, 30 Nov 2005 18:25:23 -0700

(This is a repeat with extraneous whitespace from a defective mail 
client removed)

- Mark

POSIX references: