Subject: [Tsvwg] Is there a SCTP maximum user message size?

From: Anders Torger <[email-masked]>
Date: Sat, 14 Apr 2007 09:10:23 +0200

I'm trying to understand what the maximum user message size for SCTP is 
so I can adapt buffer management accordingly.

It seems to me that it at least theoretically can be unlimited, since 
there is no user message size tag in the protocol, just a beginning 
fragment, a number of middle fragments ant then the ending fragment.

However, implementations seem to limit the size in some way related to 
the RWND, but I guess that is only to make things practical rather than 
it really is the limit enforced directly or indirectly by the protocol 
specification?

Also, it seems to me that there is no negotiation during association 
establishment that sets the maximum user message size. This means that 
the receiving side cannot know in advance how large user messages it 
needs to accept, and the stack is not allowed to abort the association 
if a "too large user message" arrives, am I right?

Thus, is it true that if the sending side implementation supports it, it 
could send a message that never ends, and the receiving stack is 
required to accept this? Since the receiving side can deliver partial 
messages using the partial delivery API, it seems like this is a 
possible scenario (no real application would use this approach of 
course, but it could potentially be used as an attack against 
applications with poor buffer management that assumes some maximum 
size, or extend buffers until it runs out of memory).

There is however no partial sending API of what I know of, so in 
practice user messages cannot become extremely large (unless a special 
stack implementation is made), but it bugs me that for a particular 
association I cannot find out what the limit really is.

/Anders Torger

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 14 Apr 2007 12:36:41 +0200

Hi Anders,

see my comments in-line.

Best regards
Michael

> fragment, a number of middle fragments ant then the ending fragment.
Correct. There is no limit in the protocol.
> it really is the limit enforced directly or indirectly by the protocol
What I see is that Linux and Solaris fail the send() call when the
length of the user message is larger than the send buffer size.
I do not see (and expect) a relation to the a_rwnd of the peer.
> if a "too large user message" arrives, am I right?
The SCTP stack should not initiate the ABORT, but if the application
code does this, SCTP has to do it. So: you are right.
> size, or extend buffers until it runs out of memory).
The BSD stack can send really large messages... and you are correct,
applications should do an appropriate buffer management.
> association I cannot find out what the limit really is.
As I said the BSD stack can send large messages. And in addition,
it supports what you call "partial sending API" and is described in
section 7.1.30 of
http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-sctpsocket-14.txt
called the explicit EOR mode. Using this mode, the user can make  
multiple
send() calls for one user message.

---

From: Randall Stewart <[email-masked]>
Date: Sat, 14 Apr 2007 06:45:38 -0400

> it really is the limit enforced directly or indirectly by the protocol 

Well.. like many things this is implementation dependent. In theory
we have the partial delivery API that can be started on the receiver
this gets out of the deadlock you are referring to.

Now it USED to be that the BSD stack would only allow a message
as large as the socket send buffer. This is no longer true. You
basically can malloc a 1 Gig buffer and send it... the BSD stack
will happily get the data all across to the peer. This came about
with the implementation of EEOR mode. Which basically allows you to
do multiple writes to the same message until you indicate a MSG_EOR.
So using EEOR mode you don't even have to have a large malloc.. you can
just do multiple writes until you send a EEOR explictly.. this would
actually be ideal for something like an FTP transfer where you were
sending disk blocks.. you set EEOR mode and then write away disk
block at a time to the last read.. then on that send.. you set the
MSG_EOR. The peer of course needs to set is PARTIAL_DELIVERY_POINT down
so he wont get a large read all at once.. and he can then read and
write to the file until it sees the MSG_EOR. Of course once you
start the EEOR you cannot write a message to another stream.. and once
the PD-API starts on the receiver you pretty much are reading that
same message until its complete... defeats the multi-streaming.. if
you will.

Of course if you are using the one-2-many model you could still
get reads off other assoc's (if you have enabled the socket option..
SCTP_FRAGMENT_INTERLEAVE)... but no matter what the place where the
large message is coming from is stuck reading that one message (and
sending on the sender side too) until its all consumed.

> if a "too large user message" arrives, am I right?

Well, if the stack is written correctly it won't have that problem.
Since it will start the PD-API. Now there are some stacks that I
know of that will just abort the association if they have a limited
message size and don't support the PD-API...  but as long as
a stack supports the PD-API then this is not a problem.

For an application one would hope that you would design it with
a pretty good idea of what messages are going to be received and
what formats are coming from the peer :-D

> required to accept this? 

Thats what EEOR mode does, yes... if both sides
a) set the PARTIAL_DELIVERY_POINT to 1 byte
and
b) Turn on EEOR mode.

Then by default, if they never send a MSG_EOR in any send .. they
look and smell like a stream of bytes being sent.

Each side will start the PD_API right away and from then
on that assoc will be stuck on that message until a MSG_EOR is
sent or the assoc is closed.

Since the receiving side can deliver partial
> size, or extend buffers until it runs out of memory).

That is the whole reason you have the ability to set down the
PARTIAL_DELIVERY_POINT to some other value. For example if I am
a app and I expect no more than 10k messages, I will set the
PARTIAL_DELIVERY_POINT to 10k +1 byte. Once I have done this
I would then only receive 10k mesgs (since the stack will not
start the PD-API until 10k+1byte.. unless of course the
local socket read buffer is less than 10k.. but lets assume
it is something more sane like 200k)...

If a PD-API begins (i.e. I see a message WITHOUT the MSG_EOR and
its 10k read) then I know its an attacker.. and I can abort
the association to him.

> association I cannot find out what the limit really is.
No thats not true.. set EEOR mode and you have just that.. a partial
sending point.. you keep sending until you specify a MSG_EOR.

This is really no different than TCP.. except the connection can
stay up.. Look at a TCP connection as sending one message. This is
in effect what happens when you do an FTP.. the sender/recv setup
a new connection and the sender just dumps the file into it.
The receiver reads it and writes everything up to the close
into the file.. ta-da.. you have transfered the file.. The connection
in effect is one big record.

The EEOR mode lets you keep the connection up and still have these
same semantics.. if you so desire them.

Of course, as I illustrated above, there is a way to know the absolute
msg size for those wanting to write an application that takes in
no more than a set size mesg... The of course should make sure
the socket receiver buffer is at least that big.. or bigger..

R

-- 
Randall Stewart
803-345-0369 <or> 815-342-5222(cell)

---

From: Anders Torger <[email-masked]>
Date: Sat, 14 Apr 2007 19:29:26 +0200

Thank you (and Michael) for the very informative replies.

There is still one thing that confuses me. Section 3.1.4 in the latest 
sockets api draft says:

  "If the SCTP stack is running low on buffers, it may partially deliver
   a message.  In this case, MSG_EOR will not be set, and more calls to
   recvmsg() will be necessary to completely consume the message.  Only
   one message at a time per stream can be partially delivered."

Does this mean:

 Alternative A)
   If recvmsg() returns a partial message, no other messages on the same
   *association* can be delivered until the started message delivery is
   complete. However, in the case of a one-to-many socket the delivery
   can be interleaved with message deliveries belonging to other
   associations (but only if SCTP_FRAGMENT_INTERLEAVE is on).

or

 Alternative B)
   If recvmsg() returns a partial message, no other messages on the same
   *stream* can be delivered until the started message delivery is
   complete. However, it may be interleaved by a delivery of a message
   belonging to another association (one-to-many socket with
   SCTP_FRAGMENT_INTERLEAVE activated) or another stream on the same
   association. It will not be interleaved with an unordered message on
   the same stream though.

It seems to me that 3.1.4 actually indicates B), and this is what 
confuses me, because I have read a lot of other things indicating A).

This is a quite important aspect, since B) means that an application 
needs one receive buffer per stream, while in A) it typically suffices 
with one per association.

Thus A) is more efficient on buffers (having one or 65535 inbound 
streams would not make any difference in application buffer space needs 
in most cases), however B) avoids head-of-line blocking better than A) 
does. With B) you could for example start a partial delivery of a 
message on the association before the complete message has arrived, and 
if then a message on another stream arrives in a single chunk the stack 
can deliver it immediately in the B) case, but it would have to store 
it in the A) case until the complete message for the partial delivery 
has arrived.

So the question is if it is A) or B) which is correct (or perhaps both 
are incorrect), and if it is enforced by the protocol standard or by 
the sockets API (that is if the sockets API introduce some head-of-line 
blocking cases which does not exist in the protocol).

/Anders Torger