Subject: [Tsvwg] SCTP socket API message size limitations

From: Mark Butler <[email-masked]>
Date: Thu, 23 Mar 2006 23:12:13 -0700

The current SCTP socket draft specifies that SO_SNDBUF "bounds the 
maximum size of data that can be sent in a single send call".  That sets 
a relatively low limit on possible message sizes.  It is common to have 
send buffer sizes limited  to 64K-128K.

However, send buffer size bounds to message sizes does have one 
advantage - it is much easier to implement atomic sendmsg operations (as 
currently specified) that way.  If a message will not fit into the send 
buffer, sendmsg has to either block until space is available or return 
with an error.  Blocking is not much of a problem except for one thing - 
if a POSIX signal is delivered to a process in the middle of an 
indefinitely blocked sendmsg operation, the signal handler has to be 
executed or a long list of POSIX mechanisms (alarms, aio, ipc) break.

In many environments the sendmsg syscall can be restarted after the 
signal is handled, but whether system calls are restarted is specified 
by the user process on a signal by signal basis via the sigaction(2) 
call.  So any implementation needing to send large messages has to have 
a policy for dealing with aborted sendmsg() operations.

One option is to use PR-SCTP to cancel the message and return -1  as if 
the message was never sent.  The problem with this is that message data 
(potentially a large amount of data) may have already been partially 
delivered on the other end.  The other is to keep the message open, 
return a partial byte count, and resume the message with the next write.

The first option is most consistent with an atomic write policy 
(SOCK_DGRAM semantics), which is what is specified now on the send 
side.  The second option is most consistent with a partial write policy 
(SOCK_SEQPACKET semantics) which is equivalent to what is specified now 
on the receive side (i.e. message termination with MSG_EOR).

So I guess the first question is:  Is SCTP (as far as this API is 
concerned) a large message protocol or a small message protocol?  If it 
is a small message protocol, partial deliveries are unnecessary.  If it 
is a large message protocol, there needs to be a way to send a message 
in multiple operations.   The standard way of doing that is with MSG_EOR.

So the second question is:  Is anyone unalterably opposed to adding an 
option to enable MSG_EOR semantics on sendmsg(2) operations?

Comments?

 - Mark B.

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 23 Mar 2006 23:47:18 -0700

Mark,

Mark Butler wrote:                             (Thu, 23 Mar 2006 23:12:13)
> send buffer sizes limited  to 64K-128K.

I see no need for the restriction.  Some implementations could have problems
with data larger than the send bufer, others will not.  Those that have a
problem can impose their own restrictions.

--X--snip--X--
> option to enable MSG_EOR semantics on sendmsg(2) operations?

I already use MSG_EOR in sendmsg(2), however, it was not traditionally used
on sendmsg, only recvmsg.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Fri, 24 Mar 2006 01:50:02 -0700

>problem can impose their own restrictions.
The restriction makes sense if the objective is atomic, small message 
writes.  It is impractical to do large writes atomically without 
blocking/ignoring all non-fatal signals for an indefinite period.

In addition socket buffer limits are generally viewed as a system 
"tunable", not something that has to be increased so a program will work 
at all.  For example, the Linux netdev folks had a discussion today 
about why they do not want to increase the send buffer limit beyond ~64K 
- basically to protect against kernel buffer consuming DoS attacks.  
With an atomic-only write policy, the system limits have to be increased 
to arbitrary sizes for applications to work at all, which conflicts with 
that objective.

Changing the default behavior for sendmsg(2) will break existing 
applications.  If there is to be portable large message support, there 
needs to be an option to get MSG_EOR semantics on writes as well as reads.

>on sendmsg, only recvmsg.
Both POSIX and SUS now specify its use on sendmsg(2) and sendto(2).

 - Mark B.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 24 Mar 2006 04:30:45 -0700

Mark,

Mark Butler wrote:                              (Fri, 24 Mar 2006 01:50:02)
>    "tunable",  not  something  that has to be increased so a program will
>    work  at  all.   For  example, the Linux netdev folks had a discussion
>    today  about  why  they  do not want to increase the send buffer limit
>    beyond ~64K - basically to protect against kernel buffer consuming DoS
>    attacks.   With an atomic-only write policy, the system limits have to
>    be increased to arbitrary sizes for applications to work at all, which
>    conflicts with that objective.

You're still thinking BSD'ish.  SVR4, for example, does not restrict the
size of the write on the basis of the high watermark for flow control.  The
maximum message size written is separate from buffer sizes.  Send buffers
are allowed to overflow by one message minus one byte regardless of message
size.  POSIX and the OpenGroup CAE does not specify the implementation for
determining when "space is not available" [sendmsg reference page].  In fact
POSIX only describes the socket "receive queue" and not the "transmit queue".

I still don't see any need for the restriction.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Fri, 24 Mar 2006 07:39:34 -0700

>POSIX only describes the socket "receive queue" and not the "transmit queue".
I agree - POSIX does not require that messages be buffer size limited.  
That is an implementation decision.  If you want atomic writes without 
buffer size limitations, you just have to block all non-fatal signals 
until enough of the message clears the buffer.

 - Mark B.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 24 Mar 2006 07:59:27 -0700

Mark,

Mark Butler wrote:                                                    (Fri, 24 Mar 2006 07:39:34)
>    I  agree  -  POSIX  does  not  require  that  messages  be buffer size
>    limited.   That  is  an  implementation  decision.  If you want atomic
>    writes  without  buffer  size  limitations, you just have to block all
>    non-fatal signals until enough of the message clears the buffer.
>     - Mark B.

My point was that you can accept a message larger than the send buffer,
atomically, but not accept another until the entire message has cleared.
DOS is averted by setting a limit on the largest message that can ever
be written.  And so, the restriction should be dropped from the draft.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Fri, 24 Mar 2006 13:27:15 -0700

>be written.  And so, the restriction should be dropped from the draft.
If it is acceptable to block non-fatal signals for long periods, sure.  
Consider an application sending a 64MB message through a send buffer 
limited to 64KB.  Normal (blocking) behavior for TCP implemenations is 
to send as many segments as possible until the congestion window fills 
up, then start filling up the send buffer, then push through new frames 
as space is freed up, until the write is completed or a signal is 
caught.   It is only the policy against blocking signals indefinitely 
that keeps a TCP write from being an atomic operation.

UDP implementations on the other hand cannot send messages larger than 
64K, never block, and never check signals..  The datagram is either 
copied completely into the buffer or an error is returned.  Simple all 
or nothing behavior.  Non-blocking writes on TCP work the same way.

The restriction per se is not that important.  It is a fait accompli - 
present in at least two major implementations,  probably others.  It is 
not likely to go away for atomic writes on those platforms unless there 
is a way to remove it without serious side effects.

What I believe is important is a standard, portable way to make the 
restriction irrelevant.  The easiest way to do that is with an option to 
enable sending messages with more than one operation.  Then 
implementations simply add an outer loop similar to their local TCP 
implementation to blocking writes, keep track of the current message 
being sent on any given association, a bit of MSG_EOR checking, and they 
are done.

The ability to portably send large messages has considerable bearing on 
defining SCTP adaptations for higher level protocols.  Any such mapping 
has to specify where message boundaries need to be.  A mapping that 
requires support for large messages will not be successful if few SCTP 
implementations can actually generate them.  Of course large message 
mappings are much more efficient than small message mappings, or 
mappings where the message boundaries are ignored.  If SCTP is to 
transport HTTP or SMTP more efficiently than TCP, it is important that 
the standard mappings be done correctly.  Message boundary agnosticism 
is not a good strategy.

Anyone else out there?

 - Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 24 Mar 2006 15:55:44 -0700

Mark,

Mark Butler wrote:                                                    (Fri, 24 Mar 2006 13:27:15)
> If it is acceptable to block non-fatal signals for long periods, sure.  

Your not getting it.  Read carefully: and implementation can accept a write
that is larger than the send buffer size and yet buffer it for transmission
atomically and without blocking.  It does not have to block waiting for
send buffer space, a single message is allowed to slop over the send buffer
by the maximum message size minus one byte.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Fri, 24 Mar 2006 19:33:16 -0700

>by the maximum message size minus one byte.
Perhaps in some environments.  However, I can't imagine the Linux 
mainline (for example) accepting code that allowed 64MB messages to be 
buffered on sockets where the system limit is 64KB. 

 - Mark B.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 24 Mar 2006 23:05:44 -0700

Mark,

Mark Butler wrote:                                                    (Fri, 24 Mar 2006 19:33:16)
>    buffered on sockets where the system limit is 64KB.

It is good that Linux does not set the standard for all.

These is no point in specifying a restriction in the API that is
implementation specific.  If Linux or other implementations would like
to restrict their message sizes in some way, it is part of that
implementation's specifics.

I don't see a problem with providing a mechanism to determine what that
maximum is; however, the maximum should be dictated by the API as bound
to the send buffer size: let the implementation do so if it has such a
restriction.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Sat, 25 Mar 2006 00:36:59 -0700

>implementation's specifics.
A well defined API should specify how a minimally compliant 
implementation behaves.  It would be excessive for the draft to 
_require_ that messages that do not fit into the buffer be rejected.  
However, it is effort well spent for the draft to warn that messages 
that do not fit into the buffer _may_ be rejected.   To the degree that 
major implementations cannot or will not change it is very helpful to 
document where potential problems are.

>to the send buffer size: let the implementation do so if it has such a

If you want such a mechanism, a change in the draft, or a different 
solution, please make your own proposal.  The fact that some 
implementations do not have a problem is irrelevant.  Major ones do, and 
cannot be changed in the manner you have suggested.  The key purpose of 
a specification like this (and yes it is a _specification_, if only an 
informal one) is to define an effective least common denominator for 
portable programming.

 I would like to hear feedback about whether including support for 
partial message writes is an extension worth pursuing.  Partial message 
writes have other advantages - increased efficiency for common message 
boundary ignorant code is a big one.  Telnet over SCTP would be 
ridiculously inefficient without it, for example, Nagle algorithm or no 
Nagle algorithm

 - Mark B.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 25 Mar 2006 02:33:27 -0700

Mark,

Mark Butler wrote:                                                                    (Sat, 25 Mar 2006 00:36:59)

> cannot be changed in the manner you have suggested. ...

Where those "major" implementations Kings and Presidents? ;)

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/