Subject: RE: recv() clarification in SCTP sockets API draft

From: "David Lehmann" <[email-masked]>
Date: Mon, 2 Aug 2010 14:53:39 -0400

I think the spec could be a little more descriptive. E.g. It could
explicitly state that if the buffer is big enough, recv() will stop at
the message boundary.  Thus, a positive return value which is less than
the given buffer length, _does_ give the receiver a message boundary
indication.

If the above is true, then the sentence, "recv() and recvfrom() cannot
distinguish message boundaries.", could use clarification, such as,
"recv() and recvfrom() cannot indicate message boundaries by means of
the MSG_EOR flag."  (It is not that recv() cannot distinguish message
boundaries; it is just that it may not be able to indicate it to the
user.)

Another area, which could use some more language, is the case of
receiving data with non-blocking one-to-one sockets.  There is some text
about one-to-one non-blocking sockets in the one-to-many sections (3.2
and 3.3).  Maybe it should be moved down to the one-to-one section.

Could a non-blocking one-to-one recv() return a partial message even
though the user gave enough buffer for the entire message?  Or will it
return -1 with errno == EWOULDBLOCK?  

--
David Lehmann
Ulticom, Inc.
856-787-2952

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> Vlad Yasevich
> Sent: Friday, July 30, 2010 2:24 PM
> To: Chris Benson
> Cc: [email-masked]
> Subject: Re: recv() clarification in SCTP sockets API draft
> 
> On 07/30/2010 02:06 PM, Chris Benson wrote:
> > Vlad,
> >
> > My understanding (I think in agreement with your description)
> > is that on an SCTP socket, recv() will stop at the first message
> > boundary or before.  The latter case would be if the receiver's
> > buffer were not large enough for ONE message.  My further
> > understanding is that the two cases (one full message, or
> > one part of one message) cannot be distinguished intrinsically
> > by the caller of recv().
> >
> > I am more familiar with a AT&T SysV streams paradigm using
> > getmsg() (<stropts.h> from IEEE Std 1003.1-2001), which
> > will also read at most one "message", but can distinguish
> > between the two cases (returning 0, or MOREDATA respectively,
> > the returned length is inside a provided parameter structure.
> > This is a clear advantage for a reader that doesn't understand
> > higher layer packetization.  Although I am aware that
> > sctpsockets is maturing, would it be advantageous to extend
> > it to provide a call that can indicate full or partial message
> > receipt?  Or is there such a mechanism within sctpsockets
> > already?  I imagine that the internal [kernel] implementation
> > remains aware of received message boundaries.
> >
> 
> Hi Chris
> 
> The spec already documents an API that provides information about
> message boundaries.  It abstract that call into sctp_recvmsg()
> that provides all the necessary information.  There are other calls
> as well that provide the same information.
> 
> The spec simply discourages the use of recv() and other calls that do
> not carry message boundary information.
> 
> -vlad
> 
> > With thanks, from Chris Benson.
> >
> > On Thu, 29 Jul 2010, Vlad Yasevich wrote:
> >
> >>>  Date: Thu, 29 Jul 2010 15:07:22 -0400
> >>>  From: Vlad Yasevich <[email-masked]>
> >>>  To:  <[email-masked]>
> >>>  Subject: Re: recv() clarification in SCTP sockets API draft
> >>>
> >>>  On 07/29/2010 11:47 AM, David Lehmann wrote:
> >>>  > Hello,
> >>>  >
> >>>  >
> >>>  >
> >>>  > Section 6.1 of http://tools.ietf.org/html/draft-ietf-tsvwg-
> sctpsocket-23
> >>>  > states:
> >>>  >
> >>>  > "SCTP is message based.  The msg buffer above in send() and
sendto()
> is
> >>>  > considered to be a single message."
> >>>  >
> >>>  >
> >>>  >
> >>>  > But then goes on to say:
> >>>  >
> >>>  > "recv() and recvfrom() cannot distinguish message boundaries."
> >>>  >
> >>>  >
> >>>  >
> >>>  > So a user on one end using send() sends two messages and the
peer
> doing
> >>>  > _one_ recv() can pick up both messages?  I would think that the
> recv()
> >>>  > would pick up a single message in a similar manner that send()
sends
> one
> >>>  > message.
> >>>  >
> >>>
> >>>  Yes, but a recv() can also pick up only a part of the first
message if
> partial
> >>>  delivery is involved.  The trouble with recv() is that it doesn't
tell
> you anything
> >>>  about message boundries and that's what the spec attempts to say.
> >>>
> >>>  >
> >>>  >
> >>>  > Yet the following statement in the draft states...
> >>>  >
> >>>  >
> >>>  >
> >>>  > "In receiving, if the buffer supplied is not large enough to
hold a
> >>>  > complete message, the receive call acts like a stream socket
and
> returns
> >>>  > as much data as will fit in the buffer."
> >>>  >
> >>>  >
> >>>  >
> >>>  > ... which implies that recv() will stop reading after it reads
the
> >>>  > "complete message" as long as the buffer is big enough, which
means
> >>>  > recv() can distinguish message boundaries.
> >>>  >
> >>>  >
> >>>  >
> >>>  > What is the correct interpretation?  Thanks.
> >>>  >
> >>>
> >>>  The interpretation is the recv() call doesn't provide you any
> information regarding
> >>>  message boundaries.  In an ideal scenario, it works just fine.
> However, if only a
> >>>  partial buffer is returned, due to insufficient user buffer or
partial
> delivery state in
> >>>  the kernel, then there is no way to tell the message boundaries,
> especially if
> >>>  message sizes change.
> >>>
> >>>  -vlad
> >>>  >
> >>>  >
> >>>  > -David
> >>>  >
> >>>  >
> >>>  >
> >>>  >
> >>>  >
> >>>
> >

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 2 Aug 2010 22:00:49 +0200

Hi David,

comments in-line.

Best regards
Michael

> the given buffer length, _does_ give the receiver a message boundary
But it would also return not the complete buffer if the kernel
is doing partial delivery.
> boundaries; it is just that it may not be able to indicate it to the
I think the point is:
The user has no indication whether a complete message is delivered
by recv()/recvfrom() or not.
Do you agree on this? Do you think is is not written clearly?
> and 3.3).  Maybe it should be moved down to the one-to-one section.
Yes, that sounds good. I'll move it.
> return -1 with errno == EWOULDBLOCK?  
Sure it can. Whenever partial delivery kicks in, for example.

Best regards
Michael

---

From: "David Lehmann" <[email-masked]>
Date: Wed, 4 Aug 2010 12:00:23 -0400

> is doing partial delivery.

I would expect the kernel partial delivery to be a rare case. i.e. I
would expect the kernel to do everything reasonably possible not to
deliver partial messages.  E.g. If the receiver buffer is 64KB in the
kernel, and my association uses at most 4KB messages, I would expect the
kernel not to deliver partial messages if I provide a 4KB buffer in
recv().  (Maybe I am assuming too much!)

> Do you agree on this? Do you think is is not written clearly?

I think it would be more clear to mention the MSG_EOR flag as noted
above.

> Sure it can. Whenever partial delivery kicks in, for example.

Given the example above, 64KB kernel buffer, 4KB user buffer, and a
maximum 4KB messages on the association, could/should it return a
partial message with a non-blocking one-to-one socket using recv()?

-David

---

From: Randy Stewart <[email-masked]>
Date: Thu, 5 Aug 2010 02:29:01 -0400

David:

in-line ---

> recv().  (Maybe I am assuming too much!)

Yes, it is rare, but what happens when it occurs? You cannot
avoid it. Its easy to make happen.. send large messages...

I doubt you will ever see it if messages are small (say < 2000  
bytes)... but
you must always handle this situation.. which means you end up with a  
real
problem if you just use recv().

> I think it would be more clear to mention the MSG_EOR flag as noted

Hmm.. I thought it was in the documents... as Scott used to say..  
"send text" to
what you think would make it clearer ;-)

> partial message with a non-blocking one-to-one socket using recv()?

Well that would be nice.. but consider the kernel perspective. It cannot
possibly know the user is only sending 4K messages (if I read
your example right).. all it may know is at one edge of its rwnd
it has a message with a "B" bit. It has no idea until it sees the "E"  
bit
how big that message will be.

In general your scenario for recv() would work ... but its very  
possible with
multi-core and threading that it might not ... In order to be completely
correct you just can't code that way in the application... sure it may
work 99.9% of the time.. but that .1% where it will NOT work is when
you are in your heaviest load and at the worst possible moment...

R

> -David

-----
Randall Stewart
[email-masked]

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 5 Aug 2010 00:56:15 -0600

Randy, Michael,

Was it your intention to deviate from the POSIX behaviour
of recv()?

POSIX tells me that:

 "The recv() function shall return the length of the message
  written to the buffer pointed to by the 'buffer' argument.
  For message-based sockets, such as SOCK_DGRAM and
  SOCK_SEQPACKET, the entire message shall be read in a single
  operation.  If a message is too long to fit in the supplied
  buffer, and MSG_PEEK is not set in the 'flags' argument, the
  excess bytes shall be discarded.  For stream-based sockets,
  such as SOCK_STREAM, message boundaries shall be ignored.  In
  this case, data shall be returned to the use as soon as it
  becomes available, and no data shall be discarded."

If recv() follows POSIX behaviour, I expect that when recv()
returns >0 that either an entire message is in the buffer, or
the message was larger than the buffer and the excess has been
(or will be) discarded.  That is one successful recv() call,
one received message.

If your implementation does not guarantee this, it does not
follow POSIX.  If the spec does not require this guarantee,
then better call it 'sctp_recv()'.

Note that POSIX also states:

 "If 'fildes" refers to a socket, read() shall be equivalent
  to recv() with no flags set."

So read() had better behave the same way too, or be called
'sctp_read()'.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 5 Aug 2010 11:24:05 +0200

> Was it your intention to deviate from the POSIX behaviour
The problem is that POSIX makes SOCK_SEQPACKET unreliable in
contrast to SOCK_STREAM. Having the same function (recv())
for the same protocol behave that differently, seems to be
wrong to me.

Please note, that you argument also applies to recvmsg().

As far as I know, Solaris, Linux (LKSCTP) and FreeBSD do not
provide the POSIX behaviour for SOCK_SEQPACKET sockets when
used with SCTP.
> then better call it 'sctp_recv()'.
We'll have a sctp_recvv(), but it can be implemented as a
wrapper around recvmsg().
> So read() had better behave the same way too, or be called
See above.

Best regards
Michael

---

From: "David Lehmann" <[email-masked]>
Date: Thu, 5 Aug 2010 10:16:42 -0400

> >>> though the user gave enough buffer for the entire message?  Or
will
> > it

Of course, the kernel does not know that the user is sending a maximum
of 4KB.
But the application programmer does! It is the application programmer
who will choose the appropriate function to call.  If the programmer
knows that
the maximum message is 4KB, he could chose to call recv() with a buffer
of
4KB + 1.  A return value between 1 and 4K inclusive should indicate that

the caller got a complete message.

> you are in your heaviest load and at the worst possible moment...

In general, a programmer creates a buffer for the maximum expected
buffer and will prefer to use recv() over recvmsg().   It is simpler
and it is the most familiar.

-David

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 5 Aug 2010 18:40:27 +0200

> the caller got a complete message.
But the programmer of the recv() call must know what the programmer
of the send() call does... SIGTRAN comes up to my mind. There is
no such limit for SIGTRAN. However, I saw implementations which were
not able to handle large messages...
> and it is the most familiar.
But what if the peer sends a message larger than you expect? Which
upper layer restricts its messages to 4K?

Best regards
Michael

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 5 Aug 2010 13:03:30 -0600

Michael,

Michael Tuexen wrote:                                                                                                                                      (Thu, 05 Aug 2010 11:24:05)
> for the same protocol behave that differently, seems to be

What do you mean?  POSIX says:

 "SOCK_SEQPACKET
       Provides sequenced, reliable, bidirectional,
       connection-mode transmission paths for records.  A record
       can be sent using one or more output operations and
       received using one or more input operations, but a single
       operation never transfers part of more than one record.
       Record boundaries are visible to the received via the
       MSG_EOR flag."

SOCK_SEQPACKET is clearly "reliable".  What do you mean by

  > contrast to SOCK_STREAM.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "David Lehmann" <[email-masked]>
Date: Thu, 5 Aug 2010 15:30:32 -0400

> not able to handle large messages...

Call me crazy, but I think there is the assumption that the receiver is
conforming to the same spec as the sender.  :-)

The fact that Sigtran or any other protocol does not specify a limit,
does not diminish the fact that all applications have limits.  Is there
a Sigtran implementation which handles a 1GB message?  How about a 1TB
message?  We can play this 'what if' argument all day long, but it
serves little value.  If a receiver gets a message beyond its limit
(whatever that may be), it will handle it in a manner appropriate for
its application.

> But what if the peer sends a message larger than you expect? 
 
Return an error to the caller, raise an alarm, kill the connection,
ignore the message, send an error message back to the sender, etc. ....
whatever the protocol/API dictates.  It is beyond the scope of the
transport layer.

> Which upper layer restricts its messages to 4K?

It doesn't matter.  It was just an example. I would imagine that there
thousands upon thousands of TCP applications which are out there, used
by companies whose protocols are not made public via the IETF or some
other standards body.  Who knows how many of these apps fall into the
scenario that was presented?  A number of these applications may benefit
by using SCTP.  The programmers, IMHO, will be looking for their trusty
old recv(), along with send(), listen(), accept(), and connect().

I guess my basic premise is that the recv() call semantics for
one-to-one sockets could use some more clarity.  I realize the authors
may want to discourage it use, but it is the function most familiar to
many programmers.  The simplicity of recv() is attractive and probably
'good enough' for a large number of applications.

In particular, the statement "recv() and recvfrom() cannot distinguish
message boundaries" is confusing and a bit misleading, IMHO.  Recv() can
distinguish message boundaries under the scenario that was presented.
>From what I am gathering, the authors meant that 'recv() and recvfrom()
cannot distinguish message boundaries in ALL scenarios, because it has
no way to return the MSG_EOR indicator.'  I am asking for clarification.
It may also be prudent to provide some guidance/pitfalls text for the
use of recv()/recvfrom() using one-to-one sockets.

...just one person's opinion. 

-David

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 5 Aug 2010 22:03:58 +0200

> SOCK_SEQPACKET is clearly "reliable".  What do you mean by
How can I receive "using one or more input operations". I can't
do that with recv(), recvfrom() or recvmsg(). They drop the
bytes of the record which do not fit into the buffer. This
makes it "unreliable". As a receiver I have no idea how big
the largest message would be. For SCTP we do no want to loose
parts of the messages.

Best regards
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 5 Aug 2010 22:24:24 +0200

> (whatever that may be), it will handle it in a manner appropriate for
This is my point. If you receive a message which is too large,
you need to read the complete message, possibly send a
message indicating that the message was too long, and 
continue to process the next message. Therefore you must
be able to find the record boundary, which means get the
MSG_EOR flag. This is not possible with recv().
> whatever the protocol/API dictates.  It is beyond the scope of the
But the application needs to detect this, skip the message which is too
big, and continue with the next message.
> old recv(), along with send(), listen(), accept(), and connect().
If they use TCP, they can handle partial delivery already, since they
can not rely on message boundaries at all.
> 'good enough' for a large number of applications.
It is OK, if you use a single stream and reconstruct message boundaries
at the application level.
> use of recv()/recvfrom() using one-to-one sockets.
Send some text. I think you ca get what you want if you set the partial delivery
point approppriately (section 7.1.21). However, you need to be able to
handle the case where you get a message longer than you expect.
> ...just one person's opinion.
Please, send some text, which we can discuss.

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 5 Aug 2010 22:44:18 +0200

> use of recv()/recvfrom() using one-to-one sockets.
Just to be clear: The following code (untested, conceptual):

offset = 0;
len = 0;
do {
	flags = 0;
	res = recvmsg(buffer + offset, sizeof(buffer) - offset, &flags);
	if (res > 0) {
		len += res;
	}
	if ((len >= sizeof(buffer)) && (flags & MSG_EOR == 0)) {
		offset = 0;
	} else {
		offset += res;
	}
} while (res > 0) && (flags & MSG_EOR == 0);

if (len <= sizeof(buffer) {
	handle_message();
} else {
	send_error_message();
}

shows how I think you need to consider MSG_EOR to be able to process
arbitrary messages, handle the ones within the expected range and send
an error message for others. Please note, that I assume no message
interleaving, which means for one-to-one style sockets, level 0 or level 1.
For level 2, the above code does not work.

Best regards
Michael

---

From: "David Lehmann" <[email-masked]>
Date: Thu, 5 Aug 2010 16:52:03 -0400

> >>> 4KB + 1.  A return value between 1 and 4K inclusive should
indicate
> > that

My point is that once you get past the receiver's limit, it may not
matter what the message boundary is.  A simple (READ: good enough) app
can take the radical step of aborting the association.

> big, and continue with the next message.

The app would detect it, in the above scenario, if recv() return 4KB +
1.
As far as the app's response, we should not assume the app will move on
to the next message.  The simple app, which KNOWS it should only get a
max of 4KB, could simply abort the association.... which may be good
enough if it KNOWS that it would never happen anyway.

I agree that an app, in which an abort is unacceptable, would have to
use recvmsg().

> can not rely on message boundaries at all.

My point was that many (most?) upper layer protocols are unknown to us.

> Send some text. I think you ca get what you want if you set the
partial
> delivery

OK.

-David

---

From: "David Lehmann" <[email-masked]>
Date: Thu, 5 Aug 2010 17:02:50 -0400

> For level 2, the above code does not work.

Whoa! Which recvmsg() are you using? On Solaris and Linux, the signature
is:
ssize_t recvmsg(int socket, struct msghdr *message, int flags);

To get the MSG_EOR flag is a little more complicated.

	void *buffer;   /* for message */
	size_t len;     /* length of buffer and max length of message */
	int n;

	...

	/* simple recv() call */
	n = recv(fd, buffer, len, 0);

	/* same functionality of the above recv(), but gets flags also
*/
	struct msghdr msg;
	struct iovec  iov;
	
	msg.msg_name = NULL;
	msg.msg_namelen= 0;
	iov.iov_base = buffer;
	iov.iov_len = len;
	msg.msg_iov = &iov;
	msg.msg_iovlen = 1;
	msg.msg_control = NULL;		/* ancillary data */
	msg.msg_controllen = 0;		/* ancillary data buffer len */
	msg.msg_flags = 0;		/* flags on received message */
	
	n = recvmsg(fd, &msg, 0);

Add this code to your loops.  If the app does not care, the one-liner
recv() is very attractive.

-David

---

From: Chris Benson <[email-masked]>
Date: Thu, 5 Aug 2010 15:21:25 -0700 (PDT)

Hi David,

Referring to David's comparison of recv() and recvmsg(),
the intent of my original comment about sctpsockets was
to wonder if there could be an extension to the simple
recv() call that also indicated MSG_EOR.  Standard recv()
has an input only argument named flags.  This would need
an input/output pointer argument, perhaps thus ...

ssize_t recv_with_eor(int sock, void *buf, size_t len, int *p_flags);

where the value of *p_flags might or might not contain (bitwise 
OR'd) MSG_EOR on return.

Upon following this discussion by those more familiar with
the sctpsockets behaviour for recvmsg() on end of message,
I guess "we" could all just write our own "recv_with_eor()"
that initializes and calls recvmsg() and returns possible EOR.

That would sort-of comply with David's "one line" coding,
but provide the distinction between full messages on one
hand, and partial messages on the other.

With thanks, from Chris.

Chris Benson, Software Engineer,
Adax Inc., Berkeley, California, USA.
    [email-masked]
    +1 510 548-7047 ext. 189
    www.adax.com

On Thu, 5 Aug 2010, David Lehmann wrote:

>>  Date: Thu, 5 Aug 2010 17:02:50 -0400
>>  From: David Lehmann <[email-masked]>
>>  To: Michael Tuexen <[email-masked]>
>>  Cc:  <[email-masked]>, Randy Stewart <[email-masked]>
>>  Subject: RE: recv() clarification in SCTP sockets API draft
>>

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 5 Aug 2010 18:42:37 -0600

Michael,

Michael Tuexen wrote:                       (Thu, 05 Aug 2010 22:03:58)
> the largest message would be. For SCTP we do no want to loose

Both recvmsg() and recv() offer MSG_PEEK.  If the application is
worried about it, it can always peek the message size and provide
a suitably sized buffer before calling recvmsg() or recv() without
MSG_PEEK.  This is one or more input operations.

If the application is not worried about it, it provides a buffer
large enough for the largest valid message, and simply calls
recv().

This does not make recvmsg() nor recv() unreliable for
SOCK_SEQPACKET: it allows the user to specify whether they care
about bytes beyond a maximum message size: which I believe
was the original point in this thread.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 6 Aug 2010 10:21:06 +0200

> ssize_t recvmsg(int socket, struct msghdr *message, int flags);
Sure, it is pseudo code. I needed a recv() like call which provides
the MSG_EOR flag. You can build a simple wrapper around recvmsg()
which does this... I just wanted to point out how you need to
check the flags.

Best regards
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 6 Aug 2010 10:27:55 +0200

> where the value of *p_flags might or might not contain (bitwise 
Sure you can do this and this might be very convenient to use.

You can implement it easily using recvmsg().

I'm hesitating to add this to the ID because of the following:
We have added over time sctp_recvmsg, sctp_sendmsg, sctp_send
and sctp_sendx. And we can not really specify PR-SCTP policies
with it. We can not continue on inventing these functions over
and over.
We are now using the following concept:
1. All functionality is accessible with sendmsg()/recvmsg() and
   CMSGs. This seems to be future proof, but not convenient.
2. We'll define sctp_sendv()/sctp_recvv() which are for convenience
   and also take a structure. The structure has a type/length
   indicator and can therefore be extended in the future. However,
   you can implement sctp_sendv()/sctp_recvv() using sendmsg()
   recvmsg().

So I would vote for implementing recv_with_eor() in your code.
It does not even depend on SCTP...

Best regards
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 6 Aug 2010 10:39:14 +0200

> MSG_PEEK.  This is one or more input operations.
But this means:
1. I need always two system calls for reading a single message.
2. I can not read messages larger than the socket receive buffer.
> was the original point in this thread.
Brian: I understand that not being within the definitions of POSIX
is something one should not do by accident. SCTP implementations (Linux,
Solaris and FreeBSD) do not drop the final part of messages when
calling recv()/recvfrom()/recvmsg() for years.
You can call that non-POSIX compliant, but if POSIX ever wants to
work on the SCTP socket API they need to deal with it. Either change
their spec for recv()/recvfrom()/recvmsg() or change the SCTP
socket API. Implementation might or might not implement their version
then.

Best regards
Michael

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 6 Aug 2010 03:46:04 -0600

Michael,

Michael Tuexen wrote:                       (Fri, 06 Aug 2010 10:39:14)
> 1. I need always two system calls for reading a single message.

Only if the app is worried about messages greater than a
defined maximum.

> 2. I can not read messages larger than the socket receive buffer.

Where does POSIX say that one cannot read a message larger
than the socket receive buffer?  The socket receive buffer
is a high water mark, not an absolute maximum message size.
That is, once exceeded, another message will not be buffered
at the socket buffer.  It does not affect reading of one
message.

> socket API. Implementation might or might not implement their version

This is why I say call it sctp_recv() and sctp_recvmsg(), because
they bear no similarity to the POSIX versions.  At least one SCTP
implementation has follows POSIX for +10 years. ;)  All socket
implementations have implemented reliable message-based POSIX
socket sematics (X.25, DECNET, OSI), for +25 years now.  You want
POSIX to change the semantics for all of them just for your view
of how SCTP should behave?  Also, do not say this was not brought
up many years ago now: it was; I did.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Randy Stewart <[email-masked]>
Date: Fri, 6 Aug 2010 06:36:19 -0400

in-line ...

> at the socket buffer.  It does not affect reading of one

So let me try to get a "grasp" on POSIX here.

If an app does a recv() and gives less than the size
of the message.. we discard the rest of the message so for
example

recv(.., 8k) <--- (9k msg in buffer)

and the app gets 8k.. and we discard the other 1k...

That does not seem too reliable but ok...

Now lets say we have a 10Meg message being sent by the
peer. In a blocking mode I guess this could work..

recv(.., 10Meg) <---

And we hold the app blocked until all of the 10Meg message
comes in.

But what about the non-blocking case? If the rwnd is around 200k (which
is high for most implementations) we end up getting
the first 200k or so of the 10Meg message.. does the kernel now need
to discard the other 9.8 Meg?

That again seems rather un-reliable for a reliable transport... and a  
lot
of work in the transport to remember to discard all the rest of the  
message
still making its way into the kernel....

Think 10Meg is unreasonable as far as message size goes? I do believe
that the folks at UBC were using 11Meg messages or so for MPI...

> up many years ago now: it was; I did.

Hmm that could be some of the problem. X.25/Decnet and OSI are hard to
find in most modern O/S's and getting harder by the day. And even more
hard is to find an ISP that will give you a X.25 feed (or decnet).. oh  
you
may still be able to find x.25 but I bet most of these are long gone..  
and
OSI is in this same boat I fear. IP has pretty much replaced it  
all.. ;-)

Maybe thats the problem when you think about it. POSIX defined these  
years
ago when messages in the 10Meg size were not even on the radar... the  
semantic
of discarding the rest of the message if the app did not read it all  
probably
made a lot of sense when you were dealing in real small messages ...  
but when
apps are sending large messages the semantic seems rather silly if not  
just
plain broken...

R

-----
Randall Stewart
[email-masked]

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 6 Aug 2010 13:48:19 +0200

Hi Brian,

comments in-line.

Best regards
Michael

> Only if the app is worried about messages greater than a
When such a maximum does exist...
> at the socket buffer.  It does not affect reading of one
Implementations I know derive the advertised receiver window
from the receive socket buffer size. They start partial delivery
when some substantial part of the receive buffer is used by a
message. But you can say nothing of this is defined by POSIX...
> implementation has follows POSIX for +10 years. ;)  All socket
OK, I see. This means that the applications were not portable.
So I think we need to clarify that the behavior described in the
ID is not what POSIX describes.
> of how SCTP should behave?  Also, do not say this was not brought
No, I don't. I'm even not sure if POSIX cares about SCTP or not.
If they want to work on it, it is their decision. I'm only
promoting what I (personally) think is right. And this is
not to use recv() and recvfrom() for SCTP...
> up many years ago now: it was; I did.
I did not say that it was not brought up or no one mentioned it
(but I was not aware of it, so thanks for pointing it out), but
I wanted to say that the ID didn't describe the POSIX behavior.

We can make that clear in the text...

---

From: "David Lehmann" <[email-masked]>
Date: Fri, 6 Aug 2010 09:11:30 -0400

> with it. We can not continue on inventing these functions over

Agreed!

>    you can implement sctp_sendv()/sctp_recvv() using sendmsg()

Sounds like a plan.

-David

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 6 Aug 2010 07:27:46 -0600

Randy,

Randy Stewart wrote:                          (Fri, 06 Aug 2010 06:36:19)
> to discard the other 9.8 Meg?

There is no difference between the blocking and non-blocking case
for POSIX recv().  Either the entire message is available, or the
call blocks in the blocking case, or returns EGAIN or EWOULDBLOCK
in the non-blocking case.

> that the folks at UBC were using 11Meg messages or so for MPI...

That would be the definition of low-performance computing: SCTP
exhibits head-of-line blocking for message sizes larger than an MTU,
because you forced it to deliver all the fragments in sequence before
doing anything else.  High-performance computing would break the
big 11meg messages down into MTU sized chunks to avoid head of line
blocking.  Nevertheless, MPI is a research thing that really needs
to receive its messages within the kernel to be of much use and would
benefit from a higher performance kernel API such as STREAMS rather
than user-space sockets.

> you may still be able to find x.25 but I bet most of these are long
> gone..  and OSI is in this same boat I fear. IP has pretty much
> replaced it  all.. ;-)

X.25 and OSI are still launching your missles, landing your plane and
processing your banking transactions.

> years ago when messages in the 10Meg size were not even on the
> radar... the  semantic of discarding the rest of the message if the
> app did not read it all  probably made a lot of sense when you were
> dealing in real small messages ...  but when apps are sending large
> messages the semantic seems rather silly if not  just plain broken...

To avoid head-of-line blocking in SCTP, no self-respecting app will
send messages larger than a couple of MTU in size, so I suppose the
POSIX model still applies.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 6 Aug 2010 07:32:25 -0600

Michael,

Michael Tuexen wrote:                       (Fri, 06 Aug 2010 13:48:19)
> message. But you can say nothing of this is defined by POSIX...

POSIX does not support a partial-delivery API for message-based sockets.

> We can make that clear in the text...

Perhaps you should call it a socket-like API then, rather than
a socket API.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Randy Stewart <[email-masked]>
Date: Fri, 6 Aug 2010 11:19:10 -0400

> Perhaps you should call it a socket-like API then, rather than

If I remember correctly it was NOT POSIX that came up with
sockets.. but Berkley... I can ask Kirk M. to confirm this, I think he  
would
remember ...

R

-----
Randall Stewart
[email-masked]

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 6 Aug 2010 17:29:49 +0200

> Perhaps you should call it a socket-like API then, rather than
I think it is a socket API, not a POSIX socket API. But the document
comes from the IETF, not from POSIX, so that seems to be acceptable.

Best regards
Michael

---

From: Kacheong Poon <[email-masked]>
Date: Fri, 06 Aug 2010 23:45:03 +0800

> large enough for the largest valid message, and simply calls

I think the context is about recv().  If an app chooses
to use a reliable message based transport protocol, I think
it is safe to assume that the app itself is designed not
to worry about message boundary since the underlying protocol
will take care of that.  And it is obvious that the app does
not want to loose data (unknowingly), otherwise it won't
choose a reliable transport.  For this kind of apps, there
is no need to have app level protocol to indicate message
boundary.  Let's put this in the context of discussion.  Can
you explain how an app can find out the message size using
MSG_PEEK with recv()?

AFAIK, recv() returns the size of data copied into the
supplied buffer.  Hence an app using recv(MSG_PEEK) cannot
really tell if a message is larger than the supplied buffer.
What do you suggest an app to do then?  Using a loop with
recv(MSG_PEEK) with larger and larger supplied buffer until
the returned value of recv() is smaller than the size of
supplied buffer?

The situation is a little bit better in recvmsg(MSG_PEEK)
in the sense that MSG_TRUNC is returned so that an app can
know that if the message is truncated or not.  But I think
it still needs a loop to find out the message size.

> was the original point in this thread.

I read this thread late so I may have missed something.
But I think your point is that POSIX standard should be
followed.  If I am not mistaken above and a loop is
necessary, then I don't think following POSIX is correct.
Why shouldn't we allow an app to use recvmsg() to retrieve
the whole message larger than the supplied buffer by
getting part of the message each time it is called?

I agree with David that most, if not all, apps know the
message size used.  But I don't think there is a need to
put such a restriction on the API.

-- 

					K. Poon.
					[email-masked]

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 6 Aug 2010 11:07:43 -0600

Kacheong,

please see inline below...

Kacheong Poon wrote:                       (Fri, 06 Aug 2010 23:45:03)
> really tell if a message is larger than the supplied buffer.

Only for stream-based.  For message-based, the size returned
is always the size of the message, which can be greater than
the buffer.  Thus

  n = recv( s, NULL, 0, MSG_PEEK );

returns "the length of the message".  For byte-stream, this
call would return zero.

> it still needs a loop to find out the message size.

recvmsg() "returns the total length of the message".  For
stream-based that is always less than the size of the buffer,
but for message-based is the true message size even if it
is larger than the buffer.

> getting part of the message each time it is called?

Sure, but it is not POSIX recvmsg() in that case, but
sctp_recvmsg().

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "David Lehmann" <[email-masked]>
Date: Fri, 6 Aug 2010 14:28:52 -0400

> put such a restriction on the API.

I think David thinks 'that most, if not all, apps know the _maximum_
message sized used'. ;-)

-David

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 7 Aug 2010 10:57:12 +0200

>  n = recv( s, NULL, 0, MSG_PEEK );
At least for a SOCK_DGRAM socket recv() returns 0 on
FreeBSD, Linux, Mac OS X, and OpenSolaris (just tested
it with a small program).
So it does not work on these platforms.
> is larger than the buffer.
Which is not true on FreeBSD, Linux, Mac OS X, and OpenSolaris.
(Just tested with a small program for SOCK_DGRAM sockets).
recvmsg() always returns the length of the message copied
to the buffer. So if you provide a 1 byte buffer for a 2 byte
message, recvmsg() returns 1 and MSG_TRUNC is set.

---

From: Randall Stewart <[email-masked]>
Date: Mon, 9 Aug 2010 09:01:28 -0400

> comes from the IETF, not from POSIX, so that seems to be acceptable.
Just to follow up..

I asked Kirk M if it was BSD or POSIX
that created the socket api.. His response is as follows
***************
Sockets were designed at Berkeley in the early 1980's (long before
POSIX got its name). Attached is the document about them released
with 4.2BSD in 1983.
***************

He also forwarded me the document mentioned.. if anyone would like
a copy they can send a direct email to me and I would be glad to send
off-list a copy of the paper...

R

------------------------------
Randall Stewart
803-317-4952 (cell)

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 9 Aug 2010 13:25:28 -0600

Michael,

Michael Tuexen wrote:                       (Fri, 06 Aug 2010 17:29:49)
> comes from the IETF, not from POSIX, so that seems to be acceptable.

That would be fine if it followed BSD, but it doesn't follow that either.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 9 Aug 2010 13:30:30 -0600

Michael,

Michael Tuexen wrote:                 (Sat, 07 Aug 2010 10:57:12)
> message, recvmsg() returns 1 and MSG_TRUNC is set.

I checked 4.3BSD and 4.4BSD OSI and they return the number of
bytes written to the buffer as well.  So it looks like I'm full
of little red ants.  Did I at least sound convincing?

OTOH, STREAMS (XTI) handles partial delivery API quite nicely
within the standard.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 10 Aug 2010 11:29:05 +0200

> of little red ants.  Did I at least sound convincing?
Good enough to get me writing the test programs...
> OTOH, STREAMS (XTI) handles partial delivery API quite nicely
OK. Unfortunately, I've never used STREAMS (XTI) as a programmer.

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 10 Aug 2010 11:38:35 +0200

> That would be fine if it followed BSD, but it doesn't follow that either.
I think the way the socket API for SCTP is designed does not perfectly fit
in what designers of the BSD sockets or the POSIX standard thought would
be right. But they did not know anything about SCTP. Thereofre some
compromises were necessary. I think it is an acceptable approach to do things
differently as long as it is made clear what the differences are (and
it makes sense). There are more differences: we also use MSG_EOR on
SOCK_STREAM sockets. Neither BSD nor POSIX considered that.

I think the important point is to make clear, what the differences are.
If someone uses SCTP sockets and knows about BSD sockets or POSIX sockets,
it should be clear to him what to expect. It can be learned...

Best regards
Michael