Subject: Re: [Tsvwg] A SCTP sockets/sctp_peeloff() issue to think about

From: Kacheong Poon <[email-masked]>
Date: Wed, 2 Oct 2002 14:35:55 -0500 (CDT)

Included message from Randall Stewart <[email-masked]>:

>----
>Hi all:
>
>I just had an interesting thought..
>
>When I do a
>
>sctp_peeloff(...)
>
>what happens to data that has not been read for the association being 
>peeled off?
>
>In most implementations you stuff the data up a socket buffer... with
>no way to reach up and grub the data out of the socket buffer.. not
>that you would want to do this anyway since this is an EXTREME layer
>violation and leads to spagetti code.....
>
>If I do nothing and data is pending on the socket for the
>association being pulled off I have an issue that I may deliver data
>out of order... depending on which socket is read first of course...
>----

Why is it a layer violation?  The sctp_peeloff() call is in the
socket layer.  What the code should do is to move the data or
notification of the specified assoc id from the old socket to
the new socket.  There are some race conditions involved, such
as interaction between the SCTP layer and socket layer, but
should be doable.  And the SCTP layer does not need to know
anything new happening in the socket layer, except for the new
socket so that it knows where to send data/notification to.  All
code should be in the socket layer.  Whether the cost is expensive
or not is another question.  This is also dependent on the OS.  I
guess it does not need to be mentioned in the socket draft.

As the name of the call implies, peeling off should mean taking
away something with it too (-:  If the implementation does not
do this, it will be a surprise to the apps...

							K. Poon.
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Randall Stewart <[email-masked]>
Date: Wed, 02 Oct 2002 14:44:20 -0500

> guess it does not need to be mentioned in the socket draft.

Well I had not considered the sctp_peeloff() as part of the sockets
layer... but even in that vein then how does the sockets code
sctp_peeloff() know which mbufs in queue (I will use bsd terms here)
are from which association..

I guess it could dig through and look for the sockaddr * that is
embedded in the mbuf's in the socket queue.. find each address and
then do a lookup.. but it would be very very expensive if there
were a LOT of data inqueue.. and worse if the app has read the
top of the data .. say because it supplied a too small of buffer..
then the address stuff is lost and the message at the top of
the socket buffer can not be deteremined as to who it belongs to..

> do this, it will be a surprise to the apps...

exactly... and how this is done is a real issue.. in order to
find which association goes with which data there must be
something embedded in the socketbuffer to point back to the
association.. this could in theroy be a embedded address MT_SONAME
or some other cookie.. but that means that the sockets layer must
be aware of some hook the sctp layer is putting in...

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: Qiaobing Xie <[email-masked]>
Date: Wed, 02 Oct 2002 17:17:19 -0500

...
> are from which association..

I thought for each piece of incoming data being read by the app, there
is always going to be some sort of an association id stored alongside
with the data all the way up to the app. Otherwise, after a read on the
fd, how will the app tell which association this piece of data belongs?

-Qiaobing

---

From: Randall Stewart <[email-masked]>
Date: Wed, 02 Oct 2002 19:53:29 -0500

> fd, how will the app tell which association this piece of data belongs?

well it is more complicated than that.. how BSD land does this (and I
think other O/S's that may be rooted in a BSD sockets implementation)
is this

When you dump a message onto a rcv socket buffer you dosomething like
sbappendaddr(....) the sockets layer copys the address into
a mbuf and puts it at the head of the mbuf chain to deliver
followed by any ancillary data followed by the data itself

These are all linked through the
m_next pointer

The next packet is done via the
n_nxtpkt pointer...(something like that may have the name off )

Now the data can be scatterd over multiple mbufs all chained together..

For grins lets say it is in 3 mbufs 100 bytes each...

so I have

M1(addr)->M2(control)->M3(data1)->M4(data2)->M5(data3)+flag MSG_EOR

                          ^
                          |
                       This mbuf will have a pkt_hdr on it.. (where I 
can
                        stash a pointer if I so want)

If I do a read(100 bytes)

I will then have left

M4(data2)->M5(data3)+flagMSG_EOR

At this point there is no back record to anything... no address ..

nothing..

The issue does not apply to UDP sockets since they are

PR_ATOMIC... i.e. if you read the 1st 100 bytes you LOSE the
rest of the message..

But our sockets api specifies we don't do that.. you
get the rest...

Which means the next read WILL NOT have an address or
control.. you have to pay attention to the flag MSG_EOR
to be sure you got it all..

So if the app above did

rcvmsg(...100 bytes...)
followed by
sctp_peeloff(...)

of that assoc.. the other 200 bytes would have no
way to know they needed moving to the new socket...

I know this is a pathological case... the app really should
not do thsi..but this is the case I am concerned with.. UNLESS EVERY
MBUF (SKBUF or whatever you call it  in your O/S) has a pointer
back to the assoc.. you can NOT assure that this won't occur.. MP safe
or not..

R

---

From: Qiaobing Xie <[email-masked]>
Date: Thu, 03 Oct 2002 12:45:25 -0500

Randall Stewart wrote:
>                        This mbuf will have a pkt_hdr on it.. (where I can

This might be the root of the problem... we may be trying to do too much
for the app. Imagine if we do the same PR_ATOMIC with SCTP socket (i.e.,
either drop the rest of the msg with a warning "data read incomplete",
or reject the read with an error asking the app to re-try with a bigger
buffer), the sctp_peeloff() problem would go away.

-Q