Subject: [Tsvwg] A SCTP sockets/sctp_peeloff() issue to think about

From: Randall Stewart <[email-masked]>
Date: Wed, 02 Oct 2002 09:10:56 -0500

Hi all:

I just had an interesting thought..

When I do a

sctp_peeloff(...)

what happens to data that has not been read for the association being 
peeled off?

In most implementations you stuff the data up a socket buffer... with
no way to reach up and grub the data out of the socket buffer.. not
that you would want to do this anyway since this is an EXTREME layer
violation and leads to spagetti code.....

If I do nothing and data is pending on the socket for the
association being pulled off I have an issue that I may deliver data
out of order... depending on which socket is read first of course...

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Jon Grimm <[email-masked]>
Date: Wed, 02 Oct 2002 10:38:31 -0500

> out of order... depending on which socket is read first of course...

Yes.  This is a problem.  LKSCTP currently has this feature/bug too,
but we can fix it by grubbing through the socket read queue (the socket
buffer is  really just a list of buffers) under locking.

Our notifications also sit in the socket read queue too, so those
too may need moved to the peeled off socket.

BTW, we must use the socket's read queue to make poll/select work.

We'll have to take a quite painful lock, but the alternative
as described above is not very acceptable to the application.

Just to put in an implementor's 2 cents,
Jon

---

From: Randall Stewart <[email-masked]>
Date: Wed, 02 Oct 2002 11:00:13 -0500

> buffer is  really just a list of buffers) under locking.

This is a very bad idea I think..a lower layer grubbing through a higher
kernel layers data structure is ALWAYS a bad idea and leads to spagetti
or worse.. badly broke when the upper layer (sockets in this case)
changes something and just happens not to tell you or change
the SCTP code.. after all this is its private domain...

> too may need moved to the peeled off socket.
true..

> BTW, we must use the socket's read queue to make poll/select work.
Of course...

> as described above is not very acceptable to the application.

Well one could always run a tracking queue to find out when
things were read.. I know BSD allows you to get a notification
when something is read ... it even tells you if it was the
End of a Message... so one could

allocate a small ds that has two pointers... one
to make a complete linked list of all elements..

have the head in the endpoint info

Then the other list head could be on the association. This would
be the second pointer that would point to the next element to be read
for this association...

Then every time a read occurs that ends a record pull the top
one off the list and unlink it..

When a guy does a peeloff you check the assoc being peeled off and
if it has something on its list.. mark a flag (not to put data up
the socket buffer.. hold it) and then return an error... i.e.
EDATA_TO_READ

Has the guy process off the data and reads it clear.. you would
be able to see the flag set on the asoc.. and then when the last
data is clear from the asoc.. you could send a notification...

I know this is a bit detailed with implementation stuff.... but
basically the concept for the user would be:

1) when doing a pelloff() it may fail with EDATA_TO_READ
2) if this occurs you must then read data until
    a: socket buffer has nothing in it..
    or
    b: you get a notify telling you its ok to peeloff
3) Then you peel it off... and of course when you do that
    any held data would need to be sent up the new socket descriptor.

We could even put some clause in the sctpsockets draft that states ..
if the implementation as the ability to move the data to
the new socket (some may have this capability) it can do that
and not implement the EDATA_TO_READ error/notification.. but
application programmers should not depend on this and should
code for EDATA_TO_READ...

What ya think?

R

---

From: Jon Grimm <[email-masked]>
Date: Wed, 02 Oct 2002 14:15:51 -0500

> the SCTP code.. after all this is its private domain...

I'm already in the SCTP specific portions of code at this point, so
Its not too painful really... I have an assoc_id that I'm comparing
against.  I just have to hold off the lower layer for a bit
while we walk the list.

> be the second pointer that would point to the next element to be read

An up/down counter should work too.  It doesn't look like I'd
care which buffer is in use, just that the upper layer is no longer
holding any of the association's buffers/notifications.

I too have a callback I can register, so the association can
do any needed accounting (such as this).

> code for EDATA_TO_READ...

Seems to work, though you should probably poll out on sctp-impl
too.

Maybe just EBUSY instead of a new error though?

I'm assuming that the proposed notification would have the associd
that is ready for peeloff?  In that way, more than one peeloff can be 
pending.

Best Regards,
Jon Grimm