Subject: Re: [Tsvwg] A SCTP sockets/sctp_peeloff() issue to think about

From: Kavitha Baratakke <[email-masked]>
Date: Thu, 03 Oct 2002 13:02:04 -0500

Message: 10
Date: Wed, 02 Oct 2002 19:45:25 -0500
From: Randall Stewart <[email-masked]>
To: venkat venkatsubra <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: [Tsvwg] A SCTP sockets/sctp_peeloff() issue to think about

venkat venkatsubra wrote:
> Randall,
> 
> Randall Stewart wrote:
> 
> 
>>venkat venkatsubra wrote:
>>
>>>We don't have this bug in AIX.
>>
>>It is not really a bug but an unspecified behaviour that
>>needs to be put down ..(in the sockets api draft)
> 
> 
> If a server application for example on
> getting a new association notification
> does a sctp_peeloff() and forks off a child
> to handle the new association, and the child
> process (which deals with only the peeled off socket)
> is not guaranteed to get the data sent over that association ,
> then it sounds like a bug to me.
> 

You still miss my point .. the behavior is NOT specified and
needs fixing in the document..

Isn't it logical however that when the user does a peeloff he must get
the all the data belonging to the peeled off association. Is it
something that needs to be explicitly documented??? 

>> >                                 We move the
>>
>>>data as well as any pending notifications
>>>(the peer could have sent some data and closed
>>>the connection) to the peeled off socket. It is MP
>>>safe too when new data (or any other chunk type)
>>>comes to this association while the peeloff related
>>>work is going on.
>>>
>>
>>A question for you then.. does your socket's implementation
>>have a single socket buffer for the upper layer to read
>>from?
> 
> 
> Yes, we have a single socket.
> 
> 
>>If so, how do you move the data without doing a layer violation
>>and plunging through the data in the socket buffer (grubbing has
>>La Monte frased it :>)?
> 
> 
> We do go into the socket buffer to move the data,
> notification, etc. to the new socket.  Also,  on a MP system
> while this is going on,  a new packet could hit the original socket
> when a pcb lookup is done. We take care of that case too by
> directing it to the new socket after the peeloff work is completed.
> 
>So how do you tell which association goes with which data... do you
>keep back pointers and such? If you use a mbuf like structure do
>you have EVERY chained mbuf assured of having this information..

>It is also possible to read 100 bytes (of a 1000 byte message) and then
>do the peel off... in the case of a BSD system this may well loose the
>M_PKTHDR (which is where any back pointers could possibly be kept) and
>thus you would have data left on the socket with no way to tell
>which association went with that data for the partially read message...

*** Which association goes with which data... we anyway need this
information
to deliver the data to the application even without peeloff don't we...
We have a MT_CONTROL mbuf before every data packet that we use to store
the
assoc information and such.

> ...
> 
>>Well I had not considered the sctp_peeloff() as part of the sockets
>>layer... but even in that vein then how does the sockets code
>>sctp_peeloff() know which mbufs in queue (I will use bsd terms here)
>>are from which association..
> 
> 
> I thought for each piece of incoming data being read by the app, there
> is always going to be some sort of an association id stored alongside
> with the data all the way up to the app. Otherwise, after a read on the
> fd, how will the app tell which association this piece of data belongs?
> 

well it is more complicated than that.. how BSD land does this (and I
think other O/S's that may be rooted in a BSD sockets implementation)
is this

When you dump a message onto a rcv socket buffer you dosomething like
sbappendaddr(....) the sockets layer copys the address into
a mbuf and puts it at the head of the mbuf chain to deliver
followed by any ancillary data followed by the data itself

These are all linked through the
m_next pointer

The next packet is done via the
n_nxtpkt pointer...(something like that may have the name off )

Now the data can be scatterd over multiple mbufs all chained together..

For grins lets say it is in 3 mbufs 100 bytes each...

so I have

M1(addr)->M2(control)->M3(data1)->M4(data2)->M5(data3)+flag MSG_EOR

                          ^
                          |
                       This mbuf will have a pkt_hdr on it.. (where I 
can
                        stash a pointer if I so want)

If I do a read(100 bytes)

I will then have left

M4(data2)->M5(data3)+flagMSG_EOR

At this point there is no back record to anything... no address ..

nothing..

*** Randall, all you want to do in this case is (we do this as well) 
M1(addr)->M2(control)->M3(data1)->M4(data2)->M5(data3)+flag MSG_EOR

This is the partial delivery case and you can catch it and handle it as
below.
Just make a copy of the control mbuf (and addr mbufs) and stick in into
the receive buffer
before M4 as below
M1(addr)->M2(control)->M4(data2)->M5(data3)+flag MSG_EOR

It is quite easy to handle in the BSD code actually... I don't know
about other implementations though...

The issue does not apply to UDP sockets since they are

PR_ATOMIC... i.e. if you read the 1st 100 bytes you LOSE the
rest of the message..

But our sockets api specifies we don't do that.. you
get the rest...

Which means the next read WILL NOT have an address or
control.. you have to pay attention to the flag MSG_EOR
to be sure you got it all..

So if the app above did

rcvmsg(...100 bytes...)
followed by
sctp_peeloff(...)

of that assoc.. the other 200 bytes would have no
way to know they needed moving to the new socket...

I know this is a pathological case... the app really should
not do thsi..but this is the case I am concerned with.. UNLESS EVERY
MBUF (SKBUF or whatever you call it  in your O/S) has a pointer
back to the assoc.. you can NOT assure that this won't occur.. MP safe
or not..

*** This would be handled just fine with the above solution. If you
needed to move
data to the new socket you would still have all the information for it

-- 
Thanks, 
Kavitha
_________________________________________________________________

Ms. Kavitha V. Baratakke     IBM Corporation
Office : (512)-838-1226      AIX TCP/IP Kernel Development & Support
T/L : 678-1226               11501, Burnet Road, Austin TX 78758
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 3 Oct 2002 14:51:22 -0600

Is it not simple to just require that the peel_off occur
before any reads on the association being peeled off?

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Kavitha Baratakke <[email-masked]>
Date: Thu, 03 Oct 2002 16:22:56 -0500

Brian,

That was just one case where we do a partial read before a peeloff...
Randy's example of where peel_off occurs after a partial read (partial
delivery) and we need to move the rest of the data to the new socket
buffer...

... sure we could make it mandatory that peeloff occurs before any
reads...although handling it otherwise is really not that complicated...

But the original source of this discussion was that if data is lying in
the socket buffer for the assocation that is peeled off, we still have
to move it to the new socket buffer after peeloff...

Kavitha

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 3 Oct 2002 15:41:58 -0600

Kavitha,

Socket buffer?  Isn't that an implementation issue?  I thought
that implementation issues were moved to another list.

--brian

---

From: Kavitha Baratakke <[email-masked]>
Date: Thu, 03 Oct 2002 16:51:51 -0500

Brian,

The discussion is about an implementation issue... but it was
being discussed on tsvwg ... not sure why :) Just contributing to
the already started discussion...

Kavitha

---

From: Randall Stewart <[email-masked]>
Date: Thu, 03 Oct 2002 17:01:28 -0500

> delivery) and we need to move the rest of the data to the new socket

Kavitha: I was NOT talking about the partial delivery api.. that is
different than when a user does a partial read...

The partial delivery api is when the sctp stack itself sends up
only part of the message .. that is different than the socket buffer
holding the whole message and the app reading only a couple of bytes...

> reads...although handling it otherwise is really not that complicated...

As long as you don't mind having sctp specifics in your sockets
layer... of course Kacheong feels it is part of the sockets layer
so then there is no problem :>

> to move it to the new socket buffer after peeloff...

correct.

and I think that should be stated in the socket draft at minimum.

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: Kavitha Baratakke <[email-masked]>
Date: Thu, 03 Oct 2002 17:35:04 -0500

> > reads...although handling it otherwise is really not that complicated...
Randall, how is partial delivery api different from user doing a partial
read as far as how we need to handle it goes. We still need to keep
track of which assoc the partial message belongs to in either case
right?

I'm confused ???

---

From: Randall Stewart <[email-masked]>
Date: Thu, 03 Oct 2002 19:12:15 -0500

Kavitha:

I think we now leave the "making something explcit in a draft"
and move in to implementation details for me to discuss
this.. I am copying sctp-impl and will respond
removing tsvwg...

I think it is best we move it there..

If there are no further objections next pass of the sockets
api will specify we must be sure to move the data..

I will answer you tommorrow A.M. and see if I
can't confuse you more :>

R

---

From: Randall Stewart <[email-masked]>
Date: Thu, 03 Oct 2002 19:14:10 -0500

Kavitha:

As I said in my previous response..

I have been talking about text changes to the socketapi draft ..yes
we have gone into the implementation side to illustrate the issue.

I think everyone will agree some wording in peeloff warning folks
to move the data is not a bad thing...

issues on drafts (at least tsvwg doc's) still need to be here

R