Subject: Re: [Tsvwg] SCTP support for Solaris

From: Michael Tüxen <[email-masked]>
Date: Mon, 24 Nov 2008 20:40:36 +0100

Hi Kacheong,

I'm adding TSVWG (since the chairs want to see the discussion) and
the coauthors...

Best regards
Michael

On Nov 24, 2008, at 6:02 PM, Kacheong Poon wrote:

> Michael Tüxen wrote:
>
>>> The old one
>>>
>>>  struct sctp_sndrcvinfo {
>>>     uint16_t sinfo_stream;
>>>     uint16_t sinfo_ssn;
>>>     uint16_t sinfo_flags;
>>>     uint32_t sinfo_ppid;
>>>     uint32_t sinfo_context;
>>>     uint32_t sinfo_timetolive;
>>>     uint32_t sinfo_tsn;
>>>     uint32_t sinfo_cumtsn;
>>>     sctp_assoc_t sinfo_assoc_id;
>>>  };
>>>
>>> The new one
>>>
>>>  struct sctp_sndrcvinfo {
>>>    uint16_t sinfo_stream;
>>>    uint16_t sinfo_ssn;
>>>    uint16_t sinfo_flags;
>>>    uint16_t sinfo_pr_policy;
>>>    uint32_t sinfo_ppid;
>>>    uint32_t sinfo_context;
>>>    uint32_t sinfo_pr_value;
>>>    uint32_t sinfo_tsn;
>>>    uint32_t sinfo_cumtsn;
>>>    sctp_assoc_t sinfo_assoc_id;
>>>  };
>> So a 16 bit field sinfo_pr_policy was added after  16 bit fields.
>> If I remember that correctly, that was OK, since you had a padding
>> there to get 32-bit alignment? So actually these 16 bit were there,
>> but not used... The sinfo_pr_value is just a new name for  
>> sinfo_timetolive.
>
>
> I think the unfortunate thing is that the above assumption
> may not be true universally.  How a compiler packs a struct
> probably depends on the platform.  And suppose it is OK such
> that it won't affect the position of other fields.  For old
> programs, the value of sinfo_pr_policy may not be defined.
I agree, but as I said: This is (from my understanding) the
result of the discussion in Kyoto.
>
>
>
>> Is this not OK for you? What you you suggest?
>
>
> Let's think more about it.  What are the other policies?
FreeBSD and MacOS X support

#define SCTP_PR_SCTP_TTL  0x0001/* Time based PR-SCTP */
#define SCTP_PR_SCTP_BUF  0x0002/* Buffer based PR-SCTP */
#define SCTP_PR_SCTP_RTX  0x0003/* Number of retransmissions based PR- 
SCTP */

>
> And maybe we need to think about whether having a fixed
> sctp_sndrcvinfo is the right way to move forward.  For
>
> example, suppose in future there is a new way to modify
> the sending behavior, what do we do to accommodate that?
> We cannot keep on modifying sctp_sndrcvinfo.
We don't... Especially if the ID becomes an RFC...
> -- 
> 						K. Poon.
> 						[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Tue, 25 Nov 2008 12:09:01 +0800

> We don't... Especially if the ID becomes an RFC...

This is the key.  It seems to me that instead of changing
sctp_sndrcvinfo, it is better to find another way to
handle this.  For example, I think Vlad asked about a
similar thing before, how to use a different key on
sending a message.  It is likely that there will be future
features which need to do the same thing.  Is it better
to have an extensible send/recv data structure?

-- 

						K. Poon.
						[email-masked]

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 24 Nov 2008 21:56:20 -0700

Kacheong,

Did you mean maybe a separate cmsg_type instead of packing
all rather unrelated options (mandatory, optional, implicit
and explicit) into a single oversized structure?  Maybe you
thought that cmsg structures were designed to be extensible?
Such that applications programmers might not have to fill
out PR-SCTP fields even on implementations that do not
provide PR-SCTP.  ;)

I suppose that is what you get when you allow protocol
designers to spec APIs: more work for application writers.

If a lesson were to be learned: break all the fields out of
the structure; make them all `int' (never use uintXX or long
or you get fiascos like socklen_t); and give them each their
own cmsg_type steering away from those already used by the
TCP, IP and SOCKET levels, and assign defaults, persistence
and user-specified sockopt settings for same so that there
is a value to be used when they are not supplied to sendmsg.
Never provide them on recvmsg unless they are of end-to-end
significance (i.e. their values were set (affected) by the
peer).  If they are not of end-to-end significance on receive,
read them with get_sockopt() instead.  If they are not of
end-to-end significance on sendmsg, simply set their values
to persist until changed by another presence in sendmsg or a
set_sockopt().  Most of these values only have to be set once
then (if at all) saving expensive transfer of ancilliary data
across the user/kernel interface (perhaps more than the data
itself).  The CMSG macros then also handle all alignment issues.

But I'm sure that would not fly with the authors of the
draft, regardless of whether it makes sense or not.

In short, I agree with you: adding especially this field in
the middle of the structure makes a poor design worse.

--brian

Kacheong Poon wrote:                       (Tue, 25 Nov 2008 12:09:01)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Kacheong Poon <[email-masked]>
Date: Tue, 25 Nov 2008 19:38:18 +0800

> out PR-SCTP fields even on implementations that do not

The above certainly is one way to do it.  IPv6 API uses
that extensively.

> or you get fiascos like socklen_t); 

Could you elaborate the issues with uintXX?  Using this
kind of construct, it is easy to support both 32 and 64 bits
apps concurrently.

> and give them each their

The SCTP_DEFAULT_SEND_PARAM provides this feature.

> read them with get_sockopt() instead.  

I guess the original sctp_sndrcvinfo more or less fits the
description above except for the sinfo_timetolive field.

> If they are not of

Any other suggestion?

-- 

						K. Poon.
						[email-masked]

---

From: Michael Tüxen <[email-masked]>
Date: Tue, 25 Nov 2008 15:50:01 +0100

> handle this.  For example, I think Vlad asked about a
This change was added from rev15 to rev 16. We are now
at rev 18...

I think we do need a way to support multiple PR-SCTP policies
and make that setting on a per message base.

When I integrated this into FreeBSD, I did it backwards compatible:
I use the lowest 8 bits of the sndrcv info flags for enumerating the
policies and leaves 24 bits for other things. Then I use the
timetolive field as the generic value with a semantic depending
on the policy.

I'm happy to put this in the ID (this is how the code currently
works in FreeBSD and Mac OS X anyway), if you agree to this.
Then we can stay with the structure from rev 15...
However, I think you objected against this when we discussed
it on Kyoto... But you agreed to the solution we have in the
ID since rev 16 but you disagree now...

I'm not sure how this discussion works... When I suggest to
improve the names of the structures Vlad (Linux implemenation),
Randy, Peter and myself (FreeBSD and Mac OS X implementation)
agree but you object. OK, so I rolled back the changes and we
keep the old names. Now you want to undo a change done between
revision 15 and 16 of the ID... Are you suggesting to
roll back anything we did in the past?

---

From: Michael Tüxen <[email-masked]>
Date: Tue, 25 Nov 2008 15:56:56 +0100

> out PR-SCTP fields even on implementations that do not
Most implementations support it (Linux, FreeBSD/Mac OS X, Solaris),
and it is much more convenient to provide a argument in the
sctp_sendmsg call compared to use the CMSG stuff... At least
this is what I think...
> designers to spec APIs: more work for application writers.
I would guess less... Most of the sctp_* functions are
for convenience. Other are for supporting multiple addresses.
> or you get fiascos like socklen_t); and give them each their
I do not understand that point.... If you do not specify the
length you have to do ugly things in the kernel depending
on your application is 32 bit or 64 bit or in the future something
different...
SO I think specifying the size of a field makes sense...
> itself).  The CMSG macros then also handle all alignment issues.
I do not understand what you are describing....

---

From: Kacheong Poon <[email-masked]>
Date: Tue, 25 Nov 2008 23:24:06 +0800

> This change was added from rev15 to rev 16. We are now

I'm sorry for not checking it out before.  I am raising
this now because I see an issue.

> and make that setting on a per message base.

I'm not saying that there is no need.  I am trying to
see how we can accommodate this.  And at the same time,
we can make future extensions using the same method.  For
example, is it worthwhile to use the same method to send
a message with different auth key?

> ID since rev 16 but you disagree now...

I have not thought about the uninitialized issue back
then.

> roll back anything we did in the past?

No.  I am raising an issue not just about sctp_sndrcvinfo.
I think this is an issue quite generic in using SCTP which
it is worthwhile to be extensible.

-- 

						K. Poon.
						[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Tue, 25 Nov 2008 23:46:10 +0800

> sctp_sendmsg call compared to use the CMSG stuff... At least

The way I understand Brian's point is the following.  There
are many uses in SCTP which require attaching different info
when sending a message.  Using the example in question, an
app wants to specify a PR policy when sending a message.
So the question to ask from an API perspective is how we
can associate this kind of info.  The sctp_sndrcvinfo is a
structure currently defined to associate a message with
certain info when sending/receiving.  But it is a fixed
structure.  So when an app wants to associate more info
when sending than sctp_sndrcvinfo can provide, what does it
need to do?

The way IPv6 makes this work is to use the cmsghdr structure.
It is the same way used in the current SCTP API draft.  It
is just that the draft only defines one such info for sending/
receiving, which is sctp_sndrcvinfo.  I believe what Brian
suggested was to define more types to be used for associating
different info, instead of putting everything in sctp_sndrcvinfo.

Note that the above does not conflict with the sctp_sendmsg()
call.  The associated info can be passed in as an argument.
The question is what that argument is such that it can contain
different info an app wants to associate with a message when
sending.  For example, if an app wants to send a message using
a specific stream (so sctp_sndrcvinfo is used) and at the same
time using a different auth key, how can it do that?  Right
now, the way to make it work is to use setsockopt() before
calling sctp_sendmsg().

As another example.  Suppose sctp_extrcvinfo is defined
without the sctp_sndrcvinfo field.  Using the above method,
we can have another SCTP level cmsghdr of type SCTP_EXTRCV.
When an app enables the SCTP_USE_EXT_RCVINFO option, the
stack will pass up two cmsghdr, one is SCTP_SNDRCV and the
other is SCTP_EXTRCV.  The second one contains the
sctp_extrcvinfo structure.

-- 

						K. Poon.
						[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Wed, 26 Nov 2008 00:08:56 +0800

Kacheong Poon wrote:

> call.  

I guess I need to be clearer.  Right now, sctp_sendmsg()
is basically a function which takes individual fields
defined in sctp_sndrcvinfo as separate arguments.  But
the list of arguments is static, meaning an app cannot
use it to associate more info when sending a message.
For example, an app cannot specify the PR policy when
calling sctp_sendmsg().  When I said "does not conflict",
I didn't mean that the method can be used with
sctp_sendmsg().  I just meant that sctp_sendmsg() can
still be used.  Having more cmsghdr SCTP level types does
not mean that sctp_sendmsg() can no longer be used.

> The associated info can be passed in as an argument.
> time using a different auth key, how can it do that?  

For clarity, note that I am not suggesting to modify
sctp_sendmsg() to take a new argument.

> Right

Note that I am not suggesting to re-define sctp_extrcvinfo.
I only tried to give another example of how cmsghdr can
be used.

-- 

						K. Poon.
						[email-masked]

---

From: Michael Tüxen <[email-masked]>
Date: Tue, 25 Nov 2008 21:03:42 +0100

> this now because I see an issue.
OK.
> a message with different auth key?
I agree that PR-SCTP and SCTP-AUTH are handled separately.
One came earlier, one came later and now we do not want to
change how everything works...
> I have not thought about the uninitialized issue back
OK, I see. Better now than never...
> it is worthwhile to be extensible.
I thought that each extension needs to describe how it
will be accessible via the socket API...

I think it is a bit later to redo the API...

One could think about providing additional ways
of make all the features available... but providing
the same thing via multiple methods is also not the
best...

---

From: Michael Tüxen <[email-masked]>
Date: Tue, 25 Nov 2008 21:13:02 +0100

> different info, instead of putting everything in sctp_sndrcvinfo.
So the idea is to be able to be able to provide a some cmghdr
structure, each one for
- The PPID
- The stream id
- Sending ordered/unordered
- the PR-SCTP policy
- the PR-SCTP value
- the SCTP-AUTH keyid
- the flags (from the sndrcv info flags)
- the context
- and a couple more that I forgot...

The experience I had is that people avoid using the the
cmghdr stuff... That is why we added sctp_sendmsg() which
was just a wrapper added to handle the cmghdr stuff...
> now, the way to make it work is to use setsockopt() before
That is correct. One solution would be to add the keyid
to sctp_sendmsg() (suggested by Vlad) but I think you
would object changing the signature of sctp_sendmsg()...
> sctp_extrcvinfo structure.

What is the list of cmsghdr structure you would like
to define?

---

From: Kacheong Poon <[email-masked]>
Date: Wed, 26 Nov 2008 19:28:41 +0800

> was just a wrapper added to handle the cmghdr stuff...

I guess the above was what Brian meant.  As in my reply
to Brian, sctp_sndrcvinfo provides the most basic SCTP
info.  So IMHO, it is fine as it stands (although there
is the timetolive or the re-defined pr_value which is
not really basic to SCTP).  We don't really need to
define new types for those fundamental things.  The
question is on newer stuff, such as PR and ASCONF.  They
are extensions to SCTP.

> would object changing the signature of sctp_sendmsg()...

As mentioned in my clarification email, I am not suggesting
to change sctp_sendmsg().  We cannot keep changing
sctp_sendmsg() when a new feature is added.

> What is the list of cmsghdr structure you would like

I think we need to first decide if this is the way we
want to move forward for associating SCTP info per
message.  I hope more people can comment on this.

-- 

						K. Poon.
						[email-masked]