Subject: Re: [tsvwg] draft-stewart-tsvwg-sctp-ndata-03.txt

From: Michael Tuexen <[email-masked]>
Date: Fri, 4 Jul 2014 22:42:15 +0200

On 05 Nov 2013, at 00:51, Karen E. Egede Nielsen <[email-masked]> wrote:

> Hi,
>  
> Please accept the folowing comments to draft-stewart-tsvwg-sctp-ndata-03.txt.
> Please use as fit.
Hi Karen,

thank you very much for your comments. I'm sorry for not responding earlier...
See my comments in-line.

Best regards
Michael
>  
> General comments:
> -----------------------
>  
> The draft draft-stewart-tsvwg-sctp-ndata-03.txt, in my opinion, addresses an important limitation of SCTP message fragmentation The
> limitation is important to have removed in order to support large message transfer by SCTP without detroying concurrent transmission of
> data with real-time characteristics.
>  
> The advanced stream scheduling modes, suggested in the draft, also seems very instrumental in order to instantiate different real time characteristics on
> different streams.
>  
> Title:  Given the scheduling aspects included in the draft, one should consider to adjust the title so that it better accommodate for the scheduling parts. Actually stream scheduling seems to be main topic of the draft with the new (or extended) data chunk format just a mean to support more flexible (interleaved) scheduling of fragments.
The is correct.
>  
> Something else:  The “N” as “New” in N-DATA seems not so future-proof. Perhaps E for word “Extended” (E-DATA) would be a better naming ?
Possibly. But we are using N-DATA in several places... Will think about it.
>  
> It should be clarified in the Introduction that the draft makes use of the Supported Extension Parameter as defined in RFC5061.
Make sense. Done.
>  
> Detailed Comments:
> ---------------------------
>  
> Section 1.1:
> 1.       It should be clarified that the draft defines a new data chunk option as well as that it defines an advanced stream scheduler function and that these are independent functions.
Correct. I agree.
> 2.       It could be clarified that the intention is for the new data chunk to be used for all data transmitted on a particular association (i.e., not only fragments).
Done.
> 3.       It could be clarified that the usage of the new data chunk will differentiable on a per association basis and that an SCTP implementation as such shall support coexistence of associations which use the existing data chunk format and associations which use the new data chunk format.
Sure. Done.
>  
> Section 3.1:
> 1.       In the third paragraph that starts with “Sender side usage.. ”, it should be clarified that the new data chunk is used for all data chunks and not for only data chunks that need fragmentation.
Fixed.
> 2.       The situation, when the peer does not support N-DATA but the user request it, should preferably be elaborated on in this section or somewhere else in the draft. Presumably then in this situation the SCTP layer simply uses the legacy data chunk with its in-build limitations. However this should be described.
Fixed.
> 3.       Information should preferably be provided to UL about whether peer supports usage of N-DATA. For a socket api compliant implementation such information could possibly be provided in an extension of SCTP_STATUS ?
I would prefer to signal it in the SCTP_ASSOC_CHANGE indicating SCTP_COMM_UP or SCTP_RESTART
by using SCTP_SUPPORTS_INTERLEAVING similar to SCTP_ASSOC_SUPPORTS_PR, SCTP_ASSOC_SUPPORTS_AUTH, ...
However, this goes into the socket API section.
> 
> Section 4.1 – minor/editorial:
>  
> 1.       The text could be clarified so that it more clearly refers to “one-to-many” and to 
> “one-to-one” sockets. The text could also better describe the behaviour of the socket option on each type.
Could you be more specific where information is missing?
> 2.       The second paragraph sentence: “An N-DATA chunk aware application should also set the fragment interleave level to 2” – should refer to appropriate option of RFC6458.

>  
> Section 4.1 – substance:
>  
> 1.       I would like to press for that the ability to change the N-DATA setting on current (established) associations is removed. It looks to open up for a number of complexities both in SCTP layer as well as in O&M layer;  E.g.,  if N-DATA feature is disabled, what about retransmissions where original was send as N-DATA. When the O&M operation is made, then how to control when it takes effect in the SCTP layer in respect to data already in SND buffer. Further the disabling only effectively can control the sending side. These details would have to be described in draft and, secondly, implemented, both of which could be somewhat of a challenge. Are there some important use-cases that demands for this “flexibility” ?
This functionality isn't there. I agree completely with you. Isn't the text in 4.1.1 on
SCTP_NDATA_ENABLE clear on this?
> 2.       I believe that one NEED to have the possibility, on a per association basis, to read whether the peer supports N-DATA or not. This regardless of whether it be an association belonging
Isn't the notification good enough? You can't query ASCONF support, Stream Reset, and so on.
This is currently only done via the notification. I'm open to discuss a generic method...
> to a one-to-many socket or a one-to-one socket. Support for read via socket api should be described in the draft. This may be provided via an extension of SCTP_STATUS (?). Optionally one may also prescribe for such information to be returned as part of an extension of RFC3873, sctpAssocTable object.
Let's talk about MIB stuff in general in a separate document. We are lacking MIB support for
all SCTP extensions. If there is a real need in this, we should think about updating the
MIB RFC. Years ago, when I implemented it for FreeBSD, I had a bunch of comments.
> 3.       Situation with association restart on SCTP level should preferably be described.
You get a new notification...
>  
> Section 4.1.2:
>  
>  
> 1.       Would it be reasonable to specify a subset of the listed scheduling options as something that an implementation that support N-DATA MUST support and others as MAY or optional.  Alternatively one may specify that at least one of the options need to be supported ?
I would just look which constants are defined... We can make a statement that not all implementation
support all.
> 2.       The SCTP_SS_FAIR_BANDWIDTH could be demanding from an implementation perspective.
As will be the SCTP_SS_WEIGHTED_FAIR_QUEUEING...
> 3.       It should be clarified that these scheduling options are applicable for sending normal DATA chunks also when peer does not support usage of N-DATA.
Yepp. I'll work on this while polishing/testing the implementation..
> 
>  
> Possible addition to the draft content: (?)
>  
> Possibly one could contemplate to include improvement of the receiver side operation also. As follows:
>  
> ·         The SCTP receive operation could be extended to allow for specification of read stream scheduling  - possibly options – to start with (RS: Read Scheduling):
> o   SCTP_RS_FIRST_COME: The simple first-come, first-serve algorithm
> is selected by using this value.  It just queues messages that are readable in the order in which they have been received by the SCTP layer.
> o   SCTP_RS_PRIORITY:  SCTP operates with a number of receive queues per association according to a number of priorities set. For each priority queue simple first-come, first-serve algorithm is used .I.e., for each priority queue messages are queued as readable in the order in which they have been received by the SCTP layer .The priority can be assigned on a per stream basis. Value 0 is default priority, higher values are higher priority (or lower), lower values are lower priority (or higher).
> Select, poll (BSD) and epoll (Linux) mechanism must be extended to support specification, and notification, of receiving readable event on a per priority RCV queue basis.
As we discussed, I think this could go into a separate document...
> 
>  
> Editorial:
>  
> Abstract: which which à which
> Section 2: FSN's are unsigned number à FSN's are unsigned numbers
Fixed.
> BR, Karen

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 5 Jul 2014 00:20:21 -0600

Michael,

I had some interest in this draft after having identified and
discussed this issue so many years ago.  I have read this version
of your draft and have some comments:

| 1.  Introduction
| 
| 1.1.  Overview
| 
|    When SCTP [RFC4960] was initially designed it was mainly envisioned
|    for transport of small signaling messages.  Late in the design stage
|    it was decided to add support for fragmentation and reassembly of
|    larger messages with the thought that someday Session Initiation
|    Protocol (SIP) [RFC3261] style signaling messages may also need to
|    use SCTP and a single MTU sized message would be too small.
|    Unfortunately this design decision, though valid at the time, did not
|    account for other applications which might send very large messages
|    over SCTP.  When such large messages are now sent over SCTP a form of
|    sender side head of line blocking becomes created within the
|    protocol.  This head of line blocking is caused by the use of the
|    Transmission Sequence Number (TSN) for two different purposes:
| 
|    1.  As an identifier for DATA chunks to provide a reliable transfer.
| 
|    2.  As an identifier for the sequence of fragments to allow
|        reassembly.
| 
|    The protocol requires all fragments of a user message to have
|    consecutive TSNs.  Therefore the sender can not interleave different
|    messages.

I think that is a quite fictional rewrite of history.  These issues
were addressed long before RFC 3960 issued and some even vocally
requested the postponement of the RFC until related issues could be
resolved (no, not me, that was a different issue).

|    The protocol requires all fragments of a user message to have
|    consecutive TSNs.  Therefore the sender can not interleave different
|    messages.

The first statement is correct; the inference is false.  SCTP places
no total ordering constraint on transmission of data chunks.

I will say that again: SCTP places no total ordering constraint on
transmission of data chunks.   Multi-homing would be impractical if
it did: it would be impractical in a modern operating system for a
sender at the transport level to ensure that a TSN was not emitted
from one interface before another.  It would be pointless anyway
given that the underlying network gives no guarantees of total
ordering.

Therefore, given that there is no total ordering constraint imposed
by SCTP on the transmission of TSNs, the fact that TSNs are assigned
in numerical sequence for a message does not prohibit a sender from
interleaving data chunks in transmission.  Therefore, there is no
head of line blocking.

So, for example,  As a sender, I get an large ordered message that I
chunk up into 2000 data chunks.  I assign the data chunks TSNs 1
through 2000 and start sending the first few, say 1, 2, 3, until I
use up my cwnd.  The rest I queue for transmission in a normal
queue.  I then get a small unordered message.  I want to expedite
this message (because SIGTRAN uses unordered messages for expedited
requests).  I assign it TSN 2001 and queue it in an expedited queue
for transmission.  When the cwnd opens I send TSNs 2001, 4, 5, 6.
SCTP does not prohibit me from doing this.  Yes, I will likely get a
gap ack, but so what?  I know I didn't send the messages in the gap
yet.  Also, creating a situation that could happen at any moment
anyway cannot be harmful.

Interleaving of transmission was never the issue.

The issue was with the interface between SCTP and the ULP and the
needs of the ULP with regard to message composition.  For the sender
to do what I just described, it must be able to sequentially assign
all of the large message's TSNs before it can assign a TSN to the
small message.  To assign TSNs to the large message, it needs to
know the total size of the message so it can calculate chunk sizes
and determine the total number of data chunks (and TSNs) required.
The problem with this is that it places the onus on the ULP to know
the total message size before passing any of it to SCTP, requiring
it to compose the entire message before writing the first byte.
This, of course, hampers stream-based compression, encoding schemes,
block cypher and a whole bunch of useful stuff that might want to
put the data into a big SCTP message.  However, the most direct way
around that is to fragment the stream in the ULP and wrap it with an
application header giving information like the byte offset within
the stream.  We discussed this concerning RDDP back in 2000.

My annoyance with having to know the total message size was for
directly using a standard Sockets (sendmsg) or STREAMS interface
(XTI) as an interface to SCTP where the caller just tells the
transport that there is "MORE" data making up the record that will
be presented later: it precluded me from sending the first part
until the later parts arrived and potentially blocking up the write
buffer.  No other transport had this problem.

What I wanted to see happen, and I discussed this back in 2000 and
went over it in detail with you and Randal at the 2006 interop was
to relax the sequential TSN assignment to simply an ordered TSN
assignment.  I will use the example again:

As a sender, I get the beginning of a large unordered message that
I chunk up into 4 data chunks.  I add these 4 data chunks to the
send queue for the stream.  I perform a transmission when cwnd
becomes available and assign TSNs 1, 2 and 3, and send the first 3
chunks in a packet and use up my cwnd.   I then get the small
unordered message which I chunk into one chunk and place on the
expedited queue.  I then get another portion of the large unordered
message which I chunk up into 4 more data chunks.  My cwnd opens
and I assign TSN 4 to the expedited chunk, and TSNs 5, 6 and 7 to
chunks from the large message.

Under the relaxed constraint on assigning TSNs, I can easily do what
is necessary and even avoid unnecessary gap acks.

The relaxed constraint does not require major modifications at the
receiver: the receiver cannot deliver the large message past a
missing TSN in either case.  Besides, a receiver implementation
should not attempt to enforce sender requirements, nor unnecessarily
rely heavily on sender behaviour.  When all of the TSNs are in
(between B and E), there is always sufficient information to
reassemble messages regardless of whether the sender interleaves
TSNs or not.

The position where HOL blocking could be reintroduced is where a
number of large messages with interleaved TSNs are being transferred
and one TSN in the middle of theme is lost, blocking them all until
it can be recovered (under some APIs).  This is not due to sequential
TSN assignment, but comes instead from relaxing the constraint and
deficiencies of the receiver API.

Your frame sequence number could assist the receiver in determining
in the event of a missing TSN, not to which message the missing TSN
belongs, but to which it does not belong: allowing the complete parts
of the other messages to be released.  But because of that, I don't
see why the FSN cannot simply be added to the DATA chunk header and
let the receiver that doesn't understand it ignore it.

| 3.2.  Receiver Side Considerations
| 
|    Upon reception of an SCTP packet containing a N-DATA chunk if the
|    message needs to be reassembled, then the receiver MUST use the FSN
|    for reassembly of the message and not the TSN.  Note that a non-
|    fragmented messages is indicated by the fact that both the 'E' and
|    'B' bits are set.  An ordered or unordered fragmented message is thus
|    identified with any message not having both bits set.

There is no reason to require the receiver to not use the TSN.  If all
TSNs have been received, the FSN is not even useful.  If the
receiver ignores the FSN even when a TSN is missing, it simply might
not deliver messages to its ULP as quickly as it might.  But there
are other considerations on delivery of messages to the ULP and
reassembly is not even necessary until the implementation goes to
deliver a message.  Hey, some API might even reassemble two of three
scatter-gather buffers and deliver the two before the missing one
even arrives.  I think it is an implementation decision.

I think this should say "may use the FSN in reassembly of messages".

| 3.1.  Sender Side Considerations
| 
|    A sender MUST NOT use the N-DATA chunk unless the user has requested
|    that use via the socket API (see Section 4).  This constraint is made

Please don't tell me I MUST use an INFORMATIONAL section referring
to an INFORMATIONAL RFC for anything.  End the sentence after "that
use".

I would consider that a ULP writing a message with the "MORE" bit
set is requesting the service, or that is simply using an API where
it can set a "MORE" bit as requesting the service, where "setting
a MORE bit" means opening any API where the ULP can send the first
portion of a message without specifying the total length of the
message.

| 4.  Socket API Considerations
| 
|    This section describes how the socket API defined in [RFC6458] is
|    extended to allow applications to use the extension described in this
|    document.
| 
|    Please note that this section is informational only.

Hey, can you put all this non-standard INFORMATIONAL stuff in a
different document?  It is much longer than the NORMATIVE part and
really doesn't contribute anything to the document.

Well, that's enough for now.  I'm sure some flurry will result from
these comments anyway so we can discuss more up the ladder.

I have one question (just to check that you actually read this far):
what caused you to try to address this issue now, 14 years after the
fact?

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 7 Jul 2014 21:46:31 +0200

> of your draft and have some comments:
Hi Brian,

thanks for reading it and providing feedback.
See my comments in-line.

Best regards
Michael
> resolved (no, not me, that was a different issue).
As far as I remember, the focus was on support for small message, not
for large ones. The limitation was know at the time of RFC 2960, but
the decision at that time was to live with the limitation... For
SIGTRAN is was not a real issue.
> no total ordering constraint on transmission of data chunks.
Formally you are right in the sense that SCTP will work as long
no matter in which sequence the DATA chunks will arrive at the
receiver. They can be reordered by the network or even be sent
by the sender in any order. This can't be distinguished by the
receiver. However:
* You can't assign a TSN for a DATA chunk of message before
  knowing how many TSNs you need for the messages already started
  to send (and assigned a TSN).
* The receiver will report TSNs not received in order, so you
  will get lot's of gap reports which might or might not trigger
  fast retransmissions and therefore having an impact on performance.
> interleaving data chunks in transmission.  Therefore, there is no
Really? Think about sending a 1 GB message on stream 0, having transferred
100MB of it. Now you send a 10 byte message on stream 1. What if you
don't know the size of the big messages yet (using the explicit EOR mode)?
> through 2000 and start sending the first few, say 1, 2, 3, until I
So you do early TSN assignment. This means that if the MTU decreases
during sending TSN up to 2000, you need to rely on IP-fragmentation.
This also only works, if you know how many chunks you need in total.
This does not work if you use the explicit EOR mode, so you can't
send messages larger than your send buffer.
> anyway cannot be harmful.
It depends on harmful. You are right, all messages will be delivered
within the ordering constraints of the sender. However, there can
be performance penalties if you can't differentiate anymore between
never send or probably lost.
> and determine the total number of data chunks (and TSNs) required.
It also needs the PMTU, which can change over time...
> it to compose the entire message before writing the first byte.
... and the assumption that the MTU doesn't change while sending
the message.
> put the data into a big SCTP message.  However, the most direct way
Not sure. If you run DTLS over SCTP, for example, the message size
limited anyway to about 16KB. However, RTCWeb uses SCTP over DTLS and
therefore the "upper layer" is the JS layer.
> assignment.  I will use the example again:
I have to say that I can't remember. Sorry. So we need to go through
it again...
> is necessary and even avoid unnecessary gap acks.
You are focussing on the "handle small unordered as expedited" issue.
The problem addressed here is the generic one:
You might even have a large message on stream 0, and a higher priority
smaller message, which still needs fragmentation, on a different stream.

I agree that you probably can come up with a solution for sending
smaller unordered messages "within" larger ordered ones, but this
doesn't solve the general problem. I think the base problem is that
the TSN is used for two things. Decoupling these will get clean
solution.
> let the receiver that doesn't understand it ignore it.
How can you add something to the DATA chunk header and get a receiver
to ignore it in case it doesn't support the extension?
> I think this should say "may use the FSN in reassembly of messages".
I'm not sure what you are arguing here... A receiver supporting the N-DATA
chunk but not using the FSN for reassembly? So you want to reuse the
old code, but use the new packet format?
> to an INFORMATIONAL RFC for anything.  End the sentence after "that
No, this is not intended. Good point. Changed to

<t>A sender MUST NOT use the N-DATA chunk unless the user
has requested that use (e.g. via the socket API, see <xref target='api'/>).

It is the upper layer which has to say "use interleaving" since it also
must handle the higher interleaving level on the receiving side. See
http://tools.ietf.org/html/rfc6458#section-8.1.20
> portion of a message without specifying the total length of the
This is possible with the socket API:
http://tools.ietf.org/html/rfc6458#section-8.1.26
> really doesn't contribute anything to the document.
After RFC 6458 was published, we started to added a socket API section
to the documents describing extensions, which describe how you could
implement an API for the extension based on the socket API. I wasn't
to bad to have both in one document. However, the API section is always
purely informational (similar to http://tools.ietf.org/html/rfc4960#section-10),
so if you don't implement the socket API, you don't need to care. If
you implement it, you can even do it differently. However, up to now,
the SCTP Linux Kernel implementation, the Solaris implementation and
the FreeBSD implementation use this API.
> these comments anyway so we can discuss more up the ladder.
Sure. I know that the document is not done yet and I hope that
we can improve it based on the discussion on the list.
> I have one question (just to check that you actually read this far):
So I passed that test, at least...
> what caused you to try to address this issue now, 14 years after the
RTCWeb is using SCTP for potentially large messages and needs support
for interleaving.