Subject: [Tsvwg] SCTP PAD chunk

From: Mark Butler <[email-masked]>
Date: Fri, 07 Apr 2006 14:28:49 -0600

I have a comment on draft-tuexen-tsvwg-sctp-padding-00.  Shouldn't the 
draft for this state that this chunk is an exception to the rule that 
control chunks always precede data chunks?   Reordering chunks to pad 
seems rather pointless.

This chunk could be used to advantage in some environments (local 
clusters mostly) to align DATA payloads on 64 bit or potentially larger 
boundaries, in which case it would have to be able to be placed between 
DATA chunks as well.

 - Mark B.

---

From: Randall Stewart <[email-masked]>
Date: Fri, 07 Apr 2006 16:59:47 -0400

> control chunks always precede data chunks?   Reordering chunks to pad 

Well.. its actually a control chunk.. alibeit a fairly useless one :-D

The main purpose for it has nothing to do with data.. but instead
its for P-MTU discovery using (see the PMTU work of Matt Mathis)...

> boundaries, in which case it would have to be able to be placed between 

Well.. it was not really thought of for that...

R

-- 
Randall Stewart
ITD - Cisco Systems Inc.
803-345-0369 <or> 815-342-5222 (cell)

---

From: Mark Butler <[email-masked]>
Date: Fri, 07 Apr 2006 15:24:35 -0600

Randall Stewart wrote:

> Mark Butler wrote:
>
>> I have a comment on draft-tuexen-tsvwg-sctp-padding-00.  Shouldn't 
>> the draft for this state that this chunk is an exception to the rule 
>> that control chunks always precede data chunks?   Reordering chunks 
>> to pad seems rather pointless.
>
>

I am familiar with its application to PMTU discovery.  It just seems 
much more natural to put padding at the end of a packet instead of at 
the beginning.

There are incidental benefits to putting it at the end.  Padding at the 
end improves L2 cache performance by minimizing the amount of useless 
data read.  It saves memory when chunk groups are pre-prepared in memory 
prior to transmission.  It can save memory in some forms of receive 
processing as well.

>> clusters mostly) to align DATA payloads on 64 bit or potentially 
>> larger boundaries, in which case it would have to be able to be 
>> placed between DATA chunks as well.
>
>

I often hear the argument - "that is not what it was intended for".  Who 
cares?  The proper question should be "what is it good for?". 

Speaking of what things are good for - what is the point of the control 
chunk before data chunk rule anyway?  It has little or no advantages - 
the primary effect of the rule is to increase implementation complexity 
and the number of packets that have to be transmitted.  If I the data 
and control chunks are transmitted into different packets, there is no 
way to guarantee their ordering.  So why care about their ordering if 
they happen to be in the same packet?

- Mark B.

---

From: Randall Stewart <[email-masked]>
Date: Fri, 07 Apr 2006 17:37:32 -0400

> much more natural to put padding at the end of a packet instead of at 

And nothing stops this from occuring if its considered control
--------
HB
--------
PAD-CHK
--------

No data present..

We could go "crazy" and define two chunks.. one considered
"control" and one considered "data"

Or one could put an exclusion in the document saying that it
is neither DATA nor CONTROL so it can appear anywhere
including near other DATA chunks..

But this would probably break a lot of implementations...

I prefer to consider it control..

> prior to transmission.  It can save memory in some forms of receive 

And what stops it from being put at the head or the tail when
sending a HB?

> cares?  The proper question should be "what is it good for?".
What it was designed for :-D

PMTU :-D

> they happen to be in the same packet?
Good question.. and I can't say as I remember why we did it
that way.. my memory fails me.. I guess its been to long ago :-0

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 7 Apr 2006 15:41:43 -0600

Mark,

Mark Butler wrote:                         (Fri, 07 Apr 2006 15:24:35)

-X--snip--X-
> and the number of packets that have to be transmitted.

Well, I found the rule afforded the reverse: that it decreases implementation
complexity and allows for processing optimizations.  Control chunks other than
SACK are rare during data transfer phase, whereas DATA chunks are not.  Only
one SACK normally occurs in a packet.  Placing the SACK in the first chunks
locates it nicely and a hard loop on DATA chunks can be performed after that.
Placing the SACK ahead of all the DATA chunks allows data buffers in the
retransmission buffer to be cleared before allocating any for the received
data chunks.

Ask yourself the reverse question: what's the good in burying a control chunk
in the middle of a bunch of data chunks?

Why not just make the padding chunk a "data" chunk.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 7 Apr 2006 15:47:44 -0600

Randall,

Randall Stewart wrote:                                             (Fri, 07 Apr 2006 17:37:32)

-X--snip--X-
> "control" and one considered "data"

Why not just one considered "data"?

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Randall Stewart <[email-masked]>
Date: Fri, 07 Apr 2006 17:57:38 -0400

> Why not just one considered "data"?

Fine by me.. but the only question I would have is this going
to break any implementations?

I know from inter-op testing that we have throughly tested
the upper bits by placing different chunks in a packet
and sending it off... but I cannot say that we have every
placed:

|control| data| bogus-chunk| data|

In a packet..

And I have a feeling that most implementations, like yours optimize
looking at the control at the beginning... aka handle the SACK etc
and then go into a data-processing loop...  which I think you are
correct.. makes it more efficeient...

The question is, if your data processing loop hits something that is
not CHUNKTYPE == 00

What will you do?

Looking at the BSD implementation real quick... I see that it
would stop processing at the first non-data chunk... its in a
while( ch == DATA) {
}

Optimized much as you stated in your other email..

It does NOT even consider other chunk types... which may not
be optimal.. but it fits the old rules... :-0

That can be fixed of course.. but if other implementations are
like this it imply's a non-backward compatability.. or at least
data chunks following the "pad-data-chunk" would be lost.

R

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 8 Apr 2006 00:05:10 +0200

Dear all,

all SCTP implementations consider only DATA chunks (ie chunks with  
chunk type == 0)
as data chunks and all others as control chunks. This is also what  
RFC 2960 states and
I think we should not change that.

Looking at the point Mark raises of aligning data not only to 32 bits  
but possibly also to
64 or even 128 bits, I think this is not only relevant inside the  
kernel but also on
the kernel/user land. Having in mind that the DATA chunk header is 16  
bytes (128 bits)
the padding can be done at the application layer layer. At the  
transport layer only
an initial padding before the DATA chunks may be required, which is  
possible with the
PAD chunk.

Best regards
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 8 Apr 2006 00:18:35 +0200

Mark,

and if you do not want to do it at the application layer you could  
add parameters
to the INIT/INIT-ACK which both endpoints can use to "negotiate" the  
alignment.
The default is 32 bits. You could negotiate for longer values and  
then just use
a longer chunk padding. This
- gives you the padding you want
- is an extension which is interoperable with older implementations  
(since they
   would report the new parameter as unsupported)

Best regards
Michael

On Apr 8, 2006, at 12:05 AM, Michael Tuexen wrote:

> Looking at the point Mark raises of aligning data not only to 32  
> bits but possibly also to
> the kernel/user land. Having in mind that the DATA chunk header is  
> 16 bytes (128 bits)

---

From: Mark Butler <[email-masked]>
Date: Fri, 07 Apr 2006 16:23:21 -0600

Randall Stewart wrote:

>>>> the draft for this state that this chunk is an exception to the 
>>>> rule that control chunks always precede data chunks?   Reordering 
>>>> chunks to pad seems rather pointless.
>>>
>>>

Is the "control" or "data" nature of an unknown chunk cast in stone - 
i.e. is it a foregone conclusion that all unknown chunks are control 
chunks?  If not, then any forward compatible implementation must allow 
arbitrary ordering for unknown chunks, and any implementation that does 
not is broken.

> I prefer to consider it control..

Why not have a third category: "other"?

>> There are incidental benefits to putting it at the end.  Padding at 
>> the end improves L2 cache performance by minimizing the amount of 
>> useless data read.  It saves memory when chunk groups are 
>> pre-prepared in memory prior to transmission.  It can save memory in 
>> some forms of receive processing as well.
>
>

Nothing.  The issue is position relative to DATA chunks, where L2 cache 
trashing is lessened when it is placed after the last DATA chunk, 
instead of prior to the first one.   Contemporary L2 caches typically 
read 128 bytes at a time, so placing the pad chunk at the end reduces 
the cache footprint demanded, because on average the pad chunk cache 
line overlap will be ~1/2 cache line, or 64 bytes of the L2 cache, 
instead of a full 128 if it is in the middle of the packet.

If the purpose of the PAD chunk is only to pad HEARTBEAT containing 
packets, that should be in the draft. Right now it says "This chunk is 
used to pad an SCTP packet to an arbitrary size".  My inclination would 
be to get some mileage out of the overhead, by tacking on data chunks 
more than 1 RTT away from regular transmission where reasonable.

Speaking of which, isn't sending a large PAD mostly packet without 
congestion control a potential problem?  Perhaps there should be some 
kind of rate limit.

>> I often hear the argument - "that is not what it was intended for".  
>> Who cares?  The proper question should be "what is it good for?".
>
I had an essentially identical PAD chunk extension intended primarily 
for data alignment purposes on high bandwidth networks.  I think it 
would be silly to have to create a PAD2 chunk.

As far as as PMTU discovery is concerned, I realize why this technique 
can be useful on broken / misconfigured networks, but shouldn't there 
also be a new ICMP packet defined that routers modify along the way with 
the MTU of the path it traverses, and then is echoed back to the sender 
when it reaches the destination?

- Mark B.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 7 Apr 2006 16:37:34 -0600

Randall,

Randall Stewart wrote:                                             (Fri, 07 Apr 2006 17:57:38)
> The question is, if your data processing loop hits something that is

At the moment?  Report and continue based on the flags in the chunk type.

> would stop processing at the first non-data chunk... its in a

I currently fast-path data in the main loop as:

  while (more chunks) {
	  if (ch == DATA) {
		  if ((rtn = sctp_recv_data()) >= 0)
			  continue;
	  } else switch (ch) {
		  case SACK:
			rtn = sctp_recv_sack();
			break;
		  ...
		  etc.
		  default:
			rtn = sctp_unrec_chunk_type();
			break;
	  }
	  if (rtn < 0)
		  return (rtn);
	  ..blah..
  }

So I automatically continue after control chunks that are marked
continue.  I decided to be forgiving in receiving control chunks
that are in the wrong place.  I always send them before data in
a packet.

> data chunks following the "pad-data-chunk" would be lost.

Why not set the report flag so that the sender can know the
behaviour of the receiver?

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 7 Apr 2006 16:40:10 -0600

Mark,

Mark Butler wrote:                                                    (Fri, 07 Apr 2006 16:23:21)

-X--snip--X-
>    arbitrary  ordering  for  unknown  chunks, and any implementation that
>    does not is broken.
> 

I went a little bit farther: following the principle in being strict in what
you send and forgiving in what you receive, I currently process the chunk
wheter it is unrecognized or not, even if it is nested in data.  It is not
efficient (because I expect and fast-path data), but it works.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Randall Stewart <[email-masked]>
Date: Fri, 07 Apr 2006 18:46:45 -0400

> efficient (because I expect and fast-path data), but it works.
This will be a good set of tests to add for the interop :-D

R

---

From: Mark Butler <[email-masked]>
Date: Fri, 07 Apr 2006 16:47:43 -0600

>retransmission buffer to be cleared before allocating any for the received
You can do it that way (on the send side) without the rule.  What 
benefit is there on the receive side?  Hard coding the definition of 
"control" and "data" chunks on the receive side is likely to break.

>in the middle of a bunch of data chunks?
Well, if you want 64 bit aligned data payloads, placing 4 byte PAD 
chunks between DATA chunks is one way to achieve it.  It is nice that we 
have 32 bit alignment now, but a decade from now we might look on 32 bit 
alignment the same way we look on 16 bit alignment today.  In some 
contexts (notably IPoIB, 10G Ethernet on 64 bit CPUs / DMA engines) that 
situation already exists.

Other than that I cannot think of any current situations where one would 
normally benefit from having an embedded control chunk, because one 
usually wants control chunks to be sent immediately.  However, tacking 
control chunks on the end of a series of one or more data chunks is 
another story entirely.   If one pre-assembles ~MTU sized series of DATA 
chunks, it is generally much more efficient to tack a SACK or other 
control chunk on the end of the series rather than at the beginning or 
in a separate packet.

>Why not just make the padding chunk a "data" chunk.
That would definitely be better than making it a "control" chunk.  I 
prefer "other" though.

 - Mark B.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 7 Apr 2006 16:50:41 -0600

Mark,

Mark Butler wrote:                                                    (Fri, 07 Apr 2006 16:47:43)
>    chunks  between DATA chunks is one way to achieve it.

There are better ways.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 7 Apr 2006 16:52:22 -0600

Randall,

Yes, it would be interesting to see how other implementations respond
and determine the extent of the issue.

--brian

Randall Stewart wrote:                                             (Fri, 07 Apr 2006 18:46:45)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Fri, 07 Apr 2006 17:04:23 -0600

> I think we should not change that.

It has been recommended here in the past that the proper way to solve 
the the inter-stream TSN dependency problem is to create a new data 
chunk.  This is the type of thing that has to be negotiated at startup 
time, but it is a good example of where the definition of what a data 
chunk is would be extended.

Given that the ordering rule is borderline pointless, it would be much 
easier to just drop it completely, and treat each chunk as if it arrived 
in its own packet, in order of course.

> an initial padding before the DATA chunks may be required, which is  

The total DATA chunk header is actually 20 bytes, which with a 12 byte 
header size, normally aligns the payload of the first DATA chunk on a 64 
bit boundary.  However, if the first data chunk payload is a multiple of 
eight bytes, the next data chunk payload will not be 64 bit aligned.  So 
a performance oriented ULP with relatively small messages and 64 bit 
aligned message sizes will end up with half of its messages unaligned, 
as the alignment alternates from chunk to chunk.  Interleaved 4 byte PAD 
chunks could be used to fix this problem.

- Mark B.