Subject: [Tsvwg] SCTP_NODELAY

From: Mark Butler <[email-masked]>
Date: Sat, 25 Mar 2006 01:02:05 -0700

Currently, the description of SCTP_NODELAY is worded strangely as well 
as underspecified.

    7.1.5. SCTP_NODELAY Turn on/off any Nagle-like algorithm. This means
    that packets are generally sent as soon as possible and no
    unnecessary delays are introduced, at the cost of more packets in
    the network. Expects an integer boolean flag.

This describes the behavior when SCTP_NODELAY is enabled, but the sense 
of the first sentence is backwards.  SCTP_NODELAY on means that any 
Nagle-like algorithm is turned _off_, not _on_.  The first sentence 
should probably read:

           7.1.5. SCTP_NODELAY  When this option is set to one, disable
    any Nagle-like algorithm.  This means that data is generally sent as
    soon as possible and no unnecessary delays are introduced, at the
    cost of more packets in the network.  When this option is set to
    zero, data transmissions may be temporarily delayed waiting for
    additional data to be transmitted in the same packet.

It shouldn't need to be specified, but any implementation that holds 
data indefinitely when SCTP_NODELAY is zero is not using a Nagle 
algorithm at all.  Any Nagle algorithm worthy of the name has an 
expiration timer.  The Nagle algorithm was developed to make Telnet type 
protocols more efficient, and fails spectacularly without such a timer.

Expiration-free behavior should never be a default, and should use a 
different option.  SCTP_CORK would be a good name. i.e. when enabled, 
never transmit anything until a packet is full.  When disabled, release 
what was previously corked, and restore normal behavior.  This behavior 
is useful for applications (e.g. web servers) that want to use the least 
number of packets, and are willing to explicitly cork and uncork to get it.

Comments?

 - Mark B.
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 25 Mar 2006 02:38:18 -0700

Mark,

Mark Butler wrote:                              (Sat, 25 Mar 2006 01:02:05)
>      7.1.5.  SCTP_NODELAY  Turn  on/off  any  Nagle-like algorithm. This
>      means  that  packets  are generally sent as soon as possible and no
>    This  describes  the  behavior  when  SCTP_NODELAY is enabled, but the
>    sense  of the first sentence is backwards.  SCTP_NODELAY on means that
>    any  Nagle-like  algorithm  is  turned  _off_,  not  _on_.   The first
>    sentence should probably read:
> 
>      any  Nagle-like  algorithm.  This means that data is generally sent
>      as  soon  as  possible and no unnecessary delays are introduced, at
>      the  cost  of more packets in the network.  When this option is set
>      to  zero, data transmissions may be temporarily delayed waiting for
>    expiration  timer.   The  Nagle algorithm was developed to make Telnet
>    type  protocols more efficient, and fails spectacularly without such a
>    timer.

That's implemenation, but you are really off base here: Nagle algorithm works
fine without an expiration timer.  Data is only held when data is already
outstanding to the peer, and, therefore, an acknowledgement is expected (and
necessary) for the connection to continue anyway.  Therefore, no separate
expiration time is necessary.

I think that you are confusing Nagle algorithm with the coalescing of
individual keystrokes performed on ttys, which, indeed does use a timer.

--brian

>    never  transmit  anything  until  a  packet  is  full.  When disabled,
>    release what was previously corked, and restore normal behavior.  This
>    behavior  is  useful  for applications (e.g. web servers) that want to
>    use  the  least  number of packets, and are willing to explicitly cork
>    and uncork to get it.
>    Comments?
>     - Mark B.

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Sat, 25 Mar 2006 08:57:03 -0700

>expiration time is necessary.
Understood.  However,  the only germane issue is that the algorithm must 
not hold data indefinitely.

 - Mark

---

From: Joe Touch <[email-masked]>
Date: Sat, 25 Mar 2006 10:45:54 -0800

Mark Butler wrote:
>     any Nagle-like algorithm. 

A citation would be nice here (e.g., 896).

> This means that data is generally sent as

I'm not sure if it matters, but the specific algorithm (last I recalled,
at least) focused on limiting the number of partially-filled segments
that were outstanding (unACK'd).

So it'd be more useful to word this:

	When this option is set to
	zero, only one partially-full data segment
	can be unacknowledged; this causes subsequent
	transmissions to be aggregated until either
	a) a full segment can be transmitted,
	b) the unacknowledged partial-segment is ACKd
	c) a timeout (typically 200ms in TCP) occurs

	NB: Nagle was intended to reduce the number of
	small packets for applications like telnet,
	but this achieves the desired effect
	only where the messages cannot be split across
	segments. SCTP_NODELAY SHOULD be set to 0 for
	interactive applications with multi-byte
	messages esp. where the source is bursty (e.g., HTTP),
	since otherwise a message may be split across segments,
	possibly resulting in unintended delays for a single
	application message exchange.

(I have no idea how this needs to be tweaked to address SCTP's unique
characteristics; if message boundaries are preserved, it may not be an
issue, but it should be discussed since TCP implementers should know
about this concern).

> number of packets, and are willing to explicitly cork and uncork to get it.

As per above, this is exactly what you do NOT want to do for web servers.

As to using the socket API to open/close the gates, that works ONLY if
the API correlates to the last message sent; I'm not clear that anything
that tightly-coupled is specified.

Joe

---

From: Mark Butler <[email-masked]>
Date: Sat, 25 Mar 2006 14:01:57 -0700

>A citation would be nice here (e.g., 896).

Agreed.  "[RFC896]" should be inserted after algorithm.

>	c) a timeout (typically 200ms in TCP) occurs
This is fine, but I think unnecessarily overspecifies the algorithm.  As 
Brian has pointed out, a Nagle algorithm (notably the original RFC896 
algorithm)  does not require a separate timer.  For API purposes, the 
main thing that matters is that data is not held indefinitely when 
SCTP_NODELAY is zero.  It should be safe for an application to set 
SCTP_NODELAY to zero and leave it that way.

>	application message exchange.
I think you meant to say "SHOULD be set to 1", otherwise the second 
sentence doesn't make any sense. In what cirucumstances the Nagle 
algorithm makes sense is application dependent and does not necessarily 
correlate with message boundaries.  If the ULP is latency sensitive and 
processes only complete messages Nagle doesn't make a lot of sense.  If 
the ULP processes partial messages and can handle ~200ms delays it is fine.

There is nothing inherent in SCTP that prevents the transfer of partial 
messages, by the way.  The most natural mapping of TCP to SCTP is to 
start a message and use it for the rest of the connection. 

>issue, but it should be discussed since TCP implementers should know
One lesson that can be learned from TCP implementations is that an 
independent timer is beneficial, even though RFC896 does not require one. 

>that tightly-coupled is specified.
On Linux, TCP_CORK behaves as I described, and was designed and is used 
to make web servers more efficient.  Typically a web server turns 
TCP_CORK on, writes the response headers, calls sendfile(2), and then 
turns TCP_CORK off.   The TCP_CORK option does not close the gates, it 
just holds data long enough to fill out a full size packet.  When it is 
cleared, any remaining data is released.

An SCTP_CORK option would work the same way, and be used for the same 
things, except on a per-association basis instead of a per socket basis 
in the case of one-to-many style sockets. 

 - Mark B.

---

From: Joe Touch <[email-masked]>
Date: Sat, 25 Mar 2006 13:51:09 -0800

> Joe Touch wrote:
...
> SCTP_NODELAY to zero and leave it that way.

I'm not a Nagle-implementation expert, so I'll trust that this will work ;-)

> sentence doesn't make any sense. 

Yup. Sorry ;-)_

> In what cirucumstances the Nagle

There was something about single-byte vs. multi-byte noted in 896 or
some subsequent papers thereon; I don't know if that's worth revisiting
in this context. That's what I was referring to.

> start a message and use it for the rest of the connection. 

Same for TCP - Nagle is an optimization to reduce header overhead and
the number of small packets in the net.

> independent timer is beneficial, even though RFC896 does not require one.

Sure - again, I'll let someone with more background here confirm the
details.

> in the case of one-to-many style sockets. 

How _CORK actually works, and whether the specification of the API can
_ensure_ that it works as intended are two different things. If this can
be spec'd such that all implementations must work correctly, that'd be
much more useful than showing a single implementation that happens to be
behave the way you want, under current measurements.

Joe

---

From: Randall Stewart <[email-masked]>
Date: Sun, 26 Mar 2006 06:27:17 -0500

> not hold data indefinitely.

As long as you are following the Nagle RFC I don't see how
that is possible.. since Nagle depends on a SACK coming
back to kick out more data.. if the peer does not return
a SACK then the association is failing anyway.. if it
does return a SACK you send out more data...

R

-- 
Randall Stewart
ITD - Cisco Systems Inc.
803-345-0369 <or> 815-342-5222 (cell)

---

From: Mark Butler <[email-masked]>
Date: Sun, 26 Mar 2006 15:08:49 -0700

Randall Stewart wrote:

> Mark Butler wrote:
>
>> However,  the only germane issue is that the algorithm must not hold 
>> data indefinitely.
>>
>

Understood.  The idea is that it would be helpful to clarify what the 
minimum requirement is for a Nagle-like algorithm, and not holding data 
indefinitely fits the bill. 

 - Mark B.

---

From: Rick Jones <[email-masked]>
Date: Mon, 27 Mar 2006 10:17:14 -0800

I suspect the reason "the nagle algorithm" doesn't have a timer of its 
own was it could piggy-back on the existing TCP retransmission timer.

if we are loading-up on the description of SCTP_NODELAY, might as well 
put-in text to the effect that setting SCTP_NODELAY is _NOT_ an 
acceptable workaround for not presenting "logically associated" data to 
the transport at one time.

rick jones

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 27 Mar 2006 14:18:50 -0700

Rick,

You mean "clearing" SCTP_NODELAY, right?

--brian

Rick Jones wrote:                           (Mon, 27 Mar 2006 10:17:14)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Rick Jones <[email-masked]>
Date: Mon, 27 Mar 2006 13:56:25 -0800

> You mean "clearing" SCTP_NODELAY, right?

I may have a sense problem.  In my TCP-trained brain, TCP_NODELAY is by 
default cleared which means Nagle is in force; setting TCP_NODELAY to a 
value of one disables the Nagle algorithm.   In TCPspace, disabling the 
Nagle algorithm is NOT an acceptable workaround for the application bug 
of not presenting logically associated data to the transport at one time.

I then ass-u-me-d that SCTP was the same way and may have been incorrect 
in that assumption?

rick jones

---

From: Mark Butler <[email-masked]>
Date: Mon, 27 Mar 2006 16:39:32 -0700

> own was it could piggy-back on the existing TCP retransmission timer.

Oddly enough most TCP implementations include a separate ~200 ms timer.  
Anyone know the reason for the change?

> acceptable workaround for not presenting "logically associated" data 
> to the transport at one time.

If the ULP uses SCTP message / record boundaries, that is currently 
impossible to avoid.  One can only send whole messages at a time.  I 
agree with the principle though.

- Mark B.

---

From: Mark Butler <[email-masked]>
Date: Mon, 27 Mar 2006 16:43:55 -0700

Rick Jones wrote:

> I may have a sense problem.  In my TCP-trained brain, TCP_NODELAY is 
> by default cleared which means Nagle is in force; setting TCP_NODELAY 
> to a value of one disables the Nagle algorithm.   In TCPspace, 
> disabling the Nagle algorithm is NOT an acceptable workaround for the 
> application bug of not presenting logically associated data to the 
> transport at one time.
>
> I then ass-u-me-d that SCTP was the same way and may have been 
> incorrect in that assumption?

SCTP_NODELAY has the same sense as TCP_NODELAY. 

 - Mark B.

---

From: Rick Jones <[email-masked]>
Date: Mon, 27 Mar 2006 15:50:56 -0800

> Anyone know the reason for the change?

Just for Nagle?  Not the normal retransmission or delayed ACK timer? 
Which stacks?

rick jones

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 27 Mar 2006 18:13:58 -0700

Rick,

Ditto.

--brian

Rick Jones wrote:                                                          (Mon, 27 Mar 2006 15:50:56)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Joe Touch <[email-masked]>
Date: Mon, 27 Mar 2006 17:22:23 -0800

> of not presenting logically associated data to the transport at one time.

There's NO association of data at the transport API in TCP. There's
order (sequence), but there is absolutely no correlation that can be
attributed to data within the boundaries of a send call vs. the segments
transmitted.

Joe

---

From: Rick Jones <[email-masked]>
Date: Mon, 27 Mar 2006 17:31:13 -0800

> attributed to data within the boundaries of a send call vs. the segments

That said, any application that does want to send "messages" over TCP 
really must (OK, perhaps should in RFC-speak) present the logically 
associated data at one time or it _will_ run afoul of at least the 
interaction between Nagle and Delayed ACKs and possibly the mapping of 
the byte-based ABC RFC to a packet-based cwnd implementation.

rick jones

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 27 Mar 2006 18:35:44 -0700

Rick,

Rick Jones wrote:                            (Mon, 27 Mar 2006 13:56:25)
> of not presenting logically associated data to the transport at one time.

I don't quite understand what you mean by that.  I know that some
applications that in fact have records rather than being a byte stream
often disable Nagle to reduce latency and to ease reassembly of records
at the reveiver; however, because SCTP preserves record boundaries, and
may bundle data chunks into SCTP packets, that is not much of an issue
for SCTP.  SCTP was designed for just such applications (smaller record
based transmissions in multiple logical streams).

Although it is rather implementation dependent, Nagle can be performed
in SCTP on two levels: packing data into data chunks, and bundling data
chunks into packets.  The former (packing data into data chunks) is
important when the data stream is in fact byte-oriented rather than
record based, i.e. the data has no defined record boundaries.

When packing data into data chunks, the implementation can detect that
there is more data belonging to the same data chunk forthcoming
(that is, belonging to the same SSN), and latency is not an issue.  In
this case, disabling the Nagle algorithm will not increase latency of
the entire record (but, perhaps the initial portions of the record if
partial delivery is performed).

When bundling data chunks into SCTP packets, it could often be that case
that additional data chunks are not forthcoming, and Nagle can increase
latency for low bandwidth applications that send sporadic records.

Therefore, I can only see disabling Nagle for SCTP when small, low
bandwidth, sporadic records are sent requiring minimum latency.  Indeed,
that might be a wide range of SCTP applications.

Nevertheless, Nagle for SCTP is optional.

Being an optional feature, unless TCP is being emulated, it should
default to set (disabling Nagle), the reverse of TCP.  When attempting
to emulate a TCP interface to the application, it should default to
clear (Nagle in effect).  Whether an implementation actually performs
Nagle, and just how it goes about doing it, must be implementation
specific.

That's how I (and probably other) implemented it too.

BTW, I also implemented an SCTP_CORK.  Both SCTP_NODELAY and SCTP_CORK
have the same option value as TCP_NODELAY and TCP_CORK.  An SO_NODELAY
and SO_CORK would have been an idea, but that ship has probably sailed.

SCTP_CORK behaves like TCP_CORK: that is, it holds pending data chunks
until released.  It is, again, more useful for TCP emulation, or when
the application has no concept of record boundaries.  I believe that
TCP_CORK is used primarily by FTP for building headers before using
sendfile() to transfer the file after the headers.  There the problem
is that the peer application could otherwise hang on the headers for a
while until the first page of the file is actually retrieved and
transmission resumed by the sender, or something like that...

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Mon, 27 Mar 2006 19:05:09 -0700

Rick Jones wrote:

> Mark Butler wrote:
>
>> Oddly enough most TCP implementations include a separate ~200 ms 
>> timer.  Anyone know the reason for the change?
>
>

Sorry, I was confusing the delayed ack timer with a Nagle-type timer.  
My mistake.

 - Mark B.

---

From: Rick Jones <[email-masked]>
Date: Mon, 27 Mar 2006 18:06:53 -0800

> based transmissions in multiple logical streams).

There are not enough bits in a word to count the times I've seen 
applications do:

write(header);
write(data);
read(response);

And then "fix" the problem by setting TCP_NODELAY.  99 times out of 10, 
an application setting TCP_NODELAY is doing so as a "fix" (aka kludge) 
to just that problem, when the real fix would be:

writev(header,data); (or sendmsg())
read(response)

When I go on about applications must present logically associated data 
to the transport in a single send call, that is what I'm talking about, 
and don't mean to imply that it would provide any message boundary 
sensing by TCP, or that the entire message would arrive in a single 
segment or anything other than just presenting the logically associated 
data to the transport at one time.  In one mailing list I've seen 
discussion of one example recently where it was making something like 
_FIVE_ separate send calls per "message" where the first four were 
roughly four bytes each or less.  That is a classic example of misuse of 
TCP_NODELAY.

I do accept that setting TCP_NODELAY _can_ be indicated when an 
application needs to have multiple "messages" outstanding, and so long 
as it doesn't do what I've described above that is fine.  There we start 
talking about a race between the RTT, the remote delayed ACK timer, and 
the remote server's service time.

> record based, i.e. the data has no defined record boundaries.

Sounds rather like classic TCP.

> latency for low bandwidth applications that send sporadic records.

That sounds like it should be controlled by a different socket option. 
SCTP_NOCHUNKDELAY or somesuch.

> and SO_CORK would have been an idea, but that ship has probably sailed.

Unless I misinterpreted what you said above, SCTP_NODELAY is "more" than 
TCP_NODELAY because it affects two, separate, SCTP behaviours rather 
than just the one.

> sendfile() to transfer the file after the headers. 

FTP sends no headers on the data connection :)  HTTP though does send 
headers, and indeed, I believe that HTTP servers running on Linux more 
or less have to use TCP_CORK because sendfile() under Linux does not 
offer the option of providing header/trailer buffers in the sendfile() 
call a la the other OSes' stacks.

> There the problem

I don't see where TCP_CORK could help that - it is holding the headers 
until the connection is uncorked, which (in my experience) didn't happen 
until after sendfile() completed anyway.  That said, I rather prefered 
the sendfile with header/trailer approach anyway (being biased of course :)

rick

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 27 Mar 2006 19:47:37 -0700

Rick,

Rick Jones wrote:                                                          (Mon, 27 Mar 2006 18:06:53)
> writev(header,data); (or sendmsg())

I still don't understand what your after...  Under blocking operation
the two sequences above are identical (see POSIX writev() reference page,
it is identical to multiple calls to write with each buffer in the iovec,
and is, in fact, no more than a convenience function).

OTOH, I believe this is what FTP uses TCP_CORK for:

 setsockopt(s, TCP_CORK, 1);
 write(header);
 sendfile(fd);

> roughly four bytes each or less.  That is a classic example of misuse of 

I thought that is what TCP_CORK is for.  TCP_NODELAY has, it seems to me,
the opposite effect.  I don't see what the following accomplishes (except
unnecessarily spewing small packets):

 setsockopt(s, TCP_NODELAY, 1);
 write(four_bytes);
 write(four_bytes);
 write(four_bytes);
 write(four_bytes);
 write(four_bytes);
 read(response);

> TCP_NODELAY because it affects two, separate, SCTP behaviours rather 

Yes, except in the special case where there is only one stream and it is
byte oriented (i.e. TCP emulation).

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Mon, 27 Mar 2006 20:04:05 -0700

Rick Jones wrote:

> And then "fix" the problem by setting TCP_NODELAY.  99 times out of 
> 10, an application setting TCP_NODELAY is doing so as a "fix" (aka 
> kludge) to just that problem, when the real fix would be:
>
> to the transport in a single send call, that is what I'm talking 
> about, and don't mean to imply that it would provide any message 
> boundary sensing by TCP, or that the entire message would arrive in a 
> single segment or anything other than just presenting the logically 
> associated data to the transport at one time.  In one mailing list 
> I've seen discussion of one example recently where it was making 
> something like _FIVE_ separate send calls per "message" where the 
> first four were roughly four bytes each or less.  That is a classic 
> example of misuse of TCP_NODELAY.

Complete agreement here.  On Linux MSG_MORE should be used instead (on 
TCP at least). More below.

> as it doesn't do what I've described above that is fine.  There we 
> start talking about a race between the RTT, the remote delayed ACK 
> timer, and the remote server's service time.
>
> Brian F. Bidulock wrote:

> SCTP_NOCHUNKDELAY or somesuch.

Comment below.

>> That's how I (and probably other) implemented it too.
That is the case for LKSCTP as well (SCTP_NODELAY default off not on).

> Unless I misinterpreted what you said above, SCTP_NODELAY is "more" 
> than TCP_NODELAY because it affects two, separate, SCTP behaviours 
> rather than just the one.

In the sctpsocket draft, SCTP_NODELAY currently only affects chunk 
bundling.  There is no consensus yet on a standard way to write partial 
messages, or catch a signal during a large message write, etc.  Messages 
are treated like UDP datagrams - they must fit in the send buffer, and a 
write operation must take all or none.

Eventually independent options should be worth considering, one for 
chunk bundling, and one for merging partial writes into the same chunk.  
SCTP_NODELAY most naturally maps to the latter.  From a POSIX API 
perspective a TCP stream is just one long record, and TCP_NODELAY causes 
partial record writes to be combined.

>> sendfile() to transfer the file after the headers.

TCP_CORK does not hold data unconditionally, it holds data until an MSS 
sized frame is full or the option is cleared.

> call a la the other OSes' stacks.

MSG_MORE is a simpler alternative to TCP_CORK on Linux these days.  You 
set MSG_MORE on the header send(2) / sendmsg(2), it is preserved through 
the next sendfile(2) call and then cleared.   Of course with SCTP, 
MSG_MORE would have ambiguous semantics - does one mean to glue writes 
together in the same message the way MSG_MORE on UDP behaves, or to 
delay transmission the way MSG_MORE on TCP behaves.  For MSG_MORE to be 
useful for SCTP, that ambiguity needs to be resolved, perhaps by 
changing its semantics depending on whether MSG_EOR is set. MSG_MORE set 
/ MSG_EOR clear.

> until the connection is uncorked, which (in my experience) didn't 
> happen until after sendfile() completed anyway.  That said, I rather 
> prefered the sendfile with header/trailer approach anyway (being 
> biased of course :)

TCP_CORK is misnamed - it doesn't cork the data stream at all.  It just 
acts like a Nagle type algorithm holds partial packets indefinitely.   
BSD's TCP_NOPUSH option does the same thing except  TCP_CORK (which is a 
Linux-ism) releases any remaining data when it is disabled, which 
TCP_NOPUSH has not traditionally done (although I have seen patches to 
make TCP_NOPUSH adopt that behavior as well).

In this case, the header would be sent as soon as the first data block 
from sendfile() arrived, in a TCP segment containing both header and 
data information.  Just like Nagle except no partial segments are sent 
out until TCP_CORK is cleared. 

 - Mark B.

---

From: Joe Touch <[email-masked]>
Date: Mon, 27 Mar 2006 19:42:02 -0800

> the byte-based ABC RFC to a packet-based cwnd implementation.

"presenting it at one time" needs to be defined; if you mean within a
reasonable period of time, that's fine, but it cannot mean 'in the same
send call'

Joe

---

From: Mark Butler <[email-masked]>
Date: Mon, 27 Mar 2006 21:06:58 -0700

>reasonable period of time, that's fine, but it cannot mean 'in the same
It is not really the place of the sctpsocket draft to define proper 
client application programming beyond the bare minimum requirements to 
use the API in a portable manner.  Perhaps one day someone could write a 
"network application programming best practices" draft.

 - Mark B.

---

From: Joe Touch <[email-masked]>
Date: Tue, 28 Mar 2006 08:01:19 -0800

...
> "network application programming best practices" draft.

Sure - in which case it should note the difference between send call
semantics for SCTP vs. TCP.

Joe

---

From: Rick Jones <[email-masked]>
Date: Tue, 28 Mar 2006 09:40:47 -0800

> reasonable period of time, that's fine, but it cannot mean 'in the same

I do mean in the same send call.  Otherwise, in the case of TCP at 
least, when those multiple sends are small, the application will run 
into Nagle+Delayed ACK, and the default application programmer behaviour 
is to simply set TCP_NODELAY and have their tiny sends spewed-out onto 
the network.

Now, I'll agree that it is probably something that cannot be codifed in 
anything except a best practices writeup.

rick jones

---

From: Joe Touch <[email-masked]>
Date: Tue, 28 Mar 2006 11:44:11 -0800

Rick Jones wrote:
> I do mean in the same send call.

That's OK for SCTP - which has the semantics of 'within a send call =
chunked', but that is not appropriate for TCP.

> Otherwise, in the case of TCP at
> into Nagle+Delayed ACK,

That depends on the scheduling of processing of the application, kernel
(socket), TCP, etc.; it is not spec'd by the TCP API.

> and the default application programmer behaviour

If that's what they do, then whether the sends are aggregated or not on
short timescales depends on the scheduler - and that's what the user
asked for.

> anything except a best practices writeup.

I don't think it's best-practices if it depends that heavily on the
scheduler.

Joe

---

From: Erblichs <[email-masked]>
Date: Tue, 28 Mar 2006 14:29:28 -0800

Group,

	As a raised TCP guy, their is no guarantee
	that a ack is going to be recieved and if
	it differs in SCTP, I missed it.

	Thus, I would expect to use maybe
	a coarse level rexmit timer.

	Mitchell Erblich
	----------------

> > expiration time is necessary.
	Not being 

> Understood.  However,  the only germane issue is that the algorithm
> must not hold data indefinitely.
> 
>  - Mark
> 
>     ---------------------------------------------------------------

---

From: Randall Stewart <[email-masked]>
Date: Tue, 28 Mar 2006 17:28:07 -0500

Mitchell:

There is always a retransmit timer running on any
data.. TCP or SCTP.. so if the ack does not show
up.. the timer fires.. and you retransmit the
data (collapsing the cwnd as well.. not that
it would be very big if Nagle is in play).

R

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 29 Mar 2006 00:30:29 +0200

Mitchell,

the Nagle algorithm delays sending of new data if there is outstanding
data. You are running already T3 timers for this outstanding data. This
timer makes sure that the outstanding data will be retransmitted until
it is acked by the peer (or the association fails due to excessive  
retransmissions)
which means that no data is outstanding anymore and the delayed data  
gets
transmitted. Therefore there is no need for a specific Nagle timer.

If this behaviour is not clear from the spec, we might need to  
clarify it
in the spec, but I find it pretty clear.

Best regards
Michael

---

From: Rick Jones <[email-masked]>
Date: Tue, 28 Mar 2006 14:31:44 -0800

> I don't think it's best-practices if it depends that heavily on the

I think we may be misunderstanding each other, and I may have entered 
into a context point in the discussion other than what I'm trying to 
suggest.  What I'm talking about are applications that do indeed want to 
exchange messages, and then the application programmer does something like:

write(appheader);
write(appbody);
read(reply);

and then complains that their application runs slowly "because of Nagle" 
  The appheader and appbody both have to be received and processed by 
the remote before the reply can be sent.  They are "logically 
associated" with one another, but the app body is held-back by Nagle 
because the connection is not idle - the appheader is already out on the 
connection and the appbody is sub-MSS.

Now, it may be that the application "said" there were distinct things 
being sent, but 99 times out of 10 it has been my experience that the 
application has "said" something other than what it really "meant" which 
was  "I want to send this message"  not "I want to send these two messages"

They then "fix" the problem by setting TCP_NODELAY rather than 
presenting the application message header and body to TCP in the same 
send call.

I see that as a "socket programming issue" distinct from applications 
that are doing:

write(wholesmallmessage);
write(otherwholesmallmessage);
...
start looking for replies

which does indeed involve the race between the successive sends of the 
wholesmallmessages, the arrival of replies with piggy-backed ACKs and 
perhaps the remote's standalone ACK timer.

Yes, there can be timing issues for the first case too, but from all the 
"there is a problem in networking" calls I have gotten that 
"write(appbody)" "always" "wins" the race with the remote's standalone 
ACK and so "loses" and sits waiting.

rick jones

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 28 Mar 2006 16:00:35 -0700

Rick,

This is what TCP_CORK does.  Tell them to use TCP_CORK.  I hear there is a
new Edition of Stevens.  Direct them to that.

--brian

Rick Jones wrote:                                                                     (Tue, 28 Mar 2006 14:31:44)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Rick Jones <[email-masked]>
Date: Tue, 28 Mar 2006 15:59:50 -0800

> new Edition of Stevens.  Direct them to that.

IIRC, TCP_CORK is linux-only, so I'll continue to suggest Stevens.  I 
just was hoping to have the mantra repeated elsewhere :)

rick jones